That's the whole point: There's no way to determine "unauthenticated" SSL versus MITM SSL. Once you have that "worked out" then you've solved the issue of self-signing in the first place.
I think I have a good idea to solve this.
What if a system similar to HSTS was implemented? https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security important sites (banks) could send a "don't allow self signed certs" header. Browser UI could change to educate users not to type personal information into self signed sites.
... Then an attacker MITMs the bank with a self-signed cert, and strips the "don't allow self-signed certs" header. Such a header would be literally useless.
HSTS already allows self-signed certificates, if the certificated is validated out-of-band.
I think if you try to responsibly design this feature in browser UI, you'll end up with the same problem.
User requests an HTTPS resource, which means the connection must be secure. The UA is unable to establish a secure connection, just as if a MITM attack was underway.
So, now, the user has just entered https://facebook.com. Currently, when this secure connection fails, the browser warns the user in no uncertain terms.
Your suggestion would be to what, exactly? Put a small, unobtrusive dot somewhere in the UI indicating "yeah, I know you asked for HTTPS, and I ignored that, but I didn't wanna really bother you"?
If you're responsible for user's safety, which may depend on the UA treating HTTPS as it should, then you've just betrayed your user.
And at any rate, by not aborting while verifying the certificate, you're already leaking potentially personal information: The URL, cookies, etc. So if you treat the user's data with respect, you don't go sending that insecurely after the user requested HTTPS.
And if you can't transmit the request, because you need to warn the user... you end up with a UI like browsers have.
It's pretty simple: An unvalidated/unauthenticated certificate looks like a MITM. Requesting an HTTPS resource indicates you want a secured connection. If the certificate is not trusted, then you don't have a secure connection.
The criteria is not self-signed, it's trusted/authenticated or not. Most self-signed certs are not trusted, and solving that solves the CA problem. But if a self-signed cert is trusted then browsers happily display the secure UI without any errors.
The idea behind HSTS is that the proper server commits to something for a significant period of time, making the later MITM fail.
In response to your previous post, the idea is that the user has to take some responsibility for their security (we can't stop them from entering their bank credentials into a friendly form on www.phish.com either), through a UI that makes it clear that the identity of the server is unauthenticated.
