OPINION - Thoughts on Google versus Symantec

The Google Chrome team recently published a proposal to remove trust in existing Symantec-issued HTTPS Certificates. This was in response to "a series of failures by Symantec Corporation to properly validate certificates". This did not come completely out of the blue and does follow a previous blog post on similar failures, and extra checks on Symantec-issued certificates (that they must use Certificate Transparency to be accepted by Chrome going forward, but even with that many were surprised at the post (not least of all Symantec themselves who issued a statement of strong objection).

Symantec are one of the largest certificate providers in the world. The exact size depends on exactly how you measure this but they are definitely in the top three, and are used by millions of sites. In addition Symantec has acquired and continues to run as separate brands, many other well known Certificate Authorities including Thawte, Geotrust and RapidSSL. So Symantec certificates are used by many sites (including Amazon and Paypal) and, ironically the Google certificates themselves are crossed signed by Symantec roots (though as Symantec do not issue from these roots Google Chrome have stated these and similar Apple cross signed roots will not have the same restrictions).

Chrome also accounts for over half of the browser usage at the time of writing. So, while the other browser vendors have not yet stated the intention to impose the same restrictions that Chrome has proposed, these restrictions would affect a massive number of site owners and users of those sites.

So is Google right to push for these restrictions? Or are Symantec "too big to fail" so doing so would cause more problems than it would solve? Will Google even follow through or are the saber rattling to make a point and force Symantec to up their game on addressing the issues they are raising?

HTTPS secures the communication between a user on a web browser, and the server. It ensures privacy of messages you are sending and receiving (no one else can read your passwords), integrity of those messages (no one can alter your message - e.g. to change the destination account of a bank transfer message) and ensures authentication of the URL (i.e. you are talking to the web address shown in the address bar). There are also some things HTTPS does not give you in terms of security (see my previous post: What does the Green Padlock really mean), in particular that the website is "safe" in any other ways other than the messaging to and from it. However, while HTTPS cannot confirm a site is safe, lack of HTTPS can certainly confirm you are not safe and you should not hand over any sensitive information like passwords or credit card details to sites without HTTPS. HTTPS depends on certificates issues by Certificate Authorities (CAs) and, they must be issued by a number of recognised CA trusted by your browser to show the green padlock.

One of the big downsides of HTTPS is that, by default, any CA recognised by the browser can issue a certificate for any website. The Internet is not split in any way to say, for example, that Symantec issue all .com certificates for example. This is great for competition, as you can get your HTTPS certificate from any number of CAs, but bad for security. I use Lets Encrypt for my HTTPS certificates on this site, but there is nothing to stop another CA, even on from the other side of the world that I have never spoken to, from issuing a certificate for my site. This would allow someone to set up a fake version of this site and trick people into using it while still maintaining the green padlock. Therefore there is a huge responsibility on these Certificate Authorities to ensure they are only issuing certificates to the actual site owners or you undermine the trust in the whole of HTTPS and therefore anything on the internet that depends on it! There are various technologies available (CAA, HPKP, Certificate Transparency) which attempt to give site owners ways of controlling which CAs can issue certificates for their sites but they are fairly new, not widely used and difficult for most site owners to understand.

CAs must check you are the owner of the URL before it allows you to get a HTTPS certificate for it. This can be done in a number of ways (e.g. e-mailing admin@yourdomain.com with a confirmation link, checking DNS records, or asking you to put a file on your website). And ever since HTTPS first started there have been bugs and issues with this. They are not common, and the overwhelming majority of HTTPS certificates are issued to the correct domains, but bugs and sometimes carelessness creeps in so certificates have been issued to parties who are not the site owner in the past. Symantec has a bit of a bad reputation for this in the security industry but it's difficult to tell if this is due to the fact they are bad at this, or just due to the scale they operate on and so are statistically more likely to have problems even if the overall percentage is relatively low. Saying that other large CAs (e.g. Comodo) don't seem to have as bad a reputation.

One of the things Symantec did wrong, which really kicked this off, was issuing a certificate for google.coma and www.google.com. When Google noticed (by the use of Symantec automatically publishing the certificate to Certificate Transparency logs), they were not impressed. Symantec claimed this was a test certificate, was not given to anyone outside of Symantec, and even went as far as firing the employees (Archived statement here as the post looks to have been removed from Symantec's website). This may seem harsh to fire people over some testing, but does indicate the seriousness and responsibility CAs need to take.

Unfortunately it did not end there, and further investigations as a result of that incident, revealed a whole host of incorrectly issued certificates. Some for sites that didn't exist (which obviously should not happen). The Google Chrome team responded with the first major restrictions and stated that any new certificates issued by Symantec (or any of it's other brands) must have been logged in Certificate Transparency (CT) logs for them to work in Chrome. These are publicly viewable logs and so can be searched by site owners (or anyone else) to see any certificates issued by their site (either by their authority or as a mistake). CT only makes sense if all CAs use it, so Chrome has been pushing CAs to use it, initially insisting it for EV certificates, and have recently announced that it will be mandatory for all certificates from October 2017 so this was more of bringing forward of the deadline for Symantec. It would provide more openness to Symantec issued certificates, caused them some extra work, but should have caused minimal disruption to site owners (though see below for more info on that!).

Unfortunately that too was not the end of it either, and further certificates were found to have been issued incorrectly. 127 certificates were issued by a number of partners of Symantec without following all the Symantec rules to ensure they should have been issued. Additionally documentation seems to be missing for a number of other certificates so Google is taking the hard line and saying that potentially 30,000 certificates were mis-issued. Except they didn't even use the word "potentially" basically saying that without supporting documentation they have been mis-issued even if they were given to the correct domain owner. Symantec have terminated the partnerships but Google are not impressed. Part of the problem is that the certificates were basically issued by the main Symantec "root certificates", so it's not like Chrome can easily identify and restrict just these certificates approved by the partners.

The Chrome team, do recognise the widespread disruption that would happen to the Internet if they just removed all Symantec owned CAs from their set of trusted certificates, but do not feel that they can ignore the issues they have uncovered. So they are proposing the following changes:

• Restricting any future certificates issued by Symantec to a maximum of 9 months validity period. As certificates are typically issues for 1-3 years this means all sites with Symantec-issued certificates, unless they are expiring and renewing this year, will need to be reissued with replacement certificates that meet these time frames. This will cause disruption to site owners and may also drive them away from Symantec to other providers who are still allowed to issue longer lasting certificates.
• Gradually over the remaining released of Chrome in 2017, they will reduce the length of time that current Symantec-issued certificate are valid for down to this 9 months limit. This will mean that considerable number of currently installed certificates will need to be replaced earlier than their expiration date. Also as they are gradually reducing this life time, this will affect different sites at different times. The proposed Algorithm takes a bit of understanding
• EV certificates will lose the extra green UI used to display the company behind that website. This will happen for at least one year, and there is no alternative for Symantec customers who want to keep EV status but to purchase new certificates from competitors.

In the blog post Google have repeatedly stated this is not "punitive", and merely necessary because the trust in the certificates have issued is suspect. However it is difficult to see this as anything but a severe reprimand to Symantec and, will undoubtedly make their customers at least consider moving to alternative CAs. While the 9 month restriction will be annoying for some, at least that can be implemented, whereas there is no work around for the EV issue other than to move to another CA. I run another site that uses a Thawte EV certificate that we spent quite a bit of money on, and I will definitely do this if this proposal is implemented. Google are therefore, in my opinion, effectively proposing putting Symantec out of the CA business as a punishment for their mistakes.

However the bigger concern is the sheer scale of the issue. Hundreds of thousands of sites will need to replace certificates, and many will be unaware of this until people start complaining that their website can't be used in the world's most popular browser. Of course you the onus will be on Symantec to contact all it's customers, and inform them of the issue, and ensure they reissue the certificates in time to prevent, but there's still thousands of man hours of work here for companies that did nothing wrong other than happen to pick one of the most popular CAs (who have obviously let themselves and their customers down here).

And it's this collateral damage that concerns me the most, as this increasing bad blood between Google and Symantec has already impacted a number of sites. Some of the security community seems to be gleefully rubbing their hands with this, having long standing issues with Symantec and undoubtedly these people are not using their certificates so having no work to do here because of this so will not feel the impact. And honestly I don't particularly care one way of another how Symantec fairs out of this - they made their bed and have to lie in it - but I do worry about site owners and users. Not everyone follows security news as much as I do and will be aware this is all going on. Even assuming they do become aware, they are going to have extra work for this despite the fact that their certificates are perfectively valid and should, ideally be treated as such.

I mentioned above that Google already imposed some restrictions on Symantec previously to force them to use Certificate Transparency. This should have been an extra effort for Symantec and seamless to website owners, but this was badly implemented by Google resulting in a 10 week time bomb after which the site could no longer be accessed at all from older versions of Chrome. While Chrome mostly auto updates, so most people wouldn't see this issue, corporate installations of Chrome do not always, and Android installs do not always auto update either. The other site I work on uses an EV Thawte certificate as mentioned above and, as a result of this issue, we started getting reports that people could not log into our mobile application on some Chrome phones despite running the same version of the app, and Android on the same phone. Eventually, after a couple of weeks of increasing numbers of reports, we figured turned out that it was because the login api call was failing due to this bug, and updating the WebView component of Android fixed it - but was a complete nightmare to figure out. Even when we knew the cause there was no fix but to ask each customer who complained to update their phone. I follow HTTPS news quite a bit, but never was aware of this bug, and even if I was would probably not have considered it immediately while investigating this issue. In the end the only reason I found it is because a developer in the team had an old version of Chrome and mentioned he couldn't connect to the website, otherwise we night still not have identified the cause. While bugs can happen with any update, it was particularly annoying to me that this bug was a direct result of this war between Google and Symantec, and was potentially added without full testing. Others have also experienced this issue, and struggled to explain it.

There is also another bug affecting Symantec EV certificates which is causing much confusion as some sites have lost the EV status already due to that, and some are assuming it's because this proposal has in fact already been implemented when it has not yet. All in all it's getting complicated and need to make sure we don't introduce more risks which are potentially bigger than the ones you are trying to mitigate.

This is a tough one to call. Obvously there has been a serious lose in trust in Symantec and, arguably, they should be subject to these and perhaps even harsher restrictions (a full revocation of their trusted status). Saying that I do think we need to look at the good it will do here. While no company should be considered "too big to fail" to ensure the integrity of HTTPS in the future does bring significant risks, if you break significant parts of the Internet and show too many HTTPS errors, then users will get used to ignoring them.

The figure of 30,000 certificates from Google has not been fully being explained either, and how likely it is that many of them are affected. Of course, some will say that the integrity of trust in HTTPS is at stake here, and this action (if not more!) needs to be taken, but I am trying to think with a practical head on here, and if it was the 127 certificates that Symantec claim, then disrupting hundreds of thousands or even millions of sites, does indeed seem overkill.

Google have aggressively pushed for improved standards in HTTPS more and more, often against the wishes of some site owners who were happy with the status quo without fully realising the security impact of that, and this is another example of this. Personally I've pretty much agreed with their stance previously, and applaud the work they do on this. However, it's a delicate balance and I do think they are perhaps doing more harm than good here. Then again, perhaps that is just because I help run one of the sites personally affected here.

With previous CA failures, browsers have restricted all future sites, or updated their revoked lists with affected certificates but have refrained from directly affecting production certificates unless absolutely necessarily, whereas this feels like we need to find a better way to restrict unaffected sites. This is being discussed on the proposal thread, but the problem is here we do not have a concrete list of sites affected, and the only people who could provide this is Symantec which obviously requires trusting them after they have failed to earn that trust during the investigation of this issue!

Perhaps the safest thing to do, is follow this proposal, despite the short-medium term pain this will bring?

Interestingly those two proposals are something Chrome has pushed a lot before so it's difficult not to see these restrictions with those thoughts in mind. Just this year, the same Google engineer that has proposed these changes, proposed restricting certificate lifetimes for all CAs to 12 months. This was rejected by the joint CA/Browser forum primarily because the CAs (quite rightly in my opinion) thought this was would be burdensome on their customers. Renewing certificates is often a very manual process, though CAs like Lets Encrypt have helped drive a push to automate this. This site has a job to auto renew certificates for example. However most CAs don't support this, and most companies would not be able to use them without considerable effort to implement this. Are these restrictions a way of pushing this through anyway? Saying that, the reasons they proposed shorter certificate lifetimes, are for very good reasons - as this reduces the effectiveness of mis-issued certificates and increases the security of their users when certificates are mis-issued like they have been here. So, in one way, it's unsurprising that their response to this particular incident is to revert back to their previous proposal.

On the other restriction, the Chrome security team (and especially this engineer!) have also stated their dislike of EV certificates in the past, arguing they provide no additional security and are unproved to be any use at all. They have stated their intention to remove EV completely. A lot of people in the security community agree with that statement, though I'm less certain and think, for the small extra price, the additional UX of EV is worth it. Either way I'm finding it difficult to reconcile the proposal's intention to remove EV, with previous statements that EV doesn't matter, and also that this is not punitive.

Of course Google own the Chrome browser and have the complete right to push their own agenda here if they want, but it does leave a bit of a question as to whether they are potentially forcing this on people, with all the disruption that causes, purely for their own ideological reasons rather than because they feel this is a necessary thing they must do to ensure the safety of their users for this particular incident as they claim. Personally I would prefer if there wasn't this conflict. The other browsers have not (yet!) proposed such restrictions nor stated whether they will follow Google's proposal if they go ahead with this so it will be interesting to see their views.

The Google Chrome security team (along with the other browsers and in particular the Mozilla security team) do phenomenal work and have really helped drive forward the use of safe HTTPS. Without them pushing security improvements, from turning off old, breakable, encryption cipher suites, to making many UI changes to explain the complex idea of a "Secure" site to users, the Internet would undoubtedly be a much more dangerous place for people to browse. The browsers also take their responsibilities to deliver a secure web as much as possible very seriously - arguably much more seriously than OS developers have in the past, or that some CAs do now.

However browser development teams also wield a huge amount of power, and must ensure they are not seen to be abusing that power. With over 50% of the market share, Google can easily be seen to be forcing their will on others. As long as this benefits users, this is a good thing, but there is always a risk when there are no easy answers and tough decisions have to be made. There is also the chance that Google drives users away from Chrome to other browsers that don't show these errors, if they get overly strict on security issues and end up making large proportions of the web usable in their browser.

One thing that the Chrome security team are to be applauded on, is the openness with which they are having this discussion, rather than making the decision unilaterally. Soliciting feedback from the community is a great way to decide if this is the right way forward, or if there are better ways of maintaining the security and integrity of HTTPS without causing harm. Then again, perhaps this public blog post, is intended as a wake up call for Symantec rather than a proposal which will really make it into the Chrome code?

I don't also want to seem like I'm putting the blame on Google here, for Symantec's failings. Symantec did not live up to their responsibilities here, and there are consequences to that. I'm just not sure this proposal is not going to cause more problems than it solves. Ultimately the title of a similar post on this topic covers it well: There Are No Winners in the Google/Symantec Feud.

Do you agree? Disagree? Let me know below your thoughts below.

This page was originally created on 30-Mar-2017 and last edited on 28-Jun-2022.