Standing on our own two feet

Author: gpff

Score: 401

Comments: 139

Date: 2020-11-06 16:21:30

Web Link

________________________________________________________________________________

gioele wrote at 2020-11-06 17:26:54:

Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership.

What I have never understood is why IdenTrust accepted to cross-sign Let’s Encrypt's root certificate.

With that move, IdenTrust basically broke the CA cartel and helped driving the price of basic certificates to zero. How did they, as a for-profit organization, justify "doing the right thing" when that meant disrupting their own market?

Anyway, kudos to IdenTrust.

dubcanada wrote at 2020-11-06 17:50:23:

IdenTrust doesn't care about random websites, they care about HIPPA, enterprise, government, securing documents and emails, etc.

closeparen wrote at 2020-11-06 19:49:49:

Does anyone in enterprise actually need publicly trusted certificates for documents and email? Seems like it's an inside-the-firewall Exchange server for internal traffic, and a white-label "secure messaging center" portal for external traffic.

zinekeller wrote at 2020-11-06 20:01:53:

IdenTrust's buisness also spans to managing private CAs for companies, which includes managing the HSM and private keys. Also, the companies who hire IdenTrust and similar companies are not that involved in technology. Also, security experts who can manage this safely is a tad harder to find and requests higher wages than your standard IT staff.

TLDR: yes, but some companies wants another company to manage their certs.

juliend2 wrote at 2020-11-06 18:03:50:

Exactly, but the funny thing is that Chrome and Firefox (Desktop at least) are no longer showing the differentiating green lock for those high-end (EV & OV) certs, but the same neutral-looking lock as those Domain-Validated (DV) certs that Let's Encrypt is issuing.

I'm very grateful for IdenTrust for having made that move. I just hope it won't hurt their business too much because of that.

andymatuschak wrote at 2020-11-06 22:14:04:

It seems like a case of “commoditize your complement” to me.

https://www.gwern.net/Complement

They offer a variety of enterprises services and can now capture more of the marginal consumer value relative to competitors still fighting for the cert issuing market.

admax88q wrote at 2020-11-06 17:33:38:

Always better to be doing the disruption rather be the one being disrupted.

Perhaps they saw the writing on the walls and wanted to boost their standing in the post LetsEncrypt world.

est31 wrote at 2020-11-06 17:44:14:

Yeah they likely got a stash of money for it while the other CAs got nothing. First mover advantage I guess. It's not that they'd have been able to prevent it anyways, as the root stores are not under their control, especially when Let's encrypt is being supported, even started, by the entities which run the root stores (Mozilla, through Firefox, and Google, through Android). Ultimately it's the entities which run and deploy the root stores who have ultimate say on which CA gets to issue certs and which doesn't.

alanfranz wrote at 2020-11-06 19:58:04:

Maybe IdenTrust will now offer an ACME compatible endpoint and offer signed, paid certs with their CA. Or another CA will.

I wonder whether IdenTrust imagined that a five year cross signed root ca would be too little a timespan to get wide adoption.

Btw... Wouldn't it be possible to just add a new root ca to android? Maybe an app could simplify delivery?

thayne wrote at 2020-11-07 04:21:26:

> Maybe an app could simplify delivery?

I'd be very surprised if an app without root privileges could install a new root certificate. If an app installed a malicious (or even just a poor quality) certificate, that would be a pretty big compromise to the OS.

What is strange to me though, is that it seems like the OS should have a mechanism to update the root certs independently of the OS itself. Then again, not updating root certs is a way to put an expiration date on a phone, forcing customers to buy more phones...

1337shadow wrote at 2020-11-07 02:18:22:

The article says firefox app comes with its own up to date certificates which they maintain outside of the os, so there's that solution apparently.

mamborambo wrote at 2020-11-07 05:47:52:

It appears the built-in time expiry period in the trust relationships between various organisations and clients will increasing become a weak point in the world in the future. Whether these moving parts are in trust only for three months, three years or more is not the issue, it is the fact that periodic resync and retrust needs to happen, means that it may result in a cascadable failure of our infrastructure in the future. Say if a nuclear disaster / earthquake take down one city, will it lead to paralysis of servers and services across the world? Perhaps it is time to start auditing the Internet and identify the weak points.

DanielDent wrote at 2020-11-06 19:57:12:

I think a fairly manageable path forward is underway which makes this a smaller issue than it first seems:

(1) Firefox already uses its own root store

(2) App developers can include additional roots in addition to the system root store:

https://developer.android.com/training/articles/security-con...

(3) Chrome is migrating to using it's own store: "Historically, Chrome has integrated with the Root Store provided by the platform on which it is running. Chrome is in the process of transitioning certificate verification to use a common implementation on all platforms where it's under application control, namely Android, Chrome OS, Linux, Windows, and macOS. Apple policies prevent the Chrome Root Store and verifier from being used on Chrome for iOS."

https://www.chromium.org/Home/chromium-security/root-ca-poli...

boris wrote at 2020-11-07 05:12:52:

I find the marketing spin in the title and the article unfortunate (standing on our own feet). The simple truth is they've decided getting another cross-signature not worth their trouble so now us users must expect some breakages the scale of which doesn't appear to be well understood.

Also, while the article focuses on the Android, my first thought was about all those outdated CentOS/Debian/etc boxes/VMs/containers that silently curl something from a cron job or such and that will all of a sudden stop working. So I expect a lot of infrastructure breakages.

gpff wrote at 2020-11-06 16:32:17:

Let’s Encrypt cross-signature with IdenTrust "DST Root X3" is ending on September 1, 2021 but 33.8% of Android devices are running versions under 7.1 which don't trust Let’s Encrypt new root certificate "ISRG Root X1"

brendanclune wrote at 2020-11-06 16:57:55:

Workaround is Firefox Mobile (because it ships with its own root certs), but that's a significant burden to place on the user.

est31 wrote at 2020-11-06 17:20:27:

Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices.

For those older devices, the only option is to install the new root certificate.

Anyways, there are billions of Android devices out there. 33% of those is a large number. You can't just tell all of them that they are wrong.

If this happens, people will move away from Let's encrypt in masses. They don't realize yet how self-harming this really is.

WorldMaker wrote at 2020-11-06 17:46:58:

Root certificate updates are a massive security issue. Blaming Let's Encrypt is blaming one of the canaries for the coal mine disaster. 33% of Android devices don't and can't get up to date root certificates is an impressive security crisis that grows worse by the year (look at the other root expirations and the crazy workarounds that for instance Netflix has been doing to still work on older Android devices). Shouldn't the blame squarely be on Google, the Android OEMs, and the phone carriers for allowing this disaster to happen in the first place?

I realize that is a tough message to get out to users and site owners are going to be in the cross-fire, but it seems better to try to work for solidarity in pointing fingers at the right direction and the right direction certainly isn't Let's Encrypt.

est31 wrote at 2020-11-06 18:29:20:

Yes the issue is really severe of most deployed Android devices not getting security updates, either at all, or the devices are used well beyond the update period.

But this is not up to Let's Encrypt to solve. They market themselves to build products for the mass market instead of small niches of the market, say, everyone who buys a new phone every year. But then they also have to treat their product like a mass market product, and if Android users still use older versions of the OS, then Let's Encrypt should adopt for that.

thayne wrote at 2020-11-07 04:33:36:

This problem isn't unique to Let's Encrypt. Let's Encrypt is not the only entity with certificates signed by DST Root X3. And as these devices get older, more root certificates will expire. What happens when all of them expire?

What is unique about Let's Encrypt, is they may have a harder time getting cross-signed by a CA that will still have a valid root cert on these devices for a significant amount of time, because, as has been pointed out in other comments, Let's Encrypt is disrupting the CA industry.

jacquesm wrote at 2020-11-06 20:28:05:

They will be happy to accept that blame, as long as you fork over your $ for a new device. Planned obsolescence can have many components, this is just one of them.

cpach wrote at 2020-11-06 18:00:07:

That makes me curious, what workarounds did Netflix employ?

WorldMaker wrote at 2020-11-06 18:13:24:

It was the BBC I was actually thinking about, and it's a part of this article (which also mentions this Let's Encrypt root change):

https://scotthelme.co.uk/impending-doom-root-ca-expiring-leg...

Previous HN discussion on that article:

https://news.ycombinator.com/item?id=23455463

Aissen wrote at 2020-11-06 18:06:08:

When you control the client, it's simple, you can do pretty much anything: embed your own HTTP stack, TLS stack, your QUIC stack, or simply your PKI, or subset of the webPKI.

kelnos wrote at 2020-11-07 04:27:35:

If you're running Android 5.0, you also haven't benefited from updates that _remove_ CAs that have since been shown to be untrustworthy. I think that's far worse than being unable to visit sites that use LetsEncrypt certificates.

It was really short-sighted of Google to make the system cert bundle something that can't be updated without a full OS update. There should be an OTA mechanism that allows it to be updated through the Play Store or through some other means that isn't reliant upon lazy device manufacturers.

thayne wrote at 2020-11-07 04:49:24:

To be fair they haven't gotten mant other security updates either. It's just a bad situation all around.

fuzxi wrote at 2020-11-06 21:22:44:

33% of devices, but per the article, only 1-5% of traffic on sites using LetsEncrypt. I don't see _any_ site owners moving to a different CA when this affects less than 5% of their visitors, who are likely the poorest fraction of their userbase , i.e. probably not many paying users.

merlyn wrote at 2020-11-06 18:54:56:

How many other root certificates are going to be expiring in the next 3 years that older Android won't have updates for as well? Probably more than 1.

notatoad wrote at 2020-11-06 18:24:51:

on androids older than 5, the browser is the old android browser instead of chrome. how many sites out there still test compatability with that?

even without having to click through security warnings, the web is horribly broken on old android devices. the overlap of sites using letsencrypt and sites that care about people using android <5 has got to be vanishingly small. this isn't going to cause a move away from letsencrypt.

toyg wrote at 2020-11-07 02:48:46:

I don’t think they have a choice. Reading between the lines, the CA who cross-signed their previous root doesn’t really want to continue doing so (or asked for a lot more money) because LE usage reached levels that are just too risky. I don’t blame them: a single bad actor found doing something particularly nefarious with a LE certificate might lose them the trust their business is literally built upon. I don’t see any other CA queueing up to help what is typically their commercial nemesis. And as they say, at some point they would have to do it anyway, might as well rip the plaster off.

t0astbread wrote at 2020-11-07 04:27:53:

Agreed, but what could someone do with a domain validation cert (as issued by LE) that would be so nefarious to damage IdenTrust's reputation?

TheKIngofBelAir wrote at 2020-11-06 19:29:25:

> Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices.

For those older devices, the only option is to install the new root certificate.

Microsoft Edge still gets updates on Android 4.4 KitKat

w3ll_w3ll_w3ll wrote at 2020-11-06 19:59:22:

I don't think Microsft Edge embeds its own root store on Android.

tacon wrote at 2020-11-06 22:12:21:

Is there enough incentive for Microsoft to add a root store to Edge by next September? How hard is it to make that addition?

jsjohnst wrote at 2020-11-07 01:03:24:

I may be wrong, but the effort to switch to your own root store is more doing it securely, than the difficulty of switching from system frameworks to your own SSL/HTTP transport layers. So to put another way, straight forward to do mediocre job, not as trivial to do a good or great job.

maxerickson wrote at 2020-11-06 17:26:34:

In the post, they advertise that they are going to continue to offer a service that provides certificates signed with the old one.

capableweb wrote at 2020-11-06 17:38:06:

That's not gonna help as the root will expire, so the devices stuck with the older root will still shows errors when trying to use them.

est31 wrote at 2020-11-06 17:36:23:

... which will work until September 2021.

veeti wrote at 2020-11-06 17:49:47:

You can still install the last version of Firefox to support Android 4.x.

baggy_trough wrote at 2020-11-06 17:43:15:

Not sure about that. Move away from Let's Encrypt to what? More likely, most smaller to medium sized sites will say forget those old Android guys.

est31 wrote at 2020-11-06 21:47:05:

I found at least Buypass offering a gratis ACME product "Buypass Go SSL". They have roots which are deployed at least since Android 4.1, which covers way more Android devices (according to the Android Studio statistics, >99%):

https://android.googlesource.com/platform/libcore/+/android-...

I'm not sure whether that particular root is being used for their Go SSL product. If so, Buypass might be a good alternative to migrate to.

From what I read though, they _do_ require an E-Mail address, so you've got to keep that in mind.

sroussey wrote at 2020-11-06 21:09:46:

Too cheap to update can be seen as too cheap to buy your product or service, so I can see this happening.

nyanpasu64 wrote at 2020-11-06 23:35:25:

Unfortunately, on my Moto G5 Plus (which is not an high-end device), I've found that Firefox on Android is slower than Chrome (specifically the Bromite fork). I think that Firefox may not be a good solution for low-spec or older phones running older Android versions.

stefan_ wrote at 2020-11-06 20:30:35:

A Workaround is not something that _the user_ does.

sbierwagen wrote at 2020-11-07 00:45:05:

Good. Those devices are unsafe, and should not be used.

mehrdadn wrote at 2020-11-07 00:52:12:

Would you like to buy new phones for their owners?

oh_sigh wrote at 2020-11-07 01:44:49:

The owners can just click through the prompt if they want...

ed25519FUUU wrote at 2020-11-07 04:08:40:

I think you might be ignorant with regards to how the rest of the world uses the internet.

tyingq wrote at 2020-11-06 22:48:24:

"https almost everywhere".

Thanks G!

jonathanoliver wrote at 2020-11-07 05:23:32:

I too have wondered why IdenTrust would want to do this. As has been mentioned in this thread, it appears they focus on enterprise-level customers, governments, medical, among others.

Generally speaking, the way that new competitors enter a given market is with low-cost options that are often inferior to established players. Then, as those entrants expand upmarket by offering better and improved products, the existing/established players abandon parts of their downmarket products to the new entrants. This cycle repeats until there's nowhere left for the established players to go. At that point, these upstarts can often replace the existing players and become the dominant ones.

I'm not sure if the above was a deliberate strategic move on the part of IdenTrust or not, but in cross-signing the Let's Encrypt certificate, it effectively killed off the potential for new players in the low-cost TLS/SSL certificate market because there's no margin in $0. [Citation needed] Further, because the purpose of Let's Encrypt is to serve the base level of the market [1] with no apparent desire (as per the parent organization which is effectively a non-profit/public-benefit organization) to expand upmarket. This move would appear to solidify (whether intentional or not) the position of larger players who cater to larger customers while keeping any potential newer players from disrupting the space.

Aside: I love Let's Encrypt and have about a dozen or so certificates issued through them that I am in charge of. They're awesome and kudos to their team for what they've been able to accomplish. When they first offered certificates, the 90-day validity period felt very restrictive. Now it feels great because the certificates are automatically rotated every 60 days per various automation tools and painful certificate renewals are very much a thing of the past for me.

[1]

https://letsencrypt.org/about/

miohtama wrote at 2020-11-06 20:34:28:

Somewhat related, but in other thread two weeks ago people complain Google has too much power over Android ecosystem:

https://news.ycombinator.com/item?id=24917918

Now, here people are suggesting Google should somehow update the old Androids.

Be damned one way or the other.

laksdjfkasljdf wrote at 2020-11-07 03:35:31:

Yes and Yes.

And both are reasonable and not excluding.

Google sells you a pocket computer with a locked down OS, not for your safety but to control the ability to run ads.

If they cared about user security, they would provide updates, no matter how "slow" (their excuse) the device gets.

If they didn't want full control to show ads (ads are downloaded by the GooglePlayServices, which is pretty much the kernel of all your android experience) then it would be trivial to install other android distributions like replicant.

hence, both a reasonable and google is evil. They want full control and do not care about (_your_) security updates.

maxmcd wrote at 2020-11-07 02:12:37:

People complaining is pretty constant, yeah. People even complain about complaining.

andyp-kw wrote at 2020-11-07 03:38:52:

True,

But we need Google to step up because we know that the manufacturers and carriers won't.

nevi-me wrote at 2020-11-06 17:46:32:

Could Google possibly be able (before were discuss willingness) to push an update to root certificate via Play Services?

I'd like to think that anyone not using Play Services (i.e. Android with no Play) is likely using a custom browser, and would heed a call to switch to Firefox.

The problem with some devices in Africa would be that many people will using older phone often don't have enough data for the big Play updates to succeed.

Teens often buy 10-100MB of data so they can use WhatsApp. (If you're from Southern Africa, and disagree with this, hit me up, you probably need to spend some time in a village ;) )

toast0 wrote at 2020-11-06 19:12:41:

No play services on an Android phone in the US probably implies willingness to tinker. No play services on an Android phone in China only implies it's an Android phone. In the developing world, it most likely implies a very low cost Android phone of Chinese origin.

Bundling things that need timely updates with the OS with no mechanism to update them individually is a design error. Things like root certificates, time zone databases, leap second information, and even TLS libraries need to be updated on a regular basis. These items should be distributed outside of the general upgrade process, even if the general upgrade process worked (which is clearly not the case). Alternatively, root certs and TLS libraries could be bundled with applications as needed. You could probably have a stable core x.509 library and cipher algorithms bundled with the OS, so that the application level TLS library can be kept small. You still need to get tzdb updates out though.

In an ideal world, large OS vendors could work with carriers to get this small set of updates zero-rated in exchange for making sure they are very small and background downloaded only at times of low network congestion.

sandworm101 wrote at 2020-11-06 19:38:20:

In an ideal world carriers wouldn't have a say in what software updates were installed on my phone. Comcast doesn't control the software on the computers it services. Why should Telus control what updates are made available for my phone?

freeone3000 wrote at 2020-11-06 20:30:22:

Because they're the ones who push updates over the cell network. Comcast absolutely controls what software you run on your modem. You can update "out of band" manually, at least on recent Android Pixel phones. Any other manufacturer could also make their updates public, but since installing the one not for your carrier band makes the phone unusable as a phone, it's not likely to be common.

ocdtrekkie wrote at 2020-11-06 19:45:46:

Because security and privacy is the compromise Google made for dominance: Letting OEMs and carriers do what they want is what sold them on Android.

I'd absolutely agree that this is a design error though: We'll be better off when Android is dead and gone as a platform.

young_unixer wrote at 2020-11-06 19:56:20:

Can't you buy hardware directly and then just put the carrier's SIM card?

floatboth wrote at 2020-11-06 19:04:51:

Starting with Android 10, Google can push updates to lots of system components (media codecs, android frameworks, tzdata, …)

https://android-developers.googleblog.com/2019/05/fresher-os...

But those on older versions are screwed. It's really a major fuck-up that Google didn't do this for root certs since the beginning of Android.

cassepipe wrote at 2020-11-06 18:43:34:

Actually I was surprised that there would be such an easy fix : Switching to Firefox. Which is apparently around 70MB, apparently affordable from what you wrote and definitely worth it if it allows you to unlock a chunk of the internet. So no need for an improbable and costly Play update.

qqii wrote at 2020-11-06 18:55:36:

That won't fix any other apps though will it? Anything that uses chrome webview for example.

trillic wrote at 2020-11-06 18:37:20:

How much does that much data typically cost??

DominikPeters wrote at 2020-11-07 03:12:25:

Here are some South Africa prices:

https://www.mtn.co.za/recharge/data

Valid for a day: 25MB for $0.32;

Valid for a week: 50MB for $0.64;

Valid for a month: 100MB for $1.28, 1GB for $6.35

kelnos wrote at 2020-11-07 04:29:43:

Does IdenTrust have another root cert that has a later expiration date, that is also included in the trust store of OSes farther back than 2016? If so, why can't LetsEncrypt ask them to start cross-signing with a different root cert?

(Obviously IdenTrust is under no obligation to do so, but since they've done this much, it's not a stretch to hope they'd do more, even if they want to charge for it.)

renewiltord wrote at 2020-11-06 18:16:29:

_The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. _

This one-third of Android devices only yields 5% of traffic? Interesting.

degenerate wrote at 2020-11-06 18:23:56:

I have a 2010 Android smartphone and it's _painfully slow_ to navigate the modern web, almost unbearable. Browsing news websites is simply not worth my time of waiting for the phone to download and process 22MB of JS, CSS, and graphics. Being on wifi makes no difference; it's the CPU choking to render all that cruft.

So yeah, 5% of traffic makes sense.

GoblinSlayer wrote at 2020-11-06 19:37:06:

In firefox (at least in desktop version) you can disable javascript and css. Not sure if they are still downloaded.

bzbarsky wrote at 2020-11-07 03:00:53:

For scripts: if script is disabled for a document, scripts are not downloaded. See

https://searchfox.org/mozilla-central/rev/a5d9abfda1e26b1207...

as of today

For stylesheets, I'm not certain how you're disabling them. Depending on how you do it, they may or may not get downloaded. The most common ways of disabling them result in them not being downloaded.

notatoad wrote at 2020-11-06 18:26:29:

there's a lot of old android phones out there that don't get used for anything other than making phone calls or sending texts.

halukakin wrote at 2020-11-07 00:32:21:

This also depends on the country. For instance in Poland android has 99% market share. So 1/3 Poland residents will experience issues.

t3rabytes wrote at 2020-11-07 01:07:11:

That’s a pretty bad assumption. Just because 99% of Polish folks use Android doesn’t mean that 33% of those phones are in the group that doesn’t have the cert. “will” is strong.

mafuy wrote at 2020-11-07 02:20:19:

That's right - it could also be even more :)

Havoc wrote at 2020-11-06 20:20:31:

Crappy expensive internet means it's not instinct to squeeze a quick browsing session into every free minute

toast0 wrote at 2020-11-06 19:21:44:

More interesting that % of traffic in this case would be unique users. But it's not surprising to me that people using Androids on older devices are using them to access the internet less; they're generally not as nice to use, but also if they were heavily used, they may have worn out by now.

tzs wrote at 2020-11-07 05:10:56:

Question: If your site certificate is signed by a chain of intermediate certificates leading to a root that your browser knows, when you browser checks all the signatures at time T does it require that all the certificates be valid at time T, or does it just require that each certificate was valid at the time it was used for signing?

vhiremath4 wrote at 2020-11-07 02:01:03:

The company I work at is in a high-growth phase and we are going to be expanding our global audience this coming year through various channels (SEO, performance marketing, sales, etc.). A 1-5% hit in potential customer traffic is not going to fly. Is my only option here to get off LetsEncrypt? A bit of a vent, but I would 100% had paid for a version of LetsEncrypt that supported their costs for the x-signature with IdenTrust, although I know and respect that would be off-brand for LE.

mafuy wrote at 2020-11-07 02:15:25:

You certainly can't go to IdenTrust now either. No matter what you do, you'll lose access to those old-phone-people eventually, and paying would only lengthen the time slightly. Just like you already lost access to people with even older android phones. Some of my family is still on androids as old as version 2.

CameronNemo wrote at 2020-11-07 03:37:51:

Is there any way you can measure how many of your users are on an unsupported Android version?

Aissen wrote at 2020-11-06 17:45:38:

This might also concern you if your app is running on older android devices, and you have a backend that uses letsencrypt certificates, while the app is relying on the system root CAs.

A fix for this could be to add Let's Encrypt's root CA to your app.

rkagerer wrote at 2020-11-07 04:53:01:

Are there no companies out there with very old root signatures that could be acquired? It seems to me like those sigs are valuable IP just like well-known domains (like example.com) - I wonder how much one is worth.

amluto wrote at 2020-11-07 04:16:06:

Someone could plausibly write a high-quality exploit for Android 7 that installs the ISRG root CA certificate.

/me runs

javajosh wrote at 2020-11-07 03:39:43:

My vote is for trusting Let's Encrypt, because SSL protects _my_ users (and myself, and each other) from MITM attacks! However, it is interesting to ask what "trust" means in this context, since I don't know the people behind Let's Encrypt, so I don't trust them in that way. What are we trusting the organizations associated with those certs to do, or not to do?

tothrowaway wrote at 2020-11-06 18:19:51:

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1.

Ouch. So anyone who wants to support older devices 3 months from now needs to add `--preferred-chain "DST Root CA X3"` to their certbot command. When that chain is completely retired in September next year, certbot appears to fallback on the default (even with that argument present).

nojvek wrote at 2020-11-07 03:17:18:

Good. Let’s Encrypt is giving a massive headstart. They’re also giving an easy escape hatch for those who can’t get things ready in time.

Good judgment.

lixtra wrote at 2020-11-06 18:12:35:

They propose to install Firefox to work around the root certificate problem on old android devices. But can’t you just manually install their root certificate on most phones?

lights0123 wrote at 2020-11-06 19:45:04:

It probably looks a lot less sketchy to your users to tell them to install a browser they've probably heard of and may have used in the past than install a root certificate. Plus, that doesn't fix the problem of other certificates expiring, only extends it.

regecks wrote at 2020-11-06 20:23:21:

>Plus, that doesn't fix the problem of other certificates expiring, only extends it.

Although I agree fully on the sketchiness part, installing the root is a fix.

ISRG Root X1 expires in 2035. Not one of these problematic Android devices will be online anymore.

iudqnolq wrote at 2020-11-06 21:58:56:

But to have the whole modern web work you'll need to install more than one root.

fuzxi wrote at 2020-11-06 21:28:28:

I'm not sure if it's the same on older versions, but on recent Android versions, that requires a rooted device.

Latty wrote at 2020-11-07 01:23:10:

Interestingly, it appears to be back in 11. You are right it wasn't possible for some set of versions, not sure how far you have to go back for it to be possible again.

They appear to have added it back with a big warning screen similar to what they do for VPNs and stuff telling users it could compromise them, which is reasonable. It was a pain you couldn't before.

Havoc wrote at 2020-11-06 20:19:30:

That seems reasonable. LE is doing good work & hardly their fault Android is a fragmented mess.

colinclerk wrote at 2020-11-06 20:16:02:

Does anyone have experiences with ZeroSSL? Caddy has been building in support so I think it could be a drop-in replacement for Caddy/CertMagic/ACMEx users.

mholt wrote at 2020-11-06 20:23:04:

ACMEz* ;)

Seconding regecks' comment.

We're gradually making ZeroSSL a default CA for Caddy.

(I am currently implementing multi-CA support into Caddy and CertMagic, so that Caddy will be able to use both Let's Encrypt and ZeroSSL for redundancy. It's the first server to support this!)

This is a good thing for the ecosystem.

mehrdadn wrote at 2020-11-07 01:09:16:

Hi! Could I ask a somewhat unrelated question about using Let's Encrypt with Caddy? I've been trying to help some folks (in education) get wildcard subdomain certificates to work on their Google Cloud machines via lego_deprecated's purported gcloud support in Caddy v2 (we've tried to follow the instructions and all), but we've been running into issues and it's been incredibly frustrating to figure out how to resolve them. I recall one of the errors we got was _"No TXT record found at _acme-challenge.subdomain.domain.tld"_, but it was hard to see all of them because most of the errors we'd see would be rate-limit errors. Things were so much easier and everything worked in Caddy v1, but ever since we upgraded to v2, we have no idea how to make it work with gcloud (the instructions haven't gotten it working for us), and there seems to be a lack of any working examples on the internet. Do you know if anyone has had success with gcloud at all? Would you have any guidance on how to proceed? Currently they're running on expired certificates and we have no idea how to renew them via Caddy, and it's not clear to me how to even do it out-of-band either.

mholt wrote at 2020-11-07 04:08:54:

To avoid Let's Encrypt rate limits, please use the staging endpoint, as documented:

-

https://letsencrypt.org/docs/staging-environment/

-

https://caddyserver.com/docs/automatic-https#testing

For more help, please ask on our forums! I don't use Google Cloud but it is more likely that somebody there does:

https://caddy.community

-- otherwise, time to roll up your sleeves and get to work, forge the answer for others, I suppose!

mehrdadn wrote at 2020-11-07 04:11:59:

Okay thanks!

yjftsjthsd-h wrote at 2020-11-06 21:42:18:

> We're gradually making ZeroSSL a default CA for Caddy.

As in, replacing LE as the default, or supplementing it? (And if the former, why?)

mholt wrote at 2020-11-07 04:06:33:

Note how I mentioned that Caddy will be the first server to support redundant ACME CAs, so we'll use both ZeroSSL, and Let's Encrypt for redundancy.

regecks wrote at 2020-11-06 20:19:00:

ZeroSSL is operated by Sectigo (Comodo). It works fine with Certbot and should work fine with any ACME client.

I did run into some random 503s occasionally, but that was pretty soon after it was launched. Maybe just a few teething issues.

wdb wrote at 2020-11-07 02:58:30:

What about iOS? No word about it in this article.

floo wrote at 2020-11-07 03:32:09:

According to apple [1] ISRG Root X1 is available all the way down to iOS 10.

So that would be every iPhone since the 5.

As far as marketshare goes iOS 13 and 12 make up 94% of devices [2].

So I am guessing its an insignificant amount of actual users.

[1]

https://support.apple.com/en-us/HT204132

[2]

https://developer.apple.com/support/app-store/

1over137 wrote at 2020-11-07 03:14:08:

Or macOS, or Windows, or the BSDs. Would be nice to see a table of which versions will have issues or not.

zinekeller wrote at 2020-11-07 04:12:09:

At least for Windows: if browser support is being considered, Firefox is the last browser that somewhat works (both IE and Chtome fails because of TLS 1.2) and it includes the ISRG root (because duh), and if an application still updates on XP there is a good chance that it uses OpenSSL or derivatives (which uses a different root set, likely Mozilla's).

Natively? Windows 7, assuming you have installed all updates before January. Microsoft can update the roots as they please (and historically issued updates up unto Windows 2000, so root updates are a demonstrable solved problem).

BSDs and Linuxes: (Usually) Uses Mozilla's trust list. Also updates separately from system updates, so unless you stick somehow with unsupported systems you already have this (and can be manually included into the trusted roots if you insist on using that outdated version).

macOS: bundled with the system updates (see caveat with Firefox independently managing roots). Here, I don't know what version is the oldest one with ISRG root certs.

cratermoon wrote at 2020-11-07 03:39:35:

The real story here is the state of Android and the devices it runs on.

paulpauper wrote at 2020-11-06 19:24:16:

i hate how google puts warnings on non-ssl sites. why doe a static page that has no forms need ssl? non-ssl worked fine for 20 years for webpages and google comes along and says noooo not good enough.

kibwen wrote at 2020-11-06 20:01:55:

Recently there was a browser zero-day observed in the wild that operated by MITM'ing HTTP connections and injecting the payload into the response. You're thinking of HTTPS as protecting what information you _send_, but it also protects what you receive; with an HTTP connection, anyone in the middle can make your browser receive anything they want.

upofadown wrote at 2020-11-07 00:43:01:

When things have gotten so bad that all you need to do to own a browser is to connect to it the last desperate measure is to try to keep all the badness away.

superdisk wrote at 2020-11-06 19:27:02:

The NSA can see what pages you read, and men in the middle can modify the page to insert malicious JS or ads or whatever without HTTPS.

kickscondor wrote at 2020-11-06 19:44:25:

Ok - next question then: why do browsers block self-signed certs? If Lets Encrypt now allows any domain to get a cert, what's the harm in a self-signed cert? Seems like a step up from plain HTTP.

closeparen wrote at 2020-11-06 19:54:37:

What you are trusting, when you trust a CA, is that it will only issue a certificate for a domain to someone it has verified as the owner. A self-signed cert could be from a man in the middle attacker.

An active MITM is a relatively more exotic threat than a passive observer. In the days when certificates cost money, there was a legitimate argument that you shouldn't need the whole elaborate active-MITM defense just to get protection against passive snooping. But now that you can get both for free... just use Let's Encrypt.

DivineEntropy wrote at 2020-11-06 21:51:15:

Small nit, verification is only meant to prove control over the domain, not ownership.

CGamesPlay wrote at 2020-11-07 02:11:14:

I feel like this is an oversight due to the path we took to arrive here. HTTP website: no warning, "insecure" mark in the URL bar. Trusted HTTPS website: no warning, "secure" mark in the URL bar. Untrusted HTTPS website: impossible to open in modern browsers by normal users. Really, a non-expired, otherwise-valid self-signed certificate should be treated the same as an HTTP website is. But since LetsEncrypt made self-signed certificates much less common, I doubt there's going to be energy put into improving the experience with them.

ocdtrekkie wrote at 2020-11-06 19:48:25:

The theory here is a self-signed cert could be from anyone (including the NSA) and you wouldn't know. Unless you explicitly trusted the certificate you were using, like enterprises do.

laughinghan wrote at 2020-11-06 20:04:04:

The man-in-the-middle can self-sign their own certs and present it to you as the site's own self-signed cert. Unless you have some way to verify self-signed certs out-of-band, they're useless.

josephg wrote at 2020-11-06 22:04:20:

They’re not useless. They stop passive adversaries, like the Australian government. They just don’t stop active adversaries.

Browser security warnings imply https > http > self signed https. The correct order of should be https > self signed https > http.

zhenyavinogrdov wrote at 2020-11-07 05:09:28:

Allowing untrusted certificates for https, and showing just a warning for them, would make it impossible for websites to ensure that their traffic is not intercepted

throwaway525142 wrote at 2020-11-07 02:24:44:

We're getting there, HTTP is going to be marked as insecure in the future as well. It's just the massive amount of HTTP sites that couldn't get marked as insecure before, due to the then resulting warning fatigue in users.

laughinghan wrote at 2020-11-07 01:28:15:

Interesting point. BTW I love your blog and your work on ShareJS :)

traceddd wrote at 2020-11-06 20:03:31:

To be fair, they probably can anyways, since the site willingly handed over their keys to Cloudflare, or rely on some Google CDN, host on EC2, etc.

It’s the second or third tier nation state actors that you can really make a difference with.

Lammy wrote at 2020-11-06 19:44:55:

The NSA can still see that just via the metadata of me making the connections necessary to load the encrypted page contents.

gsich wrote at 2020-11-06 22:27:58:

As a site owner that is somehow not my problem. It's not my fault that you have a shitty ISP. I can understand and support that argument.

I do TLS only for my stuff however.

inquirerofsorts wrote at 2020-11-06 22:52:29:

> It's not my fault that you have a shitty ISP.

Really don't think you're grasping this and lack an understanding of packet routing.

Please type:

      traceroute apache.org

It's not just your ISP that's potentially shitty, it's every single hop on that list. Even worse if the person is using an open wifi connection.

Apache will happily serve you an insecure connection, they are the 80th top site on the internet. This type of stuff should be called out by technologists and it's incredibly disappointing when its handwaved away.

poyu wrote at 2020-11-07 01:21:33:

Story time! Couple years ago I worked in a company in Asia. My boss went to China for work and bought some network equipment for the office: routers, access points etc. We didn't really need them, it's just so cheap he wanted to see how well it works. After setting them up and got it to work, we continue to use our devices as usual. But when we visit a company internal tool/dashboard page that we built ourselves, an ad banners starts to showed up on the page! We were baffled, is our server compromised? Are the computers we're using caught some malware? But it's happening on all of our devices, computers, phones, tablet. Then we start to suspect it's the new equipment that is injecting the ads! Following this, we also noticed it doesn't modify HTTPS sites and that was the last clue of the puzzle. We pulled all the new equipment and sworn to never trust anything from China that has a price tag that's too good to be true.

neurostimulant wrote at 2020-11-07 03:06:22:

ISP injecting scripts into unencrypted pages are pretty common, but a router injecting ads to all unencrypted traffics is a new low.

notriddle wrote at 2020-11-06 21:20:59:

A MITM can add a "Donate" button to news.ycombinator.com. The money doesn't actually go to Hacker News, though. It goes to the MITM.

redvenom wrote at 2020-11-07 02:41:35:

A lot of shared hosts support LetsEncrypt now. I actually installed them on mine and to my surprise I saw a massive reduction in spam attacks also.

legulere wrote at 2020-11-06 19:53:10:

Now that they’re standing on their own feet can they also offer S/MIME certificates for email encryption?

zero37z wrote at 2020-11-06 17:50:26:

now the corporates got a valid point, why you dont want to use lets encrypt?

still the 33% of the devices is a quite a large number to consider.

juliend2 wrote at 2020-11-06 18:11:00:

But like they said in the article, those 33% of Android phones represent "1-5% of the traffic" of the "large integrators" websites that LE communicated with.

stefan_ wrote at 2020-11-06 20:50:58:

Well that's an easy choice, lose 1-5% of traffic or pay $100 for a certificate from a vendor whose root doesn't expire next year?

fuzxi wrote at 2020-11-06 21:33:45:

1-5% of traffic that comes from people using devices that are at least 4 years old. Someone who can't or won't upgrade from a phone that still uses Android Marshmallow is probably not bringing in much revenue.

josephg wrote at 2020-11-06 22:09:26:

Exactly. The “easy choice” is to drop support for those users. I suspect most web developers won’t even notice.

It’s a pity - there’s no essential reason why old phones with new batteries should get worse over time. But software kills them in so many ways.

rsweeney21 wrote at 2020-11-06 18:58:59:

I don't understand the motivation for making this change now. Why not keep the universally accepted root certificate as the default chain? Why does Let's Encrypt need to switch to their own root certificate now, and cause thousands of websites to break on older devices?

I don't even control the certificate provisioning process. We use Heroku and Webflow. This is frustrating.

toast0 wrote at 2020-11-06 19:18:56:

Their cross cert expires in 10 months. Switching in January gives most people time to notice the problem while there's an easy temporary fix (switch to the soon to be expiring cross cert while you evaluate the root compatability of commercial CAs across the devices you support).

I imagine the cross cert cost a bunch of money, and they may not have the money to do that again.

laughinghan wrote at 2020-11-06 19:58:57:

Did you miss the part where the "universally accepted" root certificate is going to be universally rejected in 10 months?

tingletech wrote at 2020-11-06 19:18:53:

"the DST Root X3 root certificate that we relied on to get us off the ground is going to expire - on September 1, 2021."

onlinejk wrote at 2020-11-06 16:56:33:

Nice TL;DR, thx