I’m sorry, but has no-one heard of https://letsencrypt.org that issues certificates via API for free?
I would not be surprised if certificates at some point will be issued for each session.
I’m sorry, but have you ever needed to manage some certificates for a legacy system or something that isn’t just a simple public facing webserver?
Automation becomes complicated very quickly. And you don’t want to give DNS mutation access to all those systems to renew with DNS-01.
Ahh yes the: we can’t have self signed certificates for security reasons but also can’t open up the environment to the web, and we dont have our own CA server, trifecta.
Solution: awkward, manual, certificate import process from a 3rd party vendor.
Even if you have an internal CA, few appliances support this kind of automation. At best, they have an API, and you get to write that automation yourself for each appliance.
Knew a place where, for some devices, it was only available via a web interface. It was automated via WebDriver by a sysadmin that was losing his mind.
You can delegate to isolated nameservers with DNS-01, there’s no need to have control over the primary zone: https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation
Yes, and that is where we enter the complicated territories…
How complicated is it to have a CNAME? /s
If you think it’s just too easy but people are still discussing it, please entertain the notion that you may have oversimplified the situation in your assessment and that as assumptions become clarified you may yet soon understand a horror that apple can’t quite grok.
Did we read the same article? DNS-01 challenges require updates to DNS. This means you need an API for your DNS. This means you now have to worry about DNS permissions in your application cert workflow. We’ve just massively increased blast radius! Or you could do it manually but that’s already failed.
All of this is straightforward with infrastructure-as-code. While I don’t struggle with that, I’ve watched devs and sysadmins both stare blankly at this kind of thing for days at a time.
Updates to DNS, yes. Not necessarily to your primary zone. In other words, you don’t need access to the name servers for your highly privileged example.com zone, only the nameservers for inconsequential.example.com. With the challenge delegation you can easily narrow the scope by CNAMEing the relevant _acme-challenge enries in your primary domain once. This not only removes the need for the validator to modify your primary zone, but also scopes what subdomains it can validate, too. So the blast radius decreases.
I, too, maintain several devices that insist on having the certificates (and keys, yuck) being fed to them by hand. I automated it all, because I don’t see why a human should be in a loop of copying the secret material. Automaton is good.
It’s not the issuance that’s the headache, it’s the installation. There are more things that need valid certs than just webservers
Certbot is basically automatic, think mines on a cronjob now.
Who actually does this shit manually?
Any number of numerous appliances and hideously malformed business systems that don’t have ways to automate cert changes.
Not everyone gets to work in their simple little world of standards-following lab servers.
This has a lot of “I can use the bus perfectly fine for my needs, so we should outlaw cars” energy to it.
There are several systems, like firewalls , switches, routers, proprietary systems and so on that only has a manual process for updating, that can’t be easily automated.
With 45 day certs it will get that functionality real fast
Hah. Snake oil vendors will still sell snake oil, CEO will still be dazzled by fancy dinners and fast talking salesmen, and IT will still be tasked with keeping the crap running.
Yeah, this is like changing your password every 45 days.
AWS makes this impossible in a few places such as a fair number of ACM use-cases.
I think your cert-per-session idea is interesting. We’d need significant throughput and processing boosts to make that happen, probably at least on the order of 10X computing speeds and 10X transmission speeds across the board minimum. These operations are computationally intense and add data to the wire so, for example, a simple Lemmy server with hundreds of users slows to a crawl and a larger site eg Mastodon goes to dialup speeds or worse. You can test at home by trying to generate an x509 self-signed cert before connecting to a website every time.
Well AWS ACM already automates this, so if the renewal period gets shortened, I’m guessing that this will be updated to suit, unless I’m misunderstanding your point.
I hadn’t considered the CPU load, but that’s a fair point. I’m guessing that a suitable piece of code will utilise specialised hardware, or perhaps leverage the GPU or just in time SSL certificates will become a thing.
Does not work well for large corporate roll outs. Last I checked no way to auto-enroll on a NetScaler or an F5
I’m not familiar with either, but Duck Duck Go suggests:
https://www.citrix.com/blogs/2015/12/09/using-lets-encrypt-for-free-ssl-certs-with-netscaler/
https://techdrabble.com/citrix/netscaler/18-letsencrypt-automated-certificate-with-citrix-netscaler
https://www.f5.com/company/blog/nginx/using-free-ssltls-certificates-from-lets-encrypt-with-nginx
Part of this might be my general disdain towards sysadmins who don’t know the first thing about technology and security, but I can’t help but notice that article is weirdly biased:
Over the past couple of days, these unsung heroes who keep the internet up and running flocked to Reddit to bemoan their soon-to-be increasing workload.
Kind of weird to praise random Reddit users who might or might not actually sysadmins that much for not keeping up with the news, or put any kind of importance onto Reddit comments in the first place.
Personally, I’m much more partial to the opinions of actual security researchers and hope this passes. All publicly used services should use automated renewals with short lifespans. If this isn’t possible for internal devices some weird reason, that’s what private CAs are for.
I’m on the side of “automate it all and stop whining”, but I do think it’s important not to so readily dismiss the thoughts and opinions of those this directly affects in favour of the opinions of the security researchers pushing the change.
There are some legitimate issues with certain systems that aren’t easily automated today. The issue is with those systems needing to be modernised, but there isn’t a big push for that.
I’d be more concerned as well if this would be an over-night change, but I’d say that the rollout is slow and gradual enough that giving it more time would just lead to more procrastination instead, rather than finding solutions. Particularly for those following the news, which all sysadmins should, the reduction in certificate lifespan over time has been going on for a while now with a clear goal of automation becoming the only viable path forward.
I’ll also go out on a limb and make a guess that a not insignificant amount of people only think that their “special” case can’t be automated. I wouldn’t even be surprised if many of those could be solved by a bog-standard reverse-proxy setup.
Exactly. My “special case” took a little more care, but it works completely fine. Here’s my setup:
- TCP proxy at edge -> wireguard tunnel using SNI to route to the right service
- reverse proxy that handles all TLS for all services on its device (renewals and crypto)
- HTTP services behind a firewall that only communicate w/ proxy
I have my router configured to resolve DNS to #2, so I don’t need to hit the WAN to access local services over TLS, and it uses the exact same cert as WAN traffic and the browser is happy.
This is about as exotic as I can think of, and it still works just fine for TLS renewals, and it’s 100% automated. I do need to leave HTTP open (it only serves acme endpoints, so whatever), but I could also close that down and have the renewal process open that temporarily if needed.
The only special case I can think of is a device that rarely turns on, which is incredibly rare these days (you’d generally have an always-on gateway that uses self-signed certs or something for those devices that stay off).
Usually the systems that need to be modernized are working, so nobody wants to invest in a new system that may require retraining the people that may be impacted. Then there’s some systems with integrations that may also require replacing so the integrations can continue to work.
Even then, there’s always a good possibility that the automation fails, especially in the first few iterations of trying to sort out the kinks, and third party automation tools aren’t perfect either. That’s another tool to have to update and maintain once all is said and done.
I’m not trying to rail too hard against the changes, but the impact is especially felt by the people managing the systems, who’s most likely getting more work tacked on to their workload of putting out fires behind the scenes.
I’m not an “actual security researcher” but I was an “actual security officer” at a reeeeally large shop.
Yes, researchers are right. But they don’t dictate what else we have to let slide to allow time to work this constantly.
And neither are they on the hook for it.
They can be pedants, but they can’t do it blind.
Lame. 45 days? 10 days for DCV? How common are exploits involving old certificates anyway? And automated cert management is just another exploit target. Do they seriously think an attacker who pwns a server can’t keep the automatic renewals running?
The solution, according to Sectigo’s Chief Compliance Officer Tim Callan, is to automate certificate management — unsurprising considering the firm sells software that does just this.
Any post/article with the word “slammed” in it gets a downvote and a no-read from me. That word needs to disappear from journalism/forums/life/etc.
This is the one case where I’d make an exception. I read through the threads, it got particularly heated.
As someone who creates custom domain name applications, FUCK THEM WITH A PINEAPPLE SPIKY SIDE FIRST. This problem is on par with timezones for needless complexity and communication disasters. Companys and advertisers are now adding man in the middle certs for additional data collection/visibility. If the ciphers not cracked, changing the certs exposes significantly more failure, than letting one get a little stale.
Sysadmin used slam! It’s super effective!Why not just autorenew on a schedule?
I use Lets Encrypt, and my certs get renewed automatically without me thinking about it.
Mostly customer provided certs, high end clients make all kinds of stupid requests like the aforementioned man-in-the-middle chain sniffers, clients that refuse DNS validation, clients that require alternate domains to be updated regularly. Management is fine for mywebsite.com, but how are you solving an EV on the spoofed root prod domain, with an sso cert chain for lower environments on internal traffic that is originally provided by a client? And do you want the cs reps emailing each other your root cert and (mistakingly) the key? I’ve been given since SCARY keys by clueless support engineers. I don’t want to do this every 3 months.
Sounds like a change in company policy, because AFAIK, there’s no good reason for pretty much any of that.
Sounds like you don’t do contact negotiations, if someone will pay 2 million to appear on their root domain, you’ll sit down and figure it out for a couple hours.
Yes, I don’t, and I would honestly like to understand what use-case these customers are trying to solve. Because there’s a very good chance that they can get their preferred outcomes with a lot less manual work.
Unrelated to the topic, but I deal with a database storing timestamps.
In local time.
For systems all around the world.
You’ll see current entries timestamped 12:28 from eastern Europe followed by ones 6:28 from America and then another 11:28 from central Europe.
Without offset.
Ew. Just store UTC timestamps and do optional translation on the client using whatever the client sets up for their timezone. It’s not hard…
Oh believe me, I would change some things about that database if I could. Alas, I’m just the analyst building data models from it.
(To be fair, it’s otherwise easy to work with and for most use-cases, it doesn’t matter since they’re aggregated per month anyway, so I just load the last month’s data on the 2nd of each month. I definitely have worse patients to operate on.)
The Register is deliberately tabloid-like in style (right up to the “red top” site banner), but is good quality (at least when I read it).
They won’t write an article about science without using the word “boffins” either. It’s just their thing.
If approved, it will affect all Safari certificates, which follows a similar push by Google, that plans to reduce the max-validity period on Chrome for these digital trust files down to 90 days.
Max lifespans of certs have been gradually decreasing over the years in an ongoing effort to boost internet security. Prior to 2011, they could last up to about eight years. As of 2020, it’s about 13 months.
Apple’s proposal would shorten the max certificate lifespan to 200 days after September 2025, then down to 100 days a year later and 45 days after April 2027. The ballot measure also reduces domain control validation (DCV), phasing that down to 10 days after September 2027.
And while it’s generally agreed that shorter lifespans improve internet security overall — longer certificate terms mean criminals have more time to exploit vulnerabilities and old website certificates — the burden of managing these expired certs will fall squarely on the shoulders of systems administrators.
Over the past couple of days, these unsung heroes who keep the internet up and running flocked to Reddit to bemoan their soon-to-be increasing workload. As one noted, while the proposal “may not pass the CABF ballot, but then Google or Apple will just make it policy anyway…”
…
However, as another sysadmin pointed out, automation isn’t always the answer. “I’ve got network appliances that require SSL certs and can’t be automated,” they wrote. “Some of them work with systems that only support public CAs.”
Another added: “This is somewhat nightmarish. I have about 20 appliance like services that have no support for automation. Almost everything in my environment is automated to the extent that is practical. SSL renewal is the lone achilles heel that I have to deal with once every 365 days.”
Until next year, anyway.
spending $300 every 90 days instead of 365 days is so much better /s
i hate apple so much
Who is buying SSL certs for $300? Is this an enterprise thing? I’m using free certs on AWS. LetsEncrypt is also fine for self-hosting.
It is an enterprise thing, yes.
$300 sounds ok for an enterprise thing
It’s more of an issue when it’s every 90 days. Even worse is the labor cost to replace the certificate on everything that needs it every 90 days.
Are these genuinely being hand rolled in an enterprise environment? Unless it’s completely impossible to automate then I can’t be sympathetic to companies that are just doing it wrong.
There’s lots of equipment that can’t accept certificates automatically. If they can, it might be in a closed off way that’s difficult to impossible to reverse engineer. If you can, that’s still a lot of skill and labor, which drives up the cost. They also might find out that it would be insecure to do it automatically.
It’s way more than 300 if you want all the bells and whistles and many SANs even
I was in a meeting before the summer discussing this with Digicert we asked if you would need to pay every 90 days.
They answered that certs will still be bought at 1, 2, or 3 year intervals but can be renewed for free every 90 days.
It’s pretty obvious when you think about it really.
Smells like Apple knows something but can’t say anything. What reason would they want lifespans cut so short other than they know of an attack vector that means more than 10 days isn’t safe?
AFAIK they’re not a CA that sells certs so this can’t be some money making scheme. And they’ll be very aware how unpopular 10 day lifespans would be to services that suck and require manual download and upload every time you renew.
Smells like you didn’t read the article, it’s an ongoing trend:
Max lifespans of certs have been gradually decreasing over the years in an ongoing effort to boost internet security. Prior to 2011, they could last up to about eight years. As of 2020, it’s about 13 months.
Reducing it to one year made sense, one year down to 10 days is actually a fucking massive difference. Practically speaking, it’s a far, far bigger change than 8 years down to 1.
This isn’t just an “ongoing trend” at this point, it would be a fundamental change to the way that certificates are managed i.e. making it impossible to handle renewals manually for any decently sized business.
They never said the ongoing trend wasn’t logarithmic. By 2030 you’ll be updating certs 6-8 times a day! Please drink verification can.
Thank you for the smug response however I did indeed read the article and going from 13 months to 10 days is not a trend but a complete rearchitecture of how certificates are managed.
You have no idea how many orgs have to do this manually as their systems won’t enable it to be automated. Following a KBA once a year is fine for most (yet they still forget and websites break for a few days; this literally happened to NVD of all things a few weeks ago).
This change is a 36x increase in effort with no consideration for those who can’t renew and apply certs programmatically / through automation.
This change is a 36x increase in effort with no consideration for those who can’t renew and apply certs programmatically / through automation
Don’t worry. All that old gear is at least 45 days old - so old - and isn’t an apple product anyway probably. Ergo, support isn’t their issue and you will have to take that up with your OEM because la-la-la-laaaaa, can’t hear you. Wanna go ride bikes?
I did indeed read the article
Smells like Apple knows something but can’t say anything.
Then do explain your conspiracy theory. Sectigo could go for a money grab, otherwise… probably just forcing automation without thinking of impact, as usual.
Automated certificate lifecycle management is going to be the norm for businesses moving forward.
This seems counter-intuitive to the goal of “improving internet security”. Automation is a double-edged sword. Convenient, sure, but also an attack vector, one where malicious activity is less likely to be noticed, because actual people aren’t involved in tbe process, anymore.
We’ve got ample evidence of this kinda thing with passwords: increasing complexity requirements and lifetime requirements improves security, only up to a point. Push it too far, and it actually ends up DECREASING security, because it encourages bad practices to get around the increased burden of implementation.
Just going to mention my zero-dependency ACME (Let’s Encrypt) library: https://github.com/clshortfuse/acmejs
It runs on Chrome, Safari, FireFox, Deno, and NodeJS.
I use it to spin up my wildcard and HTTP certificates. I’ve personally automated it by having the certificate upload to S3 buckets and AWS Certificates. I wrote a helper for Name.com for DNS validation. For HTTP validation, I use HTTP PUT.
Why have this run in the browser? Why not just have it run on the server and renew in the background?
That’s what NodeJS and Deno are.
The point of the browser support means it runs on modern Web technologies and doesn’t need external binaries (eg: OpenSSL). It can literally run on any JS, even a browser.
I’m aware, but you led with Chrome, Safari, and Firefox, so it sounded like browser support was the point, so I was curious what the use-case was.
That’s still cool though. I personally would’ve just use Python, since that’s generally available everywhere I’d want to run something like this (though Python’s built-in HTTP lib isn’t nearly as nice as JS’s
fetch(), I’d wantrequests).I have just dumped code into a Chrome console and saved a cert while in a pinch. It’s not best practices of course, but when you need something fast for one-time use, it’s nice to have something immediately available.
You could make your own webpage that works in the browser (no backend) and make a cert. I haven’t published anything publicly because you really shouldn’t dump private keys in unknown websites, but nothing is stopping you from making your own.
Any machine I’m on has Python installed, so that’s my go-to.
time to shine for DANE (actually no since the world sucks)
This’ll never happen. The rest of the computing world will just say “nah, get fucked”
Slams!
In this case slams means “slightly irritated by”
Sounds like free money for all those certificate authorities out there. Imma start my own CA with blackjack and hookers.
Or… They do what they did last time the lifetime was cut down from 3-10 years down to 395 days… Just issue you a new certificate when the old one runs out and up to whatever the time period you bought it for…?
Let’s Encrypt isn’t the only CA to use ACME, you can auto renew with basically any CA that implemented it (spoiler: most of them have)
Good, certificates should be automated anyways. Much more reliable than the once yearly outages because nobody renewed the thing or forgot some systems.
Good, certificates should be automated anyways.
The problem being when that can’t be easily automated? Did you read the article?
Good incentive for the provider to fix it or go out of business.
They should be automated too.
The fact that I can’t use terraform to automatically deploy certs to network appliances is a problem.
Technically, you shouldn’t even deploy certs to network appliances or servers but they should fetch certificates automatically from a vault. I know there’s minimal support for such things right now from some vendors, but that should be fixed by those vendors.
Even Microsoft supports such solutions in Azure both with PaaS components and Windows and Linux servers (in Azure or onprem) via extensions
True.
cert-manager is an amazing tool for deploying certificates for containerized applications. There’s no standardized way to deploy those certs outside of containers without scripting it yourself though, unfortunately.
Ugh. Righteous ideas about how things should work don’t change the fact that these network appliances doing it the wrong way still have years of time left before the bean counters consider them depreciated and let us replace them. Or that we’re locked into a multi-year contract with this business system that requires updating certs through a web UI.
Yes, there are almost always workarounds and ways to still automate it in the end, but then it’s a matter of effort vs stability vs time savings.
I love automating manual sysadmin actions, it’s my primary role on my team. Still, ignoring the complications that will unavoidably arise in trying automating this for every unique setup is incredibly foolish.
By this logic, we should still be using copper phone lines, analog TV, and 3G should never get switched off. Obviously there are always budget constraints but technological progress does not wait for shitty vendors.
I work mainly in cloud and Kubernetes environments where this stuff is already automated. New vendors are often just deploying new containers into a cluster.
Oh yes, let me just contact the manufacturer for this appliance and ask them to update it to support automated certificate renewa–
What’s that? “Device is end of life and will not receive further feature updates?” Okay, let me ask my boss if I can replace i–
What? “Equipment is working fine and there is no room in the budget for a replacement?” Okay, then let me see if I can find a workaround with existing equipme–
Huh? “Requested feature requires updating subscription to include advanced management capabilities?” Oh, fuck off…
