PSA: LetsEncrypt ending expiration notification emails
(letsencrypt.org)
from ikidd@lemmy.world to selfhosted@lemmy.world on 30 Jan 15:54
https://lemmy.world/post/24919511
from ikidd@lemmy.world to selfhosted@lemmy.world on 30 Jan 15:54
https://lemmy.world/post/24919511
I think it’s a good idea, everyone should be automating this anyway.
threaded - newest
Those emails have warned me something was pooched in advance many times. I do find them useful.
Sad to see them go, but nice they mention an alternative.
Pretty much all monitoring solutions on the market track cert expiration nowadays. I get an alert when any of my certs have <5 days left
What monitoring solution do you use? I need to set something up for my own projects but haven’t gotten around to it. Any experience with Nagios?
I use NewRelic myself. They are software agnostic and only connect to your URL to get the expiration date.
If you set up LE correctly, it should never get an alert. I haven’t been alerted since I set it up, to the point that I wonder if I set up the monitor correctly.
The only thing I wish it could do is use custom ports. I have some services running on non standard ports.
I set up uptime kuma to also monitor certs this week when I got the reminder email about them stopping the email warnings, been using it for some time for uptime monitoring (mostly to see if some auto docker image update screws up my services) and the notification parts has worked nicely for that, so I’m also assuming it will work nicely for the certificates
If you have the time to spare (a few weeks perhaps, if coming from zero) to experiment and read, Prometheus and Grafana offers a lot and can be really flexible. I use a pretty simple bash script that scrapes my desired https endpoints and writes out the results to a file Prometheus (node-exporter) understands, and from there I can write alert rules in Grafana to fire off notices by email or slack.
I’ve mainly gotten false positives, myself. When I’ve added another subdomain or something and the certificate gets set up differently, so then you get 2-3 emails saying domain X will expire, but if you connect to the url you see it has 80+ days left. Setting up your own monitoring solution is probably long overdue for myself, and it’s nice I’m getting forced to do it, in a way
Setup uptimekuma
I did setup UptimeKuma for notifications on this. let’s see if it works out when the expiry arrives in a month
I think I’ll need to add notifications for my uptime kuma as well now. So far I’ve used it mostly for historical data but without the mails, I would like to get a notice
UptimeKuma looks nice. Simple, but it does what it is supposed to.
Just needs an API and an export/import feature.
I just wish I wouldn’t have to renew certs so often.
If Apple gets their way, you’ll be renewing every month:
certera.com/…/apples-proposal-to-shorten-ssl-tls-…
Fuck Apple and Microshit
You’re not supposed to do it manually.
My server does it automatically, but I have few services I can’t make to read the certs from server storage, so I have to manually copy cert content. Especially Adguard Home for some reason refuses to read my certs.
Have the same problem. But symlinks or copying them via cron solved it for me.
Yes! yes | cp -Lrf /etc/letsencrypt/live/…domain…/*.pem /var/snap/adguard-home/current
You could use a reverse proxy to terminate tls, and take the tls off of ad guard itself.
Tell that to all the embedded device manufacturers… switches, appliances, nas, etc.
There’s a whole load of things that will have a massive administrative burden if the frequency is dropped.
Skill issue.
Have you tried to automate it?
Fullchain.pem works. Privkey doesn’t. I’ve tried chmod 777 (yes, I know, just testing) and still can’t access the file.
Whole path has to be accessible, not just the file itself. All dirs above the file need to have the executable bit set that affects the user accessing the file.
I know, but for some reason Adguard can read the fullchain, not privkey. Now it works.
Its done for better security
This is still not possible in all scenarios. For example, wildcard certificates for DNS providers with no API support.
Then swap you nameservers to a DNS provider that allows that?
There are a lot of embedded systems that do not offer API support to swap out certificates. Things like switches, dvr, nas devices, etc.
Honestly in rare situations that a device like that needs to be accessible from the wild Internet I think it’d be mad to expose it directly, especially if it’s not manageable as you suggest. At the very least, I’d be leaning on a reverse proxy.
That implies though I don’t want valid certificates in my environment. I still want to make sure even on my private network I’m using valid certs. A lot of security departments require that too even if the device isn’t public facing.
Is there a hard source with evidence that this is at all needed? Because there are a lot of things that “security departments” do that amount to security theater. Like forcing arbitrary password changes org wide.
Regardless of “hard evidence” it’s still the company policy. How well does it go over if you try to say “well acktuslly…” when it comes to password changes.
Well, it went over easy, but I also gained the authority to implement or toss such policies when I took my job LMAO
In any case, I was referring to the “my environment” part since it implied you had such authority and were just choosing to emulate policies of others, ofc I don’t mean to make decisions you don’t have the authority to. Hard evidence is hard evidence though, it does give you a leg to stand on should you propose such changes
Valid certificate is anything you trust. Any CA which you can trust is no more or less secure than the one you get from LE, so for the private network you can just happily sign your own certificates and just distribute the CA to your devices.
But then you have to distribute CAs to all the devices that will reach this service, and not all devices allow that.
True. And there’s also a ton of devices around which don’t trust LetsEncrypt either. There’s always edge cases. For example, take a bit older photocopier and it’s more than likely that it doesn’t trust on anything on this planet anymore and there’s no easy way to update CA lists even if the hardware itself is still perfectly functional.
That doesn’t mean that your self-signed CA, in itself, would be technically any less secure than the most expensive Verisign certificate you can find. And yes, there’s a ton of details and nuances here and there, but I’m not going to go trough every technical detail about how certificates work. I’m not an expert on that field by any stretch even if I do know a thing or two and there’s plenty of material online to dig deep into the topic if you want to.
I’m good. I know very well there are uses cases for a self signed cert. LE is still far more practical for 99% of use cases, even internally.
I’m with you, but that’s why I’m automating certificate expiry checking somewhere else (in my home assistant install to be exact).
How are those devices affected by having no notification anymore? The manual labor exists anyway.
Most network switches and devices have a web gui to switch them out. Those can be automated.
Mine just auto renews anyway
I think thats the case for most of us. But for some like myself, it does mean I have to do the monitoring myself now. I can’t complain it was a free service. But it did warn me about a renewal problem before the cert expired, so it was a useful service for me.
If you’re using Prometheus, Blackbox exporter checks cert expiration as well
I have my home assistant check and also my nagios, better safe then sorry
Not doubting them, but I don’t understand how that’s possible.
Storing the email addresses and expiration dates takes an irrelevant amount of storage space, even if they had billions of cutomers.
Sending the emails should also not cost thousands, even if a significant amount of customers regularly let their certificates expire (which hopefull isn’t the case).
So where are the tens of thousands of yearly costs coming from?
I just realized I have no idea who pays for Let’s Encrypt. I just run the server commands, automate it, and move on.
Let’s Encrypt is run by a non-profit (Internet Security Research Group), they list their major sponsors and funders on their website.
Notable mention of Mozilla being a Platinum sponsor.
Transactional email services are about $15 per 10,000 emails. I’ll round down to $10 to consider b2b deals and let’s just say it’s $10,000 per year. That would be like idk 84k emails a month.
Keep in mind this doesn’t consider the DB hosting and the processing of expiring emails and salaries, so yeah, I could see it.
Edit: before anyone yells at me. I can’t math.
Not yelling, but pointing out, to people who also dont math, that if we assume $10 per 10k emails (or $1 per 1k, for simpler math), that’d be $84 for 84000 emails in a month, so you need to add another 0 to the figure (ie 840k emails in a month)
If they send 2 emails per subdomain per year, that could easily be 10s of millions which would make the cost per email measured in thousandths of a cent. And I could see the number of subdomains being larger by a factor of 10, maybe more.
Another angle: someone with IT experience needs to manage the system that seems emails, and other engineers need to integrate other systems with the email reminder system. The time spent on engineering could easily add up to thousands per year, if not tens of thousands.
I’m guessing their figure is based on both running costs and engineering costs.
According to their stats page, Let’s Encrypt’s certificates are used by around 500M domains.
So sendgrid checking does 2.5M emails a month for $90/month, and if call them the Cadillac provider. More than that you have to contact sales, so I’m still wondering how it’s that expensive to them
As with all things email, they probably really wanted to make sure that the mails were delivered and thus were using a commercial MTA to ensure that.
I’d wager, even at 20 or 30 or 40k a year, that’s way less than it’d cost to host infra and have at least two if not three engineers available 24/7 to maintain critical infra.
Looking at my mail, over the years I’ve gotten a couple hundred email from them around certificates and expirations (and other things), and if you assume there’s a couple million sites using these certs, I could easily see how you’d end up in a situation where this could scale in cost very very slowly, until it’s suddenly a major drain.
I manage all my certs using Cert Warden which has a dashboard that displays the expiry date. It does lack alerting, so I use Uptime-kuma to monitor the expiry dates of the certs. So not a big loss for me.
TIL Cert Warden is a thing. Looks awesome!
\sigh
PSA: If you use Cloudflare to proxy, you can get a free decade long certificate and not worry about it for awhile.
Oh, look: the NSA dangling a carrot on a line.
Hey, if you wanna put your home server out there so the first person who gets pissy at you can DDoS you off the net until your ISP decides to cancel your service, that’s a perfectly acceptable decision to make for yourself.
Just use certbot and cron.
Dietpi has an automatic letsencrypt recert service which could probably be ported since its just a whiptail script