from northernlights@lemmy.today to selfhosted@lemmy.world on 29 Nov 22:43
https://lemmy.today/post/42699672
Maybe it’s well known but I just came across journalctl-desktop-notification and I find it very useful so I thought I’d mention it. It’s basically a bash script that monitors systemd’s journal and pops up a notification when there are warnings or errors (or anything else you want to make it catch besides the default config).
What makes it so useful for the selfhoster is that it can monitor the journal on hosts your user has ssh access to with key authentication (set up in 2s with ‘ssh-copy-id’).
So case in point, this just popped up:
My reverse proxy can’t renew certs, that’s bad. For some reason netdata didn’t catch it, and the service didn’t trigger a system email that would have been forwarded to my smtp. Uptime kuma would have caught it when I would have had only a few days to fix it, but this caught it immediately, and I have 52 days to figure it out.
So you install that on your daily driver and you get these notifications on your desktop. They only have packages for Arch and Gentoo but the thing is just a batch script and a systemd unit. So to install anywhere you just download the “source”, extract it, cd to it, and run ‘sudo cp -r usr etc /’ which is exactly what the Arch package does (line 22).
Just a nifty little tool I wanted to share in case others haven’t heard of it.
Edit: I made .deb and .rpm packages so it’s a lot easier to install now :)
threaded - newest
If you want the next level up, set up a log collector like graylog or something.
I’m using netdata for now but for some reason it didn’t notify me of this one.
That seems like a flight of stairs up.
Hahaha nice. I have PTSD from teaching the interns how to search graylog and then locking up elastic with piped conditional searches.
Well yes, you take the stairs to the next level.
Gray lot requires a newer version of mongo. Mongo now requires a processor with the AVX instruction set; and my aging homelab is one gen before Sandy Bridge.
So basically no graylog for me because I ain’t got money to run that shit anymore let alone upgrade it
LGA1366? I’ve got a Xeon X5679 in mine. It was like an $80 upgrade from my i7-950 lol, and yeah no AVX
Wow! This sounds super handy! Thanks for sharing!
This is very cool but all the machines I would use this on are headless with no GUI installed. Womp womp for me.
i think the idea is install this on your PC, and it displays notifications from your headless servers journals over ssh
Oh I didn’t catch that part, that’s even better than how I understood it, thanks so much for clarifying!
This is cool but for self-hosting you probably want a more robust monitoring system capable of alerting at all times. Prometheus is what I use. It also gathers data over time and can monitor many machines.
This is great! Thank you for sharing
Assuming the uptime of your services are in any way important.
I’m not running a business here, I’ve got no big stakeholders. If something doesn’t work, at most me or someone close to me is affected. No one really cares if something is not available for a day.
I spent 0 minutes on monitoring and don’t intend to start now :D
This person fucks! 😅
Does your data matter ? There’s a data loss prevention risk and security.
If you don’t care about those either, then I guess your decision makes sense
There is still a good reason to know about problems early. Without any monitoring you will find out about problems exactly in that moment when you what to use the service that doesn’t work. Sometimes you need something quick and you don’t have time to debug and fix in that moment. If you get an alert early you can decide to fix it right away or in a few hours or tomorrow.
What do you need quick? I have a Minecraft server, a wiki for random stuff, a shopping list, a calendar sync, photo hosting, a media server and probably some other shit on there.
I can think of many situations where I’d want those quickly, but need I don’t anything.
Calendar sync? I learned that Nextcloud went to shit again when I missed an event added by my spouse to the shared calendar. Pretty important to me.
You don’t need something ever. Sometimes you just want something because the alternative is realy bad. I don’t need to eat. I want to eat because I don’t want to starve.
I want to watch a movie with my partner at the agrees time because otherwise they will be mad. I want to access my digitalized documents to send a letter in time because otherwise I will have to pay late fees. I want to access my gameserver because that’s the one time a week I get to have fun with my friends from my college time.
There are many situations where I’d rather do the thing I want instead of doing maintenance.
Yep and I have less time for all these things when I spend the time setting up monitoring.
Fixing takes the same time either way. But I barely ever have to touch my setup anyway, because usually ot doesn’t just break randomly.
I just discovered it a few weeks ago. But I did not know about the ssh part. Thanks.
Ha ha ha.
I love how lennart’s cancer tries to replicate fucking syslog and it’s this bad. What a mess the kids worship.
Please transcribe any images with text
Holy shit, this is amazing. Thank you so, so much for sharing this. I had not heard of it, and I am often stymied by journal CTL, since I don’t really know how it works. So I will most definitely be using this on my desktop and possibly on my self-hosted stuff as well. Thank you again.
I came across systemd-timer-notify, this is useful for me
oh neat tx