from nibbs@lemmy.zip to selfhosted@lemmy.world on 17 Aug 20:26
https://lemmy.zip/post/69867160
Hi there, I am hosting a service for mainly me and also some family members. I want it to become more.
Currently the chain just as follows: interwebz <-> UDM <-> npm <-> services. The services are mostly run in VMs or LXCs, npm runs in docker, accompanied by one Adguard home instance and another AGH instance in an LXC.
I now want to add f2b to this whole chain. I’ve installed f2b in the OS which runs docker (and therefore npm) (VM1) and also in the OS of my existing service (VM2). Both VMs use Ubuntu as their OS.
What I already figured out thru testing and reading is, that f2b in VM2 acts as expected when the jail is triggered, and sets the block. But nftables/ufw don’t block connections coming from external, as they are coming thru npm. f2b sees and uses the public IP for the checks/bans.
So now I am thinking how to solve that.
- Keep f2b on VM2 and let it set the bans on VM1 over SSH. That’s what some people do or so I’ve read.
- Use f2b on VM1, let it check logs on VM2 and then set the bans. That would also be possible over SSH, I’d guess. But I didn’t read up on it yet.
So, what’s your opinion? How do you fine folks handle that in your environment? Option 2 seems to be more elegant, as it would be “at the entry” and let’s me easily add additional services later. But I don’t know, if that’s a path worth digging in to.
threaded - newest
@nibbs
You have already found the core issue: a ban on VM2 is too late.
VM2 does see the real client address in the HTTP logs if NPM passes
X-Forwarded-Forcorrectly, but at the network layer every connection to VM2 still originates from NPM on VM1. Therefore an nftables/UFW rule on VM2 can only block NPM — which is not exactly the intended security feature.For your setup, I would put the actual enforcement at the ingress point:
Having Fail2Ban on VM2 execute remote firewall actions on VM1 via SSH can work, but it is basically building a small, brittle distributed ban system yourself: SSH keys, narrowly scoped sudo rules, reliable unban actions, error handling, and so on.
Alternative:
If you want detection from several VMs/services but enforcement centrally at NPM, CrowdSec is a more natural fit. Run agents where the relevant logs live, use a central LAPI, and run a bouncer at VM1/NPM. Then the components are designed to exchange decisions instead of hoping two independent Fail2Ban installations telepathically coordinate.
Also: do not use
set_real_ip_from 0.0.0.0/0just to makeX-Forwarded-Forwork. That turns a client-supplied header into an IP-spoofing API.#SelfHosting #ReverseProxy #Fail2Ban #CrowdSec #NginxProxyManager
Thank you very much for your input. I will look into the third option.
Thought of it, but did forget it over tinkering the last few days. :)
Based on a quick read it sounds like the issue will be that the bad actors won’t get dropped immediately. Meaning whatever servers and services are in front of the firewall that fail2ban configures, will still receive the performance hits. But your backend services will still be protected.
I have a server specifically set up as the ingress server. That runs fail2ban and Traefik and I have crowdsec set up with a Traefik bouncer plugin. So fail2ban runs in front of everything to catch the worst, most obviously bad stuff and drop the traffic very quickly and efficiently so it doesn’t get any further including things trying to access services other than the web services. Then crowdsec is focused on more complex threats as they flow through the reverse proxy. Seems to work the most efficiently for me, but I haven’t had a chance tondo any real analysis on how much improvement it might be over any other configuration.
Thanks for bringing it up.
Before I dove into that project, I was checking logs on npm and the service I am already exposing. It seems to me, both only get http GET requests which on npm all get a 404 response.
I don’t remember what the service does with those, but I will investigate later. As I didn’t panicked, I guess it wasn’t that bad.
There are basically no brute force attacks on the service yet and the admin account is only allowed to login from internal IPs.
One initial aspect to further harden the exposed systems were the logs in the UDM which show frequent blocked attempts of varying severity.
This may be security theater and I am not able to assess whether the UDM is a “good enough” protection of the most common attacks.
As I see it, at the moment the UDM already blocks most of the (more sophisticated) attacks and leaves only common requests for the services to handle.
Long story short, as I see it, the UDM has the role of ingress server in my setup.
I have a similar setup to you, but I run caddy instead of npm. I recently added a log analyzer to caddy and noticed a bunch of unauthorized hits I thought I could do something about..
I added a region based firewall rule for the port forward address on my UniFi gateway only allowing incoming traffic from my own country. I couldn’t do a global region block as I have some legitimate incoming traffic from the USA, so I went with a rule targeting traffic to the reverse proxy ip specifically. It still managed to let in some random bot traffic, so I also added a block region rule above the allow region rule (both to the reverse proxy) blocking countries like Russia, china, North Korea etc..
If you don’t need to allow most of the world, it is an easy fix to do this at your gateway (I note you have a udm)