Just learned how to do a reverse proxy
from randombullet@programming.dev to selfhosted@lemmy.world on 27 Jan 22:23
https://programming.dev/post/24618997
from randombullet@programming.dev to selfhosted@lemmy.world on 27 Jan 22:23
https://programming.dev/post/24618997
Just exposed Immich via a remote and reverse proxy using Caddy and tailscale tunnel. I’m securing Immich using OAuth.
I don’t have very nerdy friends so not many people appreciate this.
threaded - newest
Congratulations!
It feels really good when you learn something new and get it working the way you like.
If you want more challenges take a look at this:
Immich-public-proxy
This would be useful if you ever wanted to share albums with other people outside your tailscale network and that lack an account for your immich server.
I’m a huge fan of Caddy and I wish more people would try it. The utter simplicity of the config file is breathtaking when you compare it with Apache or Nginx. Stuff that takes twenty or thirty lines in other webservers becomes just one in Caddy.
I moved from swag to caddy and I’m glad i did. So much more simple.
I love Caddy. So easy to configure, and the automatic SSL is almost always what I need.
The only thing I don’t like about caddy is that using DNS challenge requires recompiling the program itself, and the plugins themselves can be a bit quirky. Mind you, you can easily handle this with a separate program like
legoorcertbotso not a huge deal.Wrapping my head around reverse proxy was a game changer for me. I could finally host things that are usefull outside my LAN. I use Nginx-Proxy-Manager which makes the config simple for lazy’s like me.
NPM is awesome until you have a weird error that the web GUI does not give a hint about the problem. Used it for years at this point and wouldn’t consider anything else at this point. It just works and is super simple.
+1 for NPM! Used to even do things manually, but I’m too lazy for that and NPM fulfils nearly all my use cases lol
came here to leave this exact response! 😁
Do you serve things to a public? Like a website? Because unless you’re serving a public, that’s dumb to do… and you really don’t understand the purpose of it.
If all you wanted was the ability to access services remotely, then you should have just created a WireGuard tunnel and set your phone/laptop/whatever to auto connect through it as soon as you drop your home Wifi.
A lemmy instance, a wiki, and a couple of other website type things, yes.
Publicly facing things are pretty limited, but it’s still super handy inside the LAN with Adguard Home doing DNS rewrites to point it to the reverse proxy.
I appreciate what you’re saying, though. A lot of people get in trouble by having things like Radarr etc. open to the internet through their reverse proxy.
Am I making a mistake by having my Jellyfin server proxied through nginx? The other service I set up did need to be public so I just copied the same thing when I set up Jellyfin but is that a liability even with a password to access?
Not really. Personally I’d allow the service account running jellyfin only access to read media files to avoid accidental deletion but otherwise no.
Also, jellyfin docs have a sample proxy config. You should use that. It’s a bit more in depth than a normal proxy config.
This is very short sighted. I can think of dozens of things to put on the open internet that aren’t inherently public. The majority are things for sharing with multiple people you want to have logins for. As long as the exposed endpoints are secure, there’s no inherent problem.
And yet you’ve not provided one example, hmmmm
Seriously?
Plex, Jellyfin, VaultWarden, AdGuard, Home Assistant, GameVault, any flavor of pastebin, any flavor of wiki, and the list goes on.
If you’re feeling spicy throw whatever the hell you want onto a reverse proxy and put it behind a zero trust login.
The idea that opening up anything at all through to the open internet is “dumb” is antiquated. Are there likely concerns that need to be addressed? Absolutely. But don’t make blanket statements about virtually nothing belonging on the open internet.
Used to mess around with multiple Apache Proxy Servers. When I left that job I found Docker and (amongst other things) NPM and I swear, I stared at the screen in disbelief on how easy the setup and config was. All that time we wasted on Apache, the issues, the upgrades, the nightmare in setting it all up…
If I were to do that job again I would not hesitate to use NPM 100% and stop wasting my time with that Apache Proxy mess.
Good job!
I'm still trying to understand what it is and why I would want it. I see several programs I use recommend it but I just don't get what it does and why what it does is good.
It does a couple things. It’s one service that routes requests to multiple services. So if you have radarr, sonarr, etc., you can put a reverse proxy in front and use the same ip-port to connect to all, and the proxy routes the request to the service by hostname.
If you have multiple instances of the same service for HA, it can load balance between them (though this is unlikely for a homelab).
Personally I run all my services through docker and put traefik in front, so that I don’t have to keep track of ports. It’s all by name.
It’s also nice because traefik handles HTTPS termination, so it automatically gets certs for each name, and the backing service never needs to worry about it (it’s http on the backend, but all that traffic is internal).
Thank you for the explanation. But that's it than? Just convenience with ports?
Well it IS pretty nice to be able to tell people to go to jellyfin.example.com instead of example.com:8096, but you also get security benefits for using a properly set up reverse proxy. You don’t need to keep your ports open to the whole internet, only the reverse proxy accesses them. As far as the rest of the internet is concerned, you have :443 open.
Edit: Forgot to add, Caddy and NPM and such can also automatically renew your certificates!
I know that feeling ! My first service hosted via docker + Treafik outside my lan with a wireguard tunnel felt like a big dopamine hit ! Congrats !
Now I have over 20 services and It feels trivial :( I still love the easy to read/write syntax of Treafik ,however I feel like I’m missing a lot of important networking knowledge while avoiding Nginx !
Maybe one day when I’m too bored I will switch everything to Nginx, see how it goes !
me too like last week!!! yay us!!
haven’t gotten oauth going yet but soon
Just be sure to read up on network security and set yourself up for success! Even tunnels can still be an attack surface. Always keep everything up to date! And plan for the worst case.
Can someone ELI5? I’m a noob who aspires to set up immich in the near future. I only recently started making efforts to separate myself from the cloud. So far I’ve got a wireguard server set up and I’ve disconnected both my Bambu printers from the cloud and I’m currently setting up some home assistant stuff. Pretty soon I’m hoping to set up a NAS, Immich, Plex (or similar) and replace my google nest cameras.
Pretty much I have caddy on a VPS that’s pointing to my internal IP using a tailscale tunnel. You are still exposing the web gui to the Internet so I just changed authentication to OAuth to mitigate since risk. There is still a possibility of attacks via zero days, but my immich is on a VM and I’m creating firewall rules to just allow certain ports out.
I appreciate the extra details but I still don’t know what “caddy”, “VPS”, “tailscale tunnel”, or “zero days” are, but I can look it up.
It’s hard to explain from scratch.
Caddy is a reverse proxy software that essentially redirects traffic from a certain port to another port. For example external:port => internal:port. It also enables SSL encryption meaning everything will be encrypted en route between the external and the user.
VPS is a virtual private server. Just someone else’s computer you can expose to the Internet.
Tailscale is a mesh VPN that uses wire guard as its transport. I use this to tunnel between my VPS and my Immich server to hide my home IP and to allow encrypted traffic between my Immich server and my VPS.
A zero-day (also known as a 0-day) is a vulnerability in software or hardware that is typically unknown to the vendor and for which no patch or other fix is available. The vendor thus has zero days to prepare a patch, as the vulnerability has already been described or exploited.
There’s no fix other than security through layers.
That actually helps a lot, thanks!
I’ll try to ELI5, if there’s something you don’t understand ask me.
Op has a home server where he’s running immich, that’s only accessible when he’s at home via the IP, so something like 192.168.0.3:3000, so he installed Tailscale on that server. Tailscale is a VPN (Virtual Private Network) that allows you to connect to your stuff remotely, it’s a nice way to do it because it is P2P (peer-to-peer) which means that in theory only he can access that network, whereas if he were using one of the many VPNs people use for other reasons, other people on the same VPN could access his server.
Ok, so now he can access his immich instance away from home, all he has to do is connect to the VPN on his phone or laptop and he’ll be able to access it with something like http://my_server:3000 since Tailscale adds a DNS (Domain Name System) which resolves the hostnames to whatever IP they have on the Tailscale network.
But if you want to give your family access it’s hard to explain to them that they need to connect to this VPN, so he rented a VPS (Virtual Private Server) on some company like DigitalOcean or Vultr and connected that machine to the Tailscale network. He probably also got a domain name from somewhere like namecheap, and pointed that domain name to his VPS. Só now he can access his VPS by using
ssh user@myserver.com. Now all he needs to do is have something on the VPS which redirects everything that comes to a certain address into the Tailscale machine, Caddy is a nice way to do this, but the more traditional approach is ngnix, so if he puts Caddy on that VPS a config like this:immich.myserver.com { handle { reverse_proxy my_server.tailscale.network.name:3000 } }Then any requests that come to immich.myserver.com will get redirected to the home server via Tailscale.
It is a really nice setup, plus OP also added authentication and some other stuff to make it a bit more secure against attacks directly on immich.
Quick, now lean a firewall with a good IDS
and fail2ban
I prefer wazuh. Much more powerful and preconfigured with tons of rules
Nice work! 😎
Wow, so my understanding of the terms ‘reverse proxy’ and Tailscale must be wrong then, because I thought they were mutually exclusive. I’ll go do some more research, unless someone feels like explaining how you can do both at the same time.
Also, I think the ‘Risks’ section of this page is informative:
en.m.wikipedia.org/wiki/Reverse_proxy
I think self hosting the proxy with the services at hobbyist scale mitigates most of the security risks. The single point of failure risk is another matter. I once had to effectively reverse-hack my services by uploading a Jenkins test job through an existing java project to regain access. Ever since then, I maintain a separate ddns address that’s just used for emergency ssh access.
Congrats! I just pulled off the same thing last week using cloudflare tunneling? The phrase “reverse proxy” scared me too much lol. So props to you.
Same boat (in the learning cycle that is). No idea what immich is, but I got Stirling-PDF hosting in docker. I only learned the other day that localhost, is localhost for the container. I couldn’t get a bunch of stuff running for.ever, till I learned the way I was calling things needed to be to host.docker.internal.
Tailscale?
Is this setup advisable for the CGNATED environment?
You will need a VPS as your other endpoint
Ah, I figured… I used to do this with Wireguard instead of Tailscale.
This is necessary for CGNat ISPs. That or cloudflared or ngrok or the like. Because you aren’t really routable on a CGNAT address.
In a nutshell, CGNAT users must spend money for something that people with IPv4 addresses can do for free 😔
We wouldn’t be in this mess if we switched to ipv6, but nOoOooOo… we can’t possibly do that…
Lack of routability is a feature for ISPs, not a bug.
I just finally got it this weekend when I got Matrix-synapse and Pixelfed working on the same box.
All I can say is good for you! It wasn’t easy. And it’s so powerful.
"I just finally got it this weekend when I got Matrix-synapse and Pixelfed working on the same box.
All I can say is good for you! It wasn’t easy. And it’s so powerful. "
Mind if I ask what kind of set up you used. I recently spun up a Jellyfin server and it’s working inside my network and just started dabbling with tailscale trying to see if I could figure out a way to make it accessible while the media server is connected to my VPN. (That way I could figure out a way to add media without having to drop the connection to the VPN on the local drives hopefully)
I just got this set up last week too. Same setup with caddy on a free oracle vps, tailscale on vps and home pfsense router, tailscale on pfsense advertising routes (private IPs of my docker hosted services).
CGNAT sucks 🤮
Just out of curiosity, is the tail scale part of this required? If i just reverse proxy things and have them only protected from there by the login screen of the app being shown, that’s obviously less safe. But the attackers would still need to brute force my passwords to get any access? If they did, then they could do nasty things within the app, but limited to that app. Are there other vulnerabilities I’m not thinking about?
I don’t think a tailscale tunnel helps this anyway, maybe just from standard antispoofing and geoblocks, but it still gets to the application in full eventually, when they can do what they’d do if it was directly exposed. The attack surface might be an entire API, not just your login screen. You have no idea what that first page implements that could be used to gain access. And they could request another page that has an entirely different surface.
If someone has Nextcloud exposed, I’m not stopping at the /login page that comes up by default and hitting it with a rainbow table; I’m requesting remote.php where all the access goodies are. That has a huge surface that bypasses the login screen entirely, might not be rate limited, and maybe there’s something in webdav that’s vulnerable enough that I don’t need a correct token, I just need to confuse remote.php into letting me try to pop it.
You can improve this by putting a basic auth challenge at least in front of the applications webpage. That would drastically reduce the potential endpoints.
Thanks for the insight! Does running this in a docker container help limit the damage at all? Seems like they’d only be able to access the few folders I have the container access to?
Maybe a bit, but if you’re not running rootless docker if they get out of that container they’ll have the run of your docker host. It is a lot of layers to crack, but sometimes they’ve got nothing but time, or it’s been so long since the containers been updated that its trivial. That’s why rootless docker or podman, and Watchtower are your friends.
Also, vlan off your exposed surface and build firewall rules for the VPN and LAN inbound to it, and specific outbound rules if you need those servers to reach into those networks themselves.
It’s not required, but probably OP has a home server with Immich and a VPS which exposes it to the internet. In that setup you need Tailscale for the VPS to access your home server. Sometimes you can’t directly expose your home server for different reasons, e.g. ISP doesn’t give you an external IP directly (I’ve had this, where my router would get a 10.x IP so I couldn’t port forward because the internet IP was being shared between multiple houses), or the ISP gives you a dynamic IP so there’s no guarantee that your IP won’t change next time you reset the router, etc.
Also it provides an extra layer of separation, so for example a DDOS would hit the VPS which probably has automatic countermeasures, and even if someone were to gain access to the VPS they still need an extra jump to get to the home server (obviously if they exploit something on immich they would get direct access to the home server).
Gotcha. Thanks for the insight!
It’s annoying, as I’d like to expose things for other people in my family (like Overseerr or whatever) without hassling them to also start a VPN or other stumbling block steps.
I was hoping that reverse proxy to overseerrs login screen would be safe enough. 8(
Does docker help limit things at all? I’m running my services through docker, which seems to limit the folders the container can hit. Feels like that would limit the damage someone could do even if they bypassed the login page of Overseerr or whatever app it is?
Edit: thanks for all the replies! Always more to learn and do, haha
First of all let me make this absolutely clear, docker is not expected to be secure to that level. While they try to make it hard for someone to escape a container, it’s not their main concern so expect that there are vulnerabilities that would allow an attacker to escape.
Now the second thing, the Overseer login screen might be secure enough for your case, the problem is that login is hard to do right, and Overseer are doing several other stuff as well, so they might not give it enough emphasis, and even if they do, maybe Immich devs don’t, or any one of the dozens of other services, so there are dozen of possible points of failure. Things like Authelia or Google OAuth are focused on authentication, so they do that absolutely right, and then they become the only point of failure for authentication.
To be fair, if you keep things updated it’s unlikely not having auth would be a problem. Mostly because most hackers won’t even know of your server to begin with. And most systems are secure enough for most casual hacks. But it’s an investment worth the time if you plan on making something available to the internet.
I only let things I trust are secure (e.g. ssh) have access from the internet, other services I hide behind a VPN (e.g. Tailscale).
O have a very similar setup but have a couple of questions if you don’t mind me asking, what did you used for OAuth? and where is it running? I tried athelia on the VPS but had some problems I can’t remember now and decided it wasn’t worth the time at the time, but probably should set it up.
I just use google OAuth since everyone I know has a google account. It just can’t use OAuth on private IP addresses, just FQDNs.
Authelia is great. Recently added protection for multiple domains.
Like, good for you, man.
But you should really keep your stuff inside the VPN and not expose things, it opens up a pile of potential risks that you don’t need to have. You can still use a reverse proxy inside the VPN and use your own DNS server that spits out that internal address to your devices for your various applications. If you absolutely, positively must have something exposed directly, put it on it’s own VLAN and with no access to anything you value.
I don’t even bother with the internal DNS server. I just set my A records in Cloudflare to point to the private IPs
Do the private IPs not change at all? Or can you handle that automatically?
I have next to no experience, but I’m pretty sure that wouldn’t work for me since my IP changes? Idk
You can either set a DHCP reservation in your router, or manually set the IP on the device.
When I say private IP, I’m referring to the internal IP e.g 192.168.1.X
Means internally I just go to the domain without having to remember the IP I set.
Oooh. That makes more sense, thank you.
I somehow thought you’d meant your global IP addresses, lol
Edit: i see now they’re talking about private IP, but in case you want to learn about getting a static IP for other things…
Many ISPs will give you a dynamic (changing) IP rather than a static (unchanging) IP. Just check your IP once a week for a few weeks to see if it changes.
There are some services that get around this by checking your ip regularly and updating their records automatically. This is called a dynamic DNS provider (DDNS). I used to use “noip” but since then there are quite a few like cloudflare DDNS.
Beyond that you just would want to make sure your router or whatever device is assigning IPs on your network to give a static assignment to the server. Assigning IPs is handled by a DHCP server and it would usually be your router, but if you have a pihole you might be using that as a DHCP server instead.
Between DDNS and DHCP you can make sure both your external IP and internal IP are static.
Most routers have a feature to assign static IPs to a specific MAC address. You can also tell most devices to try to take a specific IP instead of using DHCP.
There are multiple ways to set it up, but it’s very possible to set a specific device to always have the same local IP, which is usually the first step to many self-hosting scenarios.
I tired the same, but my router wants to be smart by filtering DNS responses that points to local IP. I guess whoever designed it considered it a security feature. It is a stock router from the ISP, its configuration interface is minimal, borderline to non existent.
Sounds like Cloudflare tunnels. I used that for a while, until I realized I didn’t want to be tied to Cloudflare.
Opening it up lets you use it from devices that aren’t on tailscale, or for friends and family. I have the same idea with Nebula instead of Tailscale, if I can figure it out.
I want to be able to upload/download/share my photos from anywhere in the world without using a VPN. Additionally, this satisfies the wife requirement. It works in the background without her needing her to turn on the VPN. I don’t want her to keep asking me how do I turn on the VPN? If it’s just me, then no issue, I’ll use a VPN.
You set up the VPN and it’s always on. There’s no hassle.
Unless you’re on IOS that will shut your VPN off regularly. Or you want somebody else to be able to access what you’re hosting without having to walk theme through a VPN setup they won’t understand.
I have a couple dozen customers on ios that use their camera servers via Tailscale. Never had a peep about that sort of thing.
And the last is the typical sort of “convenience” that gets people popped.
You’re hearing about it now. It’s an issue with the way iOS handles background tasks and there isn’t any way to fix it. It’s just how the OS works.
Well, apparently a bunch of farmers are smart enough to press a button without even bothering me about it.
Why would farmers not be smart enough to press buttons?
Yah, imagine my surprise, it’s almost like people are smart enough to manage a VPN without you holding their hands.
Manage, but not set up. Something tells me you had to do that part for them. And while pushing a button to start your VPN back up every time you want to access your remote service isn’t hard, it’s a nuisance that can be avoided in situations where it isn’t necessary in the first place.
Yeah, you always have to account for the wife factor. Same reason I’m using Plex instead of Jellyfin for my video hosting; I’d personally prefer Jellyfin, but the wife factor (really the mother-in-law factor, but whatever…) demands that it doesn’t require a ton of config on the user’s end. If the goal is to encourage use by your family, it can’t be fiddly or difficult to set up on their end.
To be fair, wireguard is pretty painless.
@randombullet@programming.dev
Don’t listen to this guy. You don’t have to turtle all your stuff inside a VPN if you don’t want to. Hosting services on the internet is what the internet was created for. It’s up to you whether what you want to host is exposed to the internet or not, and as long as you’re aware of the risks do what you want man. I will mention that Immich specifically might not be the best idea to expose since it’s so unstable, but that depends on your level of comfortability. Worst case scenario is somebody gets into your Immich and can see all your photos. Would this be a dealbreaker for you? If so don’t expose it publicly. Otherwise you’re perfectly fine.
Nobody said they had to. I made him aware of the risks in case he wasn’t. You seem to have an axe to grind there.
I’m not a big fan of amateur know-nothings regurgitating the same nonsense regurgitated to them by previous know-nothings, attempting to further the cycle to people finding their footing with self hosting, telling everybody what they “should” do based on their own limited understabding. It was a big problem on the self hosted reddit and up to this point has been less of a problem here.
And yet here you are, making sure this guy knows he can expose anything he wants except the specific thing you decided is troublesome like immich. Maybe you’ll be here to help him put it all back together with your wealth of knowledge and experience.
Take a hard look at yourself, you’re doing all the stuff you accuse someone else of. Maybe you aren’t always the smartest person in the room. In any case, I’m done with your shit. Go ruin someone else’s day, you ray of sunshine.
Yeah maybe you should take notes on how to relay a little bit of relevant knowledge in the context of what it is they’re trying to do, and let them decide how it fits their use case, instead of repeating broad, inaccurate generalizations dictating what people should and shouldn’t do across the board.
If you’re not going to be helpful or informative, then don’t bother chiming in at all.
Absolutely that’s what the internet was made for!
But family photos keep a bit more secure, Particularly if it’s syncing directly from your phone, I take a lot of explicit photos of my wife, but also code that I’m writing on my computer, or the kids playing, etc.
.
Excuse me what? Here’s my dumb ass navigating to "[device name]:[port] over tailscale.
I’ve tried this a couple times and I’ve always failed. I could never figure out how to get a service.domain request to my Nginx install to be proxied in the first place. I tried putting pihole on tailscale and setting that as tailscale’s DNS. It blocked ads but I couldn’t navigate to custom domains. I put NPM on tailscale hoping that was the issue. I looked for LocalDNS/CNAMES in tailscale to see if I could do it that way. Do I have to set a local machine as an exit node and do split DNS shenanigans, service.domain goes through to my local and everything else the wider web? Do I set a router node?!
Not expecting you to troubleshoot, I don’t have time to see it through anyhow. Just annoyed at myself I couldn’t figure it out and driven to try again.
Sounds like you’re ready to jam with the console cowboys
I’ve been wanting do something similar, but with Authentik. Does anyone know a good guide on this?
Yes! Authentik is a great self-hosted OAuth platform. They actually publish integration guides in their documentation.
Integrate with Immich
There is an official guide by Authentik on how to integrate with Immich. There is an official guide by Immich on how to integrate with Authentik.
.
Nice one dude, i know the pain of not having nerdy friends to share shit like this with.