from foremanguy92_@lemmy.ml to homelab@lemmy.ml on 09 Jul 10:52
https://lemmy.ml/post/32912073
(Removed from selfhost@lemmy.world since it was apparently not related to self host)
Hello, Since 2 weeks I investigate in my whole homelab the cause of an issue. Let me explain it :
I experience slow network speed when accessing my services from outside (outside local network without a VPN to connect directly into it).
Context :
My local network is all in 1Gb, my ISP download and upload speeds are 500Mb each.
Setup :
Since a while I set up a Librespeed server to be able to benchmark speed to my homelab from everywhere. When doing a direct connection from GbE pc to server, I get perfectly 1000Mb (more or less) symmetric speeds.
I was running caddy to reverse proxy it, and basically when connecting trough the domain (therefore trough the proxy) locally I got 980Mb (Surely the overhead that is caused by https and having one more hope).
(And I need to mention that when doing fast[dot]com, speedtest[dot]net or even librespeed[dot]org speedtests I got roughly 500Mb symmetric speeds, my ISP wasn’t lying)
So for now all of it is okay and working as intended.
Issue :
Here comes the problem when doing a speed test to my server (so trough speed.example.com) from a friend computer (who got 450 and 400 Mb on librespeed[dot]org servers), he got around 100Mb download (so homelab upload) and 400Mb up.
Tried other scenarios, from my home (the same where is located the homelab), I connect to the internet trough protonVPN or windscribe (all free tier).
While I was getting around 300-400Mb from librespeed[dot]org servers. I was sometimes getting 7-8Mb from speed.example.com (my libre speed install).
Even when working better I only got something like 80 down and 200 up, better but I was always really lower than popular speedtests or just a bit that could be somehow be margin error (around 40Mb on 400Mb), but was always here, it was not able to get right speeds.
Potential issue with librespeed :
Thinking that librespeed was not working good, so I tried to make a file transfer in file browser service, (Tried going trough and around caddy), getting around 100-110MB per sec without any VPN, going down to 8MB when transferring from the VPN.
Iperf got the same results around 100Mb of transfer speed going from VPN instead of 500Mb.
Maybe this isn’t really clear but I’d be really happy and respectful if someone could help me with this. Feel free to ask questions. :)
EDIT : tried to use nginx but this was even worse as even locally the speeds are not stable at all (not able to have full gigs local speeds) But don’t worry this is an other problem I will maybe resolve later
And little question, does WAN 500/500 speeds means that I can use 500Mb up and down at the same time, or that I can only use 250 each a the same time?
threaded - newest
You may want to crosspost this to !homenetworking@selfhosted.forum
Thanks I was search a community like this one :)
Glad to be of assistance.
Sounds like one of these “up to” scams by cable ISPs. With cable internet you are sharing a connection and it is often oversubscribed.
On a 500/500 you should in theory get 500 both ways the same time, but cheap routers or *BSD based ones with singlethread networking often struggle to reach these values in a speed test as the single speed test connection overloads the single cpu core.
Pretty strange because as I said when for example going to librespeed[dot]org from my desktop I get full 500 symmetrical speeds, but I don’t have them connecting to my server from outside
500/500 should be at the same time, as far as I know. Most tech seems to work that way anyway.
I wonder if your ISP is doing some kind of traffic analysis, where private to private addresses are slower than private to datacenter/commercial addresses?
Would explain the fast speeds between you and speedtest, and the VPN and speedtest? You didn’t say where the VPN endpoint was, but that might rule that out.
I’m not getting right results when doing from residential IP to mine or datacenter IP to mine. Tried to get the endpoint of the VPN in foreigners countries or the same as mine, it didn’t change anything.
Basically I get 500/500 when connecting to speedtest[dot]net without or with ProtonVPN
But when connecting to MY speedteet with ProtonVPN I only get as low as 8Mb sometimes or sometimes 100Mb, but ProtonVPN speedtest on speedtest[dot]net shows 400/400
Could be the ISP doing something more targeted, like good speeds to speedtest.net and no one else, but that seems unlikely for librespeed servers.
That’s what I thought but don’t think librespeed servers could be prioritized…
Don’t to mention that when I download stuff from internet I can easily get 60MB/s (~500Mb)
What VPN software? Is it CPU capping?
Tried ProtonVPN and Windscribe both getting really slow results, the speed test is not CPU bottleneck at all, running at around 50% when doing the heavy stuff
So, not a VPN to the homelab (tailscale/wireguard/etc)?
Google/reddit suggests windscribe can be pretty slow, and proton VPN free tier is slow. Are you getting good results through them to regular speedtest?
Don’t know if I didn’t explain correctly if my post but I already said it.
Don’t worry gonna recap it, I wanted to use a commercial VPN to access my self host speedtest from outside of my network, so I used wind scribe and protonvpn free tier.
As said both are getting pretty nice speeds, I wasn’t getting the full 500Mb like I didn’t have one, but I was getting about 300-400Mb which is nice but connecting to my speed test returned my only 10Mb sometimes or 100Mb but not at all the 300-400 from other speedtests
You probably did explain it correctly, Im not the best reader :).
You could try tailscale for a direct VPN to your server, see if that bypasses the free tier vpn issues? Tailscale will route your traffic directly between your two points, instead of via a server, so it might save some routing overhead? Its also free if that helps.
Not a viable solution as that way I cannot share to friends and it wouldn’t resolve anything as my traffic would go directly in the local network and giving it to friends would surely not resolve the issue
Are you sure? That sounds exactly like what tailscale and headscale are good for, letting your friends have near direct access to a server within your network. (Headscale is self-hosted tailscale, bypasses account limits, but is otherwise the same thing).
You setup your server as an endpoint on your tailscale network, and then give all your friends tailscale accounts to setup their devices on the same network. They’ll be able to talk directly to your server over a wireguard tunnel. (Caveat: cgnat can break tailscales tunnelling and cause your traffic to get relayed, which is slow. Headscale let’s you run the relay which will be faster, but it still sucks as bit).
I dont want people that need to access my services to download anything. But yeah maybe your right to the fact that I might overcome the problem. Even if that works I would like to have a proper solution working
Right, thats fair, I guess VPN is a red herring then, if the service is available over the internet directly.
I think I am out of ideas sadly, beyond it being some kind of weird ISP fuckery :/
Maybe do a traceroute between your mate and your server and vice-versa (without VPN), just to make sure your packets aren’t getting sent somewhere odd. Some time ago I had a friend who was physically 5km away, have his packets routed 2000km away, which kinda impacted throughput.
Gonna check
Done the trace route the packets only goes to the VPN ISP to basically my ISP, not a big deal…
Bummer. I think I am out of ideas, sorry I couldn’t be of more help.
Gonna do a bit more research but thx anyway
Try parallel connections in iperf:
If speeds improve with multiple connections: TCP window size is likely the issue.
Gonna try it, cause I already tried with basic iperf settings but not with the “-P 8” option
Tried it, it’s not better
.
I’m gonna try to disable ipv6 but why do I get bad results when connecting trough protonvpn to my speed test, since I can normally have 500/500 without a VPN making myself going from the www
It’s possible that the lower max speed could be caused by a number of things. First of all, do you have IPv6 enabled? I don’t know about your ISP, but mine has an issue where using IPv6 causes dropped packets on uploads. Disabling IPv6 in the router fixed that for me.
Another possibility is that there could be a slow peering connection between your internet provider and your friend’s, even if it’s the same company, there could be congestion between the specific nodes depending on the time of day.
Tried to disable ipv6 and it’s basically not doing anything, sadly…