from 0xCDE@infosec.exchange to homelab@lemmy.ml on 11 Jun 12:50
https://infosec.exchange/users/0xCDE/statuses/114664814317004087
Linux systems unable to reach out to internet behind pfsense while Windows work fine.
I am trying to set up my #homelab for #forensics and I have encountered a very weird issue. I am running an internal pfsense #firewall for my environment. On this firewall there are 3 interfaces: WAN and two Isolated Labs.
On the “WAN” interface, both #linux and windows systems are able to access the internet without any problems.
But on the other two lans - named Infrastructure and Lab - the Linux systems are not able to connect to the internet after booting, while the systems are able to communicate with each other. In order to get them working I need to go the respective interface and make any change there and save the page, practically refreshing the settings on the interface. After this all currently online linux systems on that interface are working as expected and connectivity is restored. As you can imagine this is a huge pain, especially when I am testing things and I need to turn on various systems at different times. If i put the systems on the WAN interface they face no issue at all, so what could be the solution here for fixing connectivity for linux systems on the other interfaces?
edit: it issue is both on pfsense 2.7.2 and 2.8.0
#dfir #proxmox #dfir #homelab #firewall #pfsense #networking #network
threaded - newest
Its hard to determine exactly where the issue is coming from, so some more information about the setup would be helpful.
In my experience, networking is very prone to small configuration errors, and setting up a homelab is going to force you to learn some great networking skills one way or another :)
@Chewt
1. pfsense is virtualized on a Proxmox VM
2. If i bring down the network interface and back up it loses connectivity once more
3. The firewall is for and internal network so the WAN is my home network. I have an ubuntu server acting as a NAS and another ubuntu server for Jellyfin. Both servers are on the same proxmox instance, just on the WAN interfaces on the pfSense so I can access them from all the computers on the local network.
ok, so to recap just to make sure I have your setup correctly: You Have a Proxmox node with 3 VMs on it: Pfsense, an Ubuntu VM, and another Ubuntu VM. The Proxmox Node has a WAN network (likely a Linux Bridge type interface called vmbr0 or something similar), and a LAN network (another bridge, but with no associated physical interface on the Proxmox host). The last step can differ depending on how exactly you set it up, but the basic idea of having two “networks” is the same. EDIT: You will have two LAN networks as per your original post Your pfsense VM has three Network Devices under the hardware tab, one connected to your WAN network, and the other two connected to your LAN networks.
In Pfsense, you have the appropriate interfaces configured and firewall rules allowing devices on the LAN interface to connect out of the WAN interface (this step in particular has a lot of moving parts)
On each of your Ubuntu VMs, there is just a single Network Device under their hardware tabs, connected to the LAN network. (although right now you say they are connected to the WAN network so you can connect to them from other computers on your home network). Once this is set up correctly, move them back to the LAN network, and maybe set up a rule that allows connections from a single device on your home network. If allowing any device on your home network to connect to your homelab is what you want, then it doesn’t make too much sense having a firewall if you aren’t going to use it ;).
Does this match with your current setup?
@Chewt
The node has others vms as well, some of them are windows vms but they are not discussed as they do not face any problems connecting to the internet no matter what interface they are connected on. The Linux VMs that are on the WAN network which is the one that has the linux bridge that is connected to a port and those systems do not have any connectivity issues.
The other two lans also have a linux bridge that is not bound to any port and the Linux systems need the interface refresh in order to access the internet. Windows systems on those networks are connected to the internet with no issues.
yes the pfsense hardware is as you describe it. (picture attached)
For the last paragraph, I need to have some devices that will be isolated from my WAN (home network) because I want to create a cybersecuirty lab and I do not want anything to "escape" to my home devices.
Plus this teaches me how to manage a network with proper segmentation etc.
ok, if you’re fine with doing a little more troubleshooting, I have a few more things to try.
You said before that the systems are able to communicate with each other on the LAN network, just not connect out via WAN. Does this include the pfsense box? That is to say are you able to
ping 10.10.10.1(assuming that’s the IP of the Infra interface on pfsense) from one of you Ubuntu VM’s when it is on the Infra network? Also what is the output of both commandsip -br aandip route? What I’m am curious about here is maybe the default gateway is not being correctly set when new devices connect to the pfsense. If this ends up being the case, it might be an issue with DHCP (or if you aren’t using DHCP then just the configuration on each Ubuntu VM). I suspect that Windows may be doing some fancy things to figure out on its own some network settings without them being explicitly set.@Chewt
yes they can still ping my firewall while not connected to the internet.
Here are two screenshots from one of the ubuntu vms on the home network and one from the kali vm on the infra network
Well that doesn’t really indicate any problems with the default gateway. My last bit of troubleshooting advice is to traceroute and see if packets get stuck at the router. And check the firewall logs in pfsense to see if you can find anything there.
Hopefully it ends up being a relatively painless problem to fix!
@Chewt found the issue in the end... i had two interfaces share the same IP and was creating a conflict on the linux hosts. I dont know why Windows was not affected by it but all good in the end
Happy to hear it! Address conflicts are always a pain, I recently ran into an issue with my VPN to my home network, where my home network and the network I was on shared the same subnet and mask, so I had to specify some custom routes to make it work. Good network design is a must to try and mitigate these things!
What do you mean “go to the respective interface and make any change”? In the firewall or client? What changes when you do this, IP address or routing table or gateway or what?
The way your describing this isn’t quite clear. I think you have something crossed. There are a series of videos by Lawrence Systems on YouTube where he walks through basic to complicated PFsense configurations. I think it’s worth watching. What your trying to do is setup two vlans I think? They probably don’t have access to the WAN which you need to allow as this is not the default.
It seems complicated but making firewall rules makes sense once you get the hang of it. The best part of home labbing is learning though it usually requires breaking some stuff in the meantime. All part of the journey. You’ll get there, have fun.
Check your route table and arp cache (for gw). Are you using dhcp?
@Gobo gw? I have tried with and without dhcp and the issue persists
Gateway. Does the system properly know how to get outside of its subnet.
@Gobo yes the gateway is configured properly. If ti wasnt the system would not have internet after a firewall refresh
Then I’d say get a pcap from the pfsense interface and see what’s going on. Is the systems Mac still in the fw arp table? . If you think it’s a system fw problem, try disabling the local fw and see if things work
@homelab
solved: There was an arp conflict, I had two interfaces sharing the same IP address, one from proxmox and one from pfsense. Once I changed the IP address from the proxmox interface everything worked!
Thanks everyone for helping!