Opnsense on Proxmox with Single NIC

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Opnsense on Proxmox with Single NIC

andysh

229 posts

Master Geek
+1 received by user: 16

#311877 22-Feb-2024 21:56

Will start with I think I am trying to do too much with what I have but hey.. the geek in me wants to try

Current setup:

ISP: 2degrees proper (not migrated from anywhere) needs VLAN 10
Router: tp-link ER605 v1 (on Omada SDN)
Switch: tp-link TL-SG2428P v1.0
Cabling: ONT -> ER605 -> Switch -> Devices (LAN/wifi)

The above setup has been working all OK in IPv4 but there are a couple of issues. VPN is not great for some use cases I have (VLAN in network always on VPN). And there is no IPv6 firewall.

So I thought of trying out OpnSense (also open to pfsense). But I don't have a spare box to run it on metal, but do have two mini PCs running Proxmox which both have a single NIC.

New Setup:

In Omada I have set up VLAN101 '2degrees' and VLAN10 'WAN VLAN'. Also default VLAN1 for LAN
Cabling: ONT -> Switch Port 1 // Switch Port 2 -> Mini PC (proxmox) -> Switch Port 2 -> devices (LAN/wifi)

Port 1 is set up as Native/untagged network of VLAN101 and tagged VLAN10
Port 2 is trunk port. Native/untagged VLAN1. VLAN101 and VLAN10 tagged

Proxmox is set up as VLAN aware
Split out VLAN1 and VLAN101 in proxmox
Turned off firewall for VLAN101 in proxmox
In OpnSense interfaces are to the above to VLANs so look like individual interfaces. Then set up a VLAN in OpnSense of VLAN10 against the WAN interface.
Normal WAN DHCP settings, basically all blank

And with the new setup, I get no WAN IP. LAN is all ok to the router. Is what I am trying to do, just impossible / to much?

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

MaxineN

Max
2049 posts

Uber Geek
+1 received by user: 1663

ID Verified

Trusted

Subscriber

#3198901 22-Feb-2024 23:02

The VLAN interface needs to have your DHCP settings. Having it on the WAN interface won't work.

This setup however is pretty ambitious. Why not use those mini PCs as a VPN server or even as a Tailscale host?

I'm not an expert but that's pretty crazy imo.

Ramblings from a mysterious lady who's into tech. Warning I may often create zingers.

bagheera

544 posts

Ultimate Geek
+1 received by user: 189

#3198973 23-Feb-2024 09:48

the wan interface needs to be vlan 1 (or whatever you wan vlan you want to be the native vlan wan to be) with vlan tag10 on the proxmod wan VM interface, on the switch that has ONT port set it to vlan 1 native (or whatever you wan vlan you want to be the native vlan wan to be) with vlan tag10 to the ont.

andysh

229 posts

Master Geek
+1 received by user: 16

#3198981 23-Feb-2024 10:12

bagheera:

the wan interface needs to be vlan 1 (or whatever you wan vlan you want to be the native vlan wan to be) with vlan tag10 on the proxmod wan VM interface, on the switch that has ONT port set it to vlan 1 native (or whatever you wan vlan you want to be the native vlan wan to be) with vlan tag10 to the ont.

Thanks for that. What I have now set up and seems to be working:

ONT -> Switch Port 1 // Switch Port 2 -> Mini PC (proxmox) -> Switch Port 2 -> devices (LAN/wifi)

Port 1 is set up as Native/untagged network of VLAN1 and tagged VLAN10
Port 2 is trunk port. Native/untagged VLAN1. VLAN10 tagged

Proxmox is VLAN aware

In pfsense (change to pfsense, more documentation around). I have got a VLAN interface on Port2 of 10. So I have got rid of anything to do with VLAN101.

Did break the wifes internet though, but think that is internal firewall rules, so weekend project, but WAN is up.

I guess, is there an issue with Port1 having VLAN1 on it which is exposed to the internet? I assume not, as all internet comes through VLAN10 and only place that is being picked up in pfsense?

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

bagheera

544 posts

Ultimate Geek
+1 received by user: 189

#3198995 23-Feb-2024 10:38

i would keep vlan 101 and have all other port native vlan 101 for you lan vlan, no tagging, or add vlan 102 and tag on the vlan 101 ports, and have a guest wifi SSID on vlan 102 and get the pfsense to route out vlan 102 no route to vlan 101 if you wifi units can do vlan tagging on a SSID

the trunk port vlan 1 tag 10 is a virtual cable to the ONT, so you want your lan off 1 and 10 vlans

andysh

229 posts

Master Geek
+1 received by user: 16

#3199001 23-Feb-2024 10:55

O yep makes sense

So VLAN 1(untagged),10(tagged)= WAN

VLAN101 (untagged) = LAN Or whatever VLAN I want, just not 1 or 10.

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

bagheera

544 posts

Ultimate Geek
+1 received by user: 189

#3199003 23-Feb-2024 10:59

andysh:

O yep makes sense

So VLAN 1(untagged),10(tagged)= WAN

VLAN101 (untagged) = LAN Or whatever VLAN I want, just not 1 or 10.

correct, so if you want to move the VM nuc to other end of the house, as long as you have a switch that can do vlan tagging, even if you only have 1 cable to it, you would trunk 1, 10, and 101 (and any other vlan you need over that trunk cable, and the switch that you plug the ONT into is set to vlan 1, tag 10 for the ONT port, your virtual WAN cable will work.

and if you plug a none vlan aware switch into a port that of a vlan aware switch that is set to untag/native vlan 101, then the dumb switch will still be on your lan vlan.

AliExpress

Shop now on AliExpress (affiliate link).

andysh

229 posts

Master Geek
+1 received by user: 16

#3199080 23-Feb-2024 12:49

Changed it up a little

WAN: VLAN 101 (untagged), 10(tagged). This is against the WAN port on the switch. WAN up all OK.

LAN: VLAN1 (untagged)

In proxmox, use trunk port, and then tag VLAN10.

Think this is better as then nothing internet facing is against VLAN1, I hope.

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77

bagheera

544 posts

Ultimate Geek
+1 received by user: 189

#3199091 23-Feb-2024 13:29

andysh:

Changed it up a little

WAN: VLAN 101 (untagged), 10(tagged). This is against the WAN port on the switch. WAN up all OK.

LAN: VLAN1 (untagged)

In proxmox, use trunk port, and then tag VLAN10.

Think this is better as then nothing internet facing is against VLAN1, I hope.

yip, that will work - things to keep in mind with vlan aware switches and vlan with broadcast etc , a normal network packet has either no vlan info in it, ie untag, then the switch is doing the work on working out where to send it, and look at the native / untag of port it comes from and send it to other ports that are the same vlan, if you have a port the native 200 with 101 tagged on it, then the switch will add a tag of 101 to the untagged packet, and send it to the port with 200 untagged / 101 tag, also when it gets a 101 tag packet from a none native 101 port, it will strip the 101 tags and send it to all the untag 101 if it a broadcast. so your WAN VLAN 101 is your wan none tagged wan vlan and should be treated as so ie wherever you need the WAN virtual cable to go, you need trunk vlan 101 & 10, and at both ends of the virtual cable - ie ONT port & VM WAN port, you set untag vlan 101 & tag VLAN 10 and you should be fine, even if the traffic flows over a trunk port that native 1, tagged 10 & 101 - switch will add 101 send over when it get to the last hop, the 101 will be strip and sent out the native 101

andysh

229 posts

Master Geek
+1 received by user: 16

#3200169 26-Feb-2024 12:57

Moved back to OpnSense with the same setup.

WAN Port: VLAN 101 (untagged), 10(tagged).

LAN Port into Proxmox Server: VLAN1 (untagged), 10(tagged),101(tagged)

Proxmox: VLAN aware, single Virtual NIC into OpnSense

OpnSense. WAN: tagged on VLAN10

All working as expected. Also have HA(ish) by running OpnSense on Proxmox backed by CEPH. Tried to get CARP working but just couldn't, as even though I have a static IP with 2degrees, afaik it only gets assigned through DHCP.

I tried to enable IDS/IPS but that blew up the CPU/RAM, might need to get a bigger Proxmox server 😁

Referrals:

Tesla: https://ts.la/andrew897313

Sharesies: https://sharesies.com/r/XRGS77