Hi, so out of curiosity, what vendor are you currently using, and is there a hint as to why L3 is not workable.

Cyril

cyril7:

Hi, so out of curiosity, what vendor are you currently using, and is there a hint as to why L3 is not workable.

Cyril

What hardware are you running at your endpoints?

We are currently using Zerotier on a multi-site network and it works like a charm, between 2 of our sites on Gig fiber we are hitting line speeds. (havelock north and hamilton)

cyril7: And even if you went layer2 you still need IPsec and that's going to be your bottle neck.

Wireguard performs better, so could be an option but it's only L3, but if speed is determining your desire for L2, then that's misguided.

Sounds like what you want is a private Wan vlan, but ideally you would still want to route across that but might not need the IPsec. Many ISP's offer this service but it's BS3, which is possibly what you already are on.

Cyril

sparkz25:

What hardware are you running at your endpoints?

We are currently using Zerotier on a multi-site network and it works like a charm, between 2 of our sites on Gig fiber we are hitting line speeds. (havelock north and hamilton)

chevrolux: Perhaps time to consider a business focused provider then?

Any good business focused ISP will be able to provide L2 tails between your sites. The likes of Voyager or Devoli probably have the most cost effective options (or any of their resellers).

martinjward:

sparkz25:

 

What hardware are you running at your endpoints?

 

 

 

We are currently using Zerotier on a multi-site network and it works like a charm, between 2 of our sites on Gig fiber we are hitting line speeds. (havelock north and hamilton)

 

Zerotier or wiregaurd are definite options. We have 10gbps capable router so would just be a matter of upgrading the fiber line & configuring.

What hardware are you running? 

Opnsense running natively would be a good option

hio77:

 

Fun option, Stick an ARM based Mikrotik at each site, Run ROS v7 and run zeroteir on a bridge.

 

Seen some pretty good results of people using exactly this in production already.

Maybe don't use this in production: https://pulsesecurity.co.nz/advisories/Zerotier-Private-Network-Access

cyril7: And even if you went layer2 you still need IPsec and that's going to be your bottle neck.

Wireguard performs better, so could be an option but it's only L3, but if speed is determining your desire for L2, then that's misguided.

Sounds like what you want is a private Wan vlan, but ideally you would still want to route across that but might not need the IPsec. Many ISP's offer this service but it's BS3, which is possibly what you already are on.

Cyril

Hi, sorry my misguided remark relates to L3 being somehow slower or an inhibitor compared to L2, this is simply not so, as mentioned IPSec which you will want to use on a VPN to protect your traffic is more likely to be the issue, regardless of providing a L2 or L3 VPN.

From what you describe, a DFAS or private WAN vlan are probably more appropriate for what you are after.

But whats the real traffic levels like, when you engineer things like this, the ideal most perfect solution to the idealist may not be the correct or approprate solution.

You currently have IPSec vpn's to other sites, is this new site going to have similar staff levels and therefore similar traffic demands? if so why the need (other than because :) for a 10G link? Have you looked at interface flow logs to see how much is required? that would be the first step. How much access to on premise servers at the main site is required, is any moving to the cloud anytime soon, if so the need for a VPN to remote sites becomes less important, well at least its capacity is less important.

Assuming the current IPSec solution was found to be adequate based on current user experience you could instead look at getting hyperfibre for the main hub site to beef up performance from that end to all remote sites, along with reviewing the performance and spec of hardware at all ends

As mentioned earlier if you do go with a DFAS or private WAN then I still recommend you do span the two sites at L2, you dont want that resource cluttered by broadcast traffic. A pair of base line L3 switches will route at wirespeed let alone plenty of full router/UTM devices.

Just as an aside I built and maintain a very large network across an estate of several hundred hecters covering dozens of building and providing for around 3k of users on a daily basis. This is all glued together with hundres of km private blown fibre and linked by a network of L2 switches all with at least one 10G link. But the L2 network only has those transport switches and a L3 device at each end point to route traffic off and on the transport link. On the odd occasion where I have had to drop a unit onto the transport link at L2 (ie span a vlan across the place for some short term requirement) the network performance is noticiably impacted, just dont do that.

Cyril

cyril7:

 

Hi, sorry my misguided remark relates to L3 being somehow slower or an inhibitor compared to L2, this is simply not so, as mentioned IPSec which you will want to use on a VPN to protect your traffic is more likely to be the issue, regardless of providing a L2 or L3 VPN.

 

But whats the real traffic levels like, when you engineer things like this, the ideal most perfect solution to the idealist may not be the correct or approprate solution.

 

You currently have IPSec vpn's to other sites, .... well at least its capacity is less important.

 

Assuming the current IPSec solution was found to be adequate based on current user experience you could instead look at getting hyperfibre for the main hub site to beef up performance from that end to all remote sites, along with reviewing the performance and spec of hardware at all ends

 

As mentioned earlier if you do go with a DFAS or private WAN then I still recommend you do span the two sites at L2, you dont want that resource cluttered by broadcast traffic. A pair of base line L3 switches will route at wirespeed let alone plenty of full router/UTM devices.

 

Just as an aside I built and maintain a very large network across an estate of several hundred hecters ... On the odd occasion where I have had to drop a unit onto the transport link at L2 (ie span a vlan across the place for some short term requirement) the network performance is noticiably impacted, just dont do that.

 

In terms of speed I am comparing the entire end to end solution ie. DFAS 10G vs 8G hyperfiber w L3 VPN type solution. Speed is not my only consideration though.

My DFAS theory is plug my new access switch in to the collapsed Dist/core and away we go.... essentially an access switch sitting at a different premise, would be interested to understand how that would be an issue?, latency doesn't appear to be an issue?

I am aiming to have multiple options, following the good, better, best theory.

Unfortunately the new site will not be like any of the others so its impossible to compare. I can see the bonded interface of one of the fileservers saturating at 18Gbps so there is definite scope for the 10G link to be saturated, it would not be a 10G link for the sake of it.

Yes an upgrade to hyperfiber will be presented as one of the options.

Looks like you really know your stuff, thanks for the great info Cyril! I will definitely consider your advice with regards to not spanning, when this gets setup.

hio77:

 

Fun option, Stick an ARM based Mikrotik at each site, Run ROS v7 and run zeroteir on a bridge.

 

Seen some pretty good results of people using exactly this in production already.

 

Maybe don't use this in production: https://pulsesecurity.co.nz/advisories/Zerotier-Private-Network-Access

 

Hmm as I read it, vulnerablilities have been found, reported and fixed, in addition he left some breadcrumbs for the developers to improve security.

I enjoyed the back of the napkin maths with regards to buying 138 RTX 3090s... 😆