How to get the best from your 10Gig home network?

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › How to get the best from your 10Gig home network?

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#311633 2-Feb-2024 21:02

I have recently upgraded the core of my home network to be 10GbE, through a variety of switches and network cards

gist of the network is outline below

PC 1
Windows 10 PC with QLogic BCM57810 network card > Qnap QSW-2104-2T Switch > Mikrotik CRS312-4C+8XG-RM Switch, all on cat 6A cabling with Cat7 patch leads.

PC2
Windows 10 PC with Intel(R) X540-AT2 network card > Qnap QSW-2104-2T Switch > Mikrotik CRS312-4C+8XG-RM Switch, all on cat 6A cabling with Cat7 patch leads

NAS
Synology RS1219+ with Mellanox CX312B ConnectX-3 network card > Mikrotik CRS312-4C+8XG-RM Switch via 1m DAC Cable.

I'm not having any issue per se, I'm just wondering how I get the best networking speeds out of my equipment. Currently getting between 4.5Gbps and 6.8Gbps on iperf3 tests, with the NAS running the iperf server, but i also try with one of the PC's as the server.

I have enabled jumbo frames on all the devices and also on the Mikrotik switch on the ports the above devices are on. The Qnap switches also support jumbo frames. I have turned off flow control on the 2 PC's as by default the Mikrotik has this turned off.

Does anyone have any suggestions on what to do next to increase performance?

BadCo

109 posts

Master Geek

#3190035 3-Feb-2024 18:54

Can that Synology even push 10gig? Is the CPU and storage good for it?

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#3190036 3-Feb-2024 19:03

BadCo: Can that Synology even push 10gig? Is the CPU and storage good for it?

Synology claims 1,310 MB/s reading and 470 MB/s writing. but as i am lead believe Iperf is RAM based and I have 16gb in my unit so it shouldnt be to much of an issue

I get similar speeds between the windows PC's so i dont think its a drive/read/write issue.

mentalinc

3241 posts

Uber Geek

Trusted

#3190038 3-Feb-2024 19:07

Why are you going through the two switches?

What happens if you remove one and then the other?

What does the Iperf PC1 to PC2 look like?

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#3190039 3-Feb-2024 19:11

mentalinc:

Why are you going through the two switches?

What happens if you remove one and then the other?

What does the Iperf PC1 to PC2 look like?

PC to PC looks the same, 4.5-6.8gig and that's through all 3 switches

the qnap switches are there as in each location where the PC is as there are other devices there, so the PC has the 10g connection along with the back haul being 10g and the other devices are on the 2.5g ports.

mentalinc

3241 posts

Uber Geek

Trusted

#3190051 3-Feb-2024 20:15

I'm getting circa 3.5Gbps between my PC and server

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 255 MBytes 2.14 Gbits/sec
[ 5] 1.00-2.00 sec 417 MBytes 3.50 Gbits/sec
[ 5] 2.00-3.00 sec 404 MBytes 3.39 Gbits/sec
[ 5] 3.00-4.00 sec 660 MBytes 5.53 Gbits/sec
[ 5] 4.00-5.00 sec 365 MBytes 3.06 Gbits/sec
[ 5] 5.00-6.00 sec 566 MBytes 4.74 Gbits/sec
[ 5] 6.00-7.00 sec 471 MBytes 3.95 Gbits/sec
[ 5] 7.00-8.00 sec 578 MBytes 4.85 Gbits/sec
[ 5] 8.00-9.00 sec 325 MBytes 2.72 Gbits/sec
[ 5] 9.00-10.00 sec 537 MBytes 4.51 Gbits/sec
[ 5] 10.00-10.23 sec 84.0 MBytes 3.06 Gbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.23 sec 4.55 GBytes 3.82 Gbits/sec sender

Pretty sure it's a Windows issue, as the esxi servers can do 9.8Gbps between each other.
But window 11 pc to a vm on the esxi are results above

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#3190053 3-Feb-2024 20:25

PC1 to NAS, PC Sending

[SUM] 0.00-10.00 sec 6.08 GBytes 5.22 Gbits/sec sender
[SUM] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec receiver

PC1 to NAS, NAS Sending

[SUM] 0.00-10.00 sec 7.46 GBytes 6.41 Gbits/sec 911 sender
[SUM] 0.00-10.00 sec 7.46 GBytes 6.41 Gbits/sec receiver

mentalinc

3241 posts

Uber Geek

Trusted

#3190055 3-Feb-2024 20:42

Setting - P 10 I get to around 8.2Gbps
How many processes are you using?

[SUM] 0.00-10.02 sec 9.59 GBytes 8.22 Gbits/sec sender
[SUM] 0.00-10.02 sec 9.59 GBytes 8.22 Gbits/sec receiver

I Don't have jumbo frames, just standard 1514.

Do you have RSS enabled on windows side?

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

mentalinc

3241 posts

Uber Geek

Trusted

#3190056 3-Feb-2024 20:46

What if you run two at the same time and add them?

Pc1 to nas and pc2 to nas at the same time?

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#3190109 3-Feb-2024 21:37

mentalinc: What if you run two at the same time and add them?

Pc1 to nas and pc2 to nas at the same time?

Had to set up another docker container on the NAS, but it gets to about 7.5gbps. Only about 70% CPU on the Synology and 25% RAM.

PC1 gets 10gig speeds when testing to servers on PC2 and the NAS.

PC2 seems to hit a hard wall at about 6.4Gbps when testing to serves on the PC and NAS, its only using about 30% cpu.

Im only using -P4 as anything more doesn't really change the speed i see

Thanks for the suggestions

mentalinc

3241 posts

Uber Geek

Trusted

#3190112 3-Feb-2024 22:08

Using the latest drivers for PC2 with the Intel NIC https://www.intel.com/content/www/us/en/products/sku/60020/intel-ethernet-controller-x540at2/downloads.html looks to only be a month or so old.?

And similar for PC1 - does it have the latest firmware and drivers from manufacturer not from Microsoft for the QLogic NIC? - Assume it's from a Dell - maybe try this https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=t3kh2

And results for below two commands:

netsh interface tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State : enabled
Receive Window Auto-Tuning Level : normal
Add-On Congestion Control Provider : default
ECN Capability : enabled
RFC 1323 Timestamps : allowed
Initial RTO : 1000
Receive Segment Coalescing State : enabled
Non Sack Rtt Resiliency : disabled
Max SYN Retransmissions : 4
Fast Open : disabled
Fast Open Fallback : enabled
HyStart : enabled
Proportional Rate Reduction : enabled
Pacing Profile : off

Get-NetTCPSetting | Select SettingName,AutoTuningLevelLocal

SettingName AutoTuningLevelLocal
----------- --------------------
Automatic
InternetCustom Normal
DatacenterCustom Normal
Compat Normal
Datacenter Normal
Internet Normal

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

Jase2985

13465 posts

Uber Geek

ID Verified

Lifetime subscriber

#3190163 4-Feb-2024 09:40

Already have the newest drivers for both network cards

PC1

C:\WINDOWS\system32>netsh interface tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State : enabled
Receive Window Auto-Tuning Level : normal
Add-On Congestion Control Provider : default
ECN Capability : disabled
RFC 1323 Timestamps : disabled
Initial RTO : 3000
Receive Segment Coalescing State : enabled
Non Sack Rtt Resiliency : disabled
Max SYN Retransmissions : 2
Fast Open : enabled
Fast Open Fallback : enabled
HyStart : enabled
Proportional Rate Reduction : enabled
Pacing Profile : off

Get-NetTCPSetting | Select SettingName,AutoTuningLevelLocal

SettingName AutoTuningLevelLocal
----------- --------------------
Automatic
InternetCustom Normal
DatacenterCustom Normal
Compat Normal
Datacenter Normal
Internet Normal

PC2

Receive-Side Scaling State : enabled
Receive Window Auto-Tuning Level : normal
Add-On Congestion Control Provider : default
ECN Capability : disabled
RFC 1323 Timestamps : disabled
Initial RTO : 1000
Receive Segment Coalescing State : enabled
Non Sack Rtt Resiliency : disabled
Max SYN Retransmissions : 4
Fast Open : enabled
Fast Open Fallback : enabled
HyStart : enabled
Proportional Rate Reduction : enabled
Pacing Profile : off

PS C:\Windows\System32\WindowsPowerShell\v1.0> Get-NetTCPSetting | Select SettingName,AutoTuningLevelLocal

SettingName AutoTuningLevelLocal
----------- --------------------
Automatic
InternetCustom Normal
DatacenterCustom Normal
Compat Normal
Datacenter Normal
Internet Normal