Advice on 10GBE between switches, server & 2 x Workstations...

Forums › LAN (ethernet/Wifi/routers/Bluetooth) › Advice on 10GBE between switches, server & 2 x Workstations...

1 | 2

fritzman

366 posts

Ultimate Geek

ID Verified

#2040648 19-Jun-2018 17:12

cyril7:

Cat6 should be good for 10G if installed correctly upto 37m, with variable potential over that to 50odd meters.

Sorry should have checked your current switch spec before commenting on that. As for aggregation, it would make sense between the switch and server/s but 10G to each work station is surely sufficient.

Edit: Further, 10Gbase-T is in my view not a nice connection, its power hungry and has a serialisation delay which makes it not suitable for some applications, but probably not a biggie for yours. I would still do DAC between servers and switch.

Cyril

No worries.. I plan on SFP+ between the two 1950 switches and the 1950's and the server... The 10GbE will be used for the run from the 1950's to the 1850's and also the two workstations.

Do you think that will be ok, or should I be looking to try and have SFP+ to feed the 1850 switches as well?

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

mentalinc

3239 posts

Uber Geek

Trusted

#2040835 20-Jun-2018 07:21

You're way over complicating this

Buy 4 of these

https://www.fs.com/products/11552.html

Buy 2 of these a the length you need (they do custom)

https://www.fs.com/uk/products/17235.html

Connector A: LC Connector B: LC Cable Jacket: LSZH Cable Diameter: 2.0mm

Buy two of these (one for each work station)

https://www.ebay.com/itm/322917300357

I used the above seller when I purchased mine

Buy one of these for the server

MCX312A-XCBT (search ebay)

You then run the fiber between the three ports

setup a static ip,

configure some host files to traffic is routed to the server of the 10GB link

All done

and should be circa $600 depending on shipping

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

freakngeek

356 posts

Ultimate Geek

#2040837 20-Jun-2018 07:24

@fritzman what else you looking at running to use all this LAN power ?

The Supermicro Mobo is very impressive, how will you fill it?
CPU's , dual Xeons
RAM, surely you'd want 3TB :)
HHD's vs SSD's vs RAID
Graphics, mulitple graphics, ie SLi, consumer or Quadro or ATI

All right up my alley

fritzman

366 posts

Ultimate Geek

ID Verified

#2040879 20-Jun-2018 09:24

Mentalinc...

Thanks for the suggestions. Unfortunately, the development of the plan to provide more and faster access to the server was taking place as the offices were being rebuilt and sadly, the building beat the need to run cables... what is there is brand new Cat6 and that is what I am forced to use. On top of that, the server rack is downstairs in its own room and the video guys are about 25m line of site, upstairs several rooms away. Fibre would have been nice and simple, but that bird has flown.

Freakngeek...

The Church during the daytime, has anywhere between 15 & 25 staff on site using SharePoint over a mix of LAN & 802.11AC (14 AP's), plus the two video-geeks, plus a Kindy with another 4 staff, plus a large commercial Cafe with all manner of punters wanting to do the usual (as much as they can) over the free WIFI, plus community groups coming in day by day using the facility and the free WIFI while there. Sundays of course, there are several hundred through, doing much the same.

The 'grunt' behind the scenes, is the server, which has been in place nearly 2 years now. It runs 2 x E5-2620 V3's, 32Gb DDR4, a single Intel (Ent) SSD running Server Essentials & 7 x 600Gb 10k SAS drives in Raid5 for all the data pool, plus a couple of single large drives, one for all the client device backups and the other for the server (O/S & Data) backups, using ShadowProtect.

The video workstations run pretty standard configs on X99 boards with Xeon 2650's from memory, 32Gb ram and M.2 drives with 1070ti cards... does all they need.

Along with the rebuild, they are putting in place a whole bunch of 55" displays live-streaming stuff, both in-house and from online, putting in a decent rack and I'm looking at adding a 12Tb drive to replace the server backup drive and re-purpose the current 6Tb drive to house older video stuff that is hardly used, but is taking up most of the 3Tb of SAS storage and making it almost full - should be able to free up at least 1-1.5Tb (I hope).

I mentioned the DT1200 somewhere above, as I should be able to grab one cheap (still waiting to confirm), and that would give about 80Tb in Raid60 with a couple of hot spares on top... a little for them, and a lot left over to become a backup target (RSync) as a business opportunity. Still trying to lock that down tho.

Hope the explanation helps, and apologies if you think I should have laid out the whole picture in the OP.

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

Yoban

447 posts

Ultimate Geek

#2041014 20-Jun-2018 13:05

This blog post could also offer some guidance https://blog.briancmoses.com/2016/06/building-a-cost-conscious-faster-than-gigabit-network.html

Tempted to have a crack at home myself.

freakngeek

356 posts

Ultimate Geek

#2041255 20-Jun-2018 17:28

@fritzman,
I take it you are using the Broadcom/LSI 3108 controller for the RAID5 array ?
Did your Mobo come with Cachecade/Fastpath key ?
Having this enabled makes quite a difference in performance of the RAID array
Using a SSD to also cache makes a difference too
If there a cachevault option another bonus in preventing data loss, as the cache is preserved

You mention a Dell MD1200 SAN
Are you getting it with the Perc H810 controller installed ?
The H810 is not needed as you can run 2x SAS cables (or 1) from the 2x SAS sockets on the Mobo
Not sure if the server run these out to external ports, you may need to run these cables out a expansion slot
Use the onboard 3108 to do all the RAID sorting , especially if Cachecade/Fastpath is enabled

I used to do a heap of playing with this stuff when money was no object

As your Mobo is so integrated with everything built in you can't be using hardly any of the PCIe slot potential
2x PCIe 16x slots with full bandwidth begging for suitable SLi quadro Video cards
When not crunching video, you could be Bitcoin mining as a sideline :)

sonyxperiageek

2959 posts

Uber Geek

Trusted

#2041295 20-Jun-2018 18:46

I always wonder where all the funding for all this comes from in a church!

Sony

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

fritzman

366 posts

Ultimate Geek

ID Verified

#2041296 20-Jun-2018 18:47

sonyxperiageek:

I always wonder where all the funding for all this comes from in a church!

That's an easy one... the members/adherents, but that's a good topic for another thread lol.

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

fritzman

366 posts

Ultimate Geek

ID Verified

#2041480 21-Jun-2018 08:47

freakngeek:

@fritzman,
I take it you are using the Broadcom/LSI 3108 controller for the RAID5 array ?
Did your Mobo come with Cachecade/Fastpath key ?
Having this enabled makes quite a difference in performance of the RAID array
Using a SSD to also cache makes a difference too
If there a cachevault option another bonus in preventing data loss, as the cache is preserved

You mention a Dell MD1200 SAN
Are you getting it with the Perc H810 controller installed ?
The H810 is not needed as you can run 2x SAS cables (or 1) from the 2x SAS sockets on the Mobo
Not sure if the server run these out to external ports, you may need to run these cables out a expansion slot
Use the onboard 3108 to do all the RAID sorting , especially if Cachecade/Fastpath is enabled

I used to do a heap of playing with this stuff when money was no object

As your Mobo is so integrated with everything built in you can't be using hardly any of the PCIe slot potential
2x PCIe 16x slots with full bandwidth begging for suitable SLi quadro Video cards
When not crunching video, you could be Bitcoin mining as a sideline :)

LSI 3108 - Yes

Cachecade - Yes & enabled (No SSD cache / always on a well checked UPS)

The MD1200 is something I'm investigating.. I have found one with 12 x 10Gb 7.2k SAS drives, but unconfirmed availability just yet, and I'm unsure if an H800/810 controller comes with it.

Are you suggesting that I abandon the hotswap in-case array and solely use the MD1200... I hadn't considered doing that.

The board doesn't have external SAS ports at all, so would need to find something to route the two cables. There must be a simple full-height adapter that has a pair of cables on the inside and female sockets on the outside.

I assume the speed gain using the MD1200 would be significant... 800Mb/s at present, to who knows what for 10 x drive Raid60 with 2 hot spares... attractive thought!

You're right... none of the ports are used, but the video encoding is all done on the 2 PC's... 10-core/20-thread cpu each, pretty adequate.

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

freakngeek

356 posts

Ultimate Geek

#2041542 21-Jun-2018 10:03

I'll have a closer look at your Server Mobo to see how its setup with SAS ports tonight and give some scenarios
Might get a little complicated with SAS drives in Sever and running MD1200

fritzman

366 posts

Ultimate Geek

ID Verified

#2041565 21-Jun-2018 10:30

freakngeek:

I'll have a closer look at your Server Mobo to see how its setup with SAS ports tonight and give some scenarios
Might get a little complicated with SAS drives in Sever and running MD1200

Thanks... yeah, I think I can grab an H810/830 for around $200/$300 and it may not be worth the hassle to not use the in-case SAS as the primary storage, and the MD1200 as required. I have in mind partitioning the MD1200 so say 10Tb is available for Video stuff and the balance (60Tb if I use Raid60 with 2 hot spares) as offsite storage for backups using RSync, as a business opportunity (for the Church).

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

freakngeek

356 posts

Ultimate Geek

#2041614 21-Jun-2018 11:41

Does the MD1200 have an expander the H810 will probably be the plain 8 SAS port version, so am thinking yes then ...

Ideally use the Servers SAS3108 controller as you have to go via this any way even with a H810 controller in MD1200
The MD1200 will optimally need 2x 4port SAS connectors (I think Mobo has 2) to work at its best but one will suffice

BUT the Mobo's 2x 4port connectors are probably also linked to the 8x single port headers on the Mobo
You may need to free up either all 8x SAS ports so you can run the 2x cables to the MD1200 for ultimate performance
Or
Free up 4x SAS ports then use a single 4port cable to the MD1200, you could run a set of SSD for Cachecade/Fastpath to kick some serious butt, but the path to MD1200 will be limited with just 1x 4port connector

Cables are expensive and make sure you get the correct ones as they are directional, VERY easy to get wrong

Do your homework, RAID and SAS are as intricate as LAN, can cost a packet if you make mistakes

At home I run a LSI 9266-8i with an expander to run 8x 2TB drives+ some other drives and 4x SSD's in RAID0 for OS
I don't have Cachecade but not really needed to backup our stuff at home, but would be nice.

fritzman

366 posts

Ultimate Geek

ID Verified

#2041621 21-Jun-2018 11:51

freakngeek:

Does the MD1200 have an expander the H810 will probably be the plain 8 SAS port version, so am thinking yes then ...

Ideally use the Servers SAS3108 controller as you have to go via this any way even with a H810 controller in MD1200
The MD1200 will optimally need 2x 4port SAS connectors (I think Mobo has 2) to work at its best but one will suffice

BUT the Mobo's 2x 4port connectors are probably also linked to the 8x single port headers on the Mobo
You may need to free up either all 8x SAS ports so you can run the 2x cables to the MD1200 for ultimate performance
Or
Free up 4x SAS ports then use a single 4port cable to the MD1200, you could run a set of SSD for Cachecade/Fastpath to kick some serious butt, but the path to MD1200 will be limited with just 1x 4port connector

Cables are expensive and make sure you get the correct ones as they are directional, VERY easy to get wrong

Do your homework, RAID and SAS are as intricate as LAN, can cost a packet if you make mistakes

At home I run a LSI 9266-8i with an expander to run 8x 2TB drives+ some other drives and 4x SSD's in RAID0 for OS
I don't have Cachecade but not really needed to backup our stuff at home, but would be nice.

I'm not sure... the guy I'm dealing with has been away and isn't back until tomorrow so more info then.

I'm not too concerned about the MD1200 yet until I am able to confirm that I own it, even if I simply sell the drives off and the rest as parts if it doesn't work out, but it does sound like a nice option.

Like you, I have an Areca 1882IX-16 full up with 3Tb drives in Raid60, with 4 x 8Tb drives for backups (Norco 20-bay case)... nice and quick without the need for ssd's.

Sons Rig: Asus TUF Gaming X-570, Ryzen 9 3900X, G.Skill neo 2x16Gb 3600's, Sabrent Rocket 1Tb M.2, Win10 Pro, Phanteks case, EVGA G5 850W.

NAS: DS1819+ - 52Tb in Raid6

My rig: HP Elitebook X360 Lappy with a 2Tb SN850.. woohoo.. I've retired!

Heat under fritzman (152-0-0)

1 | 2