Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


Zeon

3926 posts

Uber Geek
+1 received by user: 759

Trusted

#312891 27-May-2024 14:12
Send private message

Hey guys,
After years of hiring GPUs through various cloud providers for training transformer models, we are looking to build our own deep-learning rig. The major driving force for this has been availability issues although there may be some slight cost benefits also. After looking at different hardware our team would prefer 4090 GPUs. The system will have 4x 4090s so the only cooling option is water cooling.

 

While we usually deal with our own hardware, with no experience in water cooling the last thing we want is for leaks to damage what will be very expensive components. We have had some discussions with Computer Lounge to build this system so far but trying to get them to actually want to take on the project has been like drawing blood from a stone. I appreciate they have had some supply issues with their water cooling parts vendor (EKWB) but I don't have too much confidence we can progress this work with them anytime soon. We also want some customizations like a heat exchanger so we can hook the loop into our building's water cooling during the summer months which they are quite hesitant on installing as part of the loop it seems.

 

I wanted to check who else may be worth while contacting? We would prefer an NZ vendor but may be forced to look internationally.

 

We would be looking for something like this:

 

 

 

 

Heat exchanger seems to look like this:

 

Brazed Plate Heat Exchanger, 214.5 x 80.7 x 24.1mm





Speedtest 2019-10-14


View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
ratsun81
516 posts

Ultimate Geek
+1 received by user: 183


  #3235310 27-May-2024 16:44
Send private message

Looks like your after an enterprise solution but for some reason custom built? 

 

When you say hooking the system up to your buildings cooling what do you mean by that?

 

Ive done plenty of home pc watercooling i also work in IT Infrastructure and generally would be looking for something off the shelf rather than custom built... Lots of maintenance is needed for a custom loop let alone the risks of hooking up to a building heat exchanger. 

 

Perhaps check out alphacool, they arent local but do offer custom solutions and enterprise type systems. 





Quic Broadband

 

Use R212389ELFLL2 promo code for free setup at checkout.




tweake
2641 posts

Uber Geek
+1 received by user: 1137


  #3235319 27-May-2024 17:27
Send private message

i've not done much on water cooling, tho am using a pc radiator for another project at the mo, certainly never done a rackmount.

 

biggest thing is leaks. avoiding leaks and also mitigation should the worse happen. what is it going to leak on, whats in the lower racks.

 

what protection circuits, eg flow detection, auto shutdown etc. pumps typically run all the time, but radiator fans can be variable or staged depending on temps. i see a mention of putting it under vacuum so water cannot leak, but you need stiff pipes (rubber will collapse) and pumps don't like a lot of suction. pumps are better at pressure than suction. 

 

i would not touch building systems. especially a you don't know what chems they are using in the water supply which could react with your setup. cool it to the room and let the rooms cooling system deal with it. if you need extra cooling peilter etc, or hack a dehumidifier to use as a cooler so you can get the water down below room temp.


Zeon

3926 posts

Uber Geek
+1 received by user: 759

Trusted

  #3235331 27-May-2024 17:59
Send private message

ratsun81:

 

Looks like your after an enterprise solution but for some reason custom built? 

 

When you say hooking the system up to your buildings cooling what do you mean by that?

 

Ive done plenty of home pc watercooling i also work in IT Infrastructure and generally would be looking for something off the shelf rather than custom built... Lots of maintenance is needed for a custom loop let alone the risks of hooking up to a building heat exchanger. 

 

Perhaps check out alphacool, they arent local but do offer custom solutions and enterprise type systems. 

 

 

There are a few companies making such setups but all would be international purchases with no local support. I guess I'm looking for a local provider who can then provide ongoing support - especially considering the water cooling aspect. If there is a leak I can get them in to sort it as no one has much water cooling experience. I also would hope they are highly experienced to build a solid setup in the first place.

 

The building has water piping throuhgout each floor that takes heat from the air conditioning units to a cooling tower on the roof. My plan would be to put a heat exchanger on that and run a small local loop that would go to a local manifold that would then connect to each water cooled server in the server room. So in effect a heat exchanger in each servers loop to a local loop and then a heat exchanger from the local loop to the building loop.

 

We are running around 5KW currently and in the summer our server room's air cooling can barely keep up with it. If we get say 10KW extra of this GPU gear we will need to hugely increase the air conditioning and it will add significant additional electrical cost just for the cooling - maybe as much as the equipment itself is using....

 

Currently the plan is to just run the units around the office during winter and move into the server room during summer.





Speedtest 2019-10-14




ratsun81
516 posts

Ultimate Geek
+1 received by user: 183


  #3235451 27-May-2024 21:52
Send private message

Zeon:

 

ratsun81:

 

Looks like your after an enterprise solution but for some reason custom built? 

 

When you say hooking the system up to your buildings cooling what do you mean by that?

 

Ive done plenty of home pc watercooling i also work in IT Infrastructure and generally would be looking for something off the shelf rather than custom built... Lots of maintenance is needed for a custom loop let alone the risks of hooking up to a building heat exchanger. 

 

Perhaps check out alphacool, they arent local but do offer custom solutions and enterprise type systems. 

 

 

There are a few companies making such setups but all would be international purchases with no local support. I guess I'm looking for a local provider who can then provide ongoing support - especially considering the water cooling aspect. If there is a leak I can get them in to sort it as no one has much water cooling experience. I also would hope they are highly experienced to build a solid setup in the first place.

 

The building has water piping throuhgout each floor that takes heat from the air conditioning units to a cooling tower on the roof. My plan would be to put a heat exchanger on that and run a small local loop that would go to a local manifold that would then connect to each water cooled server in the server room. So in effect a heat exchanger in each servers loop to a local loop and then a heat exchanger from the local loop to the building loop.

 

We are running around 5KW currently and in the summer our server room's air cooling can barely keep up with it. If we get say 10KW extra of this GPU gear we will need to hugely increase the air conditioning and it will add significant additional electrical cost just for the cooling - maybe as much as the equipment itself is using....

 

Currently the plan is to just run the units around the office during winter and move into the server room during summer.

 

 

No-one local is going to want to touch this with a barge pole from a CGA/Warranty/service perspective.

 

Have you figured out how to deal with the regular gpu block maintenance needed, no one building this will be interested in that.

 

What about the galvanic corrosion factor when you hook into the building water system copper vs alu vs whatever else exists in the system? 

 

Water cooling is not a simple option. it has a ton of nuances and unless is planned and built properly from the start can be an utter poop show.

 

Trying to tie into building loops is fraught with issues. You would need to incorporate quick disconnects to existing pipes.

 

What about internal pipe maintenance once you have quick disconnects? 

 

Trying to get around/bypass putting in the needed cooling to your server room is going to cost you more than it saves in the long run. 

 

My advice, look at other upgrades to your existing cooling setup, good insulation upgrades or underfloor water options that will not create massive headaches. 

 

 





Quic Broadband

 

Use R212389ELFLL2 promo code for free setup at checkout.


CMTechNZ
12 posts

Geek
+1 received by user: 4


  #3244639 5-Jun-2024 10:23
Send private message

Hey Zeon,

 

That looks like a Comino workstation. I have installed something similar on a vessel that was connected to the ships chilled water loop via a liquid/liquid heat exchanger so it is possible but you would need a lot more info regarding the buildings chilled loop system etc. Usually they run year round for de-humidifying prior to heating but each system varies. Permission would need to be granted for any system modifications and it may incur charges depending on cooling requirements. The Comino equipment is industrial quality gear but comes with the associated cost. EKWB is fine for consumer gear but understand why they are hesitant to use it for these requirements. 


Zeon

3926 posts

Uber Geek
+1 received by user: 759

Trusted

  #3315488 3-Dec-2024 14:19
Send private message

Hey all, so have finally found a local guy who has done a lot of water cooling work on GPU builds. He works mainly with Alpha cool. I have just ordered the first round of GPUs. They are becoming hard to get - needed to order from Australia. Hopefully the transaction gets finalized and will keep this thread updated as progress is made on the build.

 





Speedtest 2019-10-14


 
 
 

Shop now on AliExpress (affiliate link).
Qazzy03
545 posts

Ultimate Geek
+1 received by user: 491


  #3315505 3-Dec-2024 15:16
Send private message

The 5090's are being released around Q1 2025 so Nvidia discontinued the 4090 product.

https://www.forbes.com/sites/antonyleather/2024/09/21/buy-an-nvidia-rtx-4090-before-its-too-late-shortages-predicted/

 

 


mentalinc
3384 posts

Uber Geek
+1 received by user: 1023

Trusted

  #3315515 3-Dec-2024 15:54
Send private message

Holy import duty when they land!





CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB:  Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

 

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX 


ratsun81
516 posts

Ultimate Geek
+1 received by user: 183


  #3315526 3-Dec-2024 16:25
Send private message

mentalinc:

 

Holy import duty when they land!

 

 

Amazon pre-charges the import fees, thats the 4k already charged on his screenshot...





Quic Broadband

 

Use R212389ELFLL2 promo code for free setup at checkout.


mentalinc
3384 posts

Uber Geek
+1 received by user: 1023

Trusted

  #3315532 3-Dec-2024 16:38
Send private message

That's just the GST.





CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB:  Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

 

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX 


richms
29098 posts

Uber Geek
+1 received by user: 10208

Trusted
Lifetime subscriber

  #3315535 3-Dec-2024 16:43
Send private message

mentalinc:

 

That's just the GST.

 

 

What duty/tariff applies to GPUs? Never have I had that applied.





Richard rich.ms

 
 
 

Shop now at Mighty Ape (affiliate link).
Zeon

3926 posts

Uber Geek
+1 received by user: 759

Trusted

  #3327273 3-Jan-2025 14:26
Send private message

Ordered a case from some company in Shenzen using ASRock motherboard and PCI-E daughter board. Got the first water block on a 4090. Will need to make some modifications to the lid to fit the water and power connectors (and higher card). All the GPUs are here now so waiting on the remaining waterblocks to arrive. All are Alphacool blocks.

 

 

Also now running a self contained water cooled tower with 4x 4090 and threadripper so we could get up and running. Temps are high at >80deg but it just has 2x 360mm radiators. This one will have another 2x 4090 added for 6x in total and has already had the water cooling system designed so we can swap it out with an external CDU and remove the radiators and pumps. It has the hose for this looped back in on itself for the moment:

 

 

 

 





Speedtest 2019-10-14


SomeoneSomewhere
1882 posts

Uber Geek
+1 received by user: 1086

Lifetime subscriber

  #3327301 3-Jan-2025 16:31
Send private message

I think your CPU coolers are in backwards on that rackmount build? Typically all airflow would be front to rear, and that's how the mid-case fans are mounted.

 

 

 

One concern about using chilled water is that you'll be below ambient, with all of the condensation issues that poses. You'll probably want a thermostatic valve on the chilled water line to regulate the loop water temperature around 25-30C. 

 

 

 

 


ratsun81
516 posts

Ultimate Geek
+1 received by user: 183


  #3327306 3-Jan-2025 16:44
Send private message

SomeoneSomewhere:

 

I think your CPU coolers are in backwards on that rackmount build? 

 

 

I agree the cpu fans are airflow towards the front, not likely a cooler mismount but the fans themselves.

 

And the rest of the case fans are airflow rear, looks like the factory has messed up the cpu cooler fans or the airflow sticker direction. 

 

 





Quic Broadband

 

Use R212389ELFLL2 promo code for free setup at checkout.


SomeoneSomewhere
1882 posts

Uber Geek
+1 received by user: 1086

Lifetime subscriber

  #3327307 3-Jan-2025 16:47
Send private message

It looks like the fans were probably installed into the coolers before the coolers were put in the board, so the whole cooler is likely 'backwards'. But yeah, they're probably symmetrical once you turn the fan around.


 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic








Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.