Video Card for AI for Local Data Crunching

Forums › Desktop computing › Video Card for AI for Local Data Crunching

Gurezaemon

~HONYAKKER!~
1355 posts

Uber Geek

ID Verified

Lifetime subscriber

#311001 5-Dec-2023 19:56

My company is setting up a local AI machine running PrivateGPT for language model data crunching.
Because of extreme confidentiality requirements, this needs to be on a local machine - not online. Cloud services aren't an option.

In terms of graphics cards, I know that faster is better, but we're happy if things aren't super snappy - if the machine can plod through the tasks needed, that should suffice.
We also understand that having more video RAM may be more important that sheer clock speed.

We'd like to keep the budget to no more than NZ$900, and if possible keep power consumption at ~200W or less.

Our thoughts are a RTX 4060 TI 16 Gig that I can source from Japan within our price constraints, but I'm open to any suggestions.

Get your business seen overseas - Nexus Translations

mentalinc

3242 posts

Uber Geek

Trusted

#3168558 5-Dec-2023 20:01

So you're spending $900 on the processing power?

Super concerned about how little you might be spending on everything else involved, policy, process, controls reporting etc etc etc.

What's the expected ROI, if you're only spending $900?

CPU: AMD 5900x | RAM: GSKILL Trident Z Neo RGB F4-3600C16D-32GTZNC-32-GB | MB: Asus X570-E | GFX: EVGA FTW3 Ultra RTX 3080Ti| Monitor: LG 27GL850-B 2560x1440

Quic: https://account.quic.nz/refer/473833 R473833EQKIBX

gzt

17159 posts

Uber Geek

Lifetime subscriber

#3168562 5-Dec-2023 20:18

Why source overseas? Local prices look similar.

Gurezaemon

~HONYAKKER!~
1355 posts

Uber Geek

ID Verified

Lifetime subscriber

#3168566 5-Dec-2023 20:55

mentalinc:

So you're spending $900 on the processing power?

Super concerned about how little you might be spending on everything else involved, policy, process, controls reporting etc etc etc.

What's the expected ROI, if you're only spending $900?

This is for a proof or concept machine rather than the full blown operational supercomputer stuff. We HAVE it running on a RTX 2060 and it works OK, but a bit too slow for usability. Since this is a new-ish build, we're looking for a new card, hence the question.

For ease of management, this will be in a corner of my office, totally air-gapped, and used for things like processing bilingual glossaries, etc.

Get your business seen overseas - Nexus Translations

eonsim

398 posts

Ultimate Geek

Trusted

#3168573 5-Dec-2023 22:37

It's not going to be fast running on a 4060TI but 16GB should help a bit in the size of the model you can run, note you can pick up some 4060 TI 16GBs at around that price point from Aussie https://www.mwave.com.au has a few.

I'd recommend you look at what other GPU options can be used, potentially modern Mac's using Metal or AMD cards using ROCm as well (which tend to have more RAM for the same price than NVIDIA).