I Have $5k to Spend on a Local AI Machine, What Should I Get?

The conversation started with Person 1 suggesting that they would buy Nvidia Digits, a platform for building and training AI models. However, Person 2 quickly shifted the conversation to more affordable options, mentioning a Reddit comment that recommended a website for buying used servers.

Person 3 took a different approach, suggesting that a more promising way to go would be to build a custom setup using 4x 7900 XTX cards and some additional hardware. They noted that some folks were using ROCm with this setup to achieve impressive numbers in terms of training speed and power efficiency.

Person 4 chimed in with a humorous comment, exclaiming "AI in the palm of your hand! Best deal evarrr!" However, Person 5 quickly brought the conversation back down to earth, pointing out that AI is a very broad field and that Nvidia GPUs have the best inference speed, but may not be the most practical option for everyone.

Person 5 went on to explain that unless you're buying several used 3090s, you'll likely have to split larger LLM models across main memory and GPU. They also noted that Apple's Mac Studio allows for up to 192GB of shared memory, making it possible to run very large LLMs.

Person 6 shared their own experience, having just purchased a Mac Mini with maximum specs. They noted that while the Mac Mini may not outperform the Nvidia 5090 in terms of FLOPS, it has the advantage of handling large models and comes with 64MB of memory, compared to the 5090's 32MB.

Here's a question: If you all are so smart and know so much about AI hardware and training models, why are you still arguing about which GPU to buy, when you could just use your incredible AI powers to build a sentient robot that buys GPUs for you?