I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).

I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.

Generic setup doc here: (not fully updated with GUFO): https://github.com/Selicos/AI_Tools_Public/blob/main/docs/Fe...

Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.

I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.

Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.

I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.

IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.

Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.

My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.

Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.

Ask HN: What hardware are you using to run local LLMs?