You've probably lived through this. You find a new model on Hugging Face, download 15, 20 GB of it, wait through the whole thing, fire it up, and it crawls at 2 tokens per second. Or worse, it doesn't even load, memory runs out before the first response comes back.
That's not on you. Running local AI has always relied on guesswork. You see "7B," "13B," "70B" in a model's name and assume it'll work, with no real way to know until you've already burned the time on the download.
The problem was never the model or the hardware. It was the gap between what a model needs and what your machine actually has.
What llmfit is
llmfit is an open-source terminal tool, written in Rust, created by developer Alex Jones. It asks one question, but asks it properly: which of these models will actually run well on your specific machine.
Running locally, it detects your RAM, CPU cores, and GPU VRAM (including multi-GPU setups, Apple Silicon's unified memory, AMD via ROCm, Intel Arc), cross-references those numbers against a database covering hundreds of models from dozens of providers, and returns a ranked list. Each model gets scored across four dimensions: expected quality, estimated tokens per second, how well it actually fits your available memory, and the maximum context length your machine can handle without spilling over to disk.
None of this touches the internet. The model database ships embedded in the binary, so the tool responds instantly, no external calls required.
Why this is more interesting than it sounds
The part that stands out isn't just "detects hardware," benchmark tools have done that for years. It's how it handles quantization. Instead of assuming you want Q4 or Q8, llmfit calculates memory requirements across the full quantization ladder, from Q8_0 down to Q2_K, and picks the best one that still fits what you actually have available.
The detail that keeps coming up in reports from people who've tested it is how it handles MoE architectures (mixture of experts, where only part of the parameters stay active at once). Most estimation tools treat every model as dense and calculate memory off the total parameter count, which unfairly penalizes MoE models. llmfit recognizes that architecture and adjusts the math, so a model like that shows up with a fit score much closer to reality.
It also integrates directly with Ollama and llama.cpp. Once it finds the right model, you can pull it down without ever leaving the terminal.
A real-world test
Someone who tried it on a 2019 laptop, 8GB of RAM, no dedicated GPU, got Phi-mini-MoE-instruct as the top recommendation, scoring 90 out of 100, with an estimated 40 tokens per second. Running it for real came in a bit below that estimate, but still comfortably in usable territory. On modest hardware, with zero manual tweaking, the tool nailed the right model on the first try.
The field moves too fast for any list to stand still
New models ship in this space practically every week, and a fixed database, no matter how thorough, is always playing catch-up. That's the price of any catalog moving at this pace. The community around llmfit already responded to it: a fork called "Cookbook" emerged, with more frequent curation, built specifically to keep pace with the constant drop of new models without depending solely on the original tool's release cycle.
That's the kind of signal that tends to show up around open-source tools that actually gained real traction. They start by solving one specific problem, build a community around it, and that community starts stretching what the original tool does.
For anyone starting to move away from cloud AI toward local models, having a tool that points the way before the first download already removes the biggest source of frustration in that phase.
If you want to try it yourself, llmfit is open-source and free. The code is on GitHub: github.com/AlexsJones/llmfit
Comments (0)
Comments are moderated and if they violate our Terms and Conditions of use, the comment will be deleted. Persistence in violation will result in a ban of your account.