bloomery says Mistral Large 4 can run with one 24GB RTX 3090 and host memory

A trillion-parameter model with a single consumer GPU is bloomery 0.3.12’s strongest hook—with the host doing essential work. The community inference server’s November 3 release says Mistral Large 4, a mixture-of-experts model with 52 billion active parameters, can run with one 24GB RTX 3090 using Unsloth’s UD-IQ1_S GGUF. Dense and attention weights sit on the card; routed experts stream from host memory.

That makes 24GB the GPU budget, not the full system-memory requirement. It also describes a specific low-bit build, not full-precision model quality or performance on every desktop. The release lists support for both UD-IQ1_S and Mix-256G, with --place a selecting the documented single-card route.

The reported result and its conditions

The maintainer reports 10.85 decode tokens per second and 36.01 prompt tokens per second on the RTX 3090 after a 4,096-ID prompt. The measurement uses the release’s server at serving defaults with --ctx-size unset. The notes identify binary revision 2cd109a60 and say the tag shares its source apart from README status changes. A shorter-prompt row that read experts cold from disk was excluded from the published table.

These are the project’s measurements, not an AITrending reproduction or a comparison proving superiority over another runtime. The same release reports contributor-measured Qwen improvements: for Qwen3.6-35B-A3B with multi-token prediction on an RTX 5090, 4,096-ID prompt prefill rose from 2,071.6 to 8,747.2 tokens per second, while first-token latency fell from 1,979 to 470 milliseconds. Different models, quantizations and hardware have separate results.

A server for existing clients

bloomery is an MIT-licensed Rust inference server with llama-server-compatible OpenAI and Anthropic interfaces. Model files and their licenses remain separate. Existing chat or coding-agent clients can point at the local endpoint; the server supplies inference, while the chosen agent harness supplies its workflow and tools.

The tagged README documents a release installer, Homebrew and Docker. Linux x86-64 requires AVX2, FMA and BMI2, glibc 2.34 or newer, a supported NVIDIA GPU and an R580-or-newer driver. Windows uses WSL2. A separate Apple-silicon Metal binary has its own supported models and limits; the Mistral RTX 3090 measurement should not be read as a Mac result.

For automation, the English guide documents a JSON recommendation command, a download-free dry run and a ready event before requests are sent. Gated model repositories require a Hugging Face token and acceptance of their model terms. The server binds localhost by default; the serving guide documents API-key scopes and rate limits for deployments that expose it more broadly.

The release also discloses known correctness and loading limits, including history-dependent GLM-5.3 output under --place bp and a DeepSeek sidecar loading restriction. Its notes initially marked the GB10 archive as pending; the release asset list contained that archive when reviewed. The practical news is a new, explicitly described local-serving option, with memory placement and measurement conditions visible to readers.