bloomery puts Bolt 74B on a 24GB card

bloomery 0.3.4 gives local-model users a concrete new target: Nemotron 3.5 Bolt 74B-A9B on one 24GB graphics card. The community inference server’s English release notes report 86.69 decode tokens per second on an RTX 3090 capped at 250 watts, using the named 24GB GGUF build at depth 4,096. That is the project’s measurement, not an independently reproduced AITrending result.

The release was published October 17. Its notes say the tag moved to release head 9723698f2 on October 18 to include further changes. That distinction matters for anyone comparing an earlier download with today’s documentation.

More than a chat endpoint

The Rust server presents llama-server-compatible OpenAI and Anthropic interfaces, making it relevant to existing local chat and agent clients. Version 0.3.4 also holds EmbeddingGemma 2 or the earlier 300M EmbeddingGemma beside a chat model through --embd-model and /v1/embeddings. The newer embedding model supports sampled video frames and mono 16kHz audio, according to the notes.

A separate addition lets several DGX Sparks serve one model by assigning ranges of routed experts to another Spark over RDMA, with that node paging experts from its own NVMe storage. The project reports matching single-Spark output for Qwen3.8. It explicitly warns that GLM-5.3-Flash and DeepSeek-V4.1-Flash greedy output can differ when expert placement changes activation quantization. No multi-node speed claim is made.

Check the hardware route before installing

Prebuilt releases require Linux x86-64, AVX2, glibc 2.34 or later, an NVIDIA GPU with compute capability 8.6 or later, and an R580-or-newer driver for CUDA 13. Homebrew, Docker and a release installer are documented. Windows users run inside WSL2; DGX Spark uses a source build rather than the x86-64 tarball. Native macOS and AMD GPUs are unsupported.

Bolt currently serves one slot on one card. The notes explicitly exclude 12GB cards for this Bolt configuration and refuse its two-card placement mode. Server code is MIT licensed, while model terms and hardware costs are separate. AITrending reviewed the English release and tagged README without running the engine. The useful news is a specific local-serving option with stated limits, rather than a universal speed promise.