Bonsai 2's RTX 3090 conversion reports 297 tokens/s

A compact new route to local Bonsai inference

WarlaxZ has uploaded a native NInfer conversion of Ternary Bonsai 2 27B for the RTX 3090 fork, accompanied by an eye-catching author result: 297 tokens per second on a short coding prompt using DFlash2 speculative decoding. The new artifact arrived on October 15, 2026. It converts Prism ML’s existing Bonsai model; it does not mark a new release of the underlying Qwen3.8-27B base model.

The package is a 9,520,051,456-byte file, about 8.87 GiB, named bonsai2_27b.ninfer. Alongside the ternary text tower, the card lists a vision tower, MTP and DFlash2 components, and an optional proposal head. A manifest records component revisions and source hashes, giving local-model users a concrete artifact to inspect and reproduce.

This is a separate runtime route from downloading the upstream GGUF. The output is a native .ninfer container, and older builds that do not recognize its t2_g128_fp16 weight format will refuse it. The text weights use ternary values with an FP16 scale per 128 weights, which the card describes as 2.125 bits per weight.

What the speed claim actually measures

The author reports the coding result on one 24 GB RTX 3090, running Windows and CUDA 12.8, with greedy decoding, INT8 KV cache, thinking disabled and 300 generated tokens. On that prompt, the card reports 88 tokens/s without speculation, 199 with three MTP drafts and 297 with seven DFlash2 drafts.

A separate four-prompt set covering code, networking, arithmetic and an email averaged 249.9 tokens/s with DFlash2, ranging from 173 to 325. These are the author’s indicative, few-prompt measurements. AITrending has reviewed the English documentation and provenance, but has not run the model or independently reproduced its speed or quality results.

The card also reports 6.70 GiB of device weight memory without drafter heads and 7.28 GiB with MTP. Those are weight-memory figures, not a promise that the complete runtime, context cache and every decoding configuration fit in that much VRAM.

How to try the conversion

The model card tells users to build ashalliants/ninfer-3090, then use its download-model script with bonsai2-27b. Its direct CLI example selects a 32,768-token context, INT8 KV cache and DFlash2 with seven draft tokens. The card declares Apache 2.0 licensing and provides a conversion report as well as the artifact manifest.

For RTX 3090 owners, the immediate appeal is a small, documented alternative for running the existing 27B model locally. Compatibility matters: this is not a drop-in GGUF for an arbitrary desktop client, and NInfer does not itself execute generated tool calls. The useful new development is the downloadable conversion and its reported single-card results.