ThinCtx says its compact Qwen3.5-4B fits 128k context in 4 GB of GPU memory
The new compact conversion trades slower prompt reading and system RAM for a smaller GPU footprint, using a dedicated runtime and restricted commercial licensing.
A smaller GPU budget for long conversations
Thinletter has released ThinCtx-Qwen3.5-4B-small, a compact conversion whose model card claims 128k tokens of context with 4.0 GB of GPU memory for weights and context, compared with 12.6 GB for the original model. The October 14 release targets people running long conversations or documents on local NVIDIA hardware.
That headline describes GPU memory, not the computer’s total memory requirement. The author says the full context is retained in system RAM, occupying another 1.4 GB at that length. The conversion stores weights at about 5.4 bits each and compresses the context store; its reported download is 3.63 GB. Download size and runtime memory are different measurements.
The tradeoff is explicit: ThinCtx reads long prompts more slowly than the base model. AITrending has not independently reproduced the memory figures or performance results. The card’s function-calling comparison covers the first 50 items in each BFCL non-live category, so it does not establish equal quality across every workload.
A specific runtime, not a drop-in GGUF
The documented route is pip install "thinctx>=0.3.1", followed by thinctx serve thinletter/ThinCtx-Qwen3.5-4B-small --port 8000. The server offers an OpenAI-compatible chat-completions endpoint with streaming and tool calling; HTTP basic authentication is available through --auth USER:PASS. A terminal chat command is also provided.
Requirements are Linux x86_64, CPython 3.12, an NVIDIA GPU and PyTorch 2.11. The default PyTorch wheel targets CUDA 13; the card gives a CUDA 12.8 installation option for older drivers. The author stresses that the dedicated ThinCtx runtime produces the context-memory behavior; loading these weights through another inference library does not reproduce it automatically.
The weights use PolyForm Small Business 1.0.0, with added permissions for personal non-commercial use, non-commercial research and education, and evaluation by anyone. Organizations outside its small-business qualification need a commercial license for production. Upstream Qwen’s Apache 2.0 terms still apply to its materials, but that does not make this conversion unrestricted Apache-licensed software.