STROT deletes its LLM fallback after a hate-speech test failed to justify the cost

STROT added a runtime LLM fallback, then removed it twenty minutes later. The open-source decision compiler’s October 18 release, version 0.11.1, deletes automatic escalation to a second model after its author reported that a hate-speech experiment bought little accuracy at a real per-record cost.

The change restores the project’s central pitch: use an LLM during compilation, then run a frozen decision pipeline locally. That does not mean no AI at runtime. The default pipeline still uses a small local model to classify, score or route each record; it no longer automatically asks a remote LLM to resolve uncertain cases.

The fallback did not rescue this task

Version 0.11.0 introduced escalate_to, allowing uncertain records to go to a stronger model. The published experiment used 400 test tweets from the tweet_eval hate dataset, with a distilled local model and Claude Haiku 4.5 as the escalation tier.

The author reports 63.7% combined accuracy. The local tier handled 255 tweets at 63.5% accuracy, while Haiku handled the remaining 145 at 64.1%. These are different subsets, so the two tier percentages are not a matched comparison of both models on identical inputs. The release nevertheless records the author’s conclusion that this cascade did not justify its additional runtime dependency and cost. AITrending has not independently reproduced the experiment.

A second finding makes the example more instructive than a headline accuracy score. Optimizing overall accuracy shifted predictions toward the majority class and reduced hate-class recall to 22%, according to the benchmark report. The author argues that recall or F-beta is a better objective when catching the minority class is the purpose.

Compile the task, leave uncertain decisions to the caller

STROT takes records, a task description and a bounded set of decisions. During compilation, an LLM synthesizes labels, input projections and, for routing, handlers. Known-answer examples can gate promotion of the resulting Python artifact. Compilation can involve several model calls for synthesis, repair or labeling; the “compile once” pitch describes the cached phase, not a promise of exactly one API request.

The retained target_accuracy option fits local corrections and a cutoff. Below that cutoff, the pipeline abstains and the consuming application decides what happens next. The guide describes this as an empirical policy, not a finite-sample guarantee. It also documents a separate conformal risk option whose assumptions and error definition differ.

What developers need to run it

This is a Python library and generated pipeline, rather than a plugin for a particular coding agent. The MIT-licensed package is available on PyPI; its default local setup uses strot-ai[local], Python 3.11 or later, the Julia-1 inference package and a downloaded checkpoint. Julia-1 has its own Apache-2.0 license.

Compilation needs a configured model client, such as an Anthropic or OpenAI API key, or the documented local Claude Code CLI route. Default local execution can run offline after dependencies and weights are installed. Optional hosted decision-model configurations still need provider access and can incur inference charges. Local hardware and compilation also have costs, so the project’s zero-marginal-cost framing refers to avoiding per-record service calls in that local mode.

The developer guide explicitly says its code lint is not a sandbox: compilation executes candidate artifacts against examples. Untrusted intent therefore needs external isolation. The interesting release is a narrow architectural decision, backed by an openly reported disappointing result: keep automatic escalation out of the default runtime, and make abstention an explicit application choice.