Ollama’s new local models make decisions inside everyday software

A repair shop receives a message saying a replacement part arrived damaged. Before anyone writes a reply, the request needs a destination: shipping, technical support or someone who can untangle an unusual case. In this hypothetical shop, that first decision is a small piece of work. It still depends on understanding what a customer means.

Ollama’s September 29 release makes that kind of judgment a new local AI function. Version 0.35 supports Bespoke Labs’ Nimble and two experimental Tev1 models from Together AI through a dedicated decision interface. They can return answers that ordinary software uses directly, with the computation running on the user’s machine. Ollama’s announcement.[1]

For the shop, the attraction would be a faster handoff: a message reaches the appropriate queue without a chatbot composing a paragraph along the way. A customer might never see the model. Whether that improves the service depends on the next part of the design—what happens when the message is ambiguous, the options are incomplete or a confident answer sends the job to the wrong person.

The new releases bring an emerging AI approach within reach of developers who want to run it themselves. Their documentation also exposes the details that determine whether it belongs in a real workflow: short input limits, different memory requirements, uneven accuracy and probabilities that need careful interpretation. Those details are more revealing than the appealing phrase “instant decisions.”

Set the possible answers, then ask the model to choose

TypeSafe introduced its hosted Jev model on September 15 around a restricted interface: provide information and ask for a selection, a judgment on an ordered scale or a probability that a statement is true. The company presents this as a way to put AI judgments inside software. Its launch argues that eliminating generated strings avoids type errors—answers that do not fit the structure an application expects. That is a narrower achievement than guaranteeing the judgment itself. TypeSafe’s explanation.[10]

Consider the repair request again. An application could offer “shipping,” “technical support” and “review” as its allowed destinations. It can safely read one of those labels without trying to extract a department name from a conversational response. Yet a broken part caused by incorrect installation might belong with technical support, while a box crushed in transit might belong with shipping. Restricting the labels leaves that distinction for the model to interpret.

Nimble implements the idea by scoring permitted answer tokens directly. Its reference implementation treats questions independently; it does not produce an explanation or a chain of reasoning. Bespoke’s documentation also makes clear that this interface does not provide arbitrary text extraction or nested generated documents. Nimble’s implementation.[2]

That creates a useful division of labor. The application defines the destinations and records what happened. The model interprets the message within those choices. If an employee needs to know why a parcel was routed, the original message and the routing rules remain essential evidence; the label alone cannot provide the reasoning. A service designer can make that evidence visible without asking the classifier to invent an explanation afterward.

Interest extends beyond local models. At DevDay on September 29, OpenAI announced a limited preview of a Decisions API using Luna, designed to return finite answers about supplied text or images. Broader availability was still planned. That puts a large hosted provider alongside the smaller companies exploring this interface, although their implementations and availability differ. DevDay recap.[9]

What actually fits on a local machine

Ollama lists the Nimble download at roughly 9.3–9.5 gigabytes. Tev1’s 4B version is roughly 4.4–4.5 gigabytes, while its smallest version is around 800 megabytes. These are model-file sizes, not complete estimates of the memory an application will use. Nimble downloads and Tev1 downloads.[6][7]

Bespoke’s own Nimble card illustrates the distinction: its smaller adapter download still requires the underlying Qwen3.5-9B model. The card specifies an 8,192-token prompt limit and an Apache 2.0 license. Someone comparing download sizes must therefore check whether a package is a complete usable model or an addition to one. Bespoke’s model card.[3]

Input length needs equal attention. Although Ollama’s catalogue displays a 256K context figure, Nimble’s implementation notes specify an 8,192-token working context. Tev1’s notes put its practical context at about 2,000 tokens. The full state and question set contribute to the prompt. A catalogue headline is therefore a poor basis for planning to submit a long customer history. Nimble notes and Tev1 notes.[6][7]

In the hypothetical repair shop, that means choosing the evidence for the routing question: the current complaint, relevant order details and perhaps the latest exchange. Sending everything the business knows about the customer could consume the available space while obscuring the fact that matters. Selecting that information is part of building the service, and a mistaken selection can make a perfectly well-formed model answer unhelpful.

Local execution changes the data path. Ollama says it does not see prompts or data processed locally, and its FAQ describes a setting that disables cloud features. Cloud-hosted models follow a different path through the service. For a business evaluating private-message processing, the relevant question is where its actual request goes, including any other software in the workflow. Ollama’s local and cloud explanation.[13]

The performance evidence is narrower than “fast on a laptop.” Ollama reports 91 milliseconds per Nimble decision in its Pac-Man example on an M5 Max. That is a vendor demonstration on specified hardware, rather than a measurement of the repair shop’s workload.[1]

Learning which detail changes the answer

Bespoke’s published dataset contains 2,676 training records arranged in paired examples, with 324 records reserved for a frozen evaluation. The labels are synthetic and checked by models, rather than reviewed by people. Training and evaluation source families are separated, and the documentation warns against using the frozen evaluation to choose future models. Dataset documentation.[5]

The paired construction targets an everyday challenge in interpreting requests: a small factual change can reverse a decision. Bespoke describes examples differing by one decisive fact. This encourages attention to the supplied evidence instead of simply recognizing the general subject. Nimble’s training approach.[2]

A version of that test at the repair shop would keep the message almost unchanged but replace “the box was crushed before I opened it” with “the part broke while I installed it.” Those invented examples could deserve different destinations. Another pair could change who reported the damage or whether an employee has already inspected it. Writing such cases forces the shop to decide what its labels actually mean before judging the software against them.

Together’s training account takes a broader mixture of classification and policy examples and says the fine-tuning run cost about $17, taking roughly 25 minutes. Its walkthrough then deploys the result on a dedicated hosted endpoint. The modest training figure describes that experiment; operating a service adds its own computing and maintenance costs. Together’s training account.[11]

Tev1 also retains a conventional next-token language-model head, its model card says. It is an experiment inspired by Jev’s decision interface, rather than a reproduction of Jev’s architecture. The card identifies further unfinished work: prompt injection, multilingual behavior and calibration have not been comprehensively evaluated. It also says the fine-tuned weights’ release license is being finalized; the base model’s license and the training code’s license do not settle every reuse question. Tev1 model card.[12]

The same average can conceal very different jobs

Ollama reports mean accuracy across 13 public datasets at 75.7 percent for Nimble, 73.3 percent for Tev1 4B and 63.5 percent for the smaller Tev1. The comparison covers 3,880 decisions; the local models were tested on Ollama, while the Jev comparison came from Bespoke’s hosted-API run. Evaluation table.[6]

The comparison also has a training-history qualification. Together’s training account lists MultiNLI and BoolQ among its training sources; those dataset families appear in the public evaluation suite too. That does not establish overlap between individual records, but it means the table is not equally strong evidence of unseen-task performance for every model.[11][4]

The underlying public benchmark report shows why a single average is only a starting point. Its original Nimble run matched 86.9 percent of labels on an English intent-classification subset, but 70.3 percent on a comments-toxicity subset and 39 percent on exact levels in one response-quality rubric. These are results from that earlier checkpoint and setup, not interchangeable measurements of the current Ollama release. Benchmark methods and task results.[4]

A routing decision and a quality rating demand different kinds of agreement. The shop can often establish a destination by checking which team ultimately resolved a case. Whether a reply was excellent or merely adequate may be disputed even among experienced employees. A business should define what counts as an error for its own task before importing an accuracy number from another one.

Bespoke’s benchmark authors disclose further limits: some rubric results depend on very few source articles, older datasets may have appeared during base-model training, and the reported hardware runs do not support a clean latency comparison. Those qualifications are part of the result. They help separate a reproducible research comparison from a prediction about production service.[4]

When the useful answer is “send this for review”

Ollama’s interface returns option probabilities and a separate confidence measure. Its Tev1 notes explicitly say that confidence describes how concentrated those probabilities are; it is not the chance the answer is correct. Treating a high displayed confidence as a reliability guarantee would give the number a meaning the documentation rejects. Interface notes.[7]

Independent researchers provide a useful test of the broader approach. A September 29 preprint by Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa evaluates hosted Jev 1.13 across 37 datasets. On a banking-intent task, retaining only the most confident half of its answers raised accuracy from 79.7 to 96.3 percent. That gain came with a substantial tradeoff: half the cases would need another route. The study tests Jev, so its results cannot establish the same behavior for Nimble or Tev1. Research paper.[8]

The same study found that a default 0.5 cutoff performed poorly on some binary tasks. Selecting thresholds using separate training examples improved several results, though the gains were not universal. The paper is a preprint, and its benchmark measurements still leave deployment-specific questions. It nonetheless supplies evidence for a practical idea: choosing when to defer can matter as much as choosing an answer.[8]

For the repair shop, an initial trial could leave every real assignment with employees while comparing the model’s proposed destinations against their decisions. The disagreements would show whether it confuses installation damage with shipping damage, or struggles with messages that omit the cause entirely. A later automated route could cover the cases it handles reliably, with unclear cases kept visible in a review queue. The acceptable balance would depend on how much a wrong route delays the repair.

Some parts of that workflow need no model judgment. TypeSafe’s own failure-mode documentation advises keeping arithmetic and date comparisons in code, filtering irrelevant material and making criteria explicit. Those recommendations concern Jev, but they clarify the engineering boundary the interface invites: use interpretation where language requires it and preserve exact computation where the rule is already known. Documented failure modes.[14]

A shop that already has a purchase date and a thirty-day return rule can calculate the deadline directly. Its harder question may be whether a customer is reporting transit damage, asking how to install the part or describing both. The new local models offer a compact way to tackle that question. Their value will be measured in the work that follows: whether the right person receives the request sooner, with enough evidence to help.