The speed is not
in the LoRA.
The adapter changes what the model learns. The inference objective changes how much work you ask it to do.
Field guide · Decision experiments · Adaptation and training
Two execution paths
| Generative language model | Kev-style decision model |
|---|---|
| Prefill the input once. | Prefill state and question branches once. |
| Decode token 1, then token 2 conditioned on token 1, and so on. | Read scores at the option/decision positions with the pointer head. |
| With a KV cache, prior keys/values are reused. It does not recompute the whole original prompt every step. | Softmax produces bounded distributions; application code serializes JSON. |
| Time depends on input prefill plus output length, decode throughput, batching and infrastructure. | No autoregressive output loop. Input length, options, questions, attention/memory costs and infrastructure still matter. |
A hundred generated tokens involve repeated sequential decoding steps. Eliminating that loop can materially reduce latency. But a decision request is not constant-time with respect to its input, and neither “one pass” nor “LoRA” mathematically implies 200× speedup.
What LoRA actually contributes
Low-rank adaptation learns a small weight update rather than training every base weight. It primarily reduces training parameters and optimizer state. At inference, an adapter may be merged into the base weights or applied alongside them. It does not delete the base transformer or make a 4B model into an 11M-parameter forward pass.
Kev’s specialized training objective and readout repurpose that transformer: instead of repeatedly sampling vocabulary tokens, it computes scores over the caller’s declared options. The small trained head is critical, but most of the inference work is still running the backbone on the packed input.
A cost-shape calculator, not a benchmark
Change the assumed prefill and per-token times. The arithmetic illustrates why output length matters; it does not model an actual provider or account for all runtime costs.
Formula: generative = prefill + tokens × decode; decision = prefill + illustrative 2ms readout. Shared prefill is an explicit simplifying assumption, not measured parity.
Where the evidence ends
- TypeSafe announcement: describes parallel sampling and workflow-specific comparisons, with important benchmark caveats.
- Archer Hume’s reconstruction: infers a plausible architecture from observations; not an official disclosure of Jev.
- Kev source and 0.8B model card: inspectable masks, heads, LoRA and base model choices.
- The smart if-statement: another typed decision architecture; its measured latency is not a benchmark of this page.
For a fair comparison, hold task quality, context, options, hardware and concurrency fixed. Compare end-to-end latency, including transport, and report the complete distribution—not just a vendor’s best multiplier.