Loss, softmax and calibration

Cross-entropy and the linear softmax gradient

For a discrete distribution over K classes with ground-truth one-hot target y and model logits $z \in ℝ^{K}$ , softmax converts logits into normalized probabilities:

p_{i} = \frac{e^{z_{i} / τ}}{\sum_{j = 1}^{K} e^{z_{j} / τ}}

To avoid numerical overflow in floating-point hardware, implementations subtract $z_{max} = \max_{j} z_{j}$ from all logits before exponentiating. Because $e^{(z_{i} - z_{max})} / \sum e^{(z_{j} - z_{max})} = e^{z_{i}} / \sum e^{z_{j}}$ , the mathematical distribution is identical while the maximum exponent is 0.0, eliminating Inf/NaN overflow.

The derivative of cross-entropy with respect to logits

Cross-entropy loss measures the negative log-likelihood of the true class:

L_{CE} = - \sum_{k = 1}^{K} y_{k} \log p_{k}

Differentiating with respect to logit $z_{i}$ (with $τ = 1$ ):

\frac{\partial L_{CE}}{\partial z_{i}} = - \sum_{k = 1}^{K} \frac{y_{k}}{p_{k}} \frac{\partial p_{k}}{\partial z_{i}} = - \frac{y_{i}}{p_{i}} p_{i} (1 - p_{i}) - \sum_{k \neq i} \frac{y_{k}}{p_{k}} (- p_{k} p_{i})

\frac{\partial L_{CE}}{\partial z_{i}} = - y_{i} (1 - p_{i}) + \sum_{k \neq i} y_{k} p_{i} = p_{i} (\sum_{k = 1}^{K} y_{k}) - y_{i}

Because y is a probability distribution ( $\sum y_{k} = 1$ ):

\frac{\partial L_{CE}}{\partial z_{i}} = p_{i} - y_{i}

The derivative is simply prediction minus target. The complicated exponential terms in the denominator of softmax and the division inside the logarithm cancel each other completely. That linear simplicity is why cross-entropy is the standard loss for all classification and language modeling.

Enable JavaScript to use this local playground. The explanation and sources below work without it.

Brier score, proper scoring rules, and calibration

The multiclass Brier score measures mean squared error over probability vectors:

BS = \frac{1}{N} \sum_{n = 1}^{N} \sum_{k = 1}^{K} {(p_{n k} - y_{n k})}^{2}

Both cross-entropy and Brier score are strictly proper scoring rules: they are uniquely minimized in expectation if and only if the model's reported distribution $p$ matches the true ground-truth posterior distribution $p^{*}$ . A proper scoring rule penalizes overconfident bluffing as well as unconfident hesitation.

Expected Calibration Error (ECE)

A classifier is calibrated if its predicted probability reflects real empirical accuracy: when a model assigns 80% probability to a class across 1 000 predictions, exactly 800 of those predictions should be correct. Expected Calibration Error (ECE) bins predictions by confidence:

ECE = \sum_{m = 1}^{M} \frac{| B_{m} |}{N} | acc (B_{m}) - conf (B_{m}) |

Modern deep networks are notoriously overconfident (Guo et al., 2017). Temperature scaling calibrates a trained model post-hoc by optimizing a single positive scalar τ on a held-out validation set to minimize NLL. Because τ rescales all logits uniformly, it preserves the argmax class ranking exactly while smoothing overconfident probability spikes.

Decision Models & Single-Pass Pointer Readout Heads

When software systems use an LLM to route a ticket, select a tool, or score risk, autoregressive token generation is both wasteful and brittle: the model must emit dozens of syntax tokens ({"choice": "..."}), token probabilities fragment across surface synonyms, and raw softmax probabilities are notoriously overconfident on out-of-distribution inputs.

As detailed in our Decision Models report (Jev, Kev, and Laya), a Decision Pointer Head packs the state and candidate options into a structured sequence with reserved option delimiters (‹0› … ‹K−1› or pre-mask slots), runs a single forward pass, and computes the inner product between the projected state representation h_state and each option delimiter's hidden vector h_‹k›. Crucially, it decouples Choice (which option wins under temperature-calibrated softmax) from the Act / Review Gate (which uses margin and entropy features of the raw distribution to abstain when evidence is ambiguous).

z_{k} = \frac{{(W_{q} h_{state})}^{T} (W_{k} h_{‹ k ›})}{\sqrt{d} \cdot T}, p = softmax (z)

Confidence = 0.65 p_{(1)} + 0.35 (p_{(1)} - p_{(2)}), Gate \in {act, review}

Loss, softmax and calibration

Cross-entropy and the linear softmax gradient

The derivative of cross-entropy with respect to logits

Brier score, proper scoring rules, and calibration

Expected Calibration Error (ECE)

Decision Models & Single-Pass Pointer Readout Heads

Single-Pass Option Pointer & Calibration Controls

Calibrated Option Distribution & Gate Decision

Sources and further reading