That is the problem I want the research process to solve: make the path from a source to a decision inspectable. A useful agent must find the relevant material, read it, resolve conflicts and duplicates, and show what remains uncertain. More searches or a longer report do not establish that it has done those jobs.

The papers below study different parts of this process. I use them to build a practical design proposal: give discovery, inspection, verification, and synthesis their own evidence budgets. This is an operating model to test in a business workflow, not a claim that the cited benchmarks have already validated vendor due diligence.

Start with a decision someone has to defend

For the vendor comparison, the request needs a decision date, a hierarchy of acceptable sources, and a stopping rule. Otherwise the agent has no reliable way to distinguish a finished investigation from a polished draft.

The discovery budget might require official product documentation, security and privacy materials, contractual terms, public incident records, and the relevant regulator’s guidance. The inspection budget requires the actual documents to be opened, dated, and quoted at the point supporting each claim. The verification budget assigns a second source or reviewer to claims about certifications, data location, breach history, and contractual liability. The synthesis budget produces both a comparison and an exception register.

The resulting artifact should contain at least:

1. the exact question and decision date;
2. the search plan and sources consulted;
3. an evidence ledger linking claims to inspected passages;
4. a contradiction and missing-information queue;
5. the inclusion, exclusion, and stopping rules;
6. the recommendation, with assumptions kept separate from findings.

The intended benefit is faster human review. A reviewer can concentrate on weak evidence and judgment calls instead of reconstructing the entire browsing session. The same record gives the next run a place to check which sources or facts changed.

Finding a page is different from reading it

Search-R1 and ReSearch train language models to interleave reasoning with multiple search calls. The point is not simply to attach a search API. The policy learns when a knowledge gap justifies a query and how retrieved material should change the next reasoning step.

That is a necessary capability, but a search result is not evidence yet. A result page contains rankings, titles, URLs, and snippets selected by another system. It may omit the qualifying sentence, point to an inaccessible page, or combine text that was not adjacent in the source. Treating the snippet as the source makes the research process fast and fragile.

OpenResearcher makes this distinction explicit. Its offline environment exposes three browser operations: search, open, and find. The authors report more than 97,000 synthesized trajectories over a fixed 15-million-document corpus, including trajectories with more than 100 tool calls. In their ablations, retrieving a relevant document did not guarantee a correct answer. Opening and inspecting it materially improved performance.

The same distinction appears in different forms elsewhere. SFR-DeepResearch gives a single agent web search, page browsing, and Python. DeepResearcher trains against live web interactions rather than assuming a clean closed corpus. Tongyi DeepResearch maintains an evolving report as compressed memory instead of carrying the entire history unchanged. These are different implementations of one operational fact: finding, reading, computing, and remembering are separate actions with separate failure modes.

Four budgets make the process inspectable

An evidence budget can be recorded as four related limits.

The budgets do not have to be fixed counts. A low-risk product comparison may permit one strong primary source per specification. A regulatory interpretation may require the rule text, the effective date, the responsible authority, later amendments, and jurisdiction-specific guidance. The important property is that the allocation follows the consequence of being wrong.

Separate budgets also make the cost discussion concrete. A team can ask whether another ten searches add coverage, whether another source inspection resolves a material uncertainty, or whether the synthesis stage is consuming tokens without adding evidence. Counting all tool calls together hides those differences.

Stopping is a claim about coverage

DeepSearchQA contains 900 multi-step information-seeking tasks across 17 fields. It evaluates complete answer sets, not only a single fact. The reported failure modes include premature stopping, which sacrifices recall, and hedging, which adds low-confidence items to widen coverage at the expense of precision.

That trade-off is familiar in business research. A market map can miss a small supplier that satisfies a critical constraint. It can also become unusable if every remotely related company is included. The right stopping rule therefore depends on the set definition. Before the search begins, the owner should specify:

• inclusion and exclusion criteria;
• the dimensions along which completeness matters;
• acceptable evidence for each item;
• how duplicates and subsidiaries are resolved;
• the residual uncertainty that may remain at the deadline.

The agent can then stop because the declared coverage test is satisfied, not because it has produced a plausible number of paragraphs. If the budget expires first, the output should state which branch remains incomplete.

The evaluation must follow the decision

DRACO builds 100 complex tasks across 10 domains and sources from 40 countries. Its task-specific rubrics cover factual accuracy, breadth and depth, presentation quality, and citation quality. This is stronger than one overall judge score because a system can write clearly while missing evidence, or cite extensively while misunderstanding the sources.

DR Tulu goes further during training. Its rubrics evolve with retrieved evidence and contrasting model responses, so the feedback can continue to distinguish outputs as the policy changes. Rethinking the Design of Reinforcement Learning-Based Deep Research Agents, the paper behind PokeeResearch, separately tests reward design, RL algorithm, data curation, and error-tolerant execution. The authors report gains from LLM feedback, on-policy RLOO, filtering weak training questions, and allowing malformed actions to be repaired at test time.

None of those benchmark results transfers automatically to procurement or compliance. The lesson is structural. A business acceptance test should preserve separate axes for source validity, factual support, required coverage, contradiction handling, citation placement, and presentation. If the organization collapses them into one score, a gain on formatting can conceal a loss on evidence.

Build the smallest loop that exposes failure

OpenSeeker generates difficult questions by traversing a web graph and obscuring entities that would permit keyword shortcuts. It also uses retrospective summarization to compress older observations while keeping recent evidence in detail. Both ideas are useful beyond model training: test cases should require real evidence traversal, and memory compression should preserve provenance rather than only conclusions.

A first implementation does not need a frontier model or a complicated multi-agent hierarchy. It needs a traceable loop:

plan -> search -> open -> extract -> record -> check coverage
                         ^                    |
                         +---- revise --------+

Each extracted claim receives a source URL, retrieval time, quoted or structured evidence, and a confidence or conflict state. Search results that were never opened stay marked as leads. The coverage check chooses the next unresolved branch or stops with a reason. A separate synthesis step writes only from the evidence ledger.

This design can use one model, several models, or deterministic tools. SFR-DeepResearch is a useful reminder that the control surface can remain small. Adding roles does not repair an undefined evidence standard.

At review time, start with a material claim and try to follow it back to an inspected source. Then check the unresolved questions and the reason the agent stopped. If that trail is missing, the report is not ready to support the decision, however convincing its prose.

A small adapter is useful during training, but the deployed model is another matter. TF2 merges the learned update into the original weights before serving it. My earlier note on the TF2 adapter described the update as \(W’ = W + (\alpha/r)BA\) and separated that adaptation from later quantization. One phrase still needs a sharper boundary: merging does not turn the full model into a low-rank model, and it does not make the deployed checkpoint contain only the adapter parameters.

Follow one adapted matrix from training to deployment and the trade-off becomes concrete. In exact arithmetic, the merged and unmerged forms compute the same linear map. They differ in storage, serving flexibility, and numerical behavior once finite precision or quantization enters the path. Keeping those properties separate prevents “parameter-efficient” from becoming an ambiguous claim about training, checkpoints, and inference at once.

The update is low-rank, not the resulting layer

Let the frozen matrix be \(W_0 \in \mathbb{R}^{d \times k}\). LoRA trains

\[ A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r} \]

with \(r \ll \min(d,k)\). With scale \(s = \alpha/r\), the adapted matrix is

\[ W’ = W_0 + sBA. \]

The original LoRA paper freezes \(W_0\), optimizes \(A\) and \(B\), and identifies \(BA\) as the constrained update. Because

\[ \operatorname{rank}(BA) \le r, \]

the difference \(W’ - W_0\) is low-rank. No corresponding statement says that \(W’\) is low-rank. If \(W_0\) has full rank, adding a rank-32 update will normally leave the resulting dense matrix full-rank.

That distinction controls the parameter count. A dense \(d\)-by-\(k\) matrix contains \(dk\) values. Its LoRA pair contains \(r(k+d)\) trainable values. For a square \(4096 \times 4096\) projection with rank 32:

The adapter has 1.5625% as many values as this base matrix. That is a real reduction in trainable, optimizer-managed, and task-specific parameters. It is not a 64-fold reduction in the size of the model required for inference. The base weights remain necessary either as a shared artifact or inside each merged checkpoint.

Merging changes the evaluation graph

Without merging, an evaluation-mode forward pass applies two branches:

\[ h = W_0x + sB(Ax). \]

The low-rank order matters computationally. Forming \(BA\) on every forward pass would create a dense matrix and discard the main benefit of keeping an adapter separate. Computing \(Ax\) and then \(B(Ax)\) adds roughly \(rk+dr\) multiply-accumulate operations per input vector, compared with \(dk\) for the base projection.

After merging, deployment first computes

\[ W’ \leftarrow W_0 + sBA \]

once, then uses

\[ h = W’x. \]

Distributivity makes the two expressions equal in exact arithmetic. Adapter dropout does not contradict this equivalence because dropout is disabled in evaluation mode. The LoRA paper’s deployment construction explicitly stores the sum to avoid additional inference latency, and the current PEFT LoRA guide exposes the same operation through merge_and_unload().

The practical gain is not the small arithmetic ratio alone. A separate adapter can add kernel launches, wrapper logic, extra memory reads, and synchronization around the base projection. Their effect depends on the runtime, batch size, hardware, and whether it has fused adapter kernels. A merge removes that branch entirely and returns the original dense architecture to the serving system.

The cost is flexibility. A shared base plus several small adapters can switch tasks without storing several full models. A merged checkpoint is one task-specific copy of the full backbone. PEFT’s checkpoint documentation also notes that merge_and_unload() removes PEFT-specific behavior: the result cannot disable, replace, or unmerge the adapter through those APIs.

Merging is therefore deployment repackaging. It trades adapter identity and switching for a simpler inference graph. Compression would require a separate operation such as quantization, pruning, or a smaller architecture.

Where the algebra meets floating point

The algebra does not promise bitwise equality on a computer. The unmerged path evaluates \(W_0x + sB(Ax)\); the merged path first rounds the entries of \(W_0 + sBA\), then multiplies the rounded matrix by \(x\). Floating-point addition and multiplication can round at different points even when both paths use the same nominal dtype.

For an ordinary FP16 or BF16 merge, the resulting differences should be judged against a declared tolerance and task-level output checks. A useful pre-deployment test compares logits from the adapter path and the merged path on fixed inputs before any generation sampling. Greedy decoded text alone is too coarse: two logit tensors can differ while producing the same tokens, or a tiny difference near a tie can change one token and everything after it.

The test also needs the model in evaluation mode. Otherwise adapter dropout can make repeated unmerged calls differ from each other, which says nothing about whether merging was implemented correctly.

Modules outside the LoRA matrices deserve explicit accounting. Biases or task heads may be trained and stored through configuration such as PEFT’s bias or modules_to_save. A checkpoint that preserves only \(A\) and \(B\) while omitting those trained values is not the same adapted model, and merging the LoRA layers cannot reconstruct them.

Quantized merging is a new approximation

Quantization makes the boundary more visible. If the base weights are already stored in a quantized format, a merge cannot generally add a floating-point \(BA\) directly to their integer codes. The implementation must reconstruct approximate real values, add the adapter update, and quantize the sum again.

PEFT’s current quantization guide describes exactly that sequence for supported methods: dequantize, add, then requantize. It also warns that unmerging performs another round trip and does not restore the original quantized weights. Repeated merge and unmerge cycles can accumulate rounding error.

This produces three distinct objects:

1. the quantized approximation of the frozen base, \(Q(W_0)\);
2. the runtime combination of that approximation with an unmerged adapter;
3. the quantized approximation of the merged matrix, \(Q(W_0 + sBA)\).

There is no general identity that makes the second and third objects numerically equal. Quantization is nonlinear because scales, zero points, groups, clipping, and rounding can change when the update is included. Some quantization methods do not support merging at all, and support can depend on the layer type and configuration.

For a release like TF2, the reproducible statement is not merely “LoRA was merged and the model was quantized.” The order matters. Merging into a higher-precision base and then calibrating or quantizing the result is a different export path from loading a previously quantized base and asking an adapter library to merge into it.

Preserve both the recipe and the product

A small adapter is attractive because the base can remain shared. That benefit depends on recording which base it modifies. Adapter provenance should include the exact backbone revision, tokenizer, target modules, per-layer ranks, scaling convention, trained biases or saved modules, and adapter weights.

A merged artifact needs a different record:

• the base and adapter revisions used to create it;
• the merge library and version, dtype, device, and selected adapters;
• whether the saved result still contains adapter wrappers;
• the quantization method, order, calibration data, group settings, and output dtype;
• a fixed-input logit comparison before and after merge;
• checksums for the adapter and final checkpoint.

The adapter recipe explains how the task-specific difference was encoded. The merged checkpoint records the exact dense product that was deployed. One cannot substitute for the other after quantization or when the original base revision disappears.

LoRA saves training state and task-specific storage because it represents the update with two small matrices. Merging trades that separability for the base model’s ordinary inference graph. The deployed matrix still has \(dk\) values; only its difference from the frozen base came through a rank-\(r\) path.

Selecting the cutoff does not change the fitted coefficients, but it still uses labels to choose one rule from several candidates. If the same final test labels choose the threshold and measure its performance, the reported score includes the noise that made this threshold win. The test set has become part of model selection.

This note makes that bias exact with ten fixed scores that contain no label information. Across every balanced assignment of five positive labels, the best same-sample threshold averages 65.3% accuracy. Apply the selected threshold to an independent balanced assignment and its expected accuracy is 50%. The apparent improvement comes entirely from reusing the labels.

Eleven cutoffs search one random ordering

Number ten cases from lowest score to highest score. The score order is fixed before any labels are assigned. Exactly five cases are positive, but every one of the \(\binom{10}{5}=252\) balanced labelings is equally likely. Under this construction, the scores carry no information about the labels.

A threshold between adjacent scores can predict the highest \(k\) cases as positive, for \(k=0,1,\ldots,10\). These are eleven distinct decision rules. For each labeling, choose the value of \(k\) with the largest accuracy on those same ten cases.

Exact enumeration gives this distribution:

The mean of the selected accuracies is

\[ \frac{823}{1260} \approx 0.6532. \]

One of the 252 random labelings can even be separated perfectly by a threshold: all five negatives happen to receive the lower scores and all five positives the higher scores. That event is rare, but every labeling lets the selection procedure keep its most favorable cutoff.

The enumeration needs no machine-learning library:

from itertools import combinations
from statistics import mean


n = 10
all_cases = set(range(n))
best_accuracies = []

for positive_tuple in combinations(range(n), n // 2):
    positive = set(positive_tuple)
    accuracies = []

    for k in range(n + 1):
        predicted_positive = set(range(n - k, n))
        true_positive = len(positive & predicted_positive)
        true_negative = len(
            (all_cases - positive) - predicted_positive
        )
        accuracies.append((true_positive + true_negative) / n)

    best_accuracies.append(max(accuracies))

print(mean(best_accuracies))  # 0.6531746031746032

This value is not a universal correction for threshold tuning. It belongs to ten cases, balanced labels, accuracy, and this set of eleven nested rules. Its role is to show the mechanism without simulation error.

A fixed cutoff has no advantage here

For any fixed \(k\), the highest-scored group contains \(k/2\) positives on average. The remaining \(10-k\) cases contain \((10-k)/2\) negatives on average. The expected number of correct predictions is therefore

\[ \frac{k}{2} + \frac{10-k}{2} = 5, \]

so the expected accuracy is 50% for every cutoff chosen without seeing the evaluated labels.

The selected cutoff is also fixed before an independent second labeling is revealed. Conditional on any \(k\) chosen from the first sample, its expected accuracy on that second balanced sample is still 50%. Averaging over the threshold-selection procedure does not change the result.

The gap between 65.3% and 50% is selection bias. Each individual accuracy is an unbiased estimate for its fixed rule under this construction. The maximum of the eleven noisy estimates is not an unbiased estimate of the selected rule’s future accuracy.

The thresholds are strongly dependent, so the example is not equivalent to taking the maximum of eleven independent measurements. Dependence reduces the search space, but it does not remove selection. A run of positives near one end of the random score order can still make one cutoff look useful on the sample that exposed the run.

Threshold tuning is part of fitting

The same boundary applies to a real classifier. Choosing a cutoff to maximize F1, balanced accuracy, recall under a precision constraint, or measured utility adapts a decision rule to labeled data. The operation may leave the score function unchanged, but the deployed predictor now contains a learned threshold.

Scikit-learn’s TunedThresholdClassifierCV represents that threshold as fitted state. Its documentation describes cross-validated cutoff selection and explicitly warns that using cv="prefit" with the same data for estimator fitting and threshold tuning is subject to overfitting.

That warning covers one reuse boundary. A second boundary appears at evaluation time. If cross-validation chooses the cutoff and the same cross-validation score is then presented as the final performance estimate, the selection criterion is being used to assess its own winner. Cawley and Talbot’s paper on over-fitting in model selection treats model selection as part of fitting and shows why performance evaluation must include that complete procedure.

A clean single-split experiment assigns three roles:

1. Fit preprocessing and score-model parameters on training data.
2. Choose the threshold and its objective on separate validation data.
3. Freeze both, then evaluate once on the test data.

The scikit-learn cross-validation guide gives the same separation for hyperparameter selection: settings chosen to optimize test performance leak test information into the model, so the final test set remains held out after validation or cross-validation.

Thresholds deserve the same treatment as regularization strengths, feature choices, and calibration methods. They are all decisions informed by labels.

The outer split evaluates the procedure

When one validation split would waste too much data or make the result depend heavily on one partition, nested cross-validation can estimate the complete procedure. In each outer fold:

1. Use only the outer training portion to fit preprocessing and the score model.
2. Use an inner split or inner cross-validation to choose the threshold.
3. Apply the resulting model and fixed threshold to the untouched outer test portion.

The outer predictions can then be collected for an overall confusion matrix or scored within folds according to a declared aggregation rule. Threshold selection must restart inside every outer fold. Choosing one global threshold from all out-of-fold labels before scoring those same labels would recreate the original reuse in a less visible form.

After evaluation, a final deployment model may be refit on all available development data and receive a threshold selected by the declared inner procedure. The outer-fold score estimates that procedure’s performance; it does not certify that one final numeric threshold is invariant across samples.

Threshold variability is itself useful evidence. If small changes to the validation set move the selected cutoff widely while the objective barely changes, reporting only one threshold hides an unstable choice. The metric curve around the optimum, fold-level thresholds, and confusion counts can distinguish a broad plateau from a sharp, sample-specific maximum.

Preserve the selection path

A reproducible threshold result needs more than the final number. I would retain:

• the score source, positive-class mapping, and fitted preprocessing;
• the candidate threshold rule, including ties and endpoint cutoffs;
• the selection objective and any constraints or error costs;
• the sample or inner folds used to choose the threshold;
• the independent sample or outer folds used for evaluation;
• the chosen threshold, objective curve, and final confusion counts.

Those fields separate the rule that was searched from the data that judged the search. A threshold can be tuned aggressively when the application requires it. The final test labels must remain unavailable until that tuning is finished.

The TF3 paper describes a hybrid objective: teacher-student KL divergence receives weight 1.0, while ordinary next-token cross-entropy receives weight 0.1. Those coefficients look like a complete mixing rule. They are not, because the magnitude and meaning of the KL term also depend on how logits become probabilities and how token losses are reduced.

This note isolates that reproducibility boundary. It does not re-evaluate the TF3 student or show that its reported result is wrong. It shows why a distillation record needs the softmax temperature, any temperature compensation, the KL implementation, and the exact valid-token denominator before 1.0 versus 0.1 can be reconstructed.

Temperature changes the target

For teacher logits \(v_i\) over a vocabulary and temperature \(T\), the soft target is

\[ p_i^{(T)} = \frac{\exp(v_i/T)}{\sum_j \exp(v_j/T)}. \]

The student distribution \(q_i^{(T)}\) is computed from student logits with the same temperature. A larger \(T\) moves both distributions toward uniform, exposing relative preferences that may be nearly invisible at \(T=1\).

The original knowledge-distillation paper makes this temperature part of the method. It generates teacher targets at a raised temperature, trains the student at that same temperature, then returns the student to \(T=1\) for inference. Temperature is therefore not an inference sampling option pasted onto training. It defines the target distribution being imitated.

Consider teacher logits [4, 2, 0] and student logits [3, 2, 1] for three possible next tokens:

The logits and the coefficient on KL did not change. The numerical loss fell by a factor of six because the probability distributions did. More importantly, the student received a different learning signal about the two lower-ranked tokens.

At \(T=1\), the teacher assigns the third token only 1.6% probability. At \(T=4\), it assigns 18.6%. The higher-temperature target asks the student to preserve more of the teacher’s ranking among alternatives. Whether those alternatives contain useful structure or noise is an empirical question, which is why \(T\) belongs in the experiment record.

The gradient brings a second temperature effect

For soft-target cross-entropy, the derivative with respect to student logit \(z_i\) is

\[ \frac{\partial \mathcal{L}_{\mathrm{soft}}}{\partial z_i} = \frac{1}{T}\left(q_i^{(T)}-p_i^{(T)}\right). \]

Raising \(T\) makes the two distributions softer and adds the explicit \(1/T\) factor. At sufficiently high temperature, the gradient magnitude scales approximately as \(1/T^2\). Hinton, Vinyals, and Dean therefore multiply the soft-target term by \(T^2\) when combining it with the hard-target term. The compensation keeps the two gradient contributions on roughly the same scale while \(T\) is varied.

The three-logit example shows the effect. Its unscaled soft-target gradient norm is 0.2497 at \(T=1\), 0.0980 at \(T=2\), and 0.0281 at \(T=4\). Multiplying the loss by \(T^2\) changes those norms to 0.2497, 0.3918, and 0.4490. The compensation is asymptotic, not a promise that finite-temperature norms will be identical.

A complete hybrid objective might therefore be written as

\[ \mathcal{L} = \alpha T^2\,\mathrm{KL}\!\left(p_T^{(T)}\|p_S^{(T)}\right) + \beta\,\mathrm{CE}(y,p_S^{(1)}). \]

If the implementation uses \(T=1\), the factor is one and can disappear harmlessly. If it uses \(T=4\), omitting the factor makes the soft-target gradient much smaller relative to the hard-target gradient even though \(\alpha\) and \(\beta\) are unchanged. Reporting only those two coefficients does not distinguish the cases.

Reduction chooses the token denominator

Language-model distillation produces one vocabulary distribution for every scored token. A batch can have shape [batch, sequence, vocabulary], plus a mask for padding or other ignored positions. The KL divergence first sums over vocabulary entries to give one value per token. Only then should the declared token reduction be applied.

This order matters with PyTorch. The current KLDivLoss documentation says that its input must contain log probabilities and warns that the default reduction="mean" does not return the mathematical KL divergence. That reduction averages over every tensor element, including vocabulary entries. batchmean sums all elements and divides by input.size(0).

For a classification tensor shaped [B, V], batchmean gives a sum over \(V\) followed by a mean over \(B\). For a language-model tensor shaped [B, L, V], it divides only by \(B\), so the result grows with the number of token positions \(L\). Flattening the same valid positions to [B L, V] before batchmean instead produces a per-token mean. The values can differ by a factor of \(L\) while the logits, temperature, and nominal KL coefficient remain identical.

A masked implementation can make the denominator explicit:

teacher_logp = teacher_logits.log_softmax(dim=-1)
student_logp = student_logits.log_softmax(dim=-1)

per_token_kl = (
    teacher_logp.exp() * (teacher_logp - student_logp)
).sum(dim=-1)

soft_loss = per_token_kl[valid_token_mask].mean()

With temperature, both log_softmax calls receive logits divided by \(T\), and the final soft loss receives the declared \(T^2\) multiplier if that is the chosen convention. The teacher output should be detached or computed without gradients. The hard-target cross-entropy must use the same valid positions if the coefficient ratio is intended to apply per scored token.

The code also states the direction of the divergence. \(\mathrm{KL}(p_T\|p_S)\) weights each log-ratio by the teacher probability. Reversing the arguments is a different objective. PyTorch’s API can make this easy to misread because the student log distribution is the first function argument while the teacher distribution is the target.

Vocabulary alignment is a real precondition

TF3 satisfies a structural condition that should not be treated as incidental: the teacher and student use the same tokenizer and vocabulary. Token index 17,432 therefore names the same subword on both sides, and both models produce a distribution with the same support at the same source position.

Without that alignment, a pointwise KL over logits is undefined as a transfer of token meaning. Two vocabularies can have the same size while assigning different strings to each index. They can also segment the prefix differently, leaving no one-to-one correspondence between prediction positions. Padding a shorter logit vector or matching indices by number would create a valid tensor operation and an invalid learning objective.

Cross-tokenizer distillation needs another bridge, such as probabilities over a shared byte or character space, sequence-level targets sampled from the teacher, or an explicit mapping between compatible tokens. Each changes the object being transferred. The recent note on tokenizer-dependent perplexity concerned an evaluation denominator. Here the shared tokenizer is part of the training objective’s domain.

Record the executable objective

For a language-model student, I would preserve these fields with the checkpoint:

• teacher and student revisions, tokenizer artifact, and vocabulary identity;
• temperature \(T\) for both distributions and inference temperature after training;
• KL direction, log-probability convention, and numerical implementation;
• whether the soft loss is multiplied by \(T^2\);
• hard and soft coefficients after all scaling;
• vocabulary sum, valid-token mask, and final reduction denominator;
• treatment of padding, document boundaries, and any unscored first tokens.

An executable unit test can use fixed teacher and student logits, a mask containing both valid and ignored positions, and a hand-computed loss. It should assert invariance to adding masked padding and to reshaping valid tokens before reduction. A second test can change \(T\) and verify the chosen scaling convention rather than accepting whatever magnitude a library default returns.

TF3 records the main structural facts: a frozen 51.65M teacher, a 26.45M student, a shared tokenizer, KL from the teacher distribution to the student distribution, hard-target cross-entropy, and weights of 1.0 and 0.1. The remaining details determine the executable fraction behind those weights. A distillation coefficient becomes reproducible only when its temperature and denominator travel with it.

My earlier TF3 release note reported that the resulting 51.65M-parameter model reached held-out cross-entropy near 0.89 and perplexity near 2.43. Those values are useful for following checkpoints and compression variants that share the same tokenizer and evaluation pipeline. They do not create a tokenizer-independent scale.

This note isolates the denominator. Perplexity exponentiates average negative log-likelihood per predicted token. If a tokenizer divides the same text into more pieces, it changes that average even when the model assigns the complete text exactly the same probability. A cross-tokenizer comparison therefore needs a unit tied to the text, not only to each model’s tokens.

The average is over tokens

For a token sequence \(x_1, \ldots, x_T\), let the total negative log-likelihood be

\[ L = -\sum_{t=1}^{T}\log p(x_t \mid x_{<t}). \]

Token-level cross-entropy is \(L/T\), and perplexity is

\[ \operatorname{PPL} = \exp(L/T). \]

The Hugging Face perplexity guide gives this definition and states the comparison constraint directly: tokenization affects perplexity. The exponent does not remove that dependence. It only maps average log loss back to the model’s token scale.

Consider one text with total negative log-likelihood 12 nats under two deterministic, lossless tokenizations:

Both factorizations assign the complete text probability \(e^{-12}\), about \(6.14 \times 10^{-6}\). The finer tokenization reports much lower perplexity because the same loss is divided by twice as many prediction events.

The comparison can reverse the likelihood ordering. A model with total NLL 10 over five tokens has perplexity 7.39. Another with total NLL 12 over ten tokens has perplexity 3.32. The second perplexity is lower, but its probability for the complete text is also lower: \(e^{-12}\) rather than \(e^{-10}\). The token scales differ, so the two averages do not rank the texts on a common unit.

TF3 fixes the tokenizer before reporting perplexity

The TF3 paper reports that Romanian BPE averaged 304.89 tokens per sentence and Romanian Unigram averaged 340.35 under the actual preprocessing pipeline. The latter is about 11.6% longer. TF3 then uses the Unigram tokenizer for its Transformer, Mamba, compression, and generation experiments.

That order makes the reported training curve interpretable. A decrease from one checkpoint to another uses the same vocabulary, segmentation, held-out distribution, packing rule, and token denominator. The curve describes improved next-token prediction within that fixed system.

It would be different to train one model with each tokenizer and compare only their token perplexities. Even in the artificial case where both models assigned every sentence the same total likelihood, the 11.6% token-count difference would give them different per-token losses. Lower perplexity could reflect a better model, a finer segmentation, or both.

The tokenizer is more than a list of strings. The original SentencePiece paper describes a model file that contains the vocabulary, segmentation parameters, and character-normalization behavior. Changing that file can change the token count and the text presented to the language model. A reproducible perplexity result therefore needs the tokenizer artifact and its normalization policy, not only the vocabulary size.

Normalize by a shared text unit

When two systems score the same normalized text, total negative log-likelihood remains additive across their token factorizations. It can be divided by a unit outside either tokenizer. For a text containing \(C\) declared characters,

\[ \text{bits per character} = \frac{L}{C\log 2}. \]

Bits per byte uses the byte length in the denominator instead. Either can support a cross-tokenizer comparison if both systems score the same underlying sequence and the unit is defined identically. Bytes avoid ambiguity about what counts as a Unicode character, while characters can be easier to interpret for a fixed normalization and encoding contract.

The contract matters for Romanian. Precomposed ț and t followed by a combining comma can be different code-point counts while representing canonically equivalent text. The evaluation should specify whether \(C\) counts raw code points, NFC-normalized code points, grapheme clusters, or UTF-8 bytes. Applying different normalization inside two tokenizers would mean they are no longer scoring the same input object.

This normalization does not make unlike corpora comparable. A restricted synthetic domain can be more predictable than open-domain prose regardless of the denominator. It also does not turn likelihood into a complete generation-quality measure. It only removes the token count as one source of scale mismatch.

Context policy changes the numerator

The denominator is not the only protocol choice. A finite-context model cannot condition every token on an arbitrarily long prefix. Splitting a corpus into disjoint blocks gives tokens at the start of each block little context and usually increases their loss.

The official perplexity guide recommends a strided sliding window for fixed-length models. Overlapping context is supplied to the model, but only the newly scored targets contribute to the loss. Each target must be counted once. The guide’s GPT-2 example changes from perplexity 19.44 with disjoint 1,024-token chunks to 16.44 with a stride of 512. The model and tokenizer are unchanged; the available context changes the numerator.

Packed training data adds a related boundary choice. The evaluation record should say whether prediction continues across document boundaries, whether an end-of-sequence token separates records, and which first token in each segment is unscored because it has no left context in the batch. These choices alter the set of conditional probabilities included in \(L\).

Corpus aggregation must retain the same weighting rule. If document \(i\) contributes loss \(L_i\) over \(T_i\) valid targets, token-weighted perplexity is

\[ \exp\left(\frac{\sum_i L_i}{\sum_i T_i}\right). \]

Averaging the per-document losses \(L_i/T_i\) instead gives every document equal weight. The two agree only when the documents have equal scored lengths or happen to have the same mean loss. Batch boundaries should not silently choose the estimand.

Report enough to reconstruct the fraction

A token-level perplexity result should preserve at least:

• the tokenizer artifact and character-normalization policy;
• the held-out text identity and preprocessing revision;
• total negative log-likelihood and the number of valid target tokens;
• treatment of beginning, end, padding, and ignored tokens;
• context length, stride, packing, and document-boundary policy;
• a shared-unit result such as bits per byte for cross-tokenizer comparisons.

The first five fields make the reported token perplexity reproducible. The last separates model likelihood from the tokenizer’s choice of how many prediction steps represent the text.

TF3’s perplexity near 2.43 is a result on its Romanian Unigram scale. It can compare checkpoints and variants that keep that scale fixed. The BPE and Unigram token counts answer a different question about segmentation. Combining the two requires returning to the fraction underneath perplexity: total loss in the numerator, and an explicitly chosen unit in the denominator.

Unicode normalization gives equivalent encodings a consistent representation. Its NFC form can compose a base letter and a combining mark into one character. My earlier note on restoration metrics included legacy cedillas among the positions that may need repair, but that repair needs a separate rule. Normalization can reconcile a precomposed ț with a sequence made from t and a combining comma below. It does not change a cedilla into a comma.

This note isolates the boundary with four encodings of fața, meaning the face. Two use the modern comma below and two use the legacy cedilla. NFC reduces four code-point sequences to two Unicode strings, not one. A Romanian-specific mapping can join the remaining pair, but it must be an explicit language and evaluation policy.

Four spellings form two normalization groups

The visible word can be represented with a precomposed letter or with a base letter followed by a combining mark:

U+0326 is COMBINING COMMA BELOW. U+0327 is COMBINING CEDILLA. The distinction can be hard to see in a small font, but it is present in the text.

Python exposes the Unicode normalization algorithm through unicodedata.normalize. The four cases can be checked directly:

from unicodedata import normalize


samples = {
    "modern precomposed": "fața",
    "modern decomposed": "fat\u0326a",
    "legacy precomposed": "faţa",
    "legacy decomposed": "fat\u0327a",
}

for name, value in samples.items():
    print(name, repr(normalize("NFC", value)))

The result is:

modern precomposed 'fața'
modern decomposed  'fața'
legacy precomposed 'faţa'
legacy decomposed  'faţa'

NFC makes each group internally consistent. It does not join the groups. NFKC produces the same result for these characters.

The decomposition table explains the result

Unicode Standard Annex #15 defines NFC as canonical decomposition followed by canonical composition. NFKC starts from compatibility decomposition instead. Neither operation is a visual-similarity rule or a language-specific spelling corrector.

The decisive data is in the Unicode Character Database. Its current UnicodeData.txt gives these canonical decompositions:

U+021B LATIN SMALL LETTER T WITH COMMA BELOW
    -> U+0074 LATIN SMALL LETTER T
     + U+0326 COMBINING COMMA BELOW

U+0163 LATIN SMALL LETTER T WITH CEDILLA
    -> U+0074 LATIN SMALL LETTER T
     + U+0327 COMBINING CEDILLA

The base letter is the same. The combining marks are not. There is no canonical or compatibility decomposition from the cedilla form to the comma-below form, so none of the four normalization forms is allowed to invent one.

This is a useful constraint. Normalization has stable, language-independent semantics. If it silently changed one mark into another because a particular language treats them as historical variants, it could corrupt text in a language where the cedilla is the intended character.

Romanian equivalence is a higher-level rule

The Unicode Standard’s discussion of Romanian prefers U+0219 ș for modern Romanian and identifies U+015F ş as the Turkish s with cedilla. It also records the practical complication: legacy Romanian data often contains the cedilla code point and Romanian processing should treat the two forms as equivalent.

Those statements describe a language-aware operation above Unicode normalization. A compact implementation is:

import unicodedata


ROMANIAN_LEGACY = str.maketrans({
    "Ş": "Ș",
    "ş": "ș",
    "Ţ": "Ț",
    "ţ": "ț",
})


def normalize_romanian(text):
    composed = unicodedata.normalize("NFC", text)
    return composed.translate(ROMANIAN_LEGACY)

The order matters. NFC first turns s plus U+0327 into precomposed U+015F, so the explicit table handles precomposed and decomposed legacy input with the same four entries. A final NFC call would be harmless here but is unnecessary because every replacement is already precomposed.

The function must not be presented as general Unicode cleanup. U+015F is valid Turkish text. Mapping it unconditionally in a multilingual corpus would replace the intended Turkish cedilla with a Romanian comma below. The input’s declared language, field, or corpus contract must authorize the mapping.

Byte decoding belongs before both steps. If an ISO-8859-2 byte stream was decoded with the wrong character encoding, Unicode normalization cannot reconstruct the lost byte interpretation. A replacement character such as U+FFFD is evidence that the pipeline crossed that boundary without enough information.

Measure spelling and output format separately

Suppose the reference contains fața știe, while a system returns faţa ştie. Three comparisons answer three different questions:

reference = "fața știe"
prediction = "faţa ştie"

raw_equal = prediction == reference
nfc_equal = (
    unicodedata.normalize("NFC", prediction)
    == unicodedata.normalize("NFC", reference)
)
romanian_equal = (
    normalize_romanian(prediction)
    == normalize_romanian(reference)
)

The values are False, False, and True. The raw comparison is sensitive to every code-point choice. NFC removes accidental composition differences but still requires the correct mark. The Romanian fold measures the intended letters while accepting a declared legacy representation.

For a restoration benchmark, I would keep at least two views:

• a content score after NFC and the declared Romanian legacy fold;
• an output-contract score after NFC but before the legacy fold.

The first avoids treating old corpus encoding practice as a language error. The second detects a system that keeps emitting legacy code points when the delivery contract requires modern Romanian text. Reporting only the folded score would hide that integration defect. Reporting only the strict score could turn a historical encoding convention into an apparent restoration failure.

The raw form should remain available for provenance and debugging. Dataset hashes, import logs, and audit samples need to show whether the source contained decomposed marks, cedillas, or already-modern text. Destructive normalization at ingestion can otherwise make a later metric impossible to interpret.

Test the distinctions, not only the function

A small regression table is enough to protect the policy:

modern precomposed   -> modern precomposed
modern decomposed    -> modern precomposed
legacy precomposed   -> modern precomposed, Romanian fields only
legacy decomposed    -> modern precomposed, Romanian fields only
Turkish s-cedilla    -> unchanged outside Romanian fields
invalid byte decode  -> rejected or recorded, never guessed

The test should cover uppercase Ș/Ț, lowercase ș/ț, both combining marks, and mixed strings. It should also assert that unrelated characters and string length at the grapheme level are preserved. A restoration pipeline that changes punctuation, URLs, or names has a different failure from one that chooses the wrong Romanian mark.

For a Romanian-only import, the sequence is small: decode correctly, apply NFC, then apply the declared legacy-character mapping. For a multilingual field, keep the language boundary explicit. Two letters that look almost identical are enough to make an otherwise reasonable cleanup rule change the wrong text.

My earlier note on paired and independent bootstrap designs treated each held-out item as one observation. Repeated judge calls add another level. They measure how the evaluator varies while the text, requested task, and item difficulty remain fixed. They do not add new stories or translations to the test set.

This note isolates the consequence with four items and three calls per item. The example is deliberately small and contains no TinyFabulist result. It shows why resampling twelve score rows can report much less uncertainty than resampling the four item identities that generated them.

Twelve rows can contain four independent units

Suppose two systems are scored on the same four items. Subtract system B’s score from system A’s score for each judge call. To make the dependence visible, let all three calls for an item return the same difference:

The mean difference is 1.0 whether it is computed over four item means or twelve rows. The point estimate does not expose the problem.

A row bootstrap draws twelve values independently from the twelve cells. It can draw the first copy of item 4 without the other two, as if those copies described different sampled texts. An item bootstrap draws four item identities. When item 4 is selected, its three calls travel together because they share the same experimental unit.

Enumerating both empirical bootstrap distributions gives:

The row analysis reduces the standard error by a factor of about \(\sqrt{3}\), exactly what would happen if three perfect copies had tripled the sample size. They did not. The narrower interval is manufactured by the independence assumption.

This is a compact instance of pseudoreplication: repeated measurements are analyzed as independent replicates even though they belong to the same experimental unit. Stochastic judge outputs make the copies less obvious, but they do not remove the shared item.

Four items are not enough for a defensible system comparison, and these percentile bounds are not proposed as a small-sample remedy. The construction only holds the point estimate fixed while exposing what the resampling unit changes.

Pairing and clustering preserve different relations

Pairing answers the system-comparison question. If systems A and B translated the same source sentence, their scores for that sentence must stay together. A difficult sentence can lower both scores, and the paired difference removes part of that shared item effect.

Clustering answers the repetition question. If one sentence was judged three times, those calls remain attached to that sentence when items are resampled. Flattening the table and preserving only the A/B row alignment keeps the system pair but loses the item cluster.

SciPy’s bootstrap documentation makes the first operation explicit: with paired=True, it draws one index array and applies it to every input sample. That is sufficient when each index names one item. It does not infer clusters from repeated item identifiers in a long table.

For balanced repeats, the simplest item-level estimator is:

\[ d_i = \frac{1}{R}\sum_{r=1}^{R}(s_{Air} - s_{Bir}), \qquad \hat{\Delta} = \frac{1}{N}\sum_{i=1}^{N} d_i. \]

Compute one mean difference \(d_i\) per item, then bootstrap the \(N\) item differences. Every item receives equal weight. Three calls refine the estimate for item \(i\); they do not change \(N\).

The target determines whether calls are resampled

Collapsing to item means treats the observed calls as the measurement used for each item and estimates variation across new items under that fixed procedure. That often matches the question behind a held-out evaluation: how stable is the mean system gap over another sample of comparable items?

If the target also includes fresh stochastic judge calls, a two-level bootstrap can represent both sources. First sample item identities with replacement. Within each selected item, sample its judge calls with replacement. Recompute the per-item mean and then the system gap. The hierarchical-bootstrap paper by Saravanan, Berman, and Sober describes this resampling pattern for nested measurements and publishes the simulation code used to study it.

Those procedures answer different questions. The item-only bootstrap conditions on the observed measurement procedure. The two-level bootstrap includes its finite-call variation. Neither procedure turns judge calls into additional item coverage.

The design becomes more complicated when every item is scored by every member of a judge panel. Items and judges are then crossed rather than purely nested. Whether judges should be resampled depends on whether the named checkpoints are fixed instruments or a sample from a wider population of possible judges. A generic hierarchical loop cannot decide that estimand.

Unequal repeats create an accidental weighting rule

Evaluation tables rarely stay balanced. One item may need a retry after malformed JSON, another may time out, and a third may receive extra calls during an audit. Averaging every surviving row gives more weight to items with more valid responses.

That weighting might be intended, but it should not arise from failure handling. An invalid response followed by a successful retry is one requested measurement with two linked attempts, not two votes. If each held-out item should count equally, aggregate according to a declared retry rule and compute one item contribution before resampling.

The minimum record needs enough structure to reconstruct that choice:

item_id, system_id, judge_id, judge_digest,
prompt_version, run_id, seed, presentation_order,
attempt_id, retry_of, parse_status, score

Without item_id, cluster membership is gone. Without attempt_id and retry_of, a transport failure can silently change statistical weight. Without the judge and prompt versions, repeated calls may combine different instruments under one label.

Spend repetitions on the uncertainty they can reduce

Additional judge calls can estimate within-item instability. They can reveal a position-sensitive comparison, quantify invalid-output rates, or reduce noise in an item’s aggregated score. They cannot reveal how the systems behave on an unseen kind of story or translation.

For a fixed evaluation budget, one more call on an existing item and one call on a new item therefore buy different evidence. The first measures the evaluator more precisely at a known point. The second expands item coverage. A confidence interval should preserve that distinction by resampling the identity that was actually sampled.

The ESC . Print Raster Graphics command makes that boundary less obvious than a fixed-size record. Its six-byte header declares a compression mode, horizontal and vertical densities, a band height, and a width in dots. In uncompressed mode, those fields determine the number of payload bytes directly. In run-length encoded mode, the stream carries counters whose encoded length varies with the image.

An earlier 86Box debugging note recorded the integration result: after the command was implemented, a Windows 95 Epson LQ-2500 test page changed from seven malformed PNG files and an emulator termination to one complete page. This note examines the parser invariant behind that result. The command ends when it has produced the declared raster rectangle, not after a fixed number of compressed input bytes.

The header describes the decoded image

Epson’s ESC/P Reference Manual gives the command in this form:

ESC . c v h m nL nH d1 d2 ... dk

The c byte selects uncompressed data (0) or run-length encoding (1). The v and h bytes select vertical and horizontal density as 3600 / value dots per inch. The band height m is 1, 8, or 24 dot rows. The two little-endian bytes nL and nH give the width in dots:

width = nL + 256 * nH

Pixels are packed most-significant bit first. A row therefore occupies the ceiling of the width divided by eight, and the complete decoded payload size is:

bytes_per_row = (width + 7) / 8
decoded_bytes = m * bytes_per_row

Integer division is intentional in the first expression. A 13-dot row still needs two bytes. The final three bits belong to padding rather than pixels. An eight-row band at that width contains 16 decoded bytes regardless of how well or poorly it compresses.

The merged 86Box setup code turns the header into this state. It validates the compression mode, densities, band height, and nonzero width; computes the row and band byte counts; records the current print position as the origin; and activates raster handling.

This is the first useful separation in the parser. The header defines an output rectangle. Compression only defines how subsequent input bytes reconstruct it.

RLE changes the transport length

The manual defines two kinds of RLE counter. Values from 0 through 127 mean that the following counter + 1 bytes are literals. Values from 128 through 255 mean that the next single byte must be repeated 257 - counter times.

For the 16-byte band above, this six-byte compressed payload produces the required decoded length:

02 80 00 FF F4 00

The first counter, 02, copies the following three bytes literally: 80 00 FF. The second counter, F4 or 244, repeats 00 thirteen times. Three literal bytes plus thirteen repeated bytes fill the band.

The same raster could require seventeen encoded bytes if it were sent as one literal run: a counter of 15 followed by all sixteen data bytes. Nothing in the six-byte command header declares either encoded length. Counting bytes received would make the command boundary depend on the compression ratio, which the parser does not know in advance.

The counter rule also deserves a literal implementation. A byte value of 128 requests 129 copies because 257 - 128 = 129. Treating 128 as a no-operation would silently substitute the rule from a different run-length format. The protocol’s arithmetic is a better specification than a familiar compression name.

Three states preserve the boundary

86Box consumes printer data one byte at a time. Its raster decoder uses three states:

COUNTER -> LITERAL -> COUNTER
        -> REPEAT  -> COUNTER

In COUNTER, one input byte selects a run type and length. In LITERAL, each input byte emits one decoded byte until that run is exhausted. In REPEAT, one input byte can emit as many as 129 decoded bytes. Uncompressed mode bypasses the RLE states and emits each input byte directly.

The invariant is independent of those paths:

0 <= decoded_so_far <= decoded_bytes

Every emitted byte increments decoded_so_far. When it reaches the band size computed from the header, the implementation advances the horizontal print position by width / horizontal_density and deactivates raster handling. The next input byte returns to the ordinary ESC/P interpreter.

This design makes compressed input a producer of raster bytes rather than a second command-framing scheme. Literal runs, repeat runs, and uncompressed bytes all meet at one output counter. The command boundary has one owner.

It also identifies the malformed-stream policy that needs an explicit test. A valid RLE sequence must produce exactly the declared rectangle. A counter that promises more output than remains must not be allowed to draw outside the band, and an incomplete stream must not be mistaken for a complete command. The merged change validates header fields and stops rendering at the declared output count. It does not claim a general recovery mechanism for every truncated or overlong printer stream.

Decoded order determines pixel position

The output counter carries enough information to place a byte without retaining the whole band. The rendering function derives a row and byte offset as follows:

row    = decoded_so_far / bytes_per_row
byte_x = decoded_so_far % bytes_per_row
dot_x  = byte_x * 8 + bit

Bits are tested from 0x80 down to 0x01, so the leftmost pixel in a byte is emitted first. The dot_x < width check discards padding bits in the last byte of a non-byte-aligned row. Horizontal and vertical densities then map the dot coordinates to positions on the emulator’s page bitmap.

This mapping depends only on decoded order. It gives identical pixel positions when the input is uncompressed, split across several literal runs, or expressed with repeat counters. It also prevents input packet boundaries from becoming image boundaries. Whether the emulated parallel port delivers the six bytes above continuously or with pauses does not change the reconstructed sequence.

The current print position moves only after the complete band. Advancing it for every compressed byte would produce different layout for two encodings of the same pixels. Again, decoded geometry owns the visible result.

Test framing before judging the page

The Windows 95 test page remains useful. It exercises a real driver, color selection, raster graphics, text, page completion, and the transition back from image data to later commands. One correct page establishes that the reproduced job crosses those subsystem boundaries successfully.

A smaller parser test should make the framing claim direct. For a 13-by-8 band, it can feed 16 known decoded bytes through both uncompressed mode and several RLE partitions, then compare the resulting pixels and final print position. The test should place another ESC/P command immediately after the raster payload and verify that it is parsed as a command rather than consumed as image data.

Boundary cases add information that a screenshot cannot:

• a width not divisible by eight, with set padding bits that must not draw;
• literal and repeat counters at both ends of their ranges, including counter 128;
• a run ending exactly on the final decoded byte;
• declared output that is shorter or longer than the supplied RLE expansion;
• the same decoded bytes divided into different legal runs.

The last case is the core equivalence test. If two byte streams decode to the same rectangle, compression choices must not change the page or the parser state that follows it.

The merged pull request repaired one concrete Windows 95 printing path and documented the page-level result. The deeper contract is smaller: derive the output length from the raster geometry, count bytes after decompression, and return control to the command interpreter at that exact boundary. A streaming parser stays synchronized when it measures the object the protocol actually declared.

Those numbers answer different questions. ROC AUC evaluates how well scores rank malignant cases above benign cases across possible cutoffs. Recall and precision describe the labels produced at one cutoff. A model can keep exactly the same ranking while a changed score scale moves many cases across the default cutoff.

This note isolates that distinction with six labeled cases. No medical claim follows from the toy sample. Its purpose is to show what a ranking metric preserves, what it discards, and why threshold selection belongs in the experimental protocol rather than in a default call to predict.

AUC keeps the order

Consider three negative and three positive cases with these scores:

import numpy as np
from sklearn.metrics import confusion_matrix, roc_auc_score


labels = np.array([0, 0, 0, 1, 1, 1])
scores = np.array([0.10, 0.40, 0.60, 0.55, 0.70, 0.90])
shifted = scores**3

print(roc_auc_score(labels, scores))
print(roc_auc_score(labels, shifted))

Both calls return (8/9), or approximately 0.889. There are nine positive-negative pairs. The positive case scored at 0.55 ranks above two negatives, while the positive cases at 0.70 and 0.90 each rank above all three. Eight pairs are ordered correctly.

Cubing every score is strictly increasing on this interval, so it preserves every pairwise order. The roc_auc_score documentation accepts either probability estimates or non-thresholded decision values because AUC concerns this ordering rather than one class cutoff.

The labels at a cutoff of 0.5 do not survive the transformation:

for name, values in (("original", scores), ("cubed", shifted)):
    predicted = (values >= 0.5).astype(int)
    tn, fp, fn, tp = confusion_matrix(labels, predicted).ravel()
    print(name, {"tp": tp, "fp": fp, "fn": fn, "tn": tn})

The original scores produce three true positives and one false positive. The cubed scores produce one true positive, no false positives, and two false negatives. Their AUC values are identical. AUC does not contain a preferred numeric origin or scale from which 0.5 could be recovered.

This is not a criticism of AUC. It is the property that lets AUC compare ranking performance from probability estimates and raw decision scores. The mistake is asking that ranking summary to specify an operating decision.

A threshold contains a cost judgment

A cutoff determines which errors the system accepts. On the original six scores, three candidate cutoffs give:

cutoff 0.50: TP=3, FP=1, FN=0, TN=2
cutoff 0.65: TP=2, FP=0, FN=1, TN=3
cutoff 0.85: TP=1, FP=0, FN=2, TN=3

Suppose, only for arithmetic, that one missed positive costs five units and one false alarm costs one. The three error costs are 1, 5, and 10. Reverse that ratio, so that a false alarm costs five units and a miss costs one, and the costs become 5, 1, and 2. The score order and AUC did not change. The preferred cutoff among these candidates did.

Real decisions require a defensible objective rather than invented unit costs. The relevant quantities might be review capacity, the consequence of a missed case, a minimum recall constraint, or the measured utility of downstream actions. Scikit-learn’s threshold-tuning guide makes the software boundary explicit: predict_proba or decision_function produces scores, while a threshold converts them to labels. Its usual 0.5 probability cutoff and zero decision-score cutoff are defaults, not application requirements.

Precision and recall must therefore be attached to the selected rule. Moving the cutoff changes both even when the model and every score remain fixed. Reporting “the model has 0.98 recall” without naming the positive class, cutoff, and evaluation set omits part of the classifier being evaluated.

Calibration is a separate claim

The cubed values above remain useful scores for ranking, but calling them probabilities would require evidence. A calibrated probability has a frequency interpretation: among cases assigned a value near 0.8, roughly 80% should belong to the positive class. Scikit-learn’s calibration guide tests this relation with reliability diagrams that compare mean predicted probability with observed positive frequency in bins.

A strictly increasing transformation can preserve AUC while changing calibration. That gives two independent questions:

• Does the model rank positive cases ahead of negative cases?
• Do its numeric outputs support a probability interpretation?

The second matters when a decision rule uses estimated risk, expected cost, or a probability communicated to a person. It is less necessary when a validated rule simply reviews the highest-scoring fixed number of cases, though that rule still needs evaluation under the deployment distribution.

Logistic regression does not make calibration automatic. The scikit-learn guide notes why logistic regression can be well calibrated when the model is correctly specified and regularization is appropriate. Dataset shift, regularization, feature changes, and a mismatched functional form can still break the frequency interpretation. Calibration has to be measured on data not used to fit the calibrator.

Do not spend the test set choosing the cutoff

Threshold selection is model selection even when the fitted coefficients never change. Trying many cutoffs on the final test set and keeping the one with the best F1 score adapts the reported result to that test sample.

A clean protocol separates three roles:

1. Fit preprocessing and model parameters on training data.
2. Select or tune the operating rule on validation data or within an explicit cross-validation procedure.
3. Evaluate the fixed model and fixed rule once on a held-out test set.

Scikit-learn’s TunedThresholdClassifierCV implements the second role by optimizing a specified binary metric over thresholds using cross-validation. Its documentation also warns against fitting the estimator and selecting its cutoff on the same data. Cross-validation handles the mechanics; it does not choose the right objective or positive label for the application.

The evaluation record should preserve the score source, positive-class mapping, calibration method if any, threshold objective, data used for selection, chosen cutoff, and final confusion counts. Those fields make clear which result belongs to ranking and which belongs to the operating rule.

Report the score and the rule

The earlier Wisconsin baseline’s ROC AUC of 0.995 supports a narrow statement about ranking on resamples of that dataset. Its malignant-class recall of 0.98 supports a different statement about one fitted pipeline, label mapping, and cutoff on one holdout split. Neither number determines the other.

A deployable classifier is not just a scoring function. It is a scoring function plus a declared rule for acting on the score. AUC can compare the first part across all cutoffs. Calibration can test whether the score has a probability meaning. The threshold, selected against a stated objective on separate data, completes the decision.

The convergence theorem explains why that procedure must eventually stop when a separating hyperplane exists. It does not say that all numeric representations of the same labeled examples take the same route. The bound depends on a radius and a margin measured in the coordinates supplied to the algorithm.

This note isolates that dependence with six points and one controlled change. The labels, example order, update rule, and separating decision boundary stay fixed. Only the unit of the first feature changes. The resulting run needs either 3 updates or 3,049.

The theorem counts mistakes, not epochs

Write each label as \(y_i \in \{-1, +1\}\) and include the bias by augmenting each point:

\[ z_i = (x_{i1}, x_{i2}, 1). \]

Starting from \(w=0\), the basic update is

\[ w \leftarrow w + y_i z_i \]

whenever \(y_i(w^Tz_i) \leq 0\). If a unit vector \(v\) separates every point with margin

\[ \gamma = \min_i y_i(v^Tz_i) > 0 \]

and \(R = \max_i \lVert z_i \rVert\), the algorithm makes at most

\[ \left(\frac{R}{\gamma}\right)^2 \]

updates. Novikoff’s 1962 convergence proof isolates the progress argument behind the finite bound. MIT’s perceptron theory notes state the modern \((R/\gamma)^2\) form directly.

The quantity being bounded is the number of mistaken-example updates. An epoch is only one pass through however many examples the implementation presents. A run can complete many epochs but make few updates, or concentrate several updates in one epoch.

Six labels under three rulers

Consider these points in this fixed order:

(-3,  3) -> -1
(-2,  0) -> -1
(-4, -3) -> -1
( 3,  0) -> +1
(-1,  1) -> -1
( 0,  2) -> +1

The rule

\[ x_1 + 0.3x_2 - 0.2 = 0 \]

separates them. Now replace the first coordinate with \(x’_1=sx_1\). The equivalent boundary is

\[ \frac{1}{s}x’_1 + 0.3x_2 - 0.2 = 0. \]

The decision for every point is unchanged for any positive \(s\). The perceptron does not receive the symbolic equivalence, though. It receives vectors whose first coordinate may be one hundred times larger or smaller, and its additive update uses those coordinates literally.

This exact loop, evaluated with rational arithmetic, records the effect:

from fractions import Fraction


ROWS = [
    (-3, 3, -1),
    (-2, 0, -1),
    (-4, -3, -1),
    (3, 0, 1),
    (-1, 1, -1),
    (0, 2, 1),
]


def train(scale):
    weights = [Fraction(0), Fraction(0), Fraction(0)]
    updates = 0

    for epoch in range(1, 10_000):
        mistakes = 0
        for x1, x2, label in ROWS:
            point = [scale * x1, Fraction(x2), Fraction(1)]
            score = sum(weight * value for weight, value in zip(weights, point))
            if label * score <= 0:
                weights = [
                    weight + label * value
                    for weight, value in zip(weights, point)
                ]
                updates += 1
                mistakes += 1
        if mistakes == 0:
            return updates, epoch, weights


for scale in (Fraction(100), Fraction(1), Fraction(1, 100)):
    print(scale, train(scale))

The three returned results are:

100   -> 3 updates,    3 epochs, weights = (300, 1, 1)
1     -> 3 updates,    3 epochs, weights = (3, 1, 1)
1/100 -> 3,049 updates, 822 epochs, weights = (66.77, 1, -1)

All three final weight vectors classify all six transformed points correctly. The slow run is not evidence that the labels became nonseparable. Shrinking the first coordinate made each update in that direction small, while the fixed second coordinate and bias continued to move by whole units.

The geometry changes with the representation

The same known boundary provides a valid, though not necessarily optimal, margin for each scale. In transformed coordinates its unnormalized weight vector is \((1/s, 0.3, -0.2)\). Normalizing that vector and measuring the largest augmented point gives:

s = 100:  R = 400.0125, gamma = 1.108974, bound = 130,109 updates
s =   1:  R =   5.0990, gamma = 0.376288, bound =     184 updates
s = 0.01: R =   3.1625, gamma = 0.004000, bound = 625,109 updates

These are upper bounds derived from one separator, not predictions of the observed counts and not claims that this separator has the maximum possible margin. Their looseness is visible at \(s=100\): the bound grows to 130,109 while the run still needs only three updates.

The useful part is the dependency. Rescaling one coordinate changes both the largest vector norm and the normalized margin. Uniformly multiplying every augmented coordinate by the same positive constant would multiply \(R\) and \(\gamma\) together, leaving their ratio unchanged. Changing one feature’s unit changes the Euclidean geometry seen by the update rule.

The theorem also does not promise the same final separator. It promises that some separator will be reached after finitely many mistakes under its assumptions. Example order, feature scaling, tie handling, bias representation, and learning rate can all change the path.

Scaling is part of the experiment record

For a direct implementation, the input units belong beside the seed and example order. Reporting only that the data were linearly separable omits information needed to reproduce the trace.

For library code, the estimator’s stopping rule must also be distinguished from the theorem. Scikit-learn’s Perceptron exposes max_iter, tol, and optional shuffling; its documented iteration count is a count of passes, not the raw mistake bound above.

Standardization is one defensible coordinate choice, not a theorem requirement. Scikit-learn’s StandardScaler records the training mean and variance used for each feature. Its preprocessing guidance also gives the operational constraint: learn transformations from the training subset, then apply the same transformation to validation, test, and production inputs.

The six-point construction establishes a narrow result. A change of units can leave every label and separating rule intact while changing a deterministic perceptron run by three orders of magnitude. Convergence is a property of separability; the route to convergence belongs to the representation.

The recompiler tracks pages needing that work in an intrusive eviction list. Each page record stores its own previous and next indices, while a global index names the list head. A special value in evict_prev means that the page is not in the list. Those fields describe one relation and have to agree.

A soft restart on an emulated VIA VT82C496G system exposed a state in which they did not. Conventional-memory remapping changed the backing pointers and dirty-code masks of existing page records. The same path also set evict_prev to the not-in-list value without unlinking the page. The global head could still point to that page, so the next purge reached it and then aborted because its membership test said it was absent. The merged correction deleted the two assignments that created this contradiction.

One relation had three representations

The relevant page fields are small enough to read as a contract. The page declaration defines evict_prev, evict_next, and the EVICT_NOT_IN_LIST sentinel. Membership is tested only through evict_prev:

static inline int
page_in_evict_list(page_t *page)
{
    return page->evict_prev != EVICT_NOT_IN_LIST;
}

That predicate is one representation of membership. The list head is another. The forward and backward links are a third. For a live list, at least these conditions must hold:

• the head is zero exactly when the list is empty;
• every page reachable from the head reports that it is in the list;
• adjacent pages agree about their forward and backward links;
• the stored page count equals the number of reachable pages.

The list insertion and removal functions update those representations together. Insertion rewires the old head, sets both links on the new page, advances the head, and increments the count. Removal rewires the neighbors and head before writing the sentinel and decrementing the count.

Writing only the sentinel is therefore not a smaller removal. It is a different operation with no valid list state as its result.

The guard reported the contradiction

The removal function begins with a defensive check:

if (!page_in_evict_list(page))
    fatal("page_remove_from_evict_list: not in evict list!\n");

The failure message can suggest an invalid removal request. In this case, the caller had obtained the page through valid list state. The purge path reads the global head, checks the page’s dirty translated-code mask, and eventually removes the processed page.

The contradiction was already present before removal began:

purgable_page_list_head = page_index
pages[page_index].evict_prev = EVICT_NOT_IN_LIST

The first line makes the page reachable. The second makes the membership predicate false. The guard converted a silent structural error into a reproducible abort at the next operation that needed the relation to be coherent.

This distinction matters during diagnosis. Weakening the guard would not repair the list. Forcing removal to continue would use a sentinel as a neighbor index and could corrupt more state. The useful question was not why removal rejected the page, but which earlier operation was allowed to produce an impossible page.

Remapping did not own list membership

The page-table allocation path creates the page array and initializes every evict_prev field to the sentinel. That is the right time to declare that a newly allocated page is not linked.

Conventional-memory remapping has a narrower job. The remap function reuses existing page records while changing their RAM pointers, write callbacks, and per-byte mask pointers. It does not allocate a new page table. It does not rebuild the eviction list. It therefore cannot safely reinitialize one of the list fields.

The fix removed one assignment from each of the function’s two remapping loops. No replacement list operation was needed because remapping was not supposed to change membership. Preserving the links kept the relation owned by the insertion and removal functions.

Deletion is safe here for a specific reason: the old lines had no independent state transition to express. They partially imitated initialization inside an update path. If remapping had needed to remove pages, the correct repair would have been an explicit unlink operation with defined handling for the list head, neighbors, and count.

Test the transition that exposes the invariant

The pull request records a narrow regression path: a Windows 95 guest on the 486-VIP-IO2 machine called ExitWindowsEx(EWX_REBOOT). The unmodified debug build aborted during the first guest-initiated restart. With the two assignments removed, three consecutive restarts returned to the Windows network logon dialog.

That test is useful because a soft restart preserves enough emulator state to exercise remapping against an already active recompiler. A cold start initializes the page table and list together, so it is less likely to reach the contradictory state.

The lifecycle test establishes that the reproduced restart no longer triggers the fatal removal check. It does not prove every remapping path or every machine configuration correct. A debug-only structural checker could make the invariant more direct: walk from the head, verify each backlink, reject the sentinel on reachable pages, detect cycles, and compare the visited total with purgeable_page_count.

The general lesson is smaller than “test more restarts.” When a relation is stored in several fields, each mutation must either update all of them or leave all of them alone. Reinitializing one field of a live intrusive list is not cleanup. It is corruption that waits for the next honest guard to find it.

The programming interface exposes four clock choices for each noise generator. The Philips description of the chip specifies three fixed rates and one rate controlled by a tone generator. It does not give an emulator author a bit-for-bit reference sequence. The implementation still needs a precise state transition, or the right register writes can produce the wrong sound.

That distinction mattered in an 86Box report about static sound effects. The old noise path used a 16-bit state and the wrong feedback taps. The merged correction changed it to the SAA1099’s 18-bit recurrence. Listening found the symptom; a state-space test made the correction reviewable.

Write the recurrence as a contract

The corrected implementation shifts the state left and inserts the XNOR of stages 18 and 11. In zero-based C bit positions, those taps are 17 and 10:

MASK = (1 << 18) - 1


def step(state):
    feedback = 1 ^ ((state >> 17) & 1) ^ ((state >> 10) & 1)
    return ((state << 1) | feedback) & MASK

This short form fixes four choices that prose such as “use an 18-bit LFSR” leaves open: shift direction, tap numbering, XOR versus XNOR, and the retained width. A test should fix the seed as well.

86Box starts its allocated device state at zero, so XNOR feedback is convenient. The all-zero state immediately advances; the all-ones state is the excluded lock-up state. AMD’s LFSR application note describes the general rule: a maximum-length n-bit recurrence visits 2^n - 1 states, and changing XOR to XNOR changes which uniform state locks up.

MAME’s SAA1099 implementation uses the complementary convention: XOR feedback with an all-ones initial state. Comparing its literal state words with 86Box would therefore reject a valid complement. The useful comparison is the tap relation, update direction, period, and declared output polarity.

Count every state once

For 18 stages, the maximum non-locking period is

\[ 2^{18} - 1 = 262{,}143. \]

The period test is small enough to run directly:

state = 0

for period in range(1, 1 << 18):
    state = step(state)
    if state == 0:
        break

assert period == 262_143

This checks more than the integer width. A mistyped tap can remain inside 18 bits and still enter a shorter cycle. A correct recurrence reaches its starting state after 262,143 updates and not before.

Running the same audit on the replaced 86Box recurrence gives a different structure. Starting from zero, it first advances to one, then enters a cycle of 32,767 states. The declared storage was 16 bits, but the feedback relation behaved as a shorter maximal sequence. This explains why changing only the C type would not have repaired the generator.

The merged 86Box code stores the running history in a 32-bit integer. Only the low 18 bits participate in the recurrence and output. Those low bits complete the 262,143-step cycle; the extra high bits are storage headroom, not additional generator stages.

Separate sequence from clocking

The SAA1099 register interface controls how quickly the recurrence advances. It does not select a different feedback polynomial. The Philips programming description gives fixed noise clocks of 31.25 kHz, 15.6 kHz, and 7.8 kHz, plus a tone-controlled option from roughly 61 Hz to 15.6 kHz.

A state-space test should therefore call the transition directly. If it renders audio samples instead, a clock-divider error can masquerade as a sequence error, and the test may pass or fail depending on buffer size. Timing needs its own assertions: each register choice should produce the intended number of generator updates over a known sample interval.

This separation also makes failures easier to name. Wrong taps alter the bit order and period. A wrong divider preserves the order but plays it at the wrong rate. Both can sound like bad noise, but they are not the same defect.

Test stereo with unequal values

The same 86Box change fixed a separate mixer error. In the noise-only path, the right output used the left amplitude. A listening test with equal channel levels could never expose it.

The smallest useful mixer test sets deliberately unequal values, enables noise without a tone, forces a known high noise bit, and checks the two accumulators independently. In the 86Box mixer’s integer scale, left and right amplitudes of 3 and 11 should contribute 270 and 990 respectively. Swapping, duplicating, or averaging them must fail.

That is a different invariant from the LFSR period. Keeping the tests separate prevents a stereo regression from being hidden inside a long expected audio buffer.

State what the test establishes

The period test establishes that the implemented recurrence has the intended maximum-length cycle under its declared seed convention. The tap and prefix checks establish deterministic bit order. The asymmetric mixer test establishes independent digital channel gains at the implementation boundary.

None of them proves that the recurrence is a transistor-level description of every SAA1099 revision. The public Philips material documents the programming surface and noise clock choices, while the exact recurrence is corroborated by an independent emulator implementation and by the repaired game paths. Direct hardware capture would be stronger evidence for silicon fidelity.

That boundary is useful rather than embarrassing. The tests turn an audible complaint into three narrow claims about sequence, timing, and stereo routing. A later hardware trace can challenge one claim without making the others vague.

That definition is less obvious than exact string matching. Addition is commutative: 12+34 and 34+12 are different strings but the same arithmetic fact. Testing on one after training on the other does not provide the same evidence as testing an unseen operand pair.

An audit of the 300-task benchmark found two additions whose operand-swapped twins were present in training. The exact-overlap check had passed because the strings differed. The arithmetic tasks did not.

The CalcGPT reproducibility branch now treats a holdout as a sample of semantic groups rather than a sample of formatted equations. The code change is small. Its useful part is deciding what counts as the same task, then carrying that decision through splitting, benchmark sampling, and artifact records.

Define sameness before shuffling

The group key parses each equation into integer operands and an operator. It ignores answer formatting and removes leading-zero differences before applying the operation rule:

def semantic_key(left, operator, right):
    if operator == "+":
        low, high = sorted((left, right))
        return ("+", low, high)
    return ("-", left, right)

Addition uses an unordered operand pair because it is commutative. Subtraction keeps the operands ordered. 12-7 and 7-12 are different tasks, even before restricting this experiment to nonnegative results.

Parsing before grouping also separates task identity from representation. 012+034=... and 12+34=... receive the same key. A reversed answer and a normal answer do too. That is required for the planned representation ablation: changing the serialization must not silently change which arithmetic facts enter the holdout.

The implementation lives in lib/data.py. It groups examples first, sorts the groups for a stable starting order, shuffles them with a declared seed, and assigns complete groups to training or validation. The regression test compares the sets of group keys across both partitions, not only their equation strings.

This is the arithmetic version of grouped splitting. Scikit-learn’s GroupShuffleSplit accepts domain-specific group labels for the same reason: the unit that must stay together is not always one row.

A requested fraction becomes approximate

Grouping changes the meaning of a 20% validation split. Individual examples are no longer independently movable.

If both addition orders exist, an off-diagonal group can contain two strings. A diagonal group such as 7+7 contains one. The splitter adds whole groups until it reaches its target example count, so the final fraction may cross the requested value by one group.

That is not a defect to hide with another random move. Moving one member would violate the exclusion invariant. The artifact should record the requested fraction, the realized example counts, and the realized group counts. A reproducible split owns its actual denominator, not only its seed and nominal ratio.

Sample from the group space directly

The benchmark sampler in lib/benchmark.py goes further than generating ordered tasks and rejecting leaks afterward. It indexes the semantic group space itself.

For a magnitude bucket containing n possible operand values, unordered addition has

\[ \frac{n(n+1)}{2} \]

groups. Canonical nonnegative subtraction has the same count, because each task satisfies left >= right. The combined bucket therefore has n(n + 1) semantic groups.

This gives 110 groups for one-digit operands, 8,190 for two-digit operands, and 810,900 for three-digit operands. Each training task is encoded into one of those group identifiers. The sampler removes the blocked identifiers, samples ranks uniformly from the remaining space without replacement, and maps each compressed rank back to its original identifier with a binary search.

Only after an addition group is selected does the sampler choose an operand order. This avoids giving an off-diagonal addition group twice the sampling weight of a diagonal group merely because it has two printable strings.

Rejection sampling could also avoid overlap, but it makes runtime depend on how much of the bucket is blocked and invites an unnoticed retry limit. Sampling the allowed group ranks states the capacity in advance. If 100 tasks are requested and only 99 groups remain, generation fails with that fact instead of returning a smaller benchmark.

Put the relation in the manifest

The holdout manifest records the exclusion policy, seed, operand width, sample count, task-roster hash, excluded-roster hash, and each task’s semantic key. Rendering the same roster into four text representations preserves the manifest identity.

A later report can therefore check two separate claims:

• the evaluated tasks belong to the declared benchmark roster;
• the benchmark roster is disjoint from training under the declared equivalence relation.

A dataset hash alone establishes neither. It identifies bytes, not the task relation used to decide whether those bytes leak information.

The equivalence relation is part of the claim

This rule does not make every arithmetic notion of similarity disappear. 1+4 and 2+3 share a result but remain different groups. Tasks with similar carry patterns also remain distinct. If the research question concerns unseen results, unseen carries, or extrapolation to wider operands, each needs a different holdout contract.

The present relation supports one narrow statement: no evaluated addition repeats a training operand pair in the opposite order, and no evaluated subtraction repeats the same ordered operands. The grouping and benchmark tests establish that software invariant. They do not establish that a model has learned arithmetic.

That result still requires completed runs. The split only makes clear which facts those runs are allowed to call unseen.

The current pull request does not make the model larger or publish a new score. It narrows the experiment: record which arithmetic tasks enter each split, define what counts as an unseen task, and bind later results to versioned data and model artifacts.

Exact overlap was not enough

The first held-out check excluded exact operand/operator tuples from the training data. That misses an equivalence specific to addition. If 12+34 is in training, testing 34+12 does not provide the same evidence as testing an unseen operand pair.

Under the intended canonical benchmark seed 42, an audit of the 300-task sample found two addition tasks whose operand-swapped twins were present in the committed training data. This does not show that an earlier model memorized either pair. It shows that exact disjointness could not support the stronger claim of semantic disjointness.

The proposed sampler assigns addition to a group keyed by its sorted operands; excluding either ordering excludes both. Subtraction remains directional. A regression test checks the semantic groups as well as exact task overlap. The lesson is narrower than “remove duplicates”: the split invariant must match the symmetry of the task.

Artifacts record evidence; they do not create it

A saved model needs more than weights. The proposed training path writes a tokenizer, a task-format specification, and a manifest containing the source dataset hash, grouped-split hashes, configuration, Git state, dependency versions, and training losses. Proposed evaluation reports record their own dataset, model, configuration, and artifact hashes.

Those records matter only when their relationships are checked. The proposed ablation plan constructs one semantic holdout manifest from the declared source roster, renders that roster for all four representations, and assigns the same manifest hash to every cell. Evaluation rejects a supplied manifest whose roster does not match the evaluated tasks; the summary contract checks that run provenance against the plan. A shared seed or row count alone is not accepted as that binding.

Identity checks are necessary but not sufficient. An adversarial review found that an aggregator could recognize the canonical benchmark hashes while accepting metrics over only 100 examples from a 300-task manifest, or a completed record that omitted EOS, arithmetic-stratum, throughput, and supervised-target evidence. A hash establishes which roster was intended; it does not establish that the entire roster was scored or that the required measurements were recorded. The summary validation now requires both provenance and completeness: the canonical 300-task denominator; versioned train and validation target-token counts that match the training manifest; EOS evidence; all seven declared arithmetic-stratum partitions reconciled to the primary counts; and finite, internally consistent throughput under its declared successful-completions scope. Regression tests cover the missing and inconsistent cases. This establishes a reporting contract, not a model-quality result.

The repository changes also separate fast contract checks from a real-ML smoke test. The pull-request matrix stays dependency-light across Python 3.11–3.13 and tests the extracted source distribution on Python 3.11. A scheduled or manually dispatched workflow installs the ML runtime, trains an intentionally tiny model, and checks that the tokenizer, task specification, training manifest, model configuration, and weights survive the round trip.

That smoke test establishes packaging and artifact plumbing. It is not a model quality benchmark.

Low accuracy is a result, not a crashed process

The previous evaluation command returned a failing process status when arithmetic accuracy fell below 50 percent. That is useful only when a threshold has been declared as a gate. For an ablation, a small model that scores poorly has still produced a valid outcome if loading, generation, validation, and report writing completed correctly.

The proposed command therefore separates operational failure from scientific outcome. A caller can request a --fail-under threshold explicitly; without one, weak runs remain in the experiment record instead of disappearing behind a nonzero exit status.

The proposed evaluator restricts its primary arithmetic metrics to prompts with a determined answer. Given only the first operand, many continuations are valid; scoring one arbitrary continuation as the unique target makes aggregate accuracy difficult to interpret. Those cases remain diagnostics rather than headline accuracy.

Four representations, one task roster

The planned experiment crosses fixed-width layout with answer order. Fixed-width means that both operands and the answer occupy declared-width fields:

The plan renders all four datasets from one ordered roster of normalized (left, operator, right) tasks. It declares model seeds 41, 42, and 43 in advance, with split seed 42 and benchmark seed 42 fixed across conditions. The current script validates the configuration and expands the four-by-three matrix into distinct dataset, model, and report paths; it does not launch training.

This is a whole-representation comparison, not a token-budget-matched test of operand padding. Under answer-only loss, minimal answers have variable length; fixed-width answers always contain operand_width + 1 digits, and EOS is a target in both cases. Those target counts belong in each run record. Any narrower claim about padding would require a separate matched-budget experiment.

If the runs proceed, every planned cell—including failures and low scores—belongs in the record. A completed run binds its normalized train, validation, and evaluation rosters to the declared benchmark-manifest hash. It reports exact, numerical, strict-format, EOS, throughput, and arithmetic-stratum measurements with denominators. A failed run retains its failed stage, error or exit status, configuration, Git revision, and available diagnostics rather than disappearing. Model weights can remain release assets; plans, manifests, hashes, and summaries can remain reviewable files.

Reinforcement learning is a separate experiment

Reinforcement learning is not part of the current baseline, implementation, or result. It may become a post-baseline study only after the supervised four-way matrix is complete.

The smallest useful study is a three-arm continuation from the frozen padded-reversed checkpoints at seeds 41, 42, and 43: no continuation, an equal-compute supervised continuation, and a verifier-reward policy-optimization continuation. Naming the representation before seeing the baseline avoids choosing the apparent winner after the fact. The supervised control distinguishes an RL effect from the simpler effect of giving the model more optimization.

The continuation data would use a separate roster with no canonical benchmark semantic groups. Reward would be 1 only for an exact canonical answer followed by EOS and 0 otherwise; the KL coefficient, schedule, compute budget, and stopping rules would be fixed before running. The existing 300-task benchmark would remain untouched until final evaluation. Verifier mismatch, reward exploitation, and output collapse would be recorded as failed runs rather than repaired mid-study.

This is an outcome-reward experiment because CalcGPT emits only final answers. Adding process reward would first require an intermediate-reasoning representation and would answer a different question. DeepSeekMath introduced Group Relative Policy Optimization for mathematical reasoning, while DeepSeek-R1 studied verifier-based reinforcement learning on reasoning tasks. Let’s Verify Step by Step is useful precisely because it separates process supervision from final-answer outcome supervision. Those results motivate the protocol; they do not establish that the same behavior will transfer to this small model.

Without the completed supervised matrix, executable protocol, and recorded runs, RL remains a deferred question rather than a capability claim.

What this work establishes

The pull request under review defines stronger software and reporting contracts for a controlled representation ablation. It does not establish that CalcGPT has learned a general arithmetic algorithm, generalizes to wider operands, or reaches a particular canonical accuracy. No model-quality result has been produced by this work.

Those claims require completed runs and published evidence. The pull request is the review boundary, and the branch’s research log records implementation status separately from empirical results.

On clean Romanian, a 2.4M-parameter BiLSTM reached 96.23% word accuracy. Under heavy corruption it fell to 36.76%. The dictionary baseline finished at 70.31%.

The test input changed the winner. Once the text began to resemble typing mistakes or optical character recognition (OCR) output, the clean-text ranking stopped being a useful deployment guide.

Evidence status. This comparison is preliminary thesis work and a manuscript in preparation. It has not been peer reviewed. ByT5 used 50,000 training pairs, so its result is not a budget-matched architecture comparison.

The clean table rewards constraint

On the clean CRAWLER-1000 test, the dictionary system reached 93.72% word accuracy with a 0.045 diacritic error rate. The 2.4M-parameter BiLSTM led at 96.23% and 0.033. ByT5 reached 92.00% and 0.081. A 1.7B-parameter Qwen3 model adapted with LoRA reached 48.99% and 0.545; prompted Qwen2.5 and Llama baselines were both near 70.5% word accuracy.

Parameter count did not predict the result. The constrained systems have the right inductive bias for a mostly character-preserving transformation. The decoder has far more capacity, but capacity is not the same as control.

Noise reverses the operational choice

From clean to high corruption, word accuracy changed as follows:

The Qwen model’s low-noise uptick should be treated as sampling variation until replicated. Its smaller relative drop is not a robustness victory because it starts from a much lower clean ceiling.

The practical selection is therefore conditional. For clean text, the BiLSTM leads this benchmark. For corrupted input, the dictionary is the strongest of these four by the high-noise endpoint. ByT5 loses some clean accuracy but degrades more gradually; a budget-matched training run is needed before attributing that behavior to architecture.

Put the corruption model in the contract

A diacritic-restoration benchmark should state which non-diacritic edits are present, when corruption is applied, and whether the system may alter other characters. It should report accuracy at diacritizable positions together with unwanted edits elsewhere. Otherwise, a fluent rewrite can score as restoration while changing the source.

The next experiment needs repeated corruption draws across additional declared seeds, confidence intervals for model differences, latency on the target hardware, and an exact-preservation measure. Until then, this table chooses candidates for deployment tests; it does not crown a universal Romanian restoration model.

Those two uses require different evidence. Item-level agreement asks whether the judges assign similar labels to the same output. Rank agreement asks whether their aggregated scores order whole systems similarly.

The three panel members disagreed on individual scores. After median aggregation, the panel’s TF1 system ranking tracked o4-mini at Spearman \(\rho=0.93\) and Kendall \(\tau=0.78\). The fixed panel combined Granite 4.1 30B, EXAONE 3.5 32B, and Granite 3.3 8B. Across three tasks it ran 6,180 evaluations, with another 900 bias reruns. Item-level agreement remained weak: Krippendorff’s alpha ranged from −0.34 to +0.12, and mean weighted kappa from −0.02 to +0.15.

Evidence status. The panel study was accepted by the thesis cutoff. The human-arbitration protocol remains proposal-stage and has not run.

Agreement depends on the object

For individual items, the judges often used the rubric differently. That is not surprising in literary generation and translation: an answer can preserve meaning while losing style, or read fluently while changing an event. A single ordinal score compresses those disagreements.

Aggregation changed the picture. Under the paper protocol, TF1 system rankings from the panel tracked the o4-mini reference at Spearman \(\rho=0.93\) and Kendall \(\tau=0.78\). A later composition analysis using the full panel produced \(\rho=0.952\) and \(\tau=0.822\). These are different protocols, so the two pairs should not be blended into one headline number.

The defensible operating rule is narrow: use the panel for the tested TF1 model-selection protocol and report system-level rank agreement. Do not infer from that result that its item scores are reliable filters.

Three judges are not three independent views

Panel size alone can overstate diversity. Two members come from the Granite family. Model lineage, training data, and instruction style can create correlated errors even when parameter counts differ. A panel audit therefore needs more than a list of checkpoints: family diversity, order sensitivity, rubric perturbations, and repeated runs belong in the record.

The 900 bias reruns address part of that record. They do not prove independence or eliminate shared blind spots.

The human gate is still open

The proposed workflow routes model disagreements and other ambiguous cases to human review. The thesis projects a reduction from 3,588 human judgments under a three-rater, per-task protocol to 255 judgments concentrated on 85 disagreement cases. That is a planning estimate, not a completed human study.

Until the arbitration is run, the panel is a ranking instrument with a known weak point. It can reduce the amount of work presented to people; it has not earned the right to stand in for them.

Changing only the protagonist disturbed the unchanged moral in 4.0% to 18.5% of matched pairs, depending on the semantic verifier. Changing the moral almost never displaced the protagonist: specified-character realization changed in 0.5% of pairs [0.1%, 2.8%].

Those directions occupy different cells of the response–leakage matrix. I generated 200 matched pairs per intervention, holding every other slot and the decoding setup fixed and using a common seed within each pair.

Evidence status. These are thesis-draft experiments, not a peer-reviewed result. They cover two of the six slots, one generator, 4-bit local inference, and 200 matched pairs per intervention.

Characters responded; morals sometimes followed

When the character slot changed, the new character appeared in 100% of pairs, with a 95% Wilson interval of [98.1%, 100%]. The unchanged moral’s realization also moved in 4.0% to 18.5% of pairs, depending on the semantic verifier.

The direct response is surface-checkable. The leakage estimate depends on a learned verifier deciding whether two stories still realize the same moral. Every tested verifier detected the direction, but they did not identify one defensible magnitude. The common seed removes one source of variation; it does not prove that every remaining difference was caused by the intervention.

Moral changes rarely displaced the protagonist

Changing the moral produced an 87.5–98.0% response rate under semantic verification. Exact wording was much less stable: the requested moral appeared verbatim in 33.5% of pairs [27.3%, 40.3%].

The specified character was disturbed in only 0.5% of these interventions [0.1%, 2.8%]. The asymmetry is the result; its mechanism is not identified. Character interventions sometimes altered the moral-realization verdict, while moral interventions almost never disturbed the protagonist.

A story can express the requested lesson without repeating the specification literally. Semantic response measures meaning as interpreted by a verifier; verbatim response measures surface realization. Neither should be renamed ground truth.

The asymmetric result

Both tested controls responded strongly. Their leakage did not run equally in both directions. Character changes sometimes disturbed the unchanged moral; moral changes almost never disturbed the character.

Because the moral cells depend on automated semantic verifiers, this establishes a direction that survived every tested instrument, not a human-perceived leakage rate. The remaining four slots, other generators, and human adjudication still belong to the next measurement round.

The model was asked to restore Romanian diacritics. It answered with autoResizeIframe.

That token is not a plausible restoration error. It is a web-page artifact, probably learned from scraped training text. In one experiment, it appeared in outputs from the LLMic_v2 3B checkpoint before adaptation and remained after rank-16 LoRA, a doubled schedule, and 10,000 training steps.

Evidence status. This is a thesis-draft case study of one checkpoint, one task corpus, and one adaptation setup. It does not show that LoRA generally cannot remove contamination, or that the artifact is irreversible.

The adapter learned around the failure

The base checkpoint scored 0.00% word accuracy on the clean restoration test. After adaptation it reached 1.45%. The task score moved, but remained unusable, and the distinctive artifact survived.

A separate multilingual Qwen checkpoint adapted on the same task did not show that exact output pattern. That contrast narrows the diagnosis to this checkpoint and run; it does not identify the original document or prove a single causal path through pretraining.

The important operational signal came before aggregate evaluation. A model that emits templating code into ordinary Romanian text has failed the output contract, even if later tuning raises an average score. More epochs are a poor default response when the base model’s failure is both severe and out of distribution for the task.

Contamination is also a selection problem

Dataset cleaning is usually discussed upstream, where the pretraining corpus may be inaccessible to a downstream user. Checkpoint selection is the available downstream control. A small diagnostic set can probe for markup, navigation fragments, templating identifiers, instruction leakage, and non-language tokens before adaptation consumes compute.

That screen should preserve raw generations. Normalizing or post-processing them too early can erase the best clue about provenance.

For this task, the observed artifact should be a stop condition: reject or quarantine the checkpoint, compare another base model, and only then tune hyperparameters. If the model must be studied further, treat the artifact rate as its own metric rather than burying it inside word accuracy.

The checkpoint was disqualified before the causal story was complete

The experiment did not identify the original pretraining document or test full-parameter tuning. It did settle the model-selection decision: after 10,000 adaptation steps, this checkpoint still emitted scraper debris into Romanian output. That was enough to quarantine it before another sweep.

A score has no pairing structure. A comparison does.

That difference determines the bootstrap. In TF1, candidate generators were evaluated on disjoint samples, so the model gap was resampled independently. In TF2, several systems translated the same held-out items, so the gap was resampled by item. Calling both procedures “a bootstrap confidence interval” hides the experimental design that makes each one valid.

Archive note. The 2 May date marks TF1 arXiv v2, which introduced the revised panel evaluation. The confidence-interval analysis itself was added later in the August 2026 thesis draft.

Different stories: resample independently

The TF1 historical-judge comparison between Tulu and Llama has an estimated gap of +0.02 with a 95% bootstrap interval of [−0.13, +0.17]. The open-panel comparison gives +0.12 [+0.07, +0.18]. Each model was scored on its own sampled stories. One model’s row 37 is not the counterpart of the other’s row 37, so preserving row numbers as pairs would invent dependence.

Independent resampling respects that design: draw with replacement within each model’s sample, recompute the two means, then store their difference. The interval is about the difference of two sample means, not a halo around either score.

The two TF1 intervals also make a useful substantive distinction. The historical-judge gap includes zero; the panel gap does not under this analysis. That does not make the panel ground truth. It says the measured separation is more stable under one evaluator construction.

Same items: keep the pair intact

TF2 systems translated the same items. Their errors can therefore be correlated: a difficult sentence may reduce every system’s score. Resampling each system separately would throw away that information and usually misstate the uncertainty of the difference.

The paired procedure samples item indices and carries every system score attached to an index. It produced +0.03 [−0.02, +0.08] for o3 minus GPT-4.1, and +0.31 [+0.23, +0.41] for o3 minus TF2-4B. The first interval does not establish a separation; the second does on this set and evaluator.

The missing interval

TF2-12B was reported nine hundredths below o3, but its per-item evaluation outputs were not retained. A mean and a sample size cannot recover the covariance between systems. There is no honest paired interval to compute after the fact.

Missing the item rows removed the covariance needed for a paired interval. It also ruled out parity or non-inferiority language. The repair is operational: preserve one row per evaluated item, with the input identifier, system output, judge scores, prompt version, and run metadata.

The choice between paired and independent bootstrap is made before any random resampling begins. It lives in the data lineage of the comparison.

Three million sounds exhaustive until it is divided by a trillion.

The six slots are character, trait, setting, challenge, outcome, and teaching, with one hundred values available in each. That makes \(100^6\), or one trillion, possible specifications. The released three-million-story corpus occupies at most 0.0003% of that grid.

Archive note. This is an August 2026 analysis of the April 2025 TF1 release. The date marks the artifact being examined; the argument was added to this journal later.

What three million buys

If the sampler is balanced, each individual value appears about 30,000 times: a particular character, setting, or teaching is no longer a rare event. A particular value pair across two slots appears about 300 times on average. That is enough to inspect many first-order and pairwise effects with useful replication.

The sixth-order combination is different. Even with no duplicate rows, the corpus leaves 999,997,000,000 possible specifications unseen. A claim that the data “covers the design space” would confuse a large row count with dense joint coverage.

That leaves useful work to do. Check whether every setting is represented, or whether one character is unusually associated with one moral. Those are questions about individual fields and pairs. Asking whether every interaction among all six fields has been exercised is a much larger demand.

Ask which combinations the experiment needs

The useful question is not whether the corpus is large. It is: coverage of what?

• First order: roughly 30,000 observations for each slot value.
• Second order: roughly 300 for each cross-slot value pair.
• Sixth order: at most three million occupied combinations; almost all the rest are unseen.

Those averages assume balanced independent sampling. The actual records still need frequency tables, duplicate checks, and conditional counts; an expectation is not an audit. Rare generation failures or filtering can also disturb the intended balance after sampling.

A better release record

For a controlled synthetic corpus, I would publish three coverage summaries beside the row count: per-value frequencies, cross-slot pair frequencies, and the number of distinct full specifications. They expose different failures. A balanced marginal table can coexist with hidden pairwise coupling; clean pair counts can coexist with duplicated full prompts.

Three million rows make first- and second-order projections measurable. They do not make a trillion-cell space dense.