Olmo 3 concept coverage

Rollouts. Each model writes unprompted from <|endoftext|> at temperature 1, with no top-k or top-p, although post-trained configs recommend 0.6 and 0.95. A post-trained rollout ends at the model's first <|im_end|> or <|endoftext|>. Special-token ids are removed before encoding, since Dolma's SAE inputs contain none.

χ and r. χ is a concept's coverage relative to the average concept (CG paper Eq. 2). r = ln χ_set − ln χ_train. All sets in a comparison use the same number of documents, since fewer documents alone steepen the curve. Slope statistics are over the slope concepts (G13).

Classes and lists. Per model, concepts with χ_train ≥ e⁻⁵ are ranked by r: lowest 10% = under, highest 10% = over, the rest = middle (CG paper Sec. 3.1, 5.1). Random lists: 80 per class. Extreme lists: the 50 highest and 50 lowest r, unfiltered and among concepts with at least 50 firing docs on both sides.

Name quiz. 200 list entries, 100 per group (base = stage1, main; instruct = SFT, DPO, Instruct). Each gets a name from Dolma windows and one from that model's rollout windows, and both are quizzed on the same held-out Dolma V windows (30 firing + 30 non-firing, balanced accuracy, Paulo et al. detection). Design: exp2_rollout_gap/design.md G70-G74. Window text stays off this page (G20).

Concepts are ordered by χ in Dolma train, so a concept keeps its place across graphs and frames. Graph 1: Dolma alone (black: train; gray band: held-out V, what noise alone gives). Graph 2: rollouts alone. Graph 3: both overlaid, with the 240 named concepts on hover. Graph 4: graph 3 without the dots. Bands are the 10th-90th percentile of χ in each bin of 512 ranks, lines the median. Bottom: median log(χ / χ train) in 20 bins; a line below V at the rare end (right) and above it at the common end (left) means the model's output is more spread than its data.

The same graphs for the three post-training stages (design.md G30-G37). Document counts per frame are matched over Dolma V and all five models, so they are lower than in the mid-trained and pretrained charts.

CG paper Figs. 3-4 style. (a) Concepts per χ bin on linear χ and log counts. (b) The same over log10 χ. (c) Per concept, ln(χ / χ train), over the slope concepts; dashed lines are medians. Dolma V shows the width noise alone gives. In the token-window frames, the spike at the left of (b) is the smallest possible χ (concepts firing in a single document).

Left: per concept, log10 χ in the set against log10 χ in Dolma train, with y = x (black), y = x moved by the set's median ratio (dashed) and the median in each of 20 bins (dots). Right: histograms of log10 χ: Dolma train (gray), the set as is (dotted), the set shifted by its median ratio (solid, left) and also rescaled to train's sd (solid, right). When the shifted histogram lies on train's, the histograms hide the per-concept change the scatter shows.

D_pair = mean r² (concepts paired by id). D_marg = the same after sorting each side, so it sees only the two histograms. Shape share = D_marg / D_pair; rank-shuffle share = the rest. Var(r) splits into a trend part, (b − 1)² Var(ln χ_train) from the line fit of ln χ_set on ln χ_train, and the scatter around that line; the trend share is the part a slope can describe. Excess dispersion sd = √(Var(r) − the set's noise − train's noise), noise from split halves, in ln units; V defines train's noise (G40, G42).

Bold: bias-corrected Δ (G45). Below it: raw Δ and its 95% basic bootstrap interval (G17). Each column applies one document choice to every set, V included. Length-matched: in each length bin, every set cut to the smallest count (G43). Garbled removed: docs whose mean token NLL under the encoder model is above V's 99th percentile dropped (G58). Count control: random drops to the garbled-removed count, 20 draws (G60). Count control minus all docs is what the smaller count does alone; garbled removed minus count control is what the flag does.

Per set, the 10 concepts that gained and the 10 that lost the most χ mass (41 distinct, each tested with the set where it moved most), plus 20 controls matched in frequency. Each was named twice by Claude Opus 5.5, from 40 Dolma V windows and from 40 rollout windows, and each name was scored by Claude Sonnet 5.5 on held-out windows from both sides. Gap on Dolma = Dolma name's score − rollout name's score, on Dolma windows; gap on rollouts = the reverse, on rollout windows. Agree = both gaps ≤ 0.13, the 95th percentile of the same gap between two human corpora in Exp 1. Not scored = fewer than 90 of 100 windows answered on a side (G47-G53).

Pooled rows count a concept drawn in both groups twice.

Mean score of each name, per cell

Share of Dolma names scoring below 0.6, per cell

One dot per scored check entry; hover for both names. The dashed line marks 40 docs, below which the rollout name was written from every firing doc. A large gap at few docs can come from thin evidence as well as from a change of meaning, so the two cannot be separated in that range.

Splits: rollout docs under 40 or 40+

Splits: concept below e⁻⁵ in rollouts or above

Below the diagonal: the Dolma name scored higher. Colors follow the class.

Rollout names exist only for check entries, so this table holds Dolma names. r sd and the interval come from the noise rerun; "stays" is the share of reruns in which the concept keeps its class or list.

Lists. Per model, all 65,536 concepts qualify, with no firing floor on the missing side. In Dolma, missing from rollouts: χ_train ≥ e⁻⁵ and χ_rollout < e⁻⁵. In rollouts, missing from Dolma: the reverse. Every concept fires somewhere in our Dolma sample, so missing means weak, not absent. Our sample holds at most about 41M of the 5.93T stage 1 tokens.

Per model and group. 80 random, unfiltered; 50 most extreme, unfiltered; 50 most extreme, filtered to at least 50 firing docs on the side where the concept is present. Most extreme uses r: lowest for concepts missing from rollouts, highest for concepts missing from Dolma.

Names. Claude Opus 5.5 from up to 40 windows where the concept fires, every firing doc if fewer. Names are reused when one exists from the same kind of text, and are not scored.

Themes. Opus writes the categories from the random draws' names without seeing model or group. Opus and Sonnet 5.5 then each assign every concept separately; shares use Opus. Design: exp2_rollout_gap/design.md G87-G94. Training data outside stage 1 is not in the reference, so stage 1 is the cleanest test of concepts missing from Dolma.

Each cell: main share → agreed only → without those concepts. The main result keeps every concept.

Each concept's share of its activation from flagged docs at the matched 7,985 docs. Medians over each list, against all firing concepts in that model. A list below "all firing concepts" draws less on garbled docs than a typical concept.

Lists. Per set: top 10 by χ; every concept with χ = 0; bottom 10 by χ among χ ≥ e⁻⁵. Ties by concept id. Top and bottom 10 follow the CG paper (Sec. 4.2.1); concepts below e⁻⁵ are the missing-concepts experiment above.

Stability. 100 reruns per set, each keeping a random 80% of the same docs (CG paper App. B.3). A concept at χ = 0 cannot move in such reruns, since a subset of the docs cannot add firing, so χ = 0 is checked against the about 2,000 rollouts per model outside the compared docs.

Names. Claude Opus 5.5 from up to 40 windows where the concept fires: Dolma text for Dolma's lists and for χ = 0 concepts, the model's rollouts for its top and bottom 10. Names are reused when one exists from the same kind of text, and are not scored. Design: exp2_rollout_gap/design.md G95-G100.

These concepts sit just above the cutoff, so a different sample of docs picks different ones. Shown for completeness only.