External benchmarks

haspi targets the One Million Posts Corpus; every number in the rest of these docs is measured on it. This page collects the external checks — other corpora, other languages, published baselines — that test whether the sequence-χ² reward is a property of the method or of the corpus it was built on.

Note

Neither benchmark ships with this package: the data are third-party and the scripts live in the project’s research repo (haspi-playground, iqlearn/transformer/rpmod_*.py and hatexplain_*.py). They are documented here because they are the evidence behind the headline table, not because pip install haspi reproduces them. Both use the packaged API — haspi.sequence.features.extract() and haspi.sequence.fit.fit_reward() — so what they measure is the shipped code.

Summary

benchmark	language	what it tests	result
RP-Mod	German	moderation removal against a published fine-tuned BERT, same corpus and splits	0.796 / 0.798 AUROC vs their 0.791
HateXplain	English	does the method transfer; frozen vs fine-tuned encoders; do the per-word attributions match human rationales	0.857 AUROC; 0.422 Token F1 vs human rationales

RP-Mod — against a published fine-tuned baseline

RP-Mod (Assenmacher et al., NeurIPS 2021 Datasets & Benchmarks) is a German moderation corpus of ~14k comments, balanced between accepted and rejected, shipping both a fixed 80/20 split and 10 CV folds. Because they publish a fine-tuned German BERT on exactly those splits, it is the one place where the frozen reward can be compared like-for-like against a fully fine-tuned model.

Protocol: frozen LeoLM-7b, mean-pooled at maxlen=512, then the ordinary standardise → PCA-512 → χ² head → fold pipeline. Expert = accepted, anchor = rejected. No fine-tuning, no per-corpus tuning.

method	RP-Mod AUROC
frozen LeoLM-7b + seq-χ² — their fixed 80/20 split	0.796
frozen LeoLM-7b + seq-χ² — their 10-fold CV	0.798
Assenmacher et al., best fine-tuned BERT	0.791

The fold check reproduces at max|Δ| = 1.01e-06, as on OMP.

Report the digits, not the rounding

This is the “0.80 vs 0.79” row in the headline tables. The honest form is 0.796–0.798 vs 0.791 — a margin of about 0.006, not the 0.01 that rounding to two decimals implies. The claim it supports is “matches a fine-tuned BERT”, not “beats” it.

HateXplain — cross-lingual transfer and human rationales

HateXplain (Mathew et al., AAAI 2021) is a 20k-post English corpus from Gab and Twitter. It is the interesting external test because it ships human rationales — the token spans annotators marked as the reason for their label — so it can evaluate the per-word attribution itself, not just the classification.

Setup

• Encoder: Llama-2-7b. LeoLM is a German-adapted Llama-2-7b, so its English peer is the clean control: same architecture, same size, native to the target language. This isolates “does the method transfer” from “does the encoder match the language”.

• Task: hate+offensive vs normal, majority vote over the three annotators. The 919 posts with no majority (1-1-1) are dropped, as in the paper — 19,229 remain, 59.4% positive, on the official train/val/test split (15,383 / 1,922 / 1,924).

• Rationale ground truth: a token counts when ≥ 2 of 3 annotators marked it (the paper’s majority rule). 11,070 posts carry one; rationale tokens are 14.5% of tokens in those posts.

Classification

Five arms, varying one thing at a time — encoder size, encoder generation, and whether the encoder is trained.

encoder	params	released	val	test
frozen bert-base-uncased	110M	2018	0.805	0.818
fine-tuned bert-base-uncased (full, end-to-end)	110M	2018	0.864	0.872
frozen Llama-2-7b	7B	2023	0.854	0.857
frozen gemma-4-12B	12B	2026	0.857	0.858
frozen gemma-4-31B	31B	2026	0.862	0.865

The fine-tuning arm unfreezes the whole encoder against the same χ² objective, at the paper’s own settings (maxlen=128, lr=2e-5, 2 epochs). Its head stays r = w·φ_mean + b, the folded form, so it writes an ordinary reward pickle and every attribution path below is byte-identical across all five arms.

Three readings:

• Frozen encoder scale saturates early. 110M → 7B is worth +0.039. 7B → 31B, over 4× the parameters, is worth +0.008. Whatever the linear head can read off these features, a 7B encoder already provides nearly all of it.

• It is not a model-generation effect. gemma-4-12B (2026, 12B) scores 0.858, statistically on top of Llama-2-7b (2023, 7B) at 0.857, despite being newer and larger. Within the Gemma-4 family, 12B → 31B moves +0.007. Three years of pretraining progress buys essentially nothing for this readout.

• Fine-tuning a small encoder still beats freezing a large one. A fine-tuned 110M BERT (0.872) tops every frozen encoder including the 31B (0.865). This is the opposite of the OMP result, where a frozen LeoLM beat a fully fine-tuned gbert-base — the difference being training-set size: 15,383 labelled posts here against OMP’s 552.

Note

These are frozen encoders used as feature extractors, scored by a linear head on mean-pooled hidden states. That measures the linear separability of the representation, which is a far narrower property than a model’s generative or instruction-following ability. A capable 31B model saturating here is a statement about linear readout, not about the model.

What this number does and does not say

It shows the method transfers — same objective, same linear head, a different language and corpus. It is not comparable to two things it superficially resembles:

• Not to haspi’s OMP hate figure (0.76). HateXplain gives 15,383 training posts where OMP gives 552 hate-labelled ones, and hate+offensive vs normal is a more separable split than hate vs neutral.

• Not to HateXplain’s own 0.843 (BERT) / 0.851 (BERT-HateXplain). Those are 3-class (hate / offensive / normal) AUROC; this is binary.

Plausibility — attributions vs human rationales

For every positive test post with a rationale (1,098 after requiring both classes among its words), the words are ranked by their exact contribution and compared to the human mask. Metrics follow the paper: discrete scores use its top-5 rule (footnote 9: five ≈ the mean annotation span), and with one rationale per post ERASER’s IOU-F1 reduces to the share of posts whose predicted and gold token sets overlap at IOU > 0.5.

encoder	AUROC per-post	AUROC pooled	AUPRC per-post	Token F1 @5	IOU F1 @5
frozen Llama-2-7b	0.850	0.744	0.759	0.422	0.107
frozen gemma-4-12B	0.847	0.739	0.729	0.414	0.093
frozen gemma-4-31B	0.846	0.738	0.728	0.411	0.103
frozen bert-base	0.814	0.716	0.689	0.385	0.081
fine-tuned bert-base	0.831	0.590	0.662	0.401	0.077
random	0.495–0.500	—	0.291–0.296	0.193	—

Classification and plausibility are close to decoupled here

This is the result worth taking away, and it holds along both axes that were varied.

Fine-tuning buys accuracy, not plausibility. Unfreezing BERT moves classification +0.054 (0.818 → 0.872) while its agreement with human rationales stays flat or degrades: Token F1 0.385 → 0.401, but IOU F1 0.081 → 0.077, per-post AUPRC 0.689 → 0.662, and pooled AUROC 0.716 → 0.590. The pooled-AUROC collapse is the sharpest symptom: fine-tuning sharpens post-level separation, lifting every word’s score in a hateful post together, so cross-post word ranking degrades even as within-post ranking holds.

Scale buys neither. The 31B is no better at explaining than the 7B — Token F1 0.411 vs 0.422, per-post AUPRC 0.728 vs 0.759 — despite scoring higher on classification. The best attributions on every one of the five measures come from the smallest large encoder, frozen Llama-2-7b.

Mathew et al. reach the same conclusion from the other direction: “models that perform very well in classification cannot always provide plausible and faithful rationales”.

Warning

IOU F1 is a coarse statistic here and small gaps in it should not be read as real. With one rationale per post it reduces to the share of posts whose top-5 tokens overlap the gold set at IOU > 0.5 — a thresholded count, so 0.077 vs 0.081 is well inside noise. Token F1 is the more stable of the two discrete metrics; the ordering agrees but the gaps are interpretable.

Reading the AUROC. Within a post, 0.850 is the probability that a randomly chosen human-marked word outranks a randomly chosen unmarked one. Random scores give ~0.50, so there is no small-sample inflation here (unlike AUPRC, where random averages ~0.29 against a 0.165 floor — an artefact of averaging average-precision over short posts).

Why per-post and pooled differ so much. Pooling ranks words across posts, where a post’s overall severity shifts all of its word scores together: a filler word in a violently hateful post can outrank the actual slur in a mild one. Per-post cancels that offset. Per-post is the right question for an explanation shown next to a single comment — which is how the reward is used — but always say which one is being quoted.

Against the published baselines

Only the discrete metrics are protocol-matched, so they carry the comparison. (The paper does not state whether its AUPRC is pooled or per-post; its BERT values, 0.747–0.778, sit on our per-post 0.759, but that is inference rather than a matched protocol.)

model	explanation	Token F1	IOU F1
frozen Llama-2-7b + seq-χ²	exact linear attribution	0.422	0.107
frozen gemma-4-12B + seq-χ²	exact linear attribution	0.414	0.093
frozen gemma-4-31B + seq-χ²	exact linear attribution	0.411	0.103
fine-tuned BERT + seq-χ²	exact linear attribution	0.401	0.077
frozen BERT + seq-χ²	exact linear attribution	0.385	0.081
BERT	attention	0.497	0.130
BERT	LIME	0.468	0.118
BERT-HateXplain	attention	0.411	0.120
BiRNN-HateXplain	attention	0.506	0.222
random top-5	—	0.193	—

Baselines: Mathew et al. (2021), Table 5. -HateXplain models are trained with supervision on the rationales themselves.

Two of our arms rule out the obvious explanations for the remaining gap. Our fine-tuned BERT (0.401) does not close it, so it is not that our encoder is frozen; and our 31B (0.411) does not close it either, so it is not encoder capacity. What is left is the explanation form: mean pooling gives every token weight 1/L, diluting credit on long posts in a way attention does not.

The claim this supports

The frozen linear attribution lands modestly below fine-tuned BERT’s attention and LIME explanations — clearly above random, and above BERT-HateXplain’s supervised-attention Token F1, but below plain BERT. The defensible claim is therefore:

A frozen encoder with a linear head — no fine-tuning, and no supervision on rationales — comes within a few points of a fine-tuned BERT’s explanations, while being faithful by construction and costing one forward pass.

Plausibility and faithfulness are different axes. Attention and LIME are post-hoc: they estimate what the model attended to and can diverge from what it computed. The folded linear reward does not estimate anything — its per-token contributions sum to the score exactly. “Comparable plausibility, guaranteed faithfulness, one forward pass” is the honest summary; “beats BERT on explainability” is not.

Future directions

The benchmarks above close off several explanations by elimination, and what they leave standing points at three specific pieces of the method rather than at the encoder. Each of these is a direction the results actively motivate, not a generic wish-list.

1. The mean-reward formulation, explored more fully

The sequence-χ² reward is the one-step reduction of the two-distribution IQ-Learn objective — γ = 1 in a mean-reward setting, or equivalently γ = 0 over a single-step episode. That reduction is why φ is a uniform mean over token states, and uniformity is exactly what the HateXplain plausibility result implicates: every token carries weight 1/L, so credit is diluted on long posts in a way attention is not. Our fine-tuned arm ruled out “the encoder is frozen” and our 31B arm ruled out “the encoder is too small” — the pooling is what remains.

The obvious probe is learned pooling, and it is attractive because it costs nothing in faithfulness. For any input-dependent weights \(\alpha_t\) with \(\varphi = \sum_t \alpha_t h_t\),

\[r = g \cdot \varphi + b = \sum_t \alpha_t \,(g \cdot h_t) + b\]

which is still an exact additive decomposition over tokens — the guarantee the whole explainability claim rests on survives. Worth measuring: whether learned \(\alpha_t\) closes part of the Token F1 gap to attention (0.422 → 0.497) without costing classification AUROC, and whether intermediate discounting between the token-level and one-step extremes behaves better than either endpoint.

2. Re-evaluating the full two-distribution IQ-Learn formulation

The token-level discrete agent was abandoned at ~0.55 AUROC on OMP, where the reward tracked post length more than hate semantics. But that verdict was reached on a frozen dbmdz/german-gpt2 — 124M parameters — and these benchmarks now show how much of the outcome the encoder alone decides: on HateXplain, moving from a 110M frozen encoder to a 7B one is worth +0.039 with the objective held fixed.

So the honest status of the token-level formulation is confounded, not refuted: it was tested at a representation quality we now know is far below the ceiling. Re-running the full two-distribution objective — with its temporal-difference machinery intact — on frozen 7B–31B features would separate “the TD estimator dilutes the signal” from “the encoder was too weak to support per-token credit assignment”. A negative result there would be much stronger evidence for the one-step reduction than the current one; a positive one would reopen per-token rewards, which is the setting the method was originally designed for.

3. Heads more expressive than linear

The deployed head is linear, and the OMP linear-probe ceiling (~0.74) is a property of that choice as much as of the features. The in-sample/held-out gaps here are modest — 0.881 vs 0.865 for the 31B, 0.879 vs 0.857 for the 7B — which suggests capacity is not obviously the binding constraint on this corpus, but that has not been tested directly.

The real constraint is that expressiveness trades against exact attribution. A general non-linear head destroys the property that per-token contributions sum to the score, which is what distinguishes this method from post-hoc explanation. Directions that keep the guarantee are therefore the interesting ones:

• additive (GAM-style) heads over token features — non-linear per token, still exactly decomposable;

• low-rank bilinear interactions, which decompose into pairwise terms that can still be reported;

• a non-linear head paired with an explicit, measured statement of how much faithfulness is given up — the point being to quantify the trade rather than quietly take it.

Note

Directions 1 and 3 are cheap: both reuse the cached features and the existing evaluation harness, so a result is a fit and an attribution pass away. Direction 2 is expensive — it needs the token-level trainer run against large frozen encoders — but it is the one that would settle a claim the package currently makes on evidence gathered under a much weaker encoder.

Reproducing

Both live in haspi-playground/iqlearn/transformer/, run against an installed haspi:

python -u rpmod_extract.py && python -u rpmod_eval.py          # RP-Mod

python -u hatexplain_prep.py                                   # once: build the arrays
export XLA_PYTHON_CLIENT_PREALLOCATE=false
python -u hatexplain_eval.py --encoder llama2 --verify          # frozen 7B
python -u hatexplain_eval.py --encoder bert   --verify          # frozen BERT
python -u hatexplain_finetune.py --epochs 2                     # full fine-tuning ablation

The Gemma-4 arms need a second environment: haspi pins transformers<4.40 (the last line shipping Flax, which the superseded IQ-Learn encoders need), and that pin predates Gemma-2/3/4 — 4.39’s CONFIG_MAPPING knows only gemma (Gemma 1). Only the encoder forward pass moves; the fit, the attribution formula and the metrics live in hatexplain_common.py and are shared verbatim with the arms above.

# env A: modern transformers, no haspi        | env B: haspi
python -u hatexplain_gemma4.py --model gemma4-31b features     # A
python -u hatexplain_gemma4.py --model gemma4-31b fit          # B  (the only haspi step)
python -u hatexplain_gemma4.py --model gemma4-31b attribute    # A

The 31B needs an overcommit change to load at all

gemma-4-31B ships as a single 49.8 GB safetensors shard. Under Linux’s default heuristic overcommit (vm.overcommit_memory=0) the kernel refuses any mapping larger than RAM+swap, so safe_open fails with Cannot allocate memory on a 31 GB host — regardless of GPU capacity, which is ample (60.5 GB of 94 GB in use once loaded). Set vm.overcommit_memory=1 for the load and put it back afterwards. gemma-4-12B is one 23.9 GB shard and needs no such change.

Two things worth knowing before rerunning:

• XLA_PYTHON_CLIENT_PREALLOCATE=false is not optional for the HateXplain script. It fits with JAX and then scores with the torch-side LeoLMScorer in one process; JAX preallocates ~75% of the card by default, leaving nothing for the 7B model. Without it this OOMs even on a 94 GB GPU. See GPU memory notes.

• --verify earns the plausibility numbers. The rationale evaluation recomputes the attribution with token offsets, so sub-words sum back onto HateXplain’s own whitespace tokens instead of relying on to_words()’s sentencepiece merging, which does not align reliably with a third-party tokenisation. --verify asserts that recomputation reproduces LeoLMScorer.score() on real test posts before any metric is computed, so a drift between the two fails the run rather than quietly reporting numbers from different maths.