Why: evaluating agents usually drags in a heavy framework. This is one file, zero deps, stdlib only — drop it into a repo, a CI job, or an agent loop and get numbers immediately.

Scorers

Reference-based (prediction, reference):

name	what it measures
`exact`	raw string equality
`normalized_exact`	equality after case/unicode/punctuation/whitespace folding
`token_f1`	bag-of-tokens F1 (order-insensitive)
`rouge_l`	LCS-based F1 (order-sensitive)
`contains`	normalized reference substring of prediction

Reference-free heuristics (prediction only):

name	what it flags
`nonempty`	blank output
`no_refusal`	refusal / hedging openers ("I cannot", "as an AI", ...)
`no_placeholder`	unfilled template markers ("TODO", "lorem ipsum", ...)

Reference-free scorers are heuristics, not ground truth — use them as guards, not grades.

Install

No install needed. Or:

pip install -e .

CLI

Input is JSONL, one record per line:

{"id": "q1", "prediction": "Paris is the capital of France.", "reference": "Paris"}

Run:

python eval_harness.py examples/sample.jsonl
python eval_harness.py examples/sample.jsonl --scorers normalized_exact,contains
python eval_harness.py examples/sample.jsonl --json

Output:

records: 3
  normalized_exact: 0.3333
         token_f1: 0.3611
          rouge_l: 0.3611
             mean: 0.3519

Library

import eval_harness as eh

scores = eh.score_pair("The answer is 42.", "42", ["contains", "token_f1"])
# {'contains': 1.0, 'token_f1': 0.5, 'mean': 0.75}

report = eh.evaluate(
    [{"prediction": "a b", "reference": "a b"},
     {"prediction": "a",   "reference": "a b"}],
    ["normalized_exact", "rouge_l"],
)
print(report["means"])
# {'normalized_exact': 0.5, 'rouge_l': 0.75}

evaluate() returns {"count", "means", "rows"} where rows carries per-record detail for diffing regressions.

Tests

python -m unittest discover -s tests -v

Design notes

Folding before comparison. normalize() NFKC-folds, lowercases, strips punctuation and collapses whitespace. Most real "wrong" answers are formatting, not content.
Order matters sometimes. token_f1 ignores order; rouge_l does not. Report both — a high F1 with low ROUGE-L means the right words in the wrong structure.
mean is naive. It's the unweighted mean of whatever scorers you asked for. Don't compare means across different scorer sets.

License

MIT.