sigeval

Repo: github.com/nikolas-sapa/sigeval

What it is

An LLM evaluation framework for Python that refuses to answer a question it can't answer honestly.

from sigeval import assert_eval

def test_summarizer_stays_on_topic():
    def scorer(article):
        summary = my_llm(article)
        return "refund" in summary.lower()

    # 20 samples. Passes only if the TRUE pass-rate is
    # significantly above 0.8 at 95% confidence.
    assert_eval("on_topic", scorer, sample=ARTICLE, n_samples=20, threshold=0.8)

Run it with pytest, like anything else.

Why it exists

Every other eval tool answers "did this run pass?" LLMs are non-deterministic, so that boolean flips on noise. A prompt scores 0.82 today and 0.78 tomorrow and your green build turns red for no real reason. Then people start re-running CI until it's green, which is the moment the eval stopped meaning anything.

sigeval answers a different question: is the true pass-rate significantly above my threshold? Wilson score intervals for the verdict, a two-proportion z-test for regressions. Three outcomes instead of two — PASS, FAIL, and INCONCLUSIVE (collect more samples).

INCONCLUSIVE is the whole point. A run that lands inside the interval doesn't get rounded into a decision it hasn't earned. It tells you to sample more.

What that buys you

Regression gates fire on a statistically real drop, not on sampling jitter. Sample budgeting tells you how many runs it would take to resolve a borderline result instead of leaving you to guess. And a flaky suite stops training you to ignore it.

Status

v0.1.0, pre-PyPI — install from git. Pytest plugin, LLM-judge helper, CI examples, MIT.

pip install git+https://github.com/nikolas-sapa/sigeval.git