variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores

State-of-the-art variant-effect models are accurate in cross-validation but their scores are poorly calibrated on temporal data. variant-confidence adds an auditable calibration layer on top of existing predictors — it does not train a new model.

The problem: accuracy is not trust

Protein variant-effect predictors (AlphaMissense, ESM-1v, EVE) report pathogenicity scores, but a clinician or researcher needs to know how much to trust the number, not just its rank. The gap is calibration, not accuracy:

AnnotateMissense (2026) reports MCC 0.94 in cross-validation, dropping to 0.76 on temporal ClinVar, accuracy 0.8798.
A raw score near 0.9 may not mean 90% probability. Acting on an uncalibrated score is a risk.

What it does

variant-confidence wraps an existing predictor’s score and produces a calibrated, uncertainty-aware output:

Probability calibration (AC1): Platt scaling or isotonic regression over a separate holdout. Selectable, not hardcoded.
Conformal prediction (AC1b): coverage 1−α intervals, split or Mondrian by gene.
ECE (AC2, AC9): Expected Calibration Error reported before/after calibration, with bootstrap CI and per-bin counts. Bins with too few samples are flagged as low-reliability.
Leakage-free split (AC3): temporal split by ClinVar release date with gene isolation — the same gene never appears in both train and test. This is unit-tested.
Missing-score handling (AC4): works with AlphaMissense or ESM-1v alone; emits an explicit warning instead of failing silently.
Non-deceptive reporting (AC7): every result includes interval/ECE + method + threshold, never a bare calibrated score.

Verification (clean clone, no network)

Built under a three-party governance loop: implement → independent audit in a clean clone → merge approval.

ruff check . → All checks passed.
pytest tests/ → 28 passed in 8.90s (offline fixture).

An honest bug we caught in audit

The first ECE test reported a “perfect” drop from raw ECE 0.4275 to ~0 after calibration. The audit found this was degenerate: the synthetic score generator ignored the real labels and produced an independent random true_p, so the calibrator simply collapsed to the base rate (91.5% pathogenic) — ECE≈0 by construction, not by merit. AUC of the raw score vs real labels was 0.51.

The fix derives true_p from the real label with noise, so the synthetic score is genuinely discriminative but miscalibrated. The acceptance criterion is now: ECE drops and AUC is preserved after calibration. This is in the committed code and the 28 passing tests.

What remains open (honestly)

AlphaMissense license ambiguity: the official README says CC BY 4.0, but the distributed TSV header, Ensembl VEP plugin, and EBI page state CC BY-NC-SA 4.0. The contradiction is unresolved; treat the data as restricted (non-commercial) until clarified. The software is AGPL-3.0-or-later and fully self-contained.
The end-to-end join with real AlphaMissense scores is implemented, but the flagship path is covered by an offline fixture, not a live download in CI.

Try it

pip install variant-confidence variant-confidence --method platt --offline

Stack

Python >=3.10, numpy, pandas, scikit-learn
AGPL-3.0-or-later