{"id":3676,"date":"2026-07-18T03:50:29","date_gmt":"2026-07-18T03:50:29","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/07\/18\/variant-confidence-v0-1-0-a-calibrated-confidence-layer-for-variant-effect-pathogenicity-scores\/"},"modified":"2026-07-18T03:50:29","modified_gmt":"2026-07-18T03:50:29","slug":"variant-confidence-v0-1-0-a-calibrated-confidence-layer-for-variant-effect-pathogenicity-scores","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/07\/18\/variant-confidence-v0-1-0-a-calibrated-confidence-layer-for-variant-effect-pathogenicity-scores\/","title":{"rendered":"variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores"},"content":{"rendered":"<div>\n<div><\/header>\n<div data-article-id=\"4170750\" id=\"article-body\">\n<h2> <a name=\"variantconfidence-v010-a-calibrated-confidence-layer-for-varianteffect-pathogenicity-scores\" href=\"#variantconfidence-v010-a-calibrated-confidence-layer-for-varianteffect-pathogenicity-scores\"> <\/a> variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores <\/h2>\n<blockquote>\n<p>State-of-the-art variant-effect models are accurate in cross-validation but their scores are poorly calibrated on temporal data. variant-confidence adds an auditable calibration layer on top of existing predictors \u2014 it does not train a new model.<\/p>\n<\/blockquote>\n<h2> <a name=\"the-problem-accuracy-is-not-trust\" href=\"#the-problem-accuracy-is-not-trust\"> <\/a> The problem: accuracy is not trust <\/h2>\n<p>Protein variant-effect predictors (AlphaMissense, ESM-1v, EVE) report pathogenicity scores, but a clinician or researcher needs to know <em>how much to trust the number<\/em>, not just its rank. The gap is calibration, not accuracy:<\/p>\n<ul>\n<li>AnnotateMissense (2026) reports MCC 0.94 in cross-validation, dropping to 0.76 on temporal ClinVar, accuracy 0.8798.<\/li>\n<li>A raw score near 0.9 may not mean 90% probability. Acting on an uncalibrated score is a risk.<\/li>\n<\/ul>\n<h2> <a name=\"what-it-does\" href=\"#what-it-does\"> <\/a> What it does <\/h2>\n<p>variant-confidence wraps an existing predictor&#8217;s score and produces a calibrated, uncertainty-aware output:<\/p>\n<ul>\n<li> <strong>Probability calibration (AC1):<\/strong> Platt scaling or isotonic regression over a separate holdout. Selectable, not hardcoded.<\/li>\n<li> <strong>Conformal prediction (AC1b):<\/strong> coverage <code>1\u2212\u03b1<\/code> intervals, split or Mondrian by gene.<\/li>\n<li> <strong>ECE (AC2, AC9):<\/strong> Expected Calibration Error reported before\/after calibration, with bootstrap CI and per-bin counts. Bins with too few samples are flagged as low-reliability.<\/li>\n<li> <strong>Leakage-free split (AC3):<\/strong> temporal split by ClinVar release date with gene isolation \u2014 the same gene never appears in both train and test. This is unit-tested.<\/li>\n<li> <strong>Missing-score handling (AC4):<\/strong> works with AlphaMissense or ESM-1v alone; emits an explicit warning instead of failing silently.<\/li>\n<li> <strong>Non-deceptive reporting (AC7):<\/strong> every result includes interval\/ECE + method + threshold, never a bare calibrated score.<\/li>\n<\/ul>\n<h2> <a name=\"verification-clean-clone-no-network\" href=\"#verification-clean-clone-no-network\"> <\/a> Verification (clean clone, no network) <\/h2>\n<p>Built under a three-party governance loop: implement \u2192 independent audit in a clean clone \u2192 merge approval.<\/p>\n<ul>\n<li> <code>ruff check .<\/code> \u2192 All checks passed.<\/li>\n<li> <code>pytest tests\/<\/code> \u2192 28 passed in 8.90s (offline fixture).<\/li>\n<\/ul>\n<h2> <a name=\"an-honest-bug-we-caught-in-audit\" href=\"#an-honest-bug-we-caught-in-audit\"> <\/a> An honest bug we caught in audit <\/h2>\n<p>The first ECE test reported a &#8220;perfect&#8221; drop from raw ECE 0.4275 to ~0 after calibration. The audit found this was <strong>degenerate<\/strong>: the synthetic score generator ignored the real labels and produced an independent random <code>true_p<\/code>, so the calibrator simply collapsed to the base rate (91.5% pathogenic) \u2014 ECE\u22480 by construction, not by merit. AUC of the raw score vs real labels was 0.51.<\/p>\n<p>The fix derives <code>true_p<\/code> from the real label with noise, so the synthetic score is genuinely discriminative but miscalibrated. The acceptance criterion is now: ECE drops <strong>and<\/strong> AUC is preserved after calibration. This is in the committed code and the 28 passing tests.<\/p>\n<h2> <a name=\"what-remains-open-honestly\" href=\"#what-remains-open-honestly\"> <\/a> What remains open (honestly) <\/h2>\n<ul>\n<li> <strong>AlphaMissense license ambiguity:<\/strong> the official README says CC BY 4.0, but the distributed TSV header, Ensembl VEP plugin, and EBI page state CC BY-NC-SA 4.0. The contradiction is unresolved; treat the data as restricted (non-commercial) until clarified. The software is AGPL-3.0-or-later and fully self-contained.<\/li>\n<li>The end-to-end join with real AlphaMissense scores is implemented, but the flagship path is covered by an offline fixture, not a live download in CI.<\/li>\n<\/ul>\n<h2> <a name=\"try-it\" href=\"#try-it\"> <\/a> Try it <\/h2>\n<div>\n<pre><code>pip <span>install <\/span>variant-confidence variant-confidence <span>--method<\/span> platt <span>--offline<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h2> <a name=\"stack\" href=\"#stack\"> <\/a> Stack <\/h2>\n<ul>\n<li>Python &gt;=3.10, numpy, pandas, scikit-learn<\/li>\n<li>AGPL-3.0-or-later<\/li>\n<\/ul>\n<h2> <a name=\"links\" href=\"#links\"> <\/a> Links <\/h2>\n<ul>\n<li> <strong>Repo:<\/strong> <a href=\"https:\/\/github.com\/amurlaniakea\/variant-confidence\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/amurlaniakea\/variant-confidence<\/a> <\/li>\n<li> <strong>Release v0.1.0:<\/strong> <a href=\"https:\/\/github.com\/amurlaniakea\/variant-confidence\/releases\/tag\/v0.1.0\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/amurlaniakea\/variant-confidence\/releases\/tag\/v0.1.0<\/a> <\/li>\n<\/ul>\n<hr>\n<p><em>License: AGPL-3.0-or-later \u2014 Pedro Sordo Mart\u00ednez (<a href=\"mailto:amurlaniakea@gmail.com\">amurlaniakea@gmail.com<\/a>), 2026.<\/em><\/p>\n<\/p><\/div>\n<\/article>\n<p> <!-- Bottom content skipped via SKIP_BOTTOM_CONTENT config --> <\/div>\n<p> <\/main> <\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>Fuente: <a href=\"https:\/\/dev.to\/magopredator\/variant-confidence-v010-a-calibrated-confidence-layer-for-variant-effect-pathogenicity-scores-30e2\">Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores State-of-the-art variant-effect models are accurate in cross-validation but their scores are poorly calibrated on temporal data. variant-confidence adds an auditable calibration layer on top of existing predictors \u2014 it does not train a new model. The problem: accuracy is not trust Protein variant-effect predictors (AlphaMissense, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3675,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[41],"tags":[],"class_list":["post-3676","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3676","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=3676"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3676\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/3675"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=3676"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=3676"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=3676"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}