latch

Run Latch on every pull request.

Latch posts the receipt as a comment. The same engine that powers this site ships as a command line tool and a composite GitHub Action. Both are in this repository. Nothing here runs on your code until you add it to your workflow.

1. Command line

Prove a pull request, then render the receipt as a comment. The comment command is a dry run unless you pass --post and set GITHUB_TOKEN.

Shell

export NEBIUS_API_KEY=...     # read from the environment only, never a flag
latch prove --repo . --base BASE_SHA --head HEAD_SHA --issue-file issue.md --receipt receipt.json
latch comment receipt.json      # dry run: prints the comment, posts nothing
latch comment receipt.json --post --repo OWNER/REPO --pr N   # needs GITHUB_TOKEN

2. GitHub Action

The composite Action lives in .github/actions/latch. The workflow below is the example we ship (latch/loop/github/latch.yml), minus its comments. It needs a repository secret named NEBIUS_API_KEY. Pull requests from forks get no secrets, so Latch reports an error there and fails closed.

.github/workflows/latch.yml

name: latch
on:
  pull_request:
    types: [opened, synchronize, reopened, edited]

permissions:
  contents: read
  pull-requests: write   # post or update the single Latch comment

concurrency:
  group: latch-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  latch:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # Latch needs the base and head commits
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install the project's own test dependencies
        run: python -m pip install -e . pytest
      - uses: aarav1656/latch/.github/actions/latch@main   # or ./.github/actions/latch inside the Latch repo
        with:
          mode: prove                      # "loop" lets Nemotron repair a shut gate and re-judge
          test-command: python -m pytest -q
          nebius-api-key: ${{ secrets.NEBIUS_API_KEY }}
          # post-comment: "false"          # only write the receipt and the job summary
          # fail-on-shut: "false"          # report, but do not fail the check

Inputs: mode (prove or loop), test-command, base, head, issue, max-iterations, nebius-api-key, github-token, post-comment, fail-on-shut, python-version. Outputs: verdict and receipt. See .github/actions/latch/action.yml.

3. What the comment says

Verdict
PROVEN or NOT PROVEN, with the reason when it is not proven.
Evidence
The proof test name, what happened on base and head, the revert check, and the triage verdict.
Models
A collapsed table of the model calls the receipt recorded, and the generated test source.
Update
One comment per pull request, updated in place on each push.
See it on GitHub
aarav1656/latch-demo: three python-tabulate pull requests carrying their Latch comments, a validated fix, a wrong patch kept shut and a fix whose test misses mutants. Each base and head commit matches its recorded receipt.

How Latch compares

0 of 9 wrong patches accepted.

Otter and AssertFlip report only fail-to-pass. Latch also measures the other side: nine plausible-but-wrong patches, none of which opened the gate, 6 of 9 failed by a valid proof test. Cost per proven fix: $0.22 (all 42 runs, $2.22, over 10 proven real fixes, so it carries the spend on wrong patches too). Slopgate's published $1.16 over its 14 proven rewrites is $0.08.

Latch compared with six alternatives across eight measures
MeasureLatchCodeRabbitqodo-cover / Qodo MergeAegisPRSlopgateOtter / AssertFlip
What it provesA pull request's fix changes the behaviour its issue describes, by executionNot stated (AI review comments; the Team plan generates unit tests)Generated tests pass and raise coverageThe outcome of one proposed test on the base and PR commitsA slop rewrite keeps behaviour, via characterization testsA generated test fails before the gold fix and passes after
Test anchored to the issueYes, the linked issue is the oracleNot statedNot stated (coverage-driven)Not stated (risk hypotheses from the code change)No (characterizes the original code)Yes (Otter: tests from issues; AssertFlip: bug reproduction)
Red on base, green on fix, executedYesNot statedNot stated (new tests must pass; coverage must rise)Yes, in Docker sandboxes; evidence from one seeded PRTests pass on original and rewrite, in Token Factory SandboxesYes, fail-to-pass
Revert checkYesNot statedNot statedNot statedNot statedNot stated
Mutation checkYes, 28 of 31 mutants killedNot statedNot statedNot stated in the evidence runYes, up to 8 boundary mutants per functionNot stated
Wrong-patch acceptance measured0 of 9Not statedNot statedNot statedNot stated (0 of 14 proven rewrites broke a hidden test)Not stated (fail-to-pass only)
Cost per runMedian $0.018; $0.22 per proven fixNot statedNot stated (credits at $0.012 each)$0.00145 for one run (app estimate)$1.16 for 15 instances; $0.08 per proven rewriteNot stated
PriceYour own model credit, no seat fee$24 per user a month annual (Essentials); Team $48 annual$30 a month Pro Team plan (qodo-cover is AGPL-3.0)MIT, freeNot statedPapers

Otter (31.4%) and AssertFlip (43.6%) are reported on a different benchmark with a looser pass condition, so they are context and not a head-to-head score. Latch's figures come from the recorded 42-run evaluation. Competitor cells are read from the linked pages; "Not stated" means that page does not say.

Scope