FixProof

Stop trusting AI fixes. Start verifying them.

FixProof 0.3.0 is an MIT-licensed agent skill and a command-line tool. The skill (SKILL.md) tells the agent to reproduce a bug and edit the code. The CLI (scripts/verify.py) runs the commands, stores the evidence, and chooses the status. Python 3.9 or newer is enough. The CLI imports the standard library and has no package to install.

The CLI validates the evidence itself. Comparable agent fix and verification skills leave that judgment in the prompt, or they stop once a reproduction exists:

• verification-before-completion (obra/superpowers, and the skill of the same name in rpikit) tells the agent to run a proving command and compare the output with the claim before saying the work is done. Its pattern notes also describe undoing a fix and expecting the test to fail. The skill does not execute that undo.
• ai-devkit verify (project page) is a prompt-level gate of the same shape, including a regression check the agent is asked to perform.
• reproduce-my-bug does not verify a fix. It produces a REPRO.md dossier. Its outcomes are DETERMINISTIC, RATE-BASED, and NOT-YET-REPRODUCED. The failing command in that dossier is a valid --repro value for FixProof.

This comparison stays inside those agent skills. It is not a survey of test runners, mutation testing, or benchmark harnesses. SWE-bench grading is a different kind of tool: FixProof borrows its pass-to-pass idea and applies it to each suite command.

Fabricated results are hard to pass off, and a hand edit is visible. Each record in .fixproof/runs.jsonl stores the hash of the previous record. validate recomputes that chain and recomputes the status from the records. An agent that can rewrite .fixproof/ can delete the log and write a new chain that validate accepts. Re-run check on a machine you trust when the result has to be independent of the agent. Details and the reporting path are in SECURITY.md.

Not affiliated with Anthropic, OpenAI, or Anysphere. Those names are trademarks of their owners.

Install

The skill is this repository. An agent needs SKILL.md and scripts/verify.py. There is no pip install.

The clones below were run against https://github.com/Livecristen/fixproof.git. Each clone printed Cloning into '<destination>...' and exited 0. Each --version command printed 0.3.0 and exited 0.

# Claude Code, personal skill folder
git clone https://github.com/Livecristen/fixproof.git ~/.claude/skills/fixproof
python3 ~/.claude/skills/fixproof/scripts/verify.py --version

# Codex
git clone https://github.com/Livecristen/fixproof.git ~/.codex/skills/fixproof
python3 ~/.codex/skills/fixproof/scripts/verify.py --version

# Cursor, from the project you are working in
git clone https://github.com/Livecristen/fixproof.git .cursor/skills/fixproof
python3 .cursor/skills/fixproof/scripts/verify.py --version

On Windows PowerShell 5.1, ~ in an argument to a native command such as git is not expanded. The bash commands above then clone into a folder literally named ~ inside the current directory. python3 ~/... still resolves that relative path, so --version prints the version and nothing looks wrong, but the agent never finds the skill. This was observed on 2026-10-26 with PowerShell 5.1.26100 and git 2.55.0.windows.3. Use $HOME in PowerShell instead:

# Codex (the same pattern applies to .claude\skills)
git clone https://github.com/Livecristen/fixproof.git "$HOME\.codex\skills\fixproof"
python "$HOME\.codex\skills\fixproof\scripts\verify.py" --version

That clone printed Cloning into 'C:\Users\<user>\.codex\skills\fixproof'..., and --version printed 0.3.0.

On Windows, and in the GitHub Actions images this repository uses, the interpreter command is python. The machine used to check this README has python3 and does not have python.

From a checkout, python3 scripts/verify.py --help lists init, check, restore, and validate. python3 scripts/verify.py --version prints 0.3.0. With no subcommand the process exits 64 and prints the help. python3 scripts/verify.py init with no --repro prints verify.py init: error: the following arguments are required: --repro and exits 64.

Verify a fix

Run the CLI in the project under test, or pass --dir. The reproduction and the suite run with the working directory set to the git toplevel.

1. init records the unfixed tree and runs the reproduction. The reproduction must fail because of the bug.
2. Edit the code. Keep the regression test on a test path (the globs are below).
3. check reruns the reproduction (3 times unless you pass --reruns), runs each suite command, then runs the revert test.
4. validate recomputes the report you are about to share.
5. If a revert test is interrupted, restore puts the backed-up files back.

python3 scripts/verify.py init --dry-run \
  --repro "python3 -m unittest test_slugify" \
  --suite "python3 -m unittest test_other"
python3 scripts/verify.py init \
  --repro "python3 -m unittest test_slugify" \
  --suite "python3 -m unittest test_other"
# replace the buggy implementation, then:
python3 scripts/verify.py check --dry-run
python3 scripts/verify.py check
python3 scripts/verify.py validate .fixproof/report.json
python3 scripts/verify.py restore

Those arguments were run against a temporary git repository built from examples/python. scripts/verify.py was invoked by its path in the FixProof checkout. The output is in Example.

The in-place revert test edits your working tree for the duration of one command. Before it does, it prints a WARNING with the file list and the backup path. check --dry-run prints the plan and does not run commands or modify files. check --revert-mode worktree runs the revert test in a temporary git worktree and leaves your tree untouched. Ignored files such as node_modules or a virtualenv are absent in that worktree.

init adds .fixproof/ to .git/info/exclude when that line is absent, so the evidence directory stays out of git status. It does not edit a committed .gitignore. This repository's own .gitignore also lists .fixproof/.

Options

--help on each subcommand lists the flags. Defaults below are the ones v0.3.0 stores or prints.

restore takes --dir. validate takes the report path.

--json must name a .json file inside .fixproof/, and the name must not be state.json or manifest.json. Passing a path outside that directory failed with FixProof: usage error - --json must be a .json file inside .fixproof (not state.json) and exit 64.

Child processes receive PYTHONDONTWRITEBYTECODE=1 plus these names when they are set and the name does not look like a secret: PATH, PATHEXT, HOME, USERPROFILE, LANG, LC_ALL, LC_CTYPE, SYSTEMROOT, SYSTEMDRIVE, WINDIR, COMSPEC, TEMP, TMP, TMPDIR, APPDATA, LOCALAPPDATA, PROGRAMDATA, PROGRAMFILES, NUMBER_OF_PROCESSORS, PROCESSOR_ARCHITECTURE. Names passed with --pass-env or --require-env are added even when they look like secrets.

Default test-path globs. Any --test-glob replaces this list:

test_*.py  *_test.py  tests/*  test/*  __tests__/*
*.test.js  *.test.ts  *.spec.js  *.spec.ts  *.test.mjs
*.test.cjs *.test.tsx *.spec.tsx

A path matches when the full relative path or the basename matches a glob, or when a dir/* glob names a parent directory.

Status rules

check and validate compute the status from the records. The latest check_start record begins the post-fix evidence. Baseline records from before that marker still count. Process exits:

validate does not use this table for its own process exit. It exits 0 when the file matches the records, and 3 when it does not.

VERIFIED (exit 0)

All of these are true:

• Required environment variables are set and required tools are on PATH.
• The baseline reproduction's outcome is fail: the process failed, and that failure was not classified as a timeout, a missing dependency, a disallowed executable, or zero tests.
• Every post-fix reproduction passes.
• No suite command that passed at baseline fails after the fix.
• The revert test's result is fails_without_fix, and the tree was restored.

The reason string for that case is:

baseline failed, fix passes all reruns, revert test fails without the fix, no regressions

FAILED (exit 1)

• Every post-fix reproduction fails. Reason: fix does not solve the bug: reproduction still fails (N/N runs).
• Or a suite command passed at baseline and fails after the fix. Reason: regression: suite command(s) passed before and fail after the fix: ....

The revert test is not started when the status is already FAILED.

PARTIALLY VERIFIED (exit 2)

Every post-fix reproduction passed, and there is no pass-to-fail suite regression, and at least one of these reasons is present:

• no regression suite given
• suite not fully passing/runnable after fix: ... (any suite outcome other than pass, including a timeout or a missing tool on that suite command)
• revert test not run (--no-revert, or no revert result was recorded)
• revert test not possible: ... (no non-test change, or the fix and the test share a file)
• revert test inconclusive: ... (without the fix, the reproduction timed out, was interrupted, or was classified as something besides pass or fail)

A weak suite result stays PARTIALLY VERIFIED. Timeouts and missing tools become BLOCKED when they hit the reproduction command or the environment checks, not when they hit a suite command.

UNVERIFIED (exit 3)

• Baseline outcome zero_tests: baseline reproduction ran zero tests
• Baseline outcome pass: bug not reproduced: baseline reproduction passed before the fix
• No post-fix run yet: no post-fix run recorded
• A post-fix run has outcome zero_tests: post-fix reproduction ran zero tests
• Mixed post-fix results: flaky: reproduction passed X/Y runs after the fix
• Revert result passes_without_fix: revert test: reproduction also passes WITHOUT the fix, so it does not prove the fix

check also exits 3, before it runs commands, when the hash chain in runs.jsonl is broken. validate exits 3 and prints integrity mismatch when the chain, chain_head, the status, or exit_code does not follow from the records. A copy of the sample report below, with status changed to FAILED and exit_code left at 0, produced:

FixProof validate: UNVERIFIED - integrity mismatch
  - status 'FAILED' does not follow from evidence (recomputed 'VERIFIED')

That process exited 3. Exit 0 is the exit for the recomputed status, so the exit field was not a second problem.

BLOCKED (exit 4, or exit 5 when the tree does not restore)

• The directory is not inside a git repository. init prints FixProof: BLOCKED - <dir> is not a git repository (required in v1) and exits 4. <dir> here stands in for the path that was passed.
• check finds no state.json. It exits 4 and tells you to run init.
• The baseline snapshot cannot be written.
• An environment check failed. Observed: environment: env var FIXPROOF_DOCS_MISSING_VAR missing and environment: tool fixproof-missing-tool-xyz missing. Both were status BLOCKED, exit 4, and the reproduction command was not started.
• Baseline reproduction outcome is timeout, missing_dependency, or blocked_exe. Reason: baseline reproduction: <outcome>. Observed with --no-shell --allow-exe node on a python3 command: outcome blocked_exe, process exit 126, status BLOCKED, exit 4, reason baseline reproduction: blocked_exe.
• A post-fix reproduction outcome is one of those three. Reason: post-fix reproduction: <outcomes>.
• A guard failed. A state.json that no longer matches the hash recorded at init, and a tree with unmerged paths, are guards. Nothing runs after a bad state hash.
• The revert test refuses (unsafe): unmerged paths, a symlink or a path outside the repository, a backup that cannot be written, or a backup left by an interrupted run. Reason: revert test refused: ....
• The revert test cannot put the tree back (restore_error). The status string remains BLOCKED. The process exits 5, the backup is kept, and restore puts the files back.

restore with no backup prints FixProof restore: no backup found; nothing to do and exits 0. A byte-for-byte restore exits 0 and removes that backup. A restore that still differs exits 5 and keeps the backup. An unreadable manifest exits 4. restore outside a git repository exits 4.

Two results that are easy to misread

A successful init exits 0 and prints Bug reproduced. The report file at that moment is not a fix verdict. There is no post-fix run, so the file's status is UNVERIFIED, its exit_code field is 3, and the reason is no post-fix run recorded. validate on that file prints FixProof validate: report consistent; status UNVERIFIED and exits 0. check replaces the file.

validate exits 0 when a report is internally consistent. That includes a consistent PARTIALLY VERIFIED report and a consistent UNVERIFIED report. Consistency is not a VERIFIED fix.

New records are appended to runs.jsonl. A later check prints earlier run lines together with the new ones. The status uses the baseline plus the records after the latest check_start. In this session, check --no-revert after a VERIFIED run exited 2 with reason revert test not run, and the transcript still showed the older revert line. validate then printed FixProof validate: report consistent; status PARTIALLY VERIFIED and exited 0.

Example

Each demo copies an example into a temporary git repository, runs init, applies the fixed file, runs check, and runs validate. In this documentation pass each command exited 0 (status VERIFIED, and validate accepted the report):

python3 examples/run_example.py python
python3 examples/run_example.py js-ts
python3 examples/run_example.py ts

The TypeScript example sets "type": "module" in package.json and uses Node's built-in type stripping. No npm install is required. Tested with Node 22.18.0+ / 23.6.0+ / 24.x (unflagged type stripping); CI uses 24.22.1. On 2026-10-25 python3 examples/run_example.py ts was run with official Node binaries: 22.17.1 FAILED (exit 1), 22.18.0 VERIFIED (exit 0), 23.5.0 FAILED (exit 1), 23.6.0 VERIFIED (exit 0), 24.22.1 VERIFIED (exit 0). On versions without unflagged type stripping (also Node v20.19.2), node --test price.test.ts cannot load the file before or after the fix, so the FAILED status reflects the missing Node feature, not the example.

The report below is from a separate run of the Python example with v0.3.0 (2026-10-25). tests/test_meta.py runs validate on it. run_example.py quotes the absolute interpreter path; this run used the command python3 so the stored command line has no machine path. Setup on this machine: git init (the branch name was master), commit slugify.py, test_slugify.py, and test_other.py, run init, replace slugify.py with slugify_fixed.py, run check. The git author on that repository was demo / demo@example.invalid.

init --dry-run exited 0 and did not write state.json:

FixProof DRY RUN (nothing executed, nothing written):
  repro: python3 -m unittest test_slugify
  suite: python3 -m unittest test_other
WARNING: current branch is 'master' (protected-looking branch)

init exited 0:

FixProof run [baseline_repro]: python3 -m unittest test_slugify
FixProof run [baseline_suite:0]: python3 -m unittest test_other
FixProof baseline: reproduction outcome = fail (exit 1)
Bug reproduced. Apply the fix, then run: verify.py check

check --dry-run after the fix exited 0 and did not modify the tree:

FixProof DRY RUN (nothing executed, nothing modified):
  would run repro 3x: python3 -m unittest test_slugify
  would run suite: python3 -m unittest test_other
  revert test would temporarily modify: slugify.py
WARNING: current branch is 'master' (protected-looking branch)

check exited 0. On the report: line, the temporary directory is written <repo>. That is the substitution in the transcript. The JSON that follows is the file the CLI wrote. A newline sits between the last } and the closing fence so the fence is valid; the JSON text itself is unchanged. [REDACTED] in the traceback is the CLI's redaction, not an edit made for this README. Stored output has the values of environment variables removed when the name contains KEY, TOKEN, SECRET, PASS, PWD, CREDENTIAL, AUTH, PRIVATE, DSN, COOKIE, or SESSION. The shell variable PWD matches, and on this run its value was the repository directory.

FixProof run [after_repro]: python3 -m unittest test_slugify
FixProof run [after_repro]: python3 -m unittest test_slugify
FixProof run [after_repro]: python3 -m unittest test_slugify
FixProof run [after_suite:0]: python3 -m unittest test_other
WARNING: FixProof revert test will TEMPORARILY modify 1 file(s) in your working tree: slugify.py
         Backup: .fixproof/backup-20261025-222852-768727. If this run is interrupted, run: verify.py restore
WARNING: current branch is 'master' (protected-looking branch)
FixProof run [revert_repro]: python3 -m unittest test_slugify
============================================================
FixProof 0.3.0  status: VERIFIED  (exit 0)
  - baseline failed, fix passes all reruns, revert test fails without the fix, no regressions