Portage - Autonomous Code-Migration Agent
0
Humans in the Loop
K=3 × 6
Eval Grid · Pinned Repos
100%
Fault Recovery (Fixture)
2
Interfaces · CLI + MCP
Give it a Flask repo and a recipe; it plans a per-file task DAG, rewrites each file with an LLM on a git worktree, verifies against the repo's own tests in a network-off Docker sandbox, recovers from failures under bounded budgets, and reports honestly — including when it fails.
System Overview
One core engine, two interfaces: the CLI drives fully autonomous migrations, and an MCP server hands the same verified primitives to co-pilot agents like Claude Code and Cursor. The dashboard is the proof surface — live task trees, diffs, recovery timelines, and the eval leaderboard. Below is the core durability claim, live: the worker is killed mid-migration, and a restarted worker resumes from the Postgres checkpoint instead of starting over.
reproduce: bash scripts/demo_kill_resume.sh · stricter: scripts/dod_check.sh
Why I Built It
Most "AI migration" demos are single-shot prompts with no verification story. Portage is built around the opposite claim: a migration is only real if the repo's own tests still pass, every planned file was actually migrated, and recovery cannot game the score by giving up. The governing principle is narrow + measured beats broad + unproven. One hard migration (Flask → FastAPI) instead of a catalogue of half-working recipes. An eval harness that runs the real queue/worker path, not a mocked agent loop. A failure taxonomy with SOLVED / PARTIAL / OPEN statuses and evidence, not an all-green sheet. The autonomous + eval core is the credibility engine for the MCP product: if the verify/recover loop is measured, another agent can trust verify_patch_in_sandbox over a raw sandbox.
How It Works
v1 ships one recipe: Flask → FastAPI. That target is deliberate — routing decorators, request/response handling, blueprints → routers, error handlers, and app factories need understanding, not mechanical rewriting; deterministic codemods cannot do this reliably. The architecture is recipe-pluggable; the evidence is recipe-specific by design. One core engine, two interfaces. Autonomous mode: `portage migrate <repo> --watch` drives the full graph. Co-pilot mode: Claude Code / Cursor call verify_patch_in_sandbox, repo_graph, and blast_radius over MCP — the same verified primitives the eval numbers were measured on. The dashboard is the observability and proof surface, not the front door.
A submitted job runs this LangGraph graph, checkpointing to Postgres after every node — kill the worker mid-run and a restarted worker resumes from the last completed node. When Verify fails, Recover classifies the failure and routes back: targeted rollback + regenerate to Execute, replan to Plan for planner misses, or give up to Integrate once budgets are exhausted — reporting an honest red rather than a gamed green.
CLI — Autonomous Mode
The portage console script is a thin httpx client over the REST API — it never touches the DB or queue directly, the same boundary the dashboard respects. migrate --watch streams live task transitions; status, jobs, and report --diff cover inspection. Exit codes are the eval bar: 0 means honestly green, 1 means finished but not complete-and-green, 2 means usage or infra.
MCP — Co-pilot Mode
The MCP server exposes the verified core so another AI agent can test its own work before writing to the caller's tree: verify_patch_in_sandbox copies the repo, applies a unified diff, runs the tests network-off, and returns structured pass/fail with failing test names — never mutating the caller's files. repo_graph and blast_radius give it structural awareness. Here is what a blast-radius query actually computes:
The blast_radius primitive in action: when db.py changes, Portage walks the structural code graph outward — direct callers first (hop 1), then their dependents (hop 2) — and selects only the tests that cover the impacted set. Verify uses this to iterate fast; the final honesty bar still runs the full suite. The same query is exposed to co-pilot agents as the blast_radius MCP tool.
Dashboard as Proof
Next.js App Router, REST only — the frontend never owns schema. Jobs list with launch form, job detail with live pipeline route, per-file diffs, and attempt tier/model timelines, and a public /eval leaderboard rendering per repo×scenario green rates, mean±variance, cost, and recovery straight from the runs/metrics tables.
Key Features
Checkpoint Durability
LangGraph Postgres checkpointer after every node; worker lease with heartbeat. Kill the worker mid-run and a restarted one resumes — Ingest runs once, Execute skips already-applied files via content hashes.
Bounded Recovery
Failures are classified, blamed, and rolled back surgically: targeted git checkout + regenerate, widen-on-repeat, replan on planner misses, skip-and-continue as last resort. Budgets bound everything.
Honest Scoring
Green cannot be gamed by skip-and-continue, empty diffs, or all-skipped suites. Report reloads task truth from Postgres, Integrate recomputes the diff, Verify requires passed > 0.
Measured Escalation
First N attempts use the driver model tier; later attempts escalate. Every attempt lands in attempts_log with tier, model, tokens, and USD — "how often does escalation rescue?" is a SQL query.
Pluggable Recipes
A recipe declares detection, task types, and per-task verify specs. Unknown recipes degrade safely to ingest → verify → report: tests run, nothing changed, verdict honest red.
Cost as a Metric
Every LLM call's tokens and USD recorded per attempt, summed per job, averaged per eval cell — retries and escalations included. Cost scales with recovery, and that relationship is part of the result.
Eval Headline
Suite k3-baseline: every repo×scenario cell runs K=3 times through the real queue/worker path, and green requires the full suite passing, every task done, and zero skips. Straight from the runs/metrics tables:
| Repo | Tier | Green | Test-pass | Recover | Cost | Wall |
|---|---|---|---|---|---|---|
| flask-items-fixture | baseline | 3/3 | 1.00 | 0.0 | $0.022 | 10s |
| minimal-flask-api | baseline | 2/3 | 0.67 | 0.3 | $0.013 | 10s |
| flask-restx-api | framework | 1/3 | 0.67 | 3.3 | $0.044 | 17s |
| flaskr | structural | 0/3 | 0.67 | 3.7 | $0.250 | 55s |
| watchlist | structural | 0/3 | 0.67 | 4.0 | $0.261 | 61s |
| microblog | heavy | 0/3 | 0.00 | 4.3 | $1.503 | 165s |
Finding
The reliability boundary is idiom, not size. JSON APIs migrate green at ~$0.01–0.02 with zero recovery; server-rendered apps complete their task DAGs but fail behaviorally — the named frontier is cross-file call-shape drift. Fault injection on the stable tier (bad_patch, bad_patch_until_escalation) recovers 100% green on the fixture, 3/3 each.
Technical Stack
Friction & Takeaways
Friction
- A single shared sandbox image cannot serve mutually incompatible dependency pins — four corpus candidates dropped for that reason; the unlock is per-repo sandbox images
- Skip-and-continue produced false greens (original suite passes after full rollback) — fixed by reloading task truth, recomputing diffs, and requiring full completion
- Models can "pass" by decorating every test with skip — Verify now requires passed > 0
- Export contracts pin names, not call shapes — get_db() drifting between plain function / needs-request / context manager across files is the dominant residual failure
- Flask-coupled extensions (flask_sqlalchemy, flask_restx) need per-extension sub-strategies; partially cracked, not reliable
- LLM nondeterminism means single runs are anecdotes — K-run mean±variance is mandatory, and organic flake is a finding, not noise to hide
Takeaways
- The hard thing (autonomous migrate + eval) validates the easy thing (MCP verify tool)
- Honesty bars must be structural, not aspirational — every false-green class found in the wild became a hard predicate
- Recovery is a product feature only if it is measured: fault scenarios, attempts_log, cost deltas
- Recipe rules encode observed failures cheaply; structural gaps need architecture, not more prompt text
- Cost that includes retries is the only honest cost; cheap first-pass numbers lie
- Compose + network-off sandboxes make multi-service agent systems operable without cloud lock-in
portage migrate --watch · exit 0 = honest green
verify_patch_in_sandbox · repo_graph · blast_radius
jobs · recovery timelines · public /eval
Durability, Recovery, Eval Methodology & the Failure Taxonomy
Every moving part explained: the animated architecture, graph-node lifecycle, checkpoint and lease mechanics, sandbox anti-gaming predicates, the seven recovery strategies, K-run eval methodology with non-claims, and the nine-category failure taxonomy with evidence.