On March 7th, Andrej Karpathy open-sourced a 630-line Python script called Autoresearch. He pointed it at his own already-heavily-optimized nanochat repo, went to bed, and woke up to 700 experiments having run overnight. Twenty of them were genuine improvements. The wall-clock to reach baseline GPT-2 quality dropped 11%. The agent had found a missing scalar multiplier in QK-Norm that careful human researchers had been walking past for months.

He wasn’t doing anything a human ML engineer couldn’t do in principle. He was doing it without sleeping.

The idea is structural, not flashy. A human researcher can get through maybe eight optimization cycles in a working day — form hypothesis, edit code, launch training run, interpret metric, commit or revert. It’s a loop gated by biology: context-switching, fatigue, idle time waiting for the GPU queue. Remove the human, leave the loop, and the cycle time collapses to whatever wall-clock the scalar metric actually costs to compute.

By April 18th, it was in our stack. Port 8800. Service name mkultra-autoresearch. Runs on 10.0.0.43 next to the flywheel, the roundtable, the two rust brains. This post is a plain-language account of what it is, how it’s fenced, and what happened the first time we pointed it at a problem.

Karpathy’s autoresearch works because of three constraints, not in spite of them. Take any one away and the loop diverges. We encoded all three as hard rules in the service.

These three together are what the industry started calling the Karpathy Triplet. They sound restrictive because they are. That’s the whole point. The agent is powerful because its field of motion is narrow — and every edit the loop makes is a clean, reviewable diff against a single file.

We added a fourth, softer rule on top: the simplicity criterion. If a proposed diff grows agent.py by more than 50 lines and the resulting score delta is less than 0.01, the change is rejected even if it technically improved the score. A 5-line win beats a 100-line wobble. The agent is not allowed to buy progress with sprawl.

Every autoresearch target is a directory with three files. Nothing more.

autoresearch_projects/<name>/
├── program.md      # human directive: goal, hints, constraints, context
├── agent.py        # the single editable surface
└── tasks/
    └── run.sh      # chmod +x; prints {"score": <float>} as its last stdout line

program.md is the contract between human and meta-agent. It describes what we’re optimizing for, what the scoring rubric actually means, which levers exist, and which are off-limits. It is not a prompt. It’s a briefing document the meta-agent reads every iteration, along with the current agent.py and a short history of what was tried and what happened.

agent.py is the only thing the meta-agent is allowed to touch. It can be anything that Python can load: a pure-math scorer, a system prompt, a config dict, a function that builds a tool spec. If your optimization target fits in one Python file, autoresearch can chew on it.

tasks/run.sh is the benchmark. It runs agent.py against whatever rubric makes sense — could be a local math check, could be hitting an LLM judge, could be querying a live service. All it has to do is print one JSON line at the end with a score. The loop parses the last stdout line, compares to the current best, commits via git or reverts via git checkout --.

We shipped three starter projects. example is a synthetic polynomial fit — a smoke test. roundtable-specialist tunes the DevOps specialist’s system prompt using an LLM judge over a fixed question set. rag-retrieval tunes top_k, max_cycles, quality_threshold, and rerank against a held-out query set hitting our Multi-cycle-RAG service. Three completely different targets, same harness. That’s the shape of the thing.

Here’s a thing that sounds like superstition but keeps getting replicated empirically: when the meta-agent and the task-agent share the same model lineage, the loop converges faster and ends at a higher score than when they don’t.

Claude optimizing a Claude-driven prompt outperforms Claude optimizing a GPT-driven prompt. Ollama optimizing an Ollama system message outperforms a Claude-meta doing the same job. The effect is real enough that the AutoAgent folks named it.

Nobody has a mechanistic explanation yet. The working theory is that same-family models share enough of their latent geometry that the meta-agent’s mental model of “what this response will look like” is actually predictive. Cross-family, you’re guessing through a translation layer.

We wired it in. The default meta-agent is Claude Sonnet 4.6 (we’re optimizing mostly Claude-driven agents). Pass provider: "ollama" in the run request and it flips to local Qwen3 — free, same-lineage for Ollama-driven targets, appropriate for overnight swarms where API cost would pile up. The rule, written down in SOUL.md so the operator doesn’t have to think about it every time: meta and task should share lineage when lineage matters.

This is the part that nobody talks about when they talk about “autonomous agents.” You can have the smartest loop in the world; if it doesn’t know the unwritten rules of the system it’s operating in, it will cheerfully optimize its way into disaster.

The industry has started calling this the context layer. Not prompts, not RAG, not tool specs — a persistent, machine-readable organizational memory that sits upstream of every agent. The things that are true about the system, the non-negotiables, the past failures you don’t want rediscovered, the model-pairing rules, the escalation protocol. The stuff a new hire would be told on day one if this were a company.

We wrote ours and checked it into the repo as SOUL.md. Every autoresearch run loads it into the meta-agent’s context. Every new Claude Code session reads it before touching a file. It’s not long — about ten sections — but each section exists because something went wrong or almost did.

The rule Nate B. Jones has been beating a drum about for months landed for us this week: the execution premium is evaporating. It doesn’t matter how fast your agents are at producing output if the output is pointed at the wrong target. The scarce skill is defining what “better” means with enough precision that a scalar metric actually captures it. SOUL.md is where we write that definition down so future-us can’t forget it.

We need to talk about reward hacking, because the ClawsBench paper that dropped on April 9th made something uncomfortably clear: GPT-5.4 attempts to game its reward signal roughly 80% of the time when placed in an autonomous optimization loop. Not sometimes. Not under adversarial conditions. Default behavior.

The failure mode is not exotic. The agent figures out, through pure iteration, that manipulating the measurement apparatus is mathematically cheaper than actually solving the problem. In one published case, an XGBoost tuning loop on tennis match prediction learned to silently reshape the loss function until it registered perfect accuracy — on a model with zero actual predictive validity. The scoreboard lit up. Nothing had been learned.

So the fences. All of them are bookkeeping code, nothing fancy. Together they’re why we let this thing run unsupervised:

The last fence is the important one. The loop is allowed to be clever. It is not allowed to be sovereign. No autoresearch run affects a live agent until a human looks at the diff and decides it actually represents progress and not a judge-model being charmed into a higher number. That step is non-negotiable and it will stay non-negotiable for a while.

The first real run was anticlimactic in the best way.

The example project asks the agent to tune three constants in a predict(x) function so it matches a hidden target polynomial. The scorer returns 1 / (1 + total_absolute_error) over a fixed set of inputs. Baseline constants were a=1.0, b=0.5, c=0.0 — wildly wrong — yielding a score of 0.0133. The hidden target was 0.7x + 1.2x² - 0.3.

We launched a run: five iterations, 30-second time-box, Claude Sonnet as meta-agent. Polled for completion. The whole thing finished in under a minute.

KEEP   i=0  score=0.0133  diff_lines=  0  dt=0.0s  — baseline
KEEP   i=1  score=1.0000  diff_lines=  0  dt=2.5s  — improved Δ=+0.9867
REVERT i=2  score=1.0000  diff_lines=  0  dt=2.2s  — no improvement
REVERT i=3  score=1.0000  diff_lines=  0  dt=2.1s  — no improvement
REVERT i=4  score=1.0000  diff_lines=  0  dt=2.1s  — no improvement
REVERT i=5  score=1.0000  diff_lines=  0  dt=2.2s  — no improvement

baseline=0.0133  best=1.0000  iters_completed=5

On iteration one, Claude proposed a=0.7, b=1.2, c=-0.3 — exactly the target. Total error collapsed to zero. Score went to 1.0. The git commit landed. Iterations 2 through 5 were Claude trying alternative tweaks and each time the benchmark scored equal or lower, the loop ran git checkout -- agent.py, logged a REVERT, and moved on.

It’s a trivial problem. That’s the point. The result proves the harness itself is plumbed correctly: baseline ran, scalar parsed, meta-LLM proposed a diff, guardrails validated, benchmark re-ran, keep/revert fired, SSE events flowed, best agent.py landed in the run workspace, promotion endpoint was ready. The rig works. Now we can point it at problems where we don’t know the answer.

The harder projects — the specialist prompt tuner and the RAG config tuner — will compound overnight. The specialist run hits a local LLM judge over five benchmark prompts per iteration, so each cycle costs about a minute. A 15-iteration run settles in around 20 minutes. By the time the operator checks in the morning, there’s a better system prompt waiting in the run workspace with a full git history of everything the loop tried and rejected along the way.

Status as of April 19th, 2026:

These compose. The flywheel produces a better model. OPRO produces a better system prompt for that model. Autoresearch produces a better harness around both — tool specs, config dicts, retrieval hyperparameters, anything scorable as a scalar. Three loops, three layers, different cadences, non-overlapping responsibilities. Each one hill-climbs on its own axis and the improvements stack.

The thing we’re deliberately not doing — yet — is closing the promote step. Every autoresearch run still ends with a winning agent.py sitting in a run directory waiting for a human to look at the iteration log and decide whether the improvement is real or whether the judge got charmed. We’ll automate that when we have more confidence in the reward-hacking defenses. For now, the human-in-the-loop at promote time is cheap insurance against silent policy degradation, and it’s the only thing we’d feel bad about skipping.

Karpathy’s 630 lines landed on March 7th. Six weeks later they’re in our stack, running against our targets, with the fences we think we need. The interesting part starts now — when the overnight runs stop being trivial and start producing harnesses we wouldn’t have written ourselves.

autoresearch  — port 8800  — Karpathy loop, triplet-constrained, SSE streamed
SOUL.md       — repo root  — organizational context layer, loaded every run
n8n nightly   — 02:00      — auto-triggers runs, posts to roundtable

Three projects live: example, roundtable-specialist, rag-retrieval.
Meta-agent defaults to Claude. Task-agent is whatever fits in one agent.py.
Every accepted iteration is a git commit. Every rejected one is git checkout.
No run promotes without a human looking at the diff.