Sakana means fish. Sakana AI is the Tokyo lab that keeps publishing the papers I actually want to read — not another hundred-billion-parameter flex, but ideas shaped like mechanisms: evolution as a search operator, models teaching models, tree search over answers instead of just tokens.

So we crawled their entire GitHub org. Fifty-nine repositories, sorted by what would survive contact with a basement server that also has to pay the electric bill.

Most of it doesn’t survive. The H100-only CUDA kernels don’t survive. The eight-GPU reinforcement-learned teachers don’t survive. Their shiny new multi-agent product turns out to be an API you call, not a thing you run. That’s fine. A frontier lab’s GitHub isn’t a store; it’s a quarry.

Three things fit. A tree-search library called TreeQuest, small enough to pip install and pure Python all the way down. An evolutionary program-search harness called ShinkaEvolve that speaks to local models. And a pair of hypernetworks that generate LoRA adapters in a single forward pass — parked for later, but parked deliberately, because they attack the exact reason my fine-tuning pipeline is switched off.

MKUltra’s roundtable has five LLM providers on call — Claude, a local Ollama model, Gemini, Perplexity, OpenRouter. For a year, the “heavy” tier meant: ask all five at once, let a sixth model smash the answers together. A flat debate. Democratic, expensive, and dumb in a specific way — it spends the same effort on every provider regardless of who’s actually good at the question.

Sakana’s AB-MCTS fixes the dumbness with a bandit. Every answer becomes a node in a tree. At each step the algorithm decides: go wide (draft a fresh answer) or go deep (refine the current best) — and with which provider. Providers that score well get called more. Providers that time out get learned around.

"search": {
  "algorithm": "ab-mcts-a",
  "expansions": 6,
  "best_agent": "Ollama-Heavy",
  "best_score": 0.9,
  "best_depth": 1,
  "action_counts": { "Perplexity": 2, "Router": 2,
                     "Groq-Fast": 1, "Ollama-Heavy": 1 }
}

Depth one. That’s the detail I care about: the winning answer wasn’t anyone’s first draft. It was a refinement of a draft, produced by a different model than the one that drafted it. The second live run went the other way — one provider nailed it immediately and the bandit just exploited him three times in a row. Same code, different search, per question. That’s the point.

Wiring it in took an evening, because the library asks almost nothing of you: a function that returns an answer and a score. The scores come from a judge model we already had. The whole thing degrades gracefully — if the search dies, the old flat gaggle catches the query.

First live query through the new search, and the winning answer — the one scored 0.9 by the judge, the one selected as best from the whole tree — opened like this:

Two bugs in one sentence, and I’m grateful for both, because they only show up when you run the thing for real.

Bug one: a provider had returned a 401 — an auth failure — and the search treated that error string as a legitimate node. Scored zero, sure. But zero doesn’t mean forbidden, and the bandit, exploring, picked the error as a parent and asked another model to improve it. The model did its best. Its best was a critique of an HTTP status code.

Bug two: the refiner was told, in writing, “do not mention the draft.” It mentioned the draft in its first breath — and the judge didn’t care, because nothing in the judge’s rubric said internal process leaking into a user-facing answer should cost points.

Three fixes: error nodes can no longer be refinement parents; the judge now explicitly punishes meta-commentary; and the final answer is synthesized from the top scored candidates instead of shipped raw. Second live run: clean, forty-seven seconds, no archaeology of the search process visible in the output.

The general lesson costs nothing to state and everything to remember: a scoring function is a contract, and anything you didn’t write into it is legal.

MKUltra had been running a five-service skeleton crew all month to save RAM. Time to wake everything. docker compose up -d, thirty-nine services, what could go wrong.

What went wrong, in order: a TTS container’s bind mount pointed at a file that no longer existed, so Docker had helpfully created an empty directory with the file’s name, and you cannot mount a directory onto a file, so the container died — and its death aborted the launch sequence with seventeen services still sitting in Created, never started. The prompt optimizer was crash-looping because someone (me) had added Prometheus metrics to its code without adding the client library to its image. And the voice pipeline couldn’t bind port 8080 because a completely unrelated side project had been squatting the port from a forgotten terminal tab for three days.

Each fix was small. A profile gate on the doomed container. One word added to a Dockerfile. One SIGTERM to the port squatter. But none of them were discoverable until the stack actually had to stand up from zero — which it hadn’t done in weeks. Restart-from-nothing is a test, and like all tests, it only has value if you occasionally run it.

One service stayed broken after all that: Grafana. Healthy container, correct config, answering perfectly on localhost inside its own namespace — and every request through its nginx proxy died waiting. The nginx logs filled with 499s: client gave up before upstream replied. Upstream never replied.

The debugging had the exact shape I’ve learned to distrust: every individual component is fine, only a specific pair is broken. Host to Grafana: fine. Container to Prometheus: fine. Container to Grafana: silence. Ping between them: fine. TCP between them: black hole. ICMP passing while TCP dies isn’t a service problem. That’s a firewall with an opinion about ports.

RETURN  tcp  10.0.0.0/24       multiport dports 3000,8880
RETURN  tcp  192.168.192.0/24  multiport dports 3000,8880
DROP    tcp  0.0.0.0/0         multiport dports 3000,8880

Months ago, some session hardened two published ports so only the LAN could reach them. Reasonable. But bridge traffic on a Docker host traverses that same chain, and the containers live on 172.30.0.0/16, which is not the LAN, so the rule wrote them out too. Grafana listens on port 3000 inside the network. Every SYN from its own proxy, silently dropped, since whenever that rule was written.

One inserted exemption for the container subnet and Grafana answered in green. The villain of this post isn’t the rule — the rule was doing exactly what it said. The villain is silently. A DROP leaves no log, no reset, no error. Just a proxy waiting politely forever for a packet that was executed at the border.

Now the main event. MKUltra’s ingress router is a pile of hand-tuned regexes that reads every incoming question and decides: is this code, strategy, research, data analysis, a domain specialist question, or just general? That one decision picks which agents wake up and how much the query costs. It is the most consequential hundred lines in the stack, and nobody had ever measured it.

To measure it you need labeled questions. I have 5,241 logged interactions! I have, it turns out, thirty-eight distinct questions — the other 5,203 are four cron jobs asking the same four things on a loop, forever. My corpus wasn’t a corpus. It was a heartbeat monitor with delusions of grandeur.

So we built the eval set instead of mining it: 324 synthetic questions generated label-first — fifty-four per intent class, mixed homelab-flavored and general, with deliberate borderline cases — plus the thirty-eight real survivors, labeled after the fact. Three hundred sixty-two total. A third sealed away as a test set the optimizer never gets to see.

macro-F1     0.709
accuracy     69.3%

worst class:  general — precision 0.43
              (the dumping ground: weak signal ⟶ full
               five-agent gaggle ⟶ the expensive mistake)

Wrong on almost one query in three. And wrong in the costly direction: when the regexes shrug, the query falls through to “general,” which spins up everything. The cheapest classifier in the stack was quietly making the most expensive mistake in it.

Here’s where the second Sakana idea comes in. ShinkaEvolve is evolutionary program search: keep a population of programs, use LLMs as mutation operators, evaluate every mutant, let the archive breed. Point it at the router, fitness = macro-F1 on the training split, and let it rewrite the regexes all night.

The part I want to defend is the shape of the setup, because the shape is why this is possible on a basement box at all. The mutation models are free-tier API calls and a 4.7-gigabyte local coder model. The evaluator is pure Python — twenty milliseconds per candidate, no GPU in the loop. The whole thing is a batch job: it runs, it writes its archive, it exits. No resident service, no idle VRAM, no monthly tax. My fine-tuning flywheel is switched off precisely because it’s the opposite shape — a training run that starves every co-hosted service while it lives.

It’s running as I write this. Sixty generations, two islands, a five-dollar cost ceiling it will never reach. A few hours in, the best mutant has clawed from 0.709 to 0.714 — not a headline, but the archive is young and evolution is a compounding creature. The rule for what happens after is already written down, which is the only time it’s safe to write such rules: the winner gets scored once against the sealed test set, and it replaces the production router only if it beats 0.709 there. Not on the split it trained on. On the one it never saw.

The week compresses to three moves.

Borrow mechanisms, not scale. Everything worth taking from fifty-nine frontier-lab repos fit in a pip install and a git clone. The ideas — bandit-guided answer trees, LLMs as mutation operators — transfer completely. The compute assumptions don’t have to.

Trust nothing you haven’t cold-started. The stack’s five latent failures — the ghost mount, the missing import, the squatted port, the packet-eating firewall rule — were all deployed weeks before they detonated, and all invisible until everything had to stand up from zero at once.

Measure before you optimize, seal the test set, and let the machine grind. The router was “fine” for a year because nobody looked. One afternoon of building an honest eval set turned fine into 0.709, and 0.709 into a number that evolution is, at this very moment, trying to beat while I do something else entirely.

The fish gave us the ideas. The firewall gave us the humility. The overnight run gave us the ending — it’s in the postscript.

The run finished the same day this post was drafted: ten and a half hours, fifty-four programs, two islands, and a five-dollar cost ceiling it never came close to touching.

production router   macro-F1 0.680    accuracy 66.7%
evolved router      macro-F1 0.708    accuracy 69.5%   ✅ promoted

The training gain generalized. And when I diffed the winner against the original, there was no genius in it — no exotic algorithm, no memorized answers. Evolution found diligence: strong code verbs like implement and refactor now count double, the domain-specialist multiplier crept from 1.2 to 1.4 with extra weight on healthcare and security vocabulary, research confidence went from 3.0 to 3.5. Weights nobody ever bothered to tune, because tuning them by hand would have required the eval set nobody had built.

Promotion was the boring kind of careful: grafted into the production router, parity-checked at zero mismatches across all 105 test questions, full test suite green, container restarted, live probes correct. The pre-evolution router sleeps in a backup file, unmourned.

The router that was wrong a third of the time is still wrong three times in ten. But it got measurably better overnight, unattended, for approximately nothing — and the harness that did it is sitting in a directory, reusable, pointed at nothing yet. That last part is the good part.