John Rush’s Blog

Finding the Bottlenecks in Agentic Development

2026-09-17T00:00:00+00:00

Finding the Bottlenecks in Agentic Development

Published: September 17, 2026

I’m building a new AI email client. I wanted to improve my agentic development process, so I started instrumenting it and looking for bottlenecks.

The goal wasn’t to produce a perfect benchmark or a universal formula for building software with agents. I wanted enough visibility to make better decisions about where to spend tokens, when to parallelize work, and which models should handle which jobs.

Start with Progress

First, I used programmatically verifiable benchmarks to define progress. There is a benchmark for opening emails, another for viewing attachments, another for getting new mail, and others for things like loading speed and software performance.

I grouped those benchmarks into checkpoints: vertical slices of capability that I could build and validate as I went. Each checkpoint contained several workstreams. For the “read emails” checkpoint, for example, one workstream handled opening rich-text email while another handled attachments.

This gave the agents a nicely bounded scope to build against. It also exposed an interesting dependency: the right size for a workstream depended on the model executing it. Smaller models like luna did better with smaller workstreams, while astra handled larger ones well.

Measure What Matters

To understand the impact of different approaches, I needed process metrics. Quality metrics are important, but they belong elsewhere. These metrics are about how the work gets done.

The first metric is cost. I used API cost per 10,000 changed, committed tokens—$/10kCoT—to normalize across workstreams and checkpoints with different scopes. Like lines of code, 10kCoT has obvious flaws, but it was good enough for this use. Even though I’m on a subscription, using API pricing gave me a consistent way to compare models. Empirically, it also aligned well with how my weekly limits were being used.

The next metric is generated versus committed tokens. This approximates how much model output it took to produce the code that was ultimately committed. We don’t expect every output token to become code: there are reasoning tokens, coordination between agents, reviews, status reports, and some amount of rework or mistakes. A lower ratio is not automatically better.

The important thing is to balance generated tokens against $/10kCoT. luna is so cheap that even generating ten times as many tokens costs only about 20% as much as astra. It isn’t as simple as minimizing generated tokens. That strategy can lead you in the wrong direction.

The final metric is active time versus wall-clock time. Active time measures how much wall-clock time agents are actually working. This helps identify where I might be holding things up and where parallelism could help.

There are tradeoffs here too. In my runs, the smaller models generally required more rework and review cycles. That cost time, but I traded the time for lower cost. Sometimes it is better to spend tokens to get speed—especially when Tibo says there is a reset coming at the end of the day.

These metrics let me make thoughtful tradeoffs instead of guessing at where the bottleneck is.

How It Went

The main resource constraint I’m facing is that I’m token-poor. I have a single Pro account, so I want to get as much productive use out of it as possible.

On the early checkpoints, astra handled implementation, and it was expensive. CP-17 cost $23.59 per 10kCoT. I moved to terra as the implementer and kept astra as the reviewer. That lowered CP-7’s cost to $15.60 per 10kCoT—a significant improvement—but the review was still expensive. In WS-19’s case, the review cost twice as much as the implementation itself.

That suggested a new model: have terra perform a first review-and-fix pass, followed by a final review with astra. This reduced WS-34’s cost to $9.27 per 10kCoT. Even with multiple terra reviews, terra accounted for only 58% of the total review cost, while astra accounted for 42%.

In a couple of cases, though, the terra and luna implementers did not complete all of the required benchmarks. I was spending expensive review tokens on incomplete work.

So I added a simple completeness-check step with luna. That step looked only for benchmark completeness; it did not evaluate correctness. This drove the cost of the next workstream down to $2.11 per 10kCoT, with the completeness review costing only $0.19.

Cheap checks that save expensive work are a classic system-design trick.

I was still seeing my weekly limit get consumed fairly quickly, so I looked at what was still costing a lot. In this case, it was the astra coordinator spending expensive tokens acting as a heartbeat. CP-7’s astra coordination cost $1.86 per 10kCoT, while CP-9’s terra coordinator cost about $0.15 per 10kCoT—even though it handled about three times as much coordination.

The Next Bottleneck

The best optimization is not to do the work in the first place. So I dug into why there was so much coordination.

Some architectural assumptions required the work to be serialized. A rearchitecture that removed those assumptions and built stronger, more independent components would allow more concurrency without increasing coordination proportionally.

Independent components would let me explore more complex agent-swarm topologies, apply more compute to the problem, and—most importantly—measure whether those approaches actually help or merely burn dollars.

A Nap, a Walk, and a USB Stick

2026-08-03T00:00:00+00:00

A Nap, a Walk, and a USB Stick

Published: August 3, 2026

We recently welcomed a new daughter to our family. During the late nights I’ve been streaming a few games on my iPad. Unfortunately, sometimes the stream would freeze and stutter. Something was up.

Why Can’t You Just Do It?

After I put her down for a quick nap I asked Codex:

Okay, I am trying to debug why my internet is so slow. Can you look at my network configuration? I have a pfSense router on 192.168.1.1, and I can give you the username/password for it (or you can get it from the 1Password CLI, item [REDACTED]). I also have a Wi-Fi router that I’m using, but I can also hook this computer up to Ethernet. I notice when I’m streaming a game on my iPad’s Wi-Fi, I get lots of freezes and drops, whereas that didn’t happen on a different network on vacation.

Codex started by mapping the interfaces and routes using scutil --nwi, route -n get default, networksetup -listallhardwareports, ifconfig -a, and ipconfig. It inspected the radio with system_profiler SPAirPortDataType and established baselines with ping and traceroute. Then Codex logged into pfSense via the HTTP admin console using a script that read the secret from 1Password CLI (Codex never read the password directly). It found both WAN and LAN are at 1 Gb/s full duplex with no errors or collisions.

Next it did some load testing using ping -c 250 -i 0.1 192.168.1.1 and networkQuality -v -s -I en1 -M 35. On the AP’s original channel it measured 59.9 Mb/s, 70.1 ms average AP latency, 12.8% packet loss, and 544 ms loaded responsiveness. Brutal.

It then located the Wi-Fi access point using pfSense’s ARP and DHCP pages, along with the Mac’s arp table. Codex fingerprinted the access point as a Ubiquiti via its ssh host key and as a UAP-AC-Pro using Ubiquiti’s UDP discovery protocol. Impressive. Then it narrowed the tests just to the AP and found the packet loss was happening on the first hop, before pfSense or the Internet.

These are exactly the diagnostics I’d do myself, using the same tools. Well, I don’t have to ssh fingerprint the access point. I can just look at it. But I’d have to spend a bunch of time looking up commands because I can never remember the exact incantations.

Codex suggested I try changing the AP to channel 44 at 40 MHz through the UniFi app. I made the change and had Codex retest: 32.9 Mb/s, 221 ms average AP latency, 13.3% packet loss, and 1,285 ms loaded responsiveness. Worse. I then gave it the 1Password item for the AP. It logged into the AP over ssh and saw thousands of retries and 14k transmission-queue drops. ethtool showed the AP’s wired uplink was clean at 1 Gb/s full duplex.

That’s when I realized my mistake. I need an agent to do this work. I can’t be fiddling with all this and taking care of a baby. This is a perfectly agent-shaped problem (action + verification). So I asked Codex the most important question. “Why can’t you just do it?”

Again Codex leapt into action. It ran another system_profiler SPAirPortDataType scan and found no nearby networks in the DFS block around channel 100. It then connected to the AP via ssh and made sure it supported channel 100. It then changed the AP’s channel temporarily, tested, and got 144.9 Mb/s, 29.7 ms average AP latency, 2.5% packet loss, and 142 ms loaded responsiveness. That was 4.4x the throughput of channel 44 and roughly a 9x improvement in loaded responsiveness. Awesome.

Codex then persisted the change and verified the configuration.

This is the same way I’d make the change too. Scan for open channels, test, verify. Very nice. And it all happened during a nap.

Great. What Else Should I Consider?

Naturally, I asked Codex the obvious next question about my home network: “Great. What else should I consider?” I know the pfSense router is hopelessly out of date. It’s been ages and I’ve just never wanted to spend time on it. Instead, I decided to let Codex have a crack at it while we walked to PCC.

Codex first considered updating the AP’s firmware based on the version it saw earlier. It did some research and decided not to install it because the fixes included in the update I didn’t need and there were reports online about regressions on my legacy hardware, but it did find a more important issue.

My router was running pfSense 2.4.4-p3 and that is horrifically out of date. I’ve known this for a while and I thought, why not? Let’s see how this goes. Codex and I chatted and we landed on a fresh install of OPNsense. I told Codex to figure out the install and all the configuration we needed.

It did some archaeology on the old pfSense and found an old OpenVPN configuration, some unused VLANs and a broken pfBlockerNG-devel installation. I told Codex that we didn’t need any of that anymore. Let’s start fresh. Codex got to work downloading OPNsense, getting the configuration documented and backing up the old pfSense install.

Codex did a much more thorough job than I would have. I would have just popped in a USB installer and yoloed. Instead, Codex sshed into the pfSense box, created an encrypted backup, and inspected and validated the hardware to make sure the OPNsense installation goes swimmingly. It did suggest replacing the 9 year old SSD based on power-on hours, but I pushed back. Codex then automatically dug into the SMART data and things looked fine. I guess we can’t quite setup the drinking bird to just press y yet lol.

It got everything ready, but the actual install had to wait for us to finish the walk. Only I could plug in the USB stick and boot the router. But once that was done Codex could ssh in and finish the installation. It had me unplug the USB and reboot the router, and everything came right up.

After that, Codex finished configuring firewalls, setting up and validating Quad9. It also did ad blocking with OISD via Unbound to replace my old and broken pfBlockerNG-devel. It also validated that was working and the list updates were coming in.

I asked Codex once more if there was anything else. It noticed that we were double NAT’d behind my C5500XK SmartNID. It backed up the configuration and put the SmartNID in Transparent Bridging with Tagged-201 mode. OPNsense had retained a stale private WAN lease so it reloaded WAN DHCP to get a new public lease and verified that the private 192.168.0.1 hop was gone. I was impressed that Codex fixed all that hands-off.

Of course, I had to do the final test. That night I streamed games on the iPad without any issues.

None of this is particularly challenging. This is all stuff I’ve done before. What’s exciting is how easy it is. I tell agents to fix stuff, grant permissions, add a little judgment, and plug in a USB stick. My home network gets upgraded while I spend time with my new daughter.

25 Years of Eggs

2026-02-23T00:00:00+00:00

25 Years of Eggs

Published: February 23, 2026

Everyone needs a rewarding hobby. I’ve been scanning all of my receipts since 2001. I never typed in a single price - just kept the images. I figured someday the technology to read them would catch up, and the data would be interesting.

This year I tested it. Two AI coding agents, 11,345 receipts. I started with eggs. If you can track one item across 25 years of garbled thermal prints, OCR failures, and folder typos, you can track anything.

14 days. 1.6 billion tokens. 589 egg receipts found. Here’s what the data says.

Here’s what I typed on Day 1:

Ok so let’s make a project plan. In the ~/Records/ we have a ton of receipts. Many are pdf/image/etc. I want to go through and extract the actual content of the receipts to find how much we spend on eggs. Receipts are notoriously terrible to OCR, so we might need to do something more advanced.

Codex explored my file system, found two existing SQLite databases I’d forgotten about, discovered 11,345 receipts across PDFs, emails, and images, and came back with a project plan. I said “write this out to a plan.md please.” It did. We were building within the hour.

The whole thing took 14 days. Maybe 15 hours of me actually at the keyboard - short bursts of direction-giving separated by long stretches of the agents just running. Codex ran 15 interactive sessions. Claude handled 10.

Shades of White

The oldest receipts were flatbed scans - multiple receipts per page, random orientations, white paper on a white scanner bed. Codex and I tried seven classical CV approaches to find receipt boundaries. Edge detection. Adaptive thresholding. Contour analysis. Morphological operations. Watershed segmentation. Template matching. A grid-based decomposition I pitched as “a classic HackerRank problem.”

None of them worked. The core issue: receipts are white and so is the scanner bed. I started calling it the “shades of white” problem. The cleverest attempt was inspired by removing tourists from landmark photos - stack all scans, compute the median pixel at each position, subtract to reveal edges. I thought that one was going to work. Best F1: 0.302.

We also threw macOS Vision OCR at it (via a Swift script Codex wrote on the fly), Tesseract, several other tools. I was starting to think the flatbed scans might just be a loss. Then I tried Meta’s SAM3.

One API call with text="receipt". 0.92-0.98 confidence on every boundary. Four seconds per scan. 1,873 receipts from 760 multi-receipt pages. Seven approaches in hours; SAM3 in an afternoon.

Wait - You Already Know the Answer

Receipts land at random angles, and OCR needs them upright. We tried Tesseract’s orientation detection, macOS Vision OCR, Moondream 2 and 3 - each one better than the last but none reliable enough. Then I realized that every time I pasted a receipt into our Claude conversation to debug something, it was already reading the text perfectly. Rotated, faded, didn’t matter.

Why am I building a rotation pipeline when the tool I’m talking to already solves this? So we sent all 11,345 receipts through Sonnet and Codex. Sometimes the answer is staring you right in the face.

Replacing Tesseract Overnight

Halfway through the project, Tesseract was the weak link. It read “OAT MILK” as “OATH ILK.” It dropped decimals - $4.37 became $437. On old thermal prints it produced nothing at all. Codex opened 20 of the worst ones by hand and found that some weren’t even receipts. A family photo. A postcard. A greeting card. All filed under “Receipts.”

I found PaddleOCR-VL - a 0.9B parameter vision-language model that runs locally on Apple Silicon. First test on a sample bank statement: clean, accurate text in 2.1 seconds. Tesseract was faster but dramatically noisier. Second test on a tall Fred Meyer receipt: disaster. The model entered a repetition loop, hallucinating “TILL YGRT” endlessly.

The fix turned out to be simple - split tall receipts into slices. Dynamic slicing based on aspect ratio: num_slices = max(2, round(aspect_ratio / 1.5)). Five parallel shards ran overnight. GPU pegged at 100% for 10.8 hours. In the morning: 11,345 receipts OCR’d successfully. Cleaner text for every receipt in the archive.

PaddleOCR-VL isn’t a Codex replacement - it can’t do structured extraction or follow instructions. It’s a better Tesseract. The real pipeline: receipt image → PaddleOCR-VL (local, clean text) → Codex/Claude (structured extraction).

Twelve Hours to Three

Once receipts were segmented, oriented, and OCR’d, they needed structured extraction - find the egg line items, pull prices and quantities.

It started with regex. The models love regex. Keyword matching for “egg,” money patterns for prices. Heuristics found eggs in 25/25 positive samples with 0 false positives. Not bad. But on the full corpus, false negatives piled up - Fred Meyer abbreviated codes like STO LRG BRUNN, Whole Foods truncated to EDGS, OCR mangled “EGGS” into LG EGO 12 CT. No regex catches these.

So I told Codex “we have unlimited tokens, let’s use them all,” and we pivoted to sending every receipt through Codex for structured extraction. From that one sentence, Codex came back with a parallel worker architecture - sharding, health management, checkpointing, retry logic. The whole thing. When I ran out of tokens on Codex mid-run, it auto-switched to Claude and kept going. I didn’t ask it to do that. I didn’t know it had happened until I read the logs.

But the runs kept crashing. Long CLI jobs died when sessions timed out. The script committed results at end-of-run, so early deaths lost everything. I watched it happen three times. On the fourth attempt I said “I would have expected we start a new process per batch.” That was the fix - one fresh process per batch, hard call cap, exit cleanly, resume from cache. Codex patched it, launched it in a tmux session, and the ETA dropped from 12 hours to 3. Not a hard fix. Just the kind of thing you know after you’ve watched enough overnight jobs die at 3 AM.

11,345 receipts processed. The thing that was supposed to take all night finished before I went to bed.

The Classifier That Beat Its Own Ground Truth

First I needed ground truth. I asked Claude to build me a labeling tool - keyboard-first, receipt image on the left, classification data on the right, arrow keys to navigate, single keypress to verdict. It built the whole Flask app in 22 minutes. I sat down and hand-labeled 375 receipts.

Regex found 650 receipts mentioning “egg.” Against those 375 labels: 88% recall. The misses told the story - abbreviated codes, OCR garble, truncated descriptions. No keyword search catches STO LRG BRUNN.

The fix: use those hand-labeled edge cases as few-shot examples in an LLM classifier. Twenty examples of what “eggs” looks like on a garbled thermal print from 2003. Batch 10 receipts per call. Eight parallel workers. Two hours. 11,345 receipts classified.

Final accuracy: 99%+. Every supposed “miss” by the LLM turned out to be a mislabel in the ground truth. A bicycle shop receipt the old heuristic had flagged. A barcode-only scan. Egg noodles. The classifier was more correct than my labels.

Then more QA. A second tool for eyeballing 497 weak images: Space for no-eggs, X for has-eggs. A third for data entry on 95 receipts with missing fields - numpad-optimized, auto-advancing. Four tools total, each built in minutes, each one I ground through by hand.

Quality

So how good is the data? I pulled 372 random samples and checked them by hand. Initially: 96% correct. The errors were mostly garbled OCR on old scans. One was a hallucination - the pipeline fabricated egg data for a receipt that contained no eggs at all.

Real-world data is messy:

Receipt folder typos spanning years: “Reciepts” (2016-2017) and “Recipts”
A receipt scanned backwards - Claude decoded the mirrored OCR character by character
Email receipts silently preferring text/plain over text/html, dropping pricing lines that only existed in the HTML part

Here’s what made the quality good: every time I caught something, I could show the agents what to look for and they’d go fix it everywhere. I caught a store address hiding in OCR noise: “915 Ny 45th St” was 915 NW 45th St, Seattle. I showed them the pattern, they ran a recovery pass on 40 missing-location receipts - all 40 resolved.

By the Numbers


Wall clock time	14 days (Feb 8-22, 2026)
Hands-on time	15 hours
Tokens consumed	1.6 billion
Estimated token cost	$1,591
Confirmed egg receipts	589
Total egg spend captured	$1,972
Total eggs	8,604

Codex and Claude are excellent at building tools and extracting structured data, but they couldn’t segment an image or replace an OCR engine. The right answer was a stack of specialized models - SAM3 for segmentation, PaddleOCR for text, Codex and Claude for everything else. I expected this, but it was worth trying the simple path first.

These are the days of miracle and wonder. I can’t wait to see what 30 years of eggs looks like.

Compounding Beats Roadmaps

2026-01-31T00:00:00+00:00

Compounding Beats Roadmaps

Published: January 31, 2026

Overview

Service-heavy SaaS companies are facing an organizational challenge disguised as a technical one.

These are products where a meaningful amount of value is created in use: through configuration, implementation, customer-specific workflows, and ongoing adaptation. They operate under real uncertainty. Customer needs change quickly, information is incomplete, and local context dominates outcomes.

Think implementation-heavy enterprise platforms like Salesforce or Workday, vertical SaaS with deep industry configuration, or any product where customer success, implementation consultants, and solutions engineers create significant value post-sale.

In that environment, the most important shift AI enables is organizational, not technical. It allows organizations to decentralize execution in a way that creates compounding learning and capability which outperforms traditional roadmap-driven development in high-uncertainty markets.

In this post, a skill is a reusable, versioned playbook or workflow. An agent is the software that runs skills with customer context.

AI Tools Without Organizational Change

Today, many SaaS companies are adopting AI tools. What they aren’t changing are the underlying organizational constraints:

Who has decision rights
Where authority lives
How work flows through execution

In practice, what’s happening is this: AI lowers the cost of local reasoning and local work, but organizations keep execution centralized. The same approval chains, roadmaps, and ticket queues remain in place.

The result is predictable:

With AI, the queue moves faster - but not smarter.

In service-heavy SaaS, where execution happens continuously in customer environments, centralized execution throttles the very advantage AI creates.

Example: Your implementation team sees the same integration failure pattern across 15 enterprise customers. With AI, a consultant could build a diagnostic workflow and fix script in an afternoon. Instead, it goes into a ticket. Product reviews it in two weeks. Engineering prioritizes it next quarter. Six months later, the pattern is still being debugged manually, customer by customer. The capability to solve it was available from day one. The organizational structure prevented it.

We’ve Seen This Failure Mode Before

This problem isn’t unique to software. It appears any time organizations operate in environments with high uncertainty and rapid change.

The clearest precedent comes from the military.

Historically, many militaries relied on strict centralized command-and-control. Decisions were made far from the front lines, plans were detailed in advance, and lower-level units were expected to execute precisely. This worked in stable environments, but failed badly in fluid ones.

Latency destroyed outcomes. By the time information traveled up the chain and orders came back down, reality had already changed.

The Shift to Mission Command

In response, modern military doctrine - most notably German Auftragstaktik and later U.S. mission command - shifted away from centralized decision-making.

Instead of issuing detailed instructions, leadership focused on:

Defining the objective
Communicating intent
Setting constraints

Lower-level leaders are expected to decide how to achieve the objective based on real-time conditions.

For example:

A commander specifies the goal - secure a position, protect civilians, delay an advance - without prescribing exact maneuvers
A platoon leader encountering unexpected terrain or enemy movement adapts immediately, without waiting for approval
Units that wait for permission in these environments lose tempo and often fail

Crucially, this model assumes something strong:

Leaders at the edge are not just allowed to decide - they are required to decide.

Micromanagement from afar consistently underperforms disciplined initiative at the point of contact.

Service-Heavy SaaS Now Faces the Same Conditions

Service-heavy SaaS organizations increasingly operate under similar dynamics:

Fast-moving markets
Incomplete or delayed information
Customers behaving in unexpected ways
Value created through ongoing human execution

Customer success teams, implementation consultants, solutions engineers, and support staff often see real problems weeks or months before centralized planning cycles surface them.

How has AI changed economically? Three ways:

AI dropped the cost of drafting workable solutions. Frontline teams can now build reliable workflows and automations without waiting for engineering. What used to require a sprint now takes an afternoon.

AI dropped the cost of iterating with full context. The person closest to the customer can try five approaches in a day and see which one actually works. Centralized teams working from abstracted requirements can’t move that fast.

AI dropped the cost of packaging learning into reusable artifacts. A workflow that solves a problem can be described, shared, and adapted by other teams without rewriting from scratch. The knowledge compounds instead of living in someone’s head.

The bottleneck shifted from engineering capacity to decision rights. Who’s allowed to ship? That’s the new constraint.

Centralize Intent, Decentralize Execution

The split looks like this:

Centralized:

Strategy and vision
Intent and priorities
Guardrails and constraints
Primitives and platforms

Decentralized:

Decision-making
Adaptation in customer environments
Skill construction
Local product and configuration surface
Optimization for specific customer contexts

This is the direct analog of mission command:

Leadership sets intent
Constraints are explicit
Execution authority moves to the point of contact

AI makes this economically viable for the first time in SaaS.

How Organizational Compounding Actually Works

The mechanism is simple:

Real problem appears at the customer edge
Frontline team builds or updates a skill (a reusable workflow/playbook) using centralized primitives
It’s used immediately - validated or killed by actual outcomes
If it works, other teams discover and reuse it
Useful patterns get curated into shared libraries
The organization is permanently more capable

No roadmap cycle. No ticket queue. Try it, validate it, move on.

Example: A solutions engineer notices that enterprise customers consistently struggle with SSO configuration during onboarding. Using your platform’s primitives (connector library, diagnostic scripts, deployment sandbox), they build a guided SSO workflow with automated validation. It cuts setup time from 3 days to 4 hours. Another SE sees it in the shared library, uses it for a different customer, and adds support for Okta edge cases. A third SE in a different region adapts it for SAML. Six months later, your SSO onboarding time is 90% faster across all customers, and nobody filed a ticket or waited for a sprint.

That’s compounding. The workflow gets better every time it’s used. The organization learns faster than centralized planning could ever move.

Centralized orgs don’t have this loop. They have planning cycles, not compounding.

Execution at the Point of Contact

The default response to AI has been to centralize it: form AI teams that build against abstractions, solve problems once, and produce bespoke solutions. The learning lives in tickets and individual heads. It doesn’t compound.

The alternative is execution at the point of contact.

Frontline teams - the people implementing, configuring, supporting, and operating the product with customers - are the SaaS equivalent of units in contact with reality.

They see failure immediately, understand customer context best, and can validate solutions fastest.

These teams already solve problems locally. They just solve them manually, repeatedly, without capturing the solution. AI makes it cheap to capture and reuse. The organizational question is whether you let them.

Giving these teams authority to build and ship skills (scripts, playbooks, workflows) means decisions are made where reality is observed and validated.

Engineering’s Role: Build Primitives

In this model, engineering’s role changes.

Instead of building complete, one-off solutions, engineers focus on primitives: the composable building blocks that frontline teams use to solve local problems.

Primitives are things like:

Data access layers (safe, scoped connectors to customer data)
Policy enforcement (what can run where, who can access what)
Observability (logging, tracing, debugging for workflows built at the edge)
Eval frameworks (automated testing so local solutions don’t break things)
Template libraries (proven patterns for common problems)
Deployment sandboxes (safe environments to try things before production)

Engineering sets these standards once. Frontline teams compose them hundreds of times.

This is similar to how military organizations standardize equipment and communications, not tactics.

Primitives preserve global consistency while enabling local autonomy. A regional or vertical-specific team can customize behavior without central approval or rebuilding everything from scratch.

Decentralization without primitives is chaos. Primitives make decentralization scalable.

Skills as Organizational Memory

Local skills become codified organizational memory.

In practice, these are often simple artifacts - plain text, markdown files that agents run to handle specific situations.

An example skill file might look like this:

# enterprise-migration-dedup

Handle data cleanup for enterprise migrations with >100K records containing duplicate emails from legacy CRM systems.

## Instructions

You are helping a solutions engineer clean up duplicate customer records during an enterprise migration.

**Prerequisites:**
- Customer has exported CSV from legacy CRM (Salesforce, HubSpot, MS Dynamics)
- Duplicate scan shows >5% duplicate email addresses
- Customer stakeholder available for spot-check validation

**Steps:**
1. Run `python scripts/dedup-scan.py --threshold 100000 --input customer_export.csv`
2. Review output for merge conflicts (check audit log for `MERGE_CONFLICT` flags)
3. Run merge with conservative settings: `python scripts/merge-tool.py --conservative --preserve-newest`
4. Validate results: Re-run dedup-scan, confirm <1% duplicates remaining
5. Spot-check 20 records with customer stakeholder before finalizing

**Edge cases:**
- If records show conflicting ownership, escalate to CS lead before merging
- If both records updated in last 7 days, use `--manual-review` flag (sends to staging queue for human review)

**Rollback:** Original export preserved at `backups/customer_export_TIMESTAMP.csv`

**Track record:** 47/50 migrations clean (94%) | Owner: @solutions-eng | Updated: Jan 2026

That’s it. No code, no complex system. Just the pattern, captured and reusable.

Skills like this are easy to share, easy to invalidate, and easy to improve. They’re exercised daily in real customer workflows and validated by outcomes. Learning stops being anecdotal. It becomes durable.

What Changes on Monday?

Pick one team. Your highest-context frontline group - implementation, solutions engineering, or customer success. Give them authority to build and ship skills for their customers without approval chains.

Build the minimum primitives kit: scoped data access, audit logging, a sandbox environment, and a deployment path. Don’t build everything. Build what lets them ship safely, then expand based on what they actually need.

Set explicit constraints. Write down what can’t be automated without sign-off: customer data exports, billing changes, destructive operations, whatever matches your risk tolerance. Everything else is fair game.

Stand up a shared library. GitHub repo, Notion workspace, internal wiki - the tool doesn’t matter. What matters is discoverability. If team A solves something, team B needs to find it.

Measure what compounds, not what ships. Track:

Reuse rate: how many teams adopt a skill after the first use
Time-to-mitigation: how fast edge teams resolve new customer problems
Incident rate: are skills breaking things or fixing things
Pattern graduation: which local solutions earn promotion to the platform

Product and engineering curate, they don’t approve. Review what’s getting reused. A skill used once dies. Used ten times gets attention. Used fifty times gets promoted to the platform and maintained centrally. You’re learning from what survives contact with customers, not predicting what will work from a conference room.

Run this pilot for one quarter. If it works, expand to more teams. If it doesn’t, you’ve learned what constraints were wrong. Either way, you’re moving faster than a year-long planning cycle.

Common Objections

What about compliance and security?

Your frontline teams already have access—they need it to do their jobs. Implementation consultants have admin access to customer instances. Support engineers run diagnostic queries. Solutions engineers hold API keys.

The question: do you let them codify what they learn into reusable skills with audit trails, or make them solve problems manually every time? Skills don’t grant new permissions—they make existing work reproducible and auditable.

Some things still require sign-off: customer data exports, billing changes, destructive operations. Everything else runs with guardrails built into the primitives.

Scoped access controls what each skill can touch. Mandatory audit logs track every action. Tiered rollout (sandbox → pilot → production) is required. Prove compliance through logs and access reviews.

What about shadow automation and inconsistent customer experiences?

This is the real failure mode. Teams build skills that work for their customers but create unpredictable experiences across the customer base. Or worse, build something that fails silently.

The answer is guardrails in the platform and curation based on reuse.

Guardrails are built into the primitives:

Scoped permissions: skills can only access data they’re explicitly granted
Audit logs: every action is logged automatically, no opt-in
Tiered rollout: sandbox environment required, then pilot customers, then production
Required ownership: every skill has an owner and escalation path
Rollback path: skills store their state, can be reverted if they break something
Active monitoring: anomaly detection and outcome alerts catch silent failures

A skill that tries to access customer data without permission doesn’t run. One that skips the sandbox doesn’t deploy. One without an owner doesn’t make it to the shared library.

Curation happens through a visible graduation path:

Local skill built by one team for one customer
Used 5+ times - other teams discover it in the library
Used 20+ times - product and engineering review in monthly curation meeting
Used 50+ times - promoted to the platform, maintained centrally, becomes official

Every skill has an owner. Owners get paged when their skill breaks. Kill switches exist.

Measure reuse, incidents, and time-to-mitigation. What gets used survives. What breaks things gets rolled back. What nobody uses dies.

How do you maintain quality and safety?

Manual execution breaks too—and when it does, you have no audit trail, no rollback, and no way to prevent the next person from making the same mistake.

Safety comes from bounded blast radius and fast feedback. Sandbox → pilot → production progression is mandatory. Incident rates, rollback frequency, and customer-visible regressions are tracked.

Skills that break things get rolled back immediately. Reuse acts as a filter: what survives 50 uses is proven safe.

Doesn’t this fragment the product?

Your product is already fragmented—in undocumented configurations, one-off customer scripts, and solutions that live in someone’s head. Skills make that fragmentation visible and measurable.

Local variation is discovery. What gets reused creates convergence.

The graduation path (5 uses → 20 uses → 50 uses → promoted to platform) ensures useful patterns become standardized. Fragmentation becomes signal about what should be built centrally.

Isn’t this chaos?

Chaos happens when skills are unowned and invisible. Prevent it with required ownership (every skill has an owner who gets paged), scoped permissions (can’t access what you’re not granted), audit logs (every action traced), and rollback paths.

Contain blast radius, enable experiments.

Isn’t this culturally hard?

Extremely. It changes who ships and what gets rewarded.

If you reward roadmap throughput, you’ll get tickets. Reward reuse and time-to-mitigation instead. Require ownership. Kill unowned skills. The incentives have to match the model or it fails.

The Bet

AI doesn’t just make software cheaper to build. It makes local iteration cheap and delay expensive.

Service-heavy SaaS companies that keep execution centralized will ship faster versions of yesterday’s assumptions. Their frontline teams will keep solving the same problems manually, one customer at a time. The queue moves faster, but nothing compounds.

Centralize intent and guardrails. Move authority and tooling to the point of contact. Build primitives that make edge execution safe and reusable. Start with one team for one quarter. Measure reuse, time-to-mitigation, and incident rate. Promote what compounds.

The durable advantage is turning customer execution into organizational capability. Solving a problem once means the organization is permanently better.

Compounding beats roadmaps. Decide where authority lives accordingly.

Agent-Streams: Skeptical Overseer for Long-Running Coding Agents

2026-01-01T00:00:00+00:00

Agent-Streams: Skeptical Overseer for Long-Running Coding Agents

Long-running coding agents are fantastic. I love giving them a big task, letting them run overnight, and coming back to merged code the next day.

The problem is that they often declare victory too early, miss requirements, or just make mistakes. A few days ago I asked Claude to build a tool to process a bunch of images using Moondream3. I let it run overnight. In the morning Claude said it was “done”—except the Moondream3 integration was a # TODO, and the tests were disabled. That’s when I realized I needed a better setup. DONE isn’t a state; it’s a claim that has to survive skeptical review.

Agents optimize for “done,” not correctness. That’s why you wake up to TODO stubs and disabled tests. I’ve tried techniques like detailed TODO lists and lots of tooling, but I don’t want to micromanage low-level process details. I want to give the agent a high-level spec and come back to mergeable code.

This is code review for agents: one role builds; another verifies.

So I split the workflow into two roles: a builder and an overseer. Rather than fight the builder’s tendency to declare victory early, I lean into it. Let it sprint to “done.” Then make that claim survive a skeptical review.

The overseer isn’t magically smarter; it’s independent. Fresh context means it doesn’t inherit the builder’s narrative, shortcuts, or “this should be fine” reasoning. Its job is simple: read the spec, review the diff, and try to find what’s missing.

In practice the overseer is another Claude session with a different instruction set. But it has one hard rule: it does not edit code. It only returns a verdict by writing ISSUES.md or APPROVED. If it can’t justify approval from the spec + diff (and whatever checks it runs), it files issues and hands the baton back to the builder.

Here’s what a tiny ISSUES.md excerpt looks like when the overseer finds a real regression:

## Critical: Regression
- Builder claims the failures are “pre-existing” — false.
- Evidence: `git checkout <base_sha> && uv run pytest cli/test_game_loop.py` passes; after this stream, 10 fail.
- Fix required: remove or replace the production-rate heuristic in `strategic.py`.

Spec → Builder → DONE → Skeptical Overseer → APPROVED | ISSUES.md → Fix loop → Merge

Here’s what I do now:

I start by chatting with Claude to sketch the shape of the solution, then break the work into workstreams.
For each workstream, I have the builder write a spec, ask questions, clear up ambiguities, and make sure we agree on what “done” looks like.
In a second separation-of-duties step, I review that spec with fresh context (a new Claude instance) to prevent drift and agreement bias. This also catches missing requirements long before code review.
I launch an agent-stream: it runs in its own tmux session and git worktree.
The builder works until it thinks it’s done, then creates a DONE marker file.
Once DONE exists, the skeptical overseer reviews the work and either creates an APPROVED marker file (meaning merge) or writes an ISSUES.md file (meaning keep iterating). The default review prompt assumes issues will be found.
The loop repeats until APPROVED is created, at which point the work is merged back to the original branch and the next workstream is automatically launched.

The spec is the leverage point for the whole process. It tells the builder what to build, and it tells the overseer what to verify. The spec is the trust boundary.

This lets me run autonomous, multi-hour coding sessions overnight and come back the next day with merged code and a high degree of confidence that it’s complete and correct.

I’ve built a simple tool called Agent-Streams that implements this “skeptical overseer” pattern. It’s a local orchestration framework for autonomous agent workflows that implements a “spec → build → review → fix → merge” loop. It uses tmux, git worktrees, Prometheus metrics, and streams output for visibility. It’s very simple by design.

github.com/john-b-rush/agent-streams

I think you should build your own version of this, and I also think you probably shouldn’t use mine. The valuable thing isn’t tmux or worktrees or metrics or my directory layout. The valuable thing is the trust boundary: a spec that the builder uses to build, and the overseer verifies against. Agent-Streams is just one way I enabled that boundary in my workflow, and I fully expect the exact tooling to be obsolete once Claude and Codex ship this natively.

Steal the trust boundary: builder ships; overseer verifies; DONE must survive review.

Building a Personal AI Factory (July 2025 snapshot)

2025-07-01T00:00:00+00:00

Building a Personal AI Factory (July 2025 snapshot)

Published: July 1, 2025

Overview

I keep several claude code windows open, each on its own git-worktree. o3 and sonnet 4 create plans, sonnet 3.7 or sonnet 4 execute the plan, and o3 checks the results against the original ask. Any issues found are fed back into the plan template and the code is regenerated. The factory improves itself.

Read on to see what might be useful for you.

Guiding Principle – Fix Inputs, Not Outputs

When something goes wrong, I don’t hand-patch the generated code. I don’t argue with claude. Instead, I adjust the plan, the prompts, or the agent mix so the next run is correct by construction.

If you know Factorio you know it’s all about building a factory that can produce itself. If not, picture a top-down sandbox where conveyor belts and machines endlessly craft parts because the factory must grow. Do the same thing with AI agents: build a factory of agents that can produce code, verify it, and improve themselves over time.

Basic day to day workflow - building the factory

My main interface is claude code. It’s my computer now. I also have a local mcp which runs Goose and o3. Goose only because I’ve already got it setup to use the models hosted in our Azure OpenAI subscription. Looking to improve this at some point, but it works for now.

Step 1: Planning

I’ll give a high level task to claude code, which calls over to o3 to generate a plan. o3 is a good planner and can ask a bunch of good questions to clarify the job to be done. I then have it write out a <task>-plan.md file with both my original ask and an implementation plan.

Step 2: Execution

First, sonnet 4 reads the plan, verifies it, and turns it into a task list. Next claude code execute the plan, either with sonnet 3.7 or sonnet 4 depending on the complexity of the task. Because most of my day-to-day is in clojure I tend to use sonnet 4 to get the parens right. One important instruction is to have claude write commits as it goes for each task step. This way either claude or I can revert to a previous state if something goes wrong.

Step 3: Verification → Feedback into Inputs

Once the code is generated, I have sonnet 4 verify the code against the original plan. Then I have o3 verify the code against the original plan and original ask. o3 is uncompromising. Claude wants to please, so will keep unnecessary backwards compatibility code in place. o3 will call that out and ask for it to be removed. Claude also tends to add “lint ignore flags” to the code which o3 will also call out. Having both models verify the code catches issues and saves me back and forth with claude.

Any issue sonnet 4 or o3 finds gets baked back into the plan template, not fixed inline.

Git worktrees let me open concurrent claude code instances and build multiple features at once. I still merge manually, but I’m no longer babysitting a single agent.

Why Inputs Trump Outputs

Outputs are disposable; plans and prompts compound.
Debugging at the source scales across every future task.
It transforms agents from code printers into self-improving colleagues.

Example: an agent once wrote code that would load an entire CSV into memory. I made it switch to streaming and had the agent write instructions to the plan to always use streaming for CSVs. Now, my plan checker flags any code that doesn’t use streaming for CSVs, and I don’t have to remember this in every PR review. The factory improves itself.

Scaling the factory

I’ve started to encode more complex workflows, where I have specific agents (behind mcps) for building specific tasks.

One MCP will sweep all the clojure code generated and then apply our local style rules. These rules are part of the instructions for the original plan and agent but often the generated code will have style issues. Especially once claude gets in the lint/test/debug cycle. This focused agent means we have tighter behavior and can apply our style rules consistently.

I’ve started doing this for internal libraries as well. It’s good at looking at generated code and replacing things like retries and Thread/sleep with our retry library.

I’m also building out a collection of these small agents. Each one can take a small specific task, and by composing them together I can build more complex workflows. For example, I can take an api doc, and a set of internally defined business cases and have a composition of agents build integrations, tests, and documentation for the api. This is a powerful way to build out features and integrations without having to do all the work by hand.

You don’t get there in one big step. Here’s the secret sauce: iterate the inputs

It’s essentially free to fire off a dozen attempts at a task - so I do. All agents run in parallel. When one fails, stalls, or lacks context, I feed that lesson into the next iteration. I resist the urge to fix outputs, instead I fix the inputs.

That loop is the factory: the code itself is disposable; the instructions and agents are the real asset.

Next up

I’m working on a few things to improve the factory:

Better overall coordination of the agents. I tend to kick things off manually, but I want to have a more automated way to manage the workflow and dependencies between agents.
Aligning our business docs with the agents. Changing the information we capture to be at a higher level of abstraction so that the agents can use it more effectively. This means moving away from low level implementation details and focusing on use cases.
More complex workflows. I’ve been able to build some pretty complex workflows with the current setup, but I want to push it further. This means more agents, more coordination, and more complex interactions between them.
Maximize token usage across providers. I’m pretty limited by bedrock’s token limits especially for sonnet 4. Going to need to be able to switch between the claude max plan and bedrock w/out interruption.

Wrapping Up

That’s where my factory sits today: good enough to ship code while I refill my coffee, not yet good enough to bump me off the payroll. Constraints will shift, but the core principle remains: fix inputs, not outputs.