You Cannot Automate Chaos: The Complete Guide to AI-Powered Dev Pipelines

I look at code all the time. But I would not trust myself to write it by hand anymore — and I would not trust myself to review it alone either.

That is not false modesty. It is the result of a system I have been building and refining over the past year — a system where AI agents do the exploration, AI workflows do the execution, and I stay on the loop. Not in the loop blocking every decision. Not out of the loop hoping for the best. On the loop: guiding, reviewing, course-correcting.

This post is the distilled version of everything I have learned. No theory. Just what works, what breaks, and what you need to fix before any of it matters.

70 Minutes to a Fintech App

Last week, I ran an experiment. I asked Claude Code to plan a fintech application — but not to code it. Just plan it. And I told it to use 16 agents in parallel.

Those agents explored API design patterns, authentication flows, database schemas, KYC compliance, UI frameworks, internationalization approaches, edge cases, and error handling strategies. The output: 47 files of planning documentation — the kind of specification that would take a product team days to align on.

Then I handed that plan to Gemini Flash — not even a frontier model — and said: implement this. 70 minutes later: GlassBank was live. A complete fintech onboarding experience with glassmorphic UI, identity verification, document scanning, biometric selfie capture, animated progress indicators, and PIN creation.

The realization: if the plan is good, even a mediocre executor can ship something impressive.

This is not an isolated result. The industry numbers tell the same story:

The pattern is industry-wide: good planning multiplies agent effectiveness. But that 45% security vulnerability rate means your CI pipeline is non-negotiable. The question is not whether to adopt this workflow. The question is whether your codebase is ready for it.

The Division of Labor: Agents vs Workflows

The first thing most teams get wrong is treating “AI coding” as a single thing. It is not. There are two fundamentally different modes, and confusing them will waste your time.

Agents are explorers. Think of Lewis and Clark mapping uncharted territory. When I point Claude Code at a codebase, it does not follow a script. It reads files, discovers patterns, asks questions, forms hypotheses. It is exploring. That is its strength. You want agents when you do not yet know what you are building or how the existing system works.

Workflows are executors. Think of an assembly line. When Jules — Google’s asynchronous coding agent — takes a ticket, it follows a plan step by step. It writes the validation logic. It creates the tests. It opens the PR. There is no exploration. There is execution.

The mistake is using one where you need the other. Agents are terrible at following rigid plans. Workflows are terrible at discovering what the plan should be. The sweet spot is hybrid: use agents to prototype and plan, then hand the plan to a workflow for implementation.

This is how I work every day. Claude Code (running on Opus 4.6) does the thinking. Jules does the building. And a CI pipeline sits between them and production, catching everything that falls through the cracks.

The Assembly Line

A ticket arrives. Claude Code picks it up and starts exploring: What auth patterns does this codebase use? What does the email infrastructure look like? What is the database schema? It launches multiple agents in parallel to cover more ground faster.

Then it produces a plan — not code, a plan. Files to change. Acceptance criteria. Testing strategy. Edge cases. This is the moment where I step in. I review the plan, not the code. I add the insights that require business context. I approve, and the plan becomes a ticket for Jules.

Jules takes over. It works asynchronously in a sandboxed environment. It writes the implementation, creates the tests, opens a PR. CI runs. If something fails, Jules reads the error logs and fixes the issue autonomously. No human needed for a wrong error code or a missing import.

Then Claude Code comes back — this time as a reviewer, not a planner. Running on Opus 4.6, it does a line-by-line code review. In my experience, it catches real, substantive issues more than half the time. Not style nits. Real bugs. Security holes. Logic errors.

If everything passes — CI green, review approved — it merges. If not, Jules iterates until it does.

Microsoft reports that 20-30% of their code is now AI-generated. Google says 25-30%. Those numbers will only go up. The question is not whether to adopt this workflow. The question is whether your codebase is ready for it.

You Cannot Automate Chaos

This is the line I want taped to every monitor in every startup.

You cannot automate chaos.

If your CI takes 45 minutes, agents sit idle for 45 minutes on every iteration. That is not a productivity gain. That is paying for cloud compute to stare at a progress bar. Agents iterate. Fast feedback means more iterations means better results. Slow CI means wasted compute and frustrated humans waiting for the loop to close.

If your tests flake randomly, agents will chase phantom bugs. They will spend hours trying to fix something that is not broken. You are, quite literally, gaslighting your AI. The agent sees a test fail, assumes it introduced a regression, and starts changing code that was perfectly fine. This is not hypothetical. I have watched it happen.

If your business logic has no test coverage, it does not exist as far as the agent is concerned. An agent cannot verify that it preserved behavior if there is no test defining that behavior. Untested features are invisible features. The agent will refactor right through them without a second thought.

The Automation-Ready Checklist

Before you add a single AI agent to your workflow, audit this list:

1. CI must run in under 10 minutes. Shard your tests. Cache aggressively. Run E2E against a dev server, not a production build. Every minute you shave off your CI is a minute saved on every single agent iteration, compounded across every ticket, every day. This is not a nice-to-have — it is the single highest-leverage investment you can make. A 10-minute CI loop means agents iterate 4-6 times per hour. A 45-minute loop means once. The productivity difference is not linear; it is exponential, because each iteration builds on the results of the last.

2. Zero flaky tests. Fix them or delete them. There is no middle ground. A flaky test is worse than no test when agents are involved, because it introduces false signal into the feedback loop. The agent cannot distinguish between “I broke something” and “this test is unreliable.” So it assumes the worst and starts making changes. One flaky test can send an agent down a 30-minute rabbit hole of “fixes” to code that was perfectly fine. Multiply that by every ticket, every day, and you are hemorrhaging compute and human review time.

3. 100% coverage on business logic. Not vanity coverage. Not padding with trivial assertions. Real coverage on the code paths that matter. When an agent refactors your transfer service, the tests are the contract that ensures the daily limit still works, the exchange rate still applies, the error codes still match. Without that contract, the agent has no way to verify it preserved behavior. It is flying blind, and so are you. Coverage is not a metric to satisfy your CI badge — it is the specification that makes autonomous refactoring safe.

4. TypeScript strict mode. This is level zero of the test pyramid. It is not technically a test, but it might be the most important check in your pipeline. Claude Code integrates with the TypeScript Language Server. It sees type errors in real time as it writes code. Strict mode means the agent gets instant feedback on every function signature, every interface contract, every null check. If you are not on strict mode, that is your first task. The cost of migration is measured in days; the cost of not migrating is measured in every bug that the type system would have caught.

The Test Pyramid for Agents

I think about testing differently now. The traditional test pyramid still applies, but each layer has a new purpose when agents are writing the code.

Level 0: TypeScript strict mode. Real-time type checking that catches errors as code is written. This is the fastest feedback loop — the agent does not need to wait for CI. It knows immediately when a type is wrong. Think of it as a co-pilot for the co-pilot: the type system corrects the agent before the agent even finishes writing the line. No other check in your pipeline provides feedback this fast.

Level 1: Unit tests. These verify intent. When agents refactor code, unit tests ensure that requirements are not accidentally removed. They are the documentation of what the system is supposed to do — not how it does it, but what it does. For an AI agent, unit tests are the acceptance criteria made executable. If a unit test passes, the agent knows it preserved the behavior that matters. If it fails, the agent knows exactly which requirement it violated. Without unit tests, the agent is guessing.

Level 2: Human-readable E2E tests. I write these in Given/When/Then format. “Given I am logged in as an admin, When I visit the agents dashboard, Then I should see status overview cards.” This is living documentation. When a test fails, anyone — human or AI — knows exactly which user capability broke. The agent does not need to reverse-engineer what the test was checking. E2E tests are the final safety net: they verify that the whole system works together, not just individual units. They catch integration bugs that unit tests miss.

Level 3: Smoke tests. Daily health checks against production. Simple, reliable, and they auto-create issues when they fail. When something breaks in production, you want to know immediately, not when a user files a complaint. Smoke tests close the loop — they verify that what passed in CI also works in the real environment. For an autonomous workflow, they are the trigger that starts the next cycle: production breaks, smoke test fails, issue is created, agent picks it up.

Each layer catches a different class of error. Together, they form the guardrail system that makes autonomous agents safe.

A Day in the Life: From Production Report to Merged Fix

Let me walk you through a realistic production workflow. Not simplified. Not idealized. The kind of flow that happens regularly, including the part that makes it actually work.

00:00 — Auto Issue Detection (0 min human time). A monitoring alert fires. A GitHub Action picks up the alert, auto-creates an issue with the error logs, stack traces, and affected endpoints. Before the issue is published, another action validates the content — rephrasing for clarity, checking for prompt injection in user-submitted data that made it into the logs, and ensuring the issue contains enough context for investigation.

00:01 — Investigation Agent (autonomous). Claude Code picks up the issue and starts investigating. It reads the error logs, traces the call stack, examines the affected code paths. Every question it has — “What does this middleware do?” “When was this function last changed?” “Are there related tests?” — spawns a background sub-agent to answer it. Those sub-agents can have questions of their own. The process is recursive: agents ask questions, spawn sub-agents, which ask their own questions, spawning more sub-agents, until every question is answered without raising new ones. No human is asked anything.

00:20 — Development Plan (~15 min autonomous). A single planning instance takes the fully-resolved investigation and writes a development plan. Files to change. Acceptance criteria. Testing strategy. Edge cases. If it has architectural questions — “Should we add a new error type or reuse an existing one?” — it resolves them by consulting codebase patterns and conventions. The output is a plan so thorough that the coding agent will have zero questions.

00:35 — Coding (20-30 min autonomous). The plan is so complete that the coding agent has nothing to ask. It executes: writes the fix, adds or updates tests, ensures type safety. Pure implementation, no exploration. This is the assembly-line phase — the hard thinking was done in investigation and planning.

00:55 — CI Validation (5 min). Tests, types, linting, coverage. If anything fails, the coding agent reads the error output, fixes the issue, and pushes again. One more iteration at most.

~01:00 — Code merged. Under one hour from alert to merged fix. Human time: approximately 15 minutes — reviewing the plan and the final PR.

The key insight is recursive question-answering. Traditional agents ask the human when they get stuck. This system asks itself — deeper and deeper — until no questions remain. That is what makes sub-one-hour autonomous fixes possible.

A Concrete Example: From Ticket to Merged PR

Let me trace a second scenario — a multi-step feature, not a bug fix — to show where agents excel and where humans intervene.

Ticket: “Add password reset functionality with email verification”

9:00 AM — Planning (Claude Code). I tell it to plan the feature using 8 agents. Two agents explore existing auth flows. Two explore email service configuration and template patterns. Two research security requirements — token expiry, rate limiting, OWASP guidelines. Two map the database schema for reset tokens and the existing user model. Output: a 12-file implementation plan with acceptance criteria.

9:45 AM — Human Review. I review the plan. Notice the agents missed something: “What if a user requests reset for a non-existent email?” I add to acceptance criteria: “Return same response for existing and non-existing emails to prevent enumeration.”

10:00 AM — Implementation (Jules). I hand the plan to Jules. It creates the API routes, email templates, validation schemas, unit tests, and an E2E test for the happy path.

10:30 AM — CI Fails.

FAIL src/app/api/auth/reset-confirm/route.test.ts
  ✕ rejects expired tokens (4ms)
    Expected: TOKEN_EXPIRED
    Received: INVALID_TOKEN

Jules reads the CI logs, identifies the wrong error code, fixes it.

10:45 AM — Code Review (Claude Code Opus). Review catches three issues: missing rate limiting on the reset-request endpoint, token stored in plain text (should be hashed), and no audit log for password changes. Jules addresses the first two. The third requires an architectural decision.

11:00 AM — Human Intervention Required. Jules asks: “Audit logging requires choosing between: (1) Add to existing logging table, (2) Create dedicated audit_events table, (3) Use external service.” I choose option 2. Jules continues.

11:45 AM — All CI checks pass. Claude Code approves. I do a 5-minute spot check. 12:00 PM — Merge.

Total human time: approximately 45 minutes of review and decisions. Total elapsed time: 3 hours. Traditional estimate: 2-3 days.

The agent got stuck exactly once — on an architectural decision that required business context. Everything else was automated.

The Risks You Cannot Ignore

I would be lying if I told you this is all upside. It is not.

45% of AI-generated code has security vulnerabilities. That is from Veracode’s 2025 report, and it is worth noting the figure comes from intentionally vulnerability-prone benchmark tasks — real-world rates will vary depending on context and guardrails. Prompt injection is the most common vulnerability in LLM applications according to the OWASP Top 10 for LLMs. Code review by a stronger model plus human oversight is not optional. It is the bare minimum.

Multi-agent coordination breaks down at scale. In my experience, more than four agents working on the same problem starts producing diminishing returns — though this is task-dependent and will likely improve as tooling matures. Agents work at cross-purposes. They overwrite each other’s changes. They introduce contradictions. Clear task boundaries and well-defined interfaces between agent responsibilities are your defense.

AI loves to over-engineer. Left unsupervised, agents will create abstraction layers you did not ask for, utility functions nobody needs, and architectural patterns that solve problems you do not have. I call it code bloat. Clear acceptance criteria and iteration limits are your defense. The fix is simple: be explicit about what “done” looks like, and tell the agent not to add anything beyond the requirements.

The mitigation is always the same: a trustworthy CI pipeline, code review by a stronger model, and a human who understands the system well enough to recognize when the agent is wrong.

Human on the Loop

The job is not about writing code anymore. But it is also not about blindly trusting AI to write it for you. It is about staying on the loop — close enough to course-correct, far enough to not block progress.

There is an important distinction here. I am not blocking the AI from getting better. I let it improve, and it helps me improve in return. It is a feedback loop, not a bottleneck. Just because I cannot play chess at grandmaster level does not mean I cannot build a system that teaches itself to play chess. The same principle applies to code: my value is not in writing perfect syntax. It is in designing the system that produces correct software.

Anthropic calls this context engineering — and it is the skill that matters most in 2026. Context engineering means: What files does the agent need access to? What acceptance criteria define “done”? What guardrails prevent the agent from going off the rails? How do you optimize the context window so the agent sees the 5 files that matter, not the 5,000 that do not?

This is where the “human on the loop” model proves itself. You are not making every micro-decision — you are making the macro-decisions that shape the agent’s effectiveness. Choosing which tests to write, which architectural patterns to enforce, which files to include in context. When I review a plan, I am not checking syntax. I am checking whether the agent understood the business constraint that is not written down anywhere — the one that lives in my head because I have been working with this system for months.

The prerequisite is non-negotiable: you must understand the system you are automating. If you do not understand how the basket architecture works, you will not recognize when the agent makes wrong assumptions about it. If you do not understand your auth flow, you will not catch the enumeration vulnerability in the password reset plan. If you do not understand the system, you will not recognize where the agent is lying to you.

I look at code every day. But I understand every system I automate. That is what makes this work.

Two Roles, Two Futures

I see two developer archetypes emerging.

The coder turns specs into syntax. This role is disappearing. Not because coders are bad — because the translation from “what to build” to “how to build it” is exactly what AI does best. If your value proposition is typing speed and syntax knowledge, you are competing with something that never sleeps and never forgets an API signature. The coder asks “how do I implement this?” The answer is increasingly: you do not. The agent does.

The software engineer solves problems at the systems level. This role is becoming more powerful. Understanding distributed systems, security implications, business context, architectural tradeoffs — these skills are amplified by AI, not replaced by it. The engineer asks “what should we build, and why?” — then designs the constraints, the tests, and the acceptance criteria that let agents build it correctly. The engineer who can design the right system and articulate what it should do can now ship ten times faster than before.

Value creation has decoupled from the manual labor of coding. The most productive developer I know has not written a function by hand in months. He writes requirements, reviews plans, and verifies outcomes. His output is enormous.

Start Here

If you are a CTO or engineering lead reading this, do not start by evaluating AI models. Start by evaluating your pipeline.

Step 1: Measure your CI pipeline time. If it is over 10 minutes, that is your first project. Shard tests. Add caching. Run E2E against dev servers. Every minute matters when agents are iterating. Use time on your CI runs for a week, find the bottleneck, and fix it. This will pay for itself within the first month.

Step 2: Count your flaky tests. Every flaky test is a lie in your feedback loop. Fix them or delete them. No exceptions. Agents will chase phantom bugs until you remove the false signals. Run your test suite 10 times in a row. If any test fails inconsistently, quarantine it and fix or remove it before proceeding.

Step 3: Check your TypeScript configuration. Are you running strict mode? If not, you are missing the fastest feedback loop available. This is a weekend project with massive payoff. Start with strict: true in your tsconfig, fix the errors that surface, and never look back.

Step 4: Audit your test coverage on business logic. Not overall coverage — coverage on the code paths that matter. If your transfer service, your auth flow, your payment processing are not at 100%, agents cannot safely refactor them. Use your coverage tool’s per-file report, not the aggregate number. The aggregate lies.

Step 5: Then — and only then — consider AI-assisted workflows. Add Claude Code for planning and review. Add Jules for implementation. Let the pipeline be the judge. Start with one ticket. Measure the result. Scale from there.

And if you want to see what this looks like in practice — spike.land is an open-source, AI-powered development platform. The entire source code is on GitHub. Clone it. Run it locally. See recursive agent workflows, fast CI, and 100% test coverage in action. Any agent can rewrite a component into Angular, Vue, whatever you need.

The foundation is not the AI. The foundation is engineering discipline. Get that right, and the AI becomes a force multiplier. Get it wrong, and you are just automating chaos.

And you cannot automate chaos.

Frequently Asked Questions

Can AI agents write production-ready code?

Yes, with caveats. AI agents write code that passes your CI pipeline — which means it is as production-ready as your tests require. If you have comprehensive tests, type checking, and security scans, the code that emerges is production-ready. If your CI is weak, the code quality reflects that.

How do AI coding agents handle code reviews?

Claude Code with Opus performs line-by-line reviews, checking for security issues, performance problems, code quality, and test coverage. Unlike human reviewers, it does not get fatigued by large PRs and applies consistent standards. When it finds issues, it can tag Jules to fix them automatically.

Will AI replace developers?

AI replaces tasks, not roles. Developers who spend 84% of their time on non-coding tasks now have those tasks automated. What remains is the work that requires human judgment: understanding problems, defining requirements, verifying solutions, and deciding what to build.

How do I handle AI mistakes?

The same way you handle human mistakes: with tests, code review, and CI. The question is not “will AI make mistakes?” (yes). The question is “does your workflow catch mistakes before production?” If your CI is trustworthy, mistakes get caught regardless of who made them.

This article was distilled from a podcast deep-dive conversation about AI-assisted software development. The insights were refined through that discussion, and this post captures the practical lessons in written form.