MyKE / AI SDLC road to Voyscout - the harness behind a vibe-coded product

Created Thu, 17 Sep 2026 19:00:00 +0000
2078 Words
ai sdlc ai agents claude code vibe coding local llm voyscout monitoring netdata github actions sre

AI SDLC road to Voyscout - the harness behind a vibe-coded product

This summer I built Voyscout, a flight and holiday price tracker. It is my first bigger side project after a long time, and it is primarily vibe-coded: AI agents wrote almost all of the code. I described what I wanted, reviewed, and made the decisions.

Vibe-coded projects have a reputation for falling apart quickly. What kept this one stable was not the model. It was everything around the model: the software development life cycle. I work as an SRE, so I treated the agents like a new team. A team needs rules, gates, runbooks, monitoring and a way to escalate. Agents need the same.

This post is a look under the lid. Principles, not secrets.

Some numbers

The repository was created on June 2, 2026. As of September 17, 2026:

  • 2,933 merged pull requests: about 620 features, 680 fixes and 1,000 documentation PRs (build logs, decisions, runbooks), the rest tests, CI and ops
  • 73 GitHub Actions workflows, 45 of them scheduled (data collection, canaries, quality checks, deploys)
  • 393 Python test files and 83 JavaScript test files, including end-to-end suites for the web app
  • 22 codified agent skills
  • a Python runtime with zero third-party dependencies, and a web app on Cloudflare Workers and D1

Agents wrote and shipped nearly all of it. Today the project mostly lives its own life and I manage it from a distance.

The big picture

                 owner: decisions, thresholds, veto
                                |
      +-------------------------+-------------------------+
      |                         |                         |
 implementer                 monitor                 supervisor       <- agent seats
 (ships PRs)             (health ticks)         (reviews the others)
      |                         |                         |
      +---------- skills / hooks / allowlists ------------+           <- harness
                                |
   gates: lint, tests, e2e, review ladder (local model first)         <- CI
                                |
       product: deterministic pipeline, no LLM inside                 <- prod
                                |
   sensors: canaries, data quality, security hygiene                  <- monitoring
                                |
        product alerts -> ops channel
        host + agent health -> Netdata + watchdog -> separate channel

1. Keep the AI out of the product’s decision path

Voyscout tells people BUY or WAIT. There is no LLM in that path. The daily pipeline is deterministic, so every verdict is reproducible and auditable. An early version let a model pick the deals. I killed it.

AI builds and operates the software. It does not invent the numbers.

2. Write the rules down, then make them mechanical

  • Invariants - a registry of “honesty rules” the product must never break. A price that was not verified is never shown as verified. An unknown cost is never treated as free. Fail loud, silence is the enemy.
  • Decisions - a log of owner decisions that bind the code. Agents do not re-litigate them.
  • Docs with a mechanical twin - when a doc table and the code drift apart, a test fails.

The biggest lesson: a rule written in prose one hop away does not arrive while the agent is typing. When the same mistake repeats, the rule becomes a tool or a hook. One example is a hook that refuses a command which would cut off a test verdict, like pytest | tail -5. The agent gets the fix in the refusal message.

3. A harness, not a chat window

The agents run Claude Code inside a dev container, never directly on the host. If you want a sandbox for your own agents, start with my Claude Code Sandbox: a Dev Container template with an allowlist-only proxy, so an agent with full shell access can reach only the domains you allow.

On top of the container, the repository carries its own harness:

  • Skills - codified procedures: ship a PR, promote a change from local to dev to prod, triage an incident, run a health check, add a data provider.
  • Hooks - guardrails that run before a tool call and can refuse it.
  • Allowlists - read-only and idempotent tools are pre-approved, so an unattended session does not sit on a permission dialog at 3 AM. Anything that writes to production is pinned out by a test. The pressure on an allowlist only goes one way, so a test has to argue the other side.
  • Vendor-neutral procedures - the important ones are plain markdown. Any agent from any vendor, or a human, runs the same steps.

4. Seats, not one super-agent

Long-running agent sessions (“seats”), each in its own git worktree with its own loop. The three core roles:

  • Implementer - builds and ships.
  • Monitor - runs a health tick every few hours. It is expected to act, not just report.
  • Supervisor - read-only over the other two. It looks for red runs nobody escalated, issues closed without evidence, merges that skipped review, and agents going in circles.

On top of that, a periodic audit by the strongest available model. It works in an isolated clone and adversarially verifies what the agents reported about their own work: promises versus delivery.

5. A local model in the loop

Recently I started mixing local inference into the pipeline: a Qwen3.8 27B model served by llama.cpp on an RTX 3090 at home. It has two jobs.

First reader in the review ladder. The local model reads every gated diff first. A model from a different vendor is the next rung, and a frontier model judges only what was found. The reason is cost. A full frontier review reads roughly 120K tokens per PR. Verifying one finding costs about 1K. With dozens of PRs a day, that decides whether the whole thing fits into a subscription.

Digesting big raw output. A 766 kB CI log (about 68K tokens) came back as a 349 token summary in 83 seconds, with zero cloud quota spent and nothing dumped into a long-lived agent context.

The rules around it:

  • Fail closed - only a literal NO FINDINGS counts as clean. No verdict means not clean, and the diff goes to the next rung.
  • Never wait on it - the box is not 24/7. If it is down, the ladder falls back automatically.
  • Findings can be false - each one is verified against the code, then fixed or refuted with evidence. Cheap does not mean lenient.
  • Keep a ledger - every call is logged (who, which model, purpose, seconds, tokens, outcome). Availability and usage are two different questions. The box was “up” for a whole day while almost nothing used it, and only the ledger showed that.

6. The same gates for agents as for humans

Protected main branch, squash merges, required checks. Lint, unit tests, end-to-end tests. Every fix ships with the regression test that would have caught it. Secret scanning runs at commit time, before anything reaches GitHub. Green checks do not mean mergeable until the review is addressed. Nobody softens a gate to get a merge. Production changes go local, then dev, then prod, with a verification step per stage.

7. Autonomy is earned in stages

An ops triage agent reads the sensors every morning, writes a briefing and prepares draft PRs for known recipes. It still runs in shadow mode, and its guardrails are structural, not just a prompt:

  • minimal token permissions - it cannot close issues or dispatch workflows
  • a tool allowlist
  • a push wrapper that takes no arguments and can only push its own branch

Its draft PRs are merged by a human or by a dead man’s switch: a 48 hour veto window, and a hold label stops it. The kill switch is disabling one workflow. The sensors lose nothing, because the agent is only a consumer of them.

8. Monitor the product with boring sensors

Deterministic checks come first: canaries, data quality checks, a consistency differ between what the web shows and what the mail says, mail observability, a watchdog that recovers a dropped scheduled run, and a daily black-box security check (SPF and DMARC, security headers, TLS, anonymous access stays denied).

An assurance matrix answers four questions for every core function: what watches it, how do we learn about a failure, does it heal itself, and where is the fix path. Alerts go to one ops channel. A runbook index maps every alert signature to a three-line triage.

A green Actions feed is not the same thing as a healthy system.

9. Monitor the harness too

Agents are long-running processes on a dedicated machine, and they die quietly. A session crashes. A login expires. A scheduled loop silently times out. The context fills up. A rate limit window closes. Or the worst one: the agent is alive, ticking, answering, and delivering nothing.

So there is a small watchdog: plain bash under launchd, every five minutes. No AI, no API calls. A monitor that runs inside an agent session cannot report that the session died.

It checks that each seat exists and an agent is really running in it, that the loop actually fires (it measures the effect, not the config), context usage, rate limit windows, token refresh, and whether the main branch is moving at all. Plus the host: reboot, disk, thermals, swap, load. It also probes the GPU box: health, temperature, and whether calls actually complete.

Netdata watches the host. A small custom plugin feeds the watchdog’s signals into it as charts with health alarms, because Netdata knows CPU and disk but has no idea what an agent loop is. Host and agent alerts go to a separate channel from product alerts. A host alarm never explains a product symptom, so the two must not mix.

The alerting rules are SRE classics:

  • confirm a finding twice before paging
  • cooldown with exponential backoff - a critical alert nobody answered for 8 hours does not need a ninth copy overnight
  • recovery with hysteresis, so a flapping detector stays quiet
  • never alert on a fact, only on a fault - a GPU box that is switched off on purpose is not an incident
  • one daily digest of everything that stayed quiet - quiet does not mean invisible
  • a self-test that proves the detectors can still fire

10. Memory lives in the repo

One build log file per day with a handover contract between sessions. Knowledge notes. An orientation map, so a fresh agent cold-starts from about seven files. Context is the real cost, so tokens and CI minutes are measured and budgeted like any other resource.

What went wrong along the way

It was not a straight road. A few failures that shaped the rules above:

  • The silent review skip. The review gate was “skip, do not block” when the reviewer was unavailable. Its login expired, reviews were skipped, and PRs merged unreviewed. Nothing stopped, it only went to a log nobody reads. Now an uncovered merge files its own follow-up issue, and the watchdog tracks reviewer logins.
  • Alive but not delivering. After 25 PRs in 20 hours one session drifted into “all green, holding” and stayed there. Every liveness check saw a healthy agent. The only signal that tells the difference is movement on the main branch.
  • Too much monitoring. Hourly health ticks produced zero findings in 64 quiet runs. The cadence went to every 2 hours, then every 4. Measure the monitor too.
  • Alert noise. One week of watchdog alerts was 329 messages, and 145 of them were recoveries, most within 10 minutes. That is where confirmation runs, hysteresis and the quiet digest came from.
  • Context rot. A long session re-reads hundreds of thousands of tokens on every turn and gets worse at following its own rules. Sessions rotate, and handovers are written down.

What is left for me

Decisions, thresholds, anything outward-facing, and the veto. I read the morning briefing, answer the few questions only an owner can answer, and sometimes put a hold label on a PR.

Takeaways

  1. Vibe coding scales when the engineering moves from the code into the process around it.
  2. Keep the model out of the product’s critical path whenever a deterministic pipeline can do the job.
  3. Prose rules decay. Turn repeated mistakes into hooks, tools and tests.
  4. Give agents roles and let one of them watch the others.
  5. Guardrails must be structural: permissions, allowlists, wrappers. A prompt is not a fence.
  6. A local model is a good first reader and log digester, as long as it fails closed and its usage is measured.
  7. Monitor the product, and monitor the agents with something that is not an agent.

The result is running at voyscout.com.