How Fire-and-Forget Writes Lost 57 Verdicts

In brief

Fire-and-forget database writes silently lost 57 of 63 deliberation verdicts over six weeks. The root cause was three interacting failures: Supabase .insert() returns errors without throwing, Promise.race 5s timeouts killed persistence mid-flight, and foreign key inserts ran before parent rows existed. The detection came from a governance dashboard showing an impossible $0.088 average cost. Three lines fixed all three failure modes. One gate now prevents it from ever happening again.

The governance dashboard showed $0.088. Average deliberation cost. For 63 debates.

The real number was closer to $0.80. Fifty-seven verdicts in the database had NULL in the actual_cost column. No errors. No crash reports. No missing data alerts. Just six weeks of silently dropped persistence — discovered only because a dashboard number was too wrong to ignore.

Cost table showing NULL actual_cost across 57 of 63 deliberation records

Key Takeaways

Fire-and-forget database writes silently lost 57 of 63 deliberation verdicts over six weeks of production operation
Three interacting failures: Promise.race 5s timeouts, FK cascade violations from unordered inserts, and Supabase .catch() that never fired on PromiseLike returns
Detection came from a governance dashboard showing an impossible $0.088 average cost — not from error logs, tests, or user reports
The fix was three lines: error destructuring, 10s timeout, and ordered parent-before-child inserts. One pre-code gate check now prevents it mechanically.

What was the code actually doing?

Every debate followed the same path. Models ran. Verdicts generated. Then a persistSafely() wrapper fired off the Supabase insert.

// The Before — three failure modes, one function
async function persistSafely<T>(op: () => PromiseLike<T>): Promise<void> {
  try {
    await Promise.race([
      op(),
      new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), 5000))
    ]);
  } catch {
    // Silent. No retry. No alert. No audit.
  }
}

Three things wrong. Each killed verdicts on its own. Together, they lost 90% of persistence.

Why did `.catch()` silently swallow every failure?

Supabase .insert() returns { data, error }. It does not throw. If you don't destructure error, you never see the failure.

The code was chaining .catch() directly on the call. But Supabase returns a PromiseLike, not a full Promise. The .catch() never attached. The error object sat in the resolved value, unread, while the catch handler waited for a rejection that never came.

57 verdicts. Zero error logs.

// The Hardened State
const { data, error } = await supabase.from('verdicts').insert(payload);
if (error) throw error;

One line. Destructure. Check. That's it.

Why did the timeout kill verdicts mid-flight?

The Promise.race had a 5-second timeout. Most persist operations completed in under a second. But verdict inserts — large JSON with claims, confidence scores, dissent, and sources — sometimes exceeded five seconds under load.

The race resolved. Timeout fired. persistSafely caught the timeout error and returned. But the Supabase insert continued in the background. It eventually succeeded. The verdict WAS in the database.

The catch handler marked it as failed anyway.

Browser showed: "1 persistence operation failed." Database showed: perfectly valid verdict. The only thing broken was the user's confidence that the system worked.

Fix: 10 seconds. No Promise.race for fire-and-forget. If a timeout is needed, make it generous and check the result after, not race against it.

Why did foreign key inserts silently fail?

Round inserts fired without await. Response inserts referenced round_id before the round row existed.

// Three rounds, zero awaits
[round1, round2, round3].forEach(round => {
  insertRound(round); // Fire. Forget.
  round.responses.forEach(response => {
    insertResponse(response.round_id, response); // FK violation. Silent.
  });
});

The database rejected every response insert. Foreign key constraint: the parent row didn't exist yet. No error surfaced. The .catch() on insertResponse was the same broken pattern as the verdict path.

Fix: await parent inserts before children. Dependency order is not optional.

How was it caught?

Not by tests. Not by error monitoring. Not by a user complaint.

The governance dashboard showed average deliberation cost: $0.088.

That number is impossible. The cheapest model in the council costs more than that per token. A single debate should average $0.80-$1.00. The dashboard was computing the mean of 6 valid rows and 57 NULLs — dividing by 63, not by 6.

A manual database query confirmed it. SELECT count(*) FROM verdicts WHERE actual_cost IS NULL. Fifty-seven rows. Six weeks of silently dropped data. The code had been writing to actual_cost for two months. No migration ever created the column. It was ghost-writing into the void.

The code was writing to a column that didn't exist. Supabase returned { error } but the fire-and-forget wrapper dropped it. The persistSafely timeout fired before the full verdict JSON finished writing for large debates. And the round-before-debate FK cascade blocked every response insert.

Three failures. One discovery moment. A number that was too wrong to ignore.

What we'd build differently

The transactional outbox pattern exists for exactly this failure mode. Write the event to a table in the same transaction as the business write. A separate process reads pending events and dispatches them. If the process crashes between the database write and the dispatch — the event is already persisted. Nothing is lost. The same failure mode cost Hypothesis 21,500 annotations when their RabbitMQ queue silently dropped indexing jobs. Supabase explicitly warns that .insert() returns { data, error } without throwing — the error must be destructured, never assumed.

That's the architectural maturity milestone this represents. Log #1 described the adversarial pipeline. Log #2 documents what broke in the infrastructure layer. The outbox pattern is where we're headed — but it requires new infrastructure, and this project is a solo operation. The three-line fix closed every crash window for now.

The gate that now exists

This failure added a check to the pre-code gate pipeline. It verifies that actual_cost columns are populated before build clearance. It runs on every commit. It catches the NULL pattern before a deployment leaves the machine.

The code was fixed in three lines. The detection came from a dashboard number that was too wrong to ignore. The prevention is mechanical now — grep-verifiable, not self-reported.

The same number that caught the data loss is now the signal that drives the execution gates. Log #4 shows how three yield points in the engine now guard every debate — pre-flight projection, round boundary, and quorum detection. The circuit breaker that stops a runaway model is the same principle that caught the 57 lost verdicts. Catch it before it compounds.

Fire-and-forget is not a pattern. It's a bet. And the house always wins.

FAQ

What does fire-and-forget mean in database terms?

Calling a write operation without awaiting its result or checking for errors. The code moves on immediately — if the write fails, nothing catches it.

Why did Promise.race cause data loss?

A 5-second timeout wrapped around persistence calls. When Supabase took longer than 5s for large verdicts, the race resolved with a timeout error before the actual insert completed. The catch handler marked the verdict as failed, but the write succeeded silently.

How was this caught?

The governance dashboard showed an average deliberation cost of $0.088 — a number so impossibly low it triggered a full audit. Manual database queries revealed 57 NULL rows.

What prevents this from happening again?

Three code fixes: error destructuring on every Supabase mutation, 10-second timeout instead of 5, and ordered parent-before-child inserts. Plus a pre-code gate check that verifies actual_cost columns are populated before build clearance.