Post-mortem: silent loss of 57/63 deliberation verdicts

The headline finding: nothing in this code path can report failure. There is no bug that
"broke" persistence — persistence was never observable, so the first available signal was
a derived business metric moving in the favourable direction. That is why it ran for six weeks.

Below, each distinct failure mode. I've marked the two places where I'm inferring beyond what
the snippet shows.

---

## F1. The Supabase client does not reject on database errors

Root cause. supabase.from('verdicts').insert(payload) returns a PostgrestBuilder that resolves to
{ data, error }. A constraint violation, RLS denial, NOT NULL violation, or schema-cache miss
resolves successfully with a populated error field. It does not throw.

Mechanism. try { await Promise.race([op(), ...]) } catch {} can only catch transport-level faults
and the synthetic timeout. Every database-reported failure sails through the await as a successful
resolution. Even if the catch block had been fully instrumented with logging and retries, it would
have fired zero times for the FK violations annotated in the pipeline. The error object was
constructed by PostgREST, returned over the wire, deserialized, and dropped on the floor.

This is the single most important finding: the catch {} is a red herring for the FK class of
failures. The code is blind even before it is silent.

Fix. Never let a PostgREST result cross a function boundary unchecked.

```ts
const { data, error } = await supabase.from('verdicts').insert(payload).select().single();
if (error) throw new PersistError('verdicts.insert', error, payload);
```

Or enforce it globally with .throwOnError() and a lint rule banning bare .insert(/.update( without
.throwOnError() or a destructured error check.

## F2. catch {} — the swallowed exception

Root cause. Error handling designed to satisfy "the request must not crash" rather than "the record
must not be lost."

Mechanism. For the failures that do reject (timeouts, DNS, socket resets, 5xx, 429), the exception
is discarded with no log line, no counter, no trace event, no dead-letter write, and no rethrow.
Worse: because the promise is swallowed rather than rejected, Node's unhandledRejection diagnostic —
the one automatic safety net that would have printed something to stderr — is suppressed. The empty
catch actively removes the last passive detection mechanism.

Fix. The catch block must be load-bearing: structured log with the payload's idempotency key,
verdict_persist_failure_total counter increment with an error-class label, durable dead-letter
write, and rethrow (or return a failure result — see F3). An empty catch on a durability path
should be a CI-blocking lint error (no-empty with allowEmptyCatch: false).

## F3. Promise<void> return type erases the outcome

Root cause. The signature is the defect, independent of the body. persistSafely<T>(op: () =>
PromiseLike<T>): Promise<void> declares that the result of persistence is unrepresentable.
The generic T is accepted and thrown away.

Mechanism. A caller who did everything right — await persistSafely(...) — still cannot branch
on success. There is no correct way to use this API. The type system guarantees the caller is
uninformed.

Fix. Make the outcome part of the type, so ignoring it is a visible choice:

```ts
type PersistResult<T> = { ok: true; data: T } | { ok: false; error: PersistError; payload: unknown };
async function persist<T>(op: () => PromiseLike<T>): Promise<PersistResult<T>>
```

Combine with a linter rule for unused return values on this function.

## F4. Fire-and-forget: the promise is never awaited

Root cause. persistSafely(...) and insertRound(round) are invoked as statements. No await,
no collection into Promise.all.

Mechanism. Two independent losses:
1. Runtime termination. On any serverless/edge host (Vercel, Lambda, Cloudflare Workers),
   returning the HTTP response ends or freezes the execution context. In-flight fetches are
   aborted mid-flight or the microtasks that would issue them never run. The insert vanishes
   with no client-side and no server-side trace.
2. Lazy builder never dispatched. PostgrestBuilder is a lazy thenable — the HTTP request is
   only issued when .then() is called. If insertRound returns the builder without awaiting it,
   no request is ever sent at all. The PromiseLike<T> in the signature is a tell that the
   author knew they were handling thenables, not promises.
   (Uncertain: depends on insertRound's body, which isn't shown. If it awaits internally,
   only mechanism 1 applies.)

Fix. await every persistence call. If work must genuinely outlive the response, use the
platform's explicit primitive (ctx.waitUntil / after()), never a bare un-awaited promise.
Enforce with @typescript-eslint/no-floating-promises.

## F5. forEach discards async callbacks

Root cause. Array.prototype.forEach has return type void; it does not await its callback
and cannot be made to.

Mechanism. All three rounds and every nested response are launched simultaneously with no
sequencing and no backpressure. The enclosing function returns before any of them settle.
This also creates a burst of concurrent connections that can trip pool exhaustion or
PostgREST rate limits — raising failure probability exactly in the window where failures
are invisible.

Fix. for...of with await where ordering matters; await Promise.all(...) where it doesn't.
Batch the responses into a single insert([...]) call rather than N calls.

## F6. Causal ordering violation -> guaranteed FK violation

Root cause. Child rows referencing round_id are inserted before the parent round row is
committed — and, if round_id is DB-generated (identity/serial), response.round_id cannot
hold a valid value at that point under any timing.

Mechanism. Postgres rejects with 23503 foreign_key_violation. Per F1, that arrives as a
resolved { error } and is discarded. Every response row for every debate is lost,
deterministically — the code comment "// FK violation. Silent." is accurate and was
apparently known.

Fix. Either (a) generate UUID primary keys client-side so children can be constructed
before the parent is durable, or (b) do the whole write in one server-side transaction.
(b) is correct here — see F7.

## F7. No transaction: partial writes are representable

Root cause. Verdict, rounds, and responses are independent unrelated INSERTs. There is
no atomic unit corresponding to "a debate."

Mechanism. Any subset can succeed. The database ends up in states that the domain model
says are impossible: a verdict with no rounds, rounds with no responses, orphaned children.
This is the direct enabler of the 57 half-written rows.

Fix. One Postgres function, one round trip, one transaction:

```sql
create function insert_debate(p jsonb) returns uuid
language plpgsql as $$ ... $$;  -- inserts verdict + rounds + responses, or raises
```

Called as await supabase.rpc('insert_debate', { p: payload }).throwOnError(). All-or-nothing;
FK ordering handled inside the transaction; one failure signal.

## F8. The timeout race leaks the operation and the timer

Root cause. Promise.race implements abandonment, not cancellation.

Mechanism. Three separate defects:
- When the 5s timer wins, the insert is still running. It may commit afterwards. The code
  has recorded "failure" for a write that succeeded — so any retry would double-write, and
  any reconciliation is unreliable. A timeout here means unknown, not failed, and the code
  cannot express that.
- setTimeout is never cleared. On the success path a live 5s timer remains, keeping a handle
  on the event loop, delaying process exit and (on serverless) delaying freeze. At scale this
  is a slow leak of timer objects.
- The losing branch's later rejection becomes an unattached rejection once race has settled.

Fix. Real cancellation plus cleanup. Make all writes idempotent (F9) so an ambiguous timeout
can be safely retried.

## F9. No retry, no idempotency key

Root cause. Single-attempt semantics on a network write, with no deduplication mechanism
that would make retries safe.

Mechanism. One transient blip — a Supabase restart, a 429, a socket reset — permanently
destroys an artifact that cost roughly $0.80 of model spend and cannot be recomputed
identically. The lost value is primarily the unrecoverable audit records for a deliberation
system.

Fix. Deterministic debate_id UUID generated at debate start; upsert(payload, { onConflict:
'debate_id' }); bounded exponential backoff with jitter, retrying only classifiable-transient
errors (network, 429, 5xx, 40001 serialization failure). Never retry 23505/23503/22P02 —
those are bugs, and retrying them just burns budget.

## F10. Nullable schema on business-critical columns

Root cause. verdicts.verdict, verdicts.cost, etc. permit NULL. The database — the last
authority that could have refused bad data — was configured to accept it.

Mechanism. This is why the failure manifested as 57 NULL rows rather than 57 missing rows.
The insert reached Postgres and was accepted. Two candidate paths, both fully consistent
with the evidence:
- (a) Undefined fields silently dropped. If any payload field was undefined at serialization
  time — e.g. cost derived from a value that was itself an un-awaited promise, per F4 —
  JSON.stringify omits the key entirely. PostgREST then inserts a row without that column,
  and Postgres fills the default: NULL. No error is generated at any layer. This produces
  exactly "row present, content NULL."
- (b) Two-phase write. An initial skeleton insert followed by an UPDATE ... SET verdict,
  cost that timed out (F8) or failed (F1/F2), leaving the skeleton behind.

(Uncertain: distinguishing (a) from (b) requires the payload-construction code and the
migration history. (a) is more likely given the observed metric arithmetic in F11 — the rows
have a valid id/created_at but NULL cost, which is the signature of key omission rather than
a failed update, though both fit.)

Fix. Push the invariant into the schema, where it cannot be bypassed. Also validate the
payload against a Zod schema before the call, so undefined is a caught type error rather
than an omitted column. After this change the 57 failures become 57 loud 23502 not_null_
violation errors.

## F11. The dashboard metric masked the loss as an improvement

Root cause. Mixing a NULL-skipping aggregate with a NULL-counting one: SUM(cost) / COUNT(*).

Mechanism. SUM ignores NULLs; COUNT(*) does not. With 6 real rows at roughly $0.80 and
57 NULL rows: 4.8 / 63 ≈ $0.076, and the reported $0.088 falls out of the same arithmetic
with slightly different per-verdict costs. The divisor was inflated by the very rows that
represented data loss. The metric therefore reported a roughly 10x cost reduction — a number
nobody investigates, because it looks like an optimization landed. Data loss was rendered as
a win.

Fix. Use AVG(cost_usd), which skips NULLs and would have stayed at roughly $0.80 throughout.
Separately publish a data-quality gauge, COUNT(*) - COUNT(cost_usd), and alert on > 0. Any
dashboard tile computing an average over a nullable column should be treated as a latent incident.

## F12. No reconciliation, no write-confirmation telemetry

Root cause. No metric anywhere counted attempted writes, only observed rows. There was no
independent source of truth to compare against.

Mechanism. Six weeks with no signal. The system had no way to know that 63 debates had run
but 6 had persisted, because "a debate ran" was never recorded independently of "a verdict
row exists."

Fix. Emit verdict_persist_attempt_total / _success_total / _failure_total{error_class}; alert
when success/attempt < 1.0 over any 1h window — for a low-volume, high-value path the correct
SLO is 100%, not 99.9%. Add a nightly reconciliation job comparing debate-start events against
count(*) from verdicts where cost_usd is not null, and page on divergence.

## F13. No durable artifact log: the loss was made irreversible

Root cause. The model output existed only in process memory until the structured DB write.
There was no upstream capture.

Mechanism. Because the raw verdict text was never written anywhere else, a persistence failure
is terminal rather than replayable. This converts a recoverable bug into 57 permanently
destroyed records.

Fix. Transactional outbox / write-ahead artifact: append the raw model response to append-only
durable storage (object store or a local outbox table) before attempting the structured insert,
keyed by debate_id. A replay worker drains it. With this in place, all thirteen other bugs cost
you a backfill script instead of your audit trail.

## F14. The name is the design error

persistSafely defines "safe" as "does not throw." For a durability primitive, safety means
"does not lose data." The function delivers the exact inverse of what its name promises,
which is why callers used it as a fire-and-forget statement — the API told them that was fine.

Fix. Delete it. Replace with persistDebate(payload): Promise<PersistResult<Verdict>> that
throws or returns an explicit failure. If a swallowing variant is genuinely needed somewhere,
name it persistIgnoringErrorsUnsafe and require a justification comment.

---

## Corrected shape

```ts
type PersistResult<T> =
  | { ok: true; data: T }
  | { ok: false; error: PersistError };

async function persist<T>(
  label: string,
  op: (signal: AbortSignal) => PromiseLike<{ data: T | null; error: unknown }>,
  payload: unknown,
): Promise<PersistResult<T>> {
  const MAX = 4;
  let lastErr: unknown;

  for (let attempt = 1; attempt <= MAX; attempt++) {
    const ac = new AbortController();
    const timer = setTimeout(() => ac.abort(), 5_000);
    try {
      const { data, error } = await op(ac.signal);
      if (error) throw new PersistError(label, error, payload);
      metrics.inc('persist_success_total', { label });
      return { ok: true, data: data as T };
    } catch (err) {
      lastErr = err;
      if (!isTransient(err) || attempt === MAX) break;
      await sleep(backoffJitter(attempt));
    } finally {
      clearTimeout(timer);
    }
  }

  const error = PersistError.wrap(label, lastErr, payload);
  logger.error({ err: error, label, payload }, 'persist failed');
  metrics.inc('persist_failure_total', { label, class: error.class });
  await deadLetter.write({ label, payload, error: error.serialize() }); // replayable
  return { ok: false, error };
}

// One atomic call for the whole debate.
const debateId = crypto.randomUUID();
await artifacts.put(debateId, rawModelOutput);          // F13: capture before write

const result = await persist(
  'insert_debate',
  (signal) => supabase.rpc('insert_debate', {
    p: DebateSchema.parse({ debate_id: debateId, verdict, cost_usd, rounds }),
  }).abortSignal(signal),
  { debateId },
);

if (!result.ok) throw result.error;   // fail loudly, upstream decides
```

Plus: NOT NULL + CHECK (cost_usd > 0) + UNIQUE (debate_id) on the table; insert_debate as a
single plpgsql transaction; no-floating-promises and no-empty (catch included) as CI errors;
AVG(cost_usd) on the dashboard with a NULL-count gauge beside it.

---

## Backfill

The 57 rows are not recoverable from the database. Check for salvage in provider-side logs
(OpenAI/Anthropic request history retains completions for a period) and in any application
log that captured responses before the write. Then DELETE FROM verdicts WHERE cost_usd IS
NULL after the NOT NULL migration is staged, so the constraint can actually be applied — and
record the deletion count as the confirmed loss figure.

## Flagged uncertainties

1. F4 mechanism 2 (lazy builder never dispatched) depends on insertRound's implementation,
   which isn't shown.
2. F10 path (a) vs (b) — whether NULLs came from undefined-key omission or a failed
   second-phase UPDATE requires the payload builder and migration history. Both are consistent
   with the evidence; the fix (NOT NULL + pre-validation) closes both.

Everything else — F1, F2, F3, F5, F6, F7, F8, F9, F11, F12, F13 — is determinable from the
snippet and the reported symptoms.

---
Model: anthropic/claude-opus-5
Prompt tokens: 524
Completion tokens: 11,173
Total tokens: 11,697