The Hard Part of AI Engineering: APIs, Retries and Idempotency

The model succeeded. The request failed.

One of the most expensive failure modes in an AI product is surprisingly ordinary: the backend accepts a generation request, the external model starts doing work, the client never receives confirmation, and the user retries. From the browser, nothing happened. From the provider's perspective, a billable operation may already be underway.

Building MailTrace and Instant Deck has forced me to think about reliability at the boundaries between HTTP, persistence, background workers and third-party APIs. The interesting questions are not whether I can issue a POST request or receive a webhook. They are what each successful response actually guarantees, how operations are identified, and what happens when the result of an external side effect is unknown.

MailTrace: an API endpoint is not a complete user journey

MailTrace has a Svelte frontend and a Python/FastAPI backend that coordinates email-connected workflows, ownership-scoped records and AI-assisted processing. A timeline view, for example, needs more than a GET request returning an array: it needs authentication, tenant and case ownership checks, stable ordering, consistent resource identifiers, and error responses that the UI can distinguish from genuinely empty data.

I have seen the difference between backend capabilities that exist and frontend journeys that are properly wired to those capabilities. A 200 response does not prove that the right user received the right records, and a rendered component does not prove that its API contract is honoured.

For a senior engineer, the design question is where the contract is specified and tested. For a founder, the question is whether a customer can complete the journey from end to end.

GET /api/cases/{case_id}/timeline
Authorization: Bearer <access_token>

200 OK
{ "events": [...], "next_cursor": null }

403 Forbidden  // no access
404 Not Found  // missing or intentionally concealed resource

Webhooks: delivery is not execution

In an event-driven integration, a third-party system may POST a notification after a call finishes or a message changes. The receiver should verify the sender, validate the event schema, persist an event identity, acknowledge delivery promptly and execute downstream work separately.

The key complication is that delivery can be repeated. A webhook endpoint that creates a CRM task directly on every request risks duplicate side effects. A database-level unique event key, transactional inbox record and retryable worker make the delivery boundary easier to reason about. Acknowledging an event is not the same as successfully completing its associated business workflow.

This is an engineering pattern relevant to MailTrace's communications workflows. It is not a claim that all of MailTrace's provider integrations currently implement this exact webhook architecture.

POST /integrations/events
  → verify signature and timestamp
  → validate event payload
  → INSERT event_id UNIQUE (on conflict: existing)
  → acknowledge receipt

Background worker
  → claim unprocessed event
  → run the domain action
  → mark outcome / retry / investigate

Instant Deck: protect user intent across retries

Instant Deck coordinates extraction, analysis, generation, design and rendering. These are not cheap, atomic operations. If a user clicks Generate twice while the first operation is still running, creating two independent jobs could waste provider spend and produce competing versions.

The repository documents a concrete operation-idempotency contract: (deck_id, user_id, idempotency_key). Its resolve_operation logic is designed to return or replay a matching operation rather than blindly scheduling another. The frontend is expected to reuse the same intent key while the command is active. Backing this with a uniqueness constraint and locking matters because two requests can race before either sees the other.

This is not exactly-once execution across the entire pipeline. It is a mechanism for controlling duplicate submission at a particular boundary. The project's recovery audit also records that double-click acceptance in production still needs proof.

def submit_generation(deck_id, user_id, idempotency_key):
    # Unique: (deck_id, user_id, idempotency_key)
    operation = resolve_operation(
        deck_id=deck_id,
        user_id=user_id,
        idempotency_key=idempotency_key,
    )
    return {"operation_id": operation.id, "status": operation.status}

The harder problem: unknown provider outcomes

A local idempotency key cannot make an external AI provider behave transactionally. Imagine the provider completes generation but the network connection drops before my worker receives its response. Retrying could initiate a second billable generation; not retrying could leave the user's operation unresolved.

Instant Deck's audit differentiates known failures from network-unknown provider outcomes. Its provider attempts have unique identities, and uncertain results require reconciliation rather than assuming a replay is safe. This boundary is more interesting than the basic definition of idempotency: you can prevent two local operation records and still have an ambiguous external side effect.

Depending on provider capabilities, an engineering team might use provider-supported idempotency keys, queryable request IDs, reconciliation processes or compensating actions. None of these should be advertised as exactly-once guarantees without validating the actual external contract.

Async workflows: accepted is not completed

For a short database lookup, synchronous HTTP is reasonable. For a long-running AI pipeline, it is often better to validate the request, persist operation state, enqueue durable work and return an operation ID. The UI then tracks progress through a status API or a supported push mechanism.

Instant Deck has encountered timeouts and rendering inconsistencies, illustrating why job state, retries, immutable output versions and render verification need explicit treatment. An HTTP 202 Accepted says a request was accepted for processing; it does not mean the slides exist or have passed visual validation.

A robust product distinguishes pending, running, failed, unknown and completed outcomes, and ties customer-visible state to durable evidence of completion.

Putting the boundaries together

Consider a hypothetical AI voice workflow. The agent requests appointment availability through a REST API. The customer confirms a booking. The system uses an idempotent booking command to prevent retries from creating two appointments. When the call ends, a webhook sends an event; the integration verifies and deduplicates it before scheduling CRM follow-up work. Analytics and summaries run asynchronously.

These principles apply to both communications intelligence and AI generation systems, but this example is an architectural illustration, not a claimed shipped voice-agent integration. The interesting work is deciding where to persist identities, where to put trust boundaries and how to recover from partial completion.

What I now distinguish when evaluating AI products

I increasingly separate three milestones: the model returned an answer; the application completed the user's workflow; and the workflow behaves predictably when networks, workers and providers fail. Those are different engineering achievements.

MailTrace has highlighted API contract drift and the difference between implemented backend logic and functioning frontend journeys. Instant Deck has pushed me to treat generation as a durable operation with explicit replay protection and provider uncertainty. Neither challenge can be solved merely by changing the prompt or upgrading the LLM.

The question I keep returning to is how to handle ambiguous provider outcomes when exactly-once execution is not available. Reconciliation, provider-side idempotency and carefully bounded retries are product decisions as much as engineering decisions.