In this guide, you build Receipt Relay, a FastAPI application that sends a transactional receipt through SMTPFast and makes that entire pipeline visible. You start with domain verification and a direct API smoke test, then add safe email rendering, delivery polling, signed webhooks, and tests that never send a real message.

TL;DR

• SMTPFast's Connect to Cloudflare flow creates the DKIM, SPF, DMARC, and MAIL FROM records for you.
• You do not need a normal inbound MX record or an existing mailbox just to send transactional email.
• The SMTPFast dashboard currently asks only for an API-key name. Dashboard-created keys have broad access, so keep them server-side and separate them by environment.
• POST /emails returns a correlation ID, not proof of delivery. Use that ID to retrieve the delivery trace.
• Keep delivery status separate from engagement. A tracking-pixel request is an open signal, not proof that a human read the message.
• Verify webhook HMAC signatures over the raw body before parsing JSON, and deduplicate events before processing them.

Prerequisites

• Python 3.11 or later
• Git
• An SMTPFast account
• A domain you control and access to its DNS configuration
• An inbox you control for the live test
• Basic familiarity with FastAPI and HTTP APIs
• Optional: Docker for the container section

This walkthrough uses a Cloudflare-managed domain because SMTPFast provides a one-click setup for it. Other DNS providers work too; you add the same records manually.

The 200 is only the first hop

Receipt Relay has one narrow job. A user enters a customer name, recipient, order reference, item, amount, and currency. FastAPI validates those fields, renders HTML and plain-text versions of a receipt, and calls SMTPFast. The browser receives the email ID and follows its delivery trace.

There are four boundaries in the flow:

1. Browser to FastAPI. Only receipt fields and an optional demo access code cross this boundary.
2. FastAPI to SMTPFast. The backend adds the API key and submits the email.
3. SMTPFast to the recipient server. The asynchronous delivery work happens here.
4. SMTPFast back to FastAPI. Signed webhook events report lifecycle changes without requiring an open browser.

The SMTPFast email ID connects all four boundaries. Treat it as a correlation key, not an inbox confirmation.

Set up SMTPFast before writing code

Prove the provider works before introducing application code. That gives you a clean line between DNS or account problems and bugs in your FastAPI integration.

1. Add your sending domain

Sign in to SMTPFast, open the domain area, and add the domain you want to send from. You can use a root domain such as example.com, or a subdomain such as mail.example.com if you want transactional mail isolated from other systems.

The exact from address used later must belong to this domain:

receipts@example.com

2. Connect the domain to Cloudflare

When SMTPFast detects Cloudflare nameservers, the domain page displays Connect to Cloudflare:

1. Click Connect to Cloudflare.
2. Review the domain and proposed records in the Cloudflare tab.
3. Approve the change.
4. Return to SMTPFast.
5. Click Verify Now.

Cloudflare creates the records for you. SMTPFast's current setup includes:

• Three DKIM CNAME records for cryptographic signing
• An SPF TXT record authorizing the sending service
• A DMARC TXT record describing how receivers handle authentication failures
• An MX record on a bounce subdomain for MAIL FROM processing
• An SPF TXT record on that bounce subdomain

SMTPFast documents the current one-click flow and each record's purpose in its Domains documentation.

If you do not use Cloudflare, copy the records shown by SMTPFast into your DNS provider exactly as displayed. Do not reuse values from another domain. DKIM hostnames are generated for your SMTPFast domain.

There are two common manual-setup mistakes. First, keep DKIM CNAMEs DNS-only rather than proxying them. Second, publish one SPF record per hostname:

# Wrong: two SPF policies on example.com
example.com  TXT  "v=spf1 include:_spf.google.com ~all"
example.com  TXT  "v=spf1 include:amazonses.com ~all"

# Right: merge both senders into one policy
example.com  TXT  "v=spf1 include:_spf.google.com include:amazonses.com ~all"

3. Wait for verification

DNS changes are often visible quickly, but the underlying sending identity can take a few minutes to finish verifying. If the domain stays pending:

1. Confirm the records exist on the correct domain.
2. Check that all three DKIM CNAMEs are not proxied.
3. Confirm there is only one SPF record on each hostname.
4. Click Verify Now again.
5. Allow more time if SMTPFast says the records are visible but verification is still in progress.

Do not debug application code until the domain is verified. SMTPFast rejects an otherwise valid request when its from address uses an unverified domain.

4. Create the API key

Open the API Keys page and click Create API Key. The current dashboard asks for one value: a descriptive key name.

Use a name that identifies the application and environment, such as receipt-relay-local. Click Create Key, copy the generated value immediately, and store it in a password manager or secret store. SMTPFast only displays the complete key when it is created.

The dashboard does not currently show a scope selector. SMTPFast's Authentication documentation says dashboard-created keys default to all scopes, while keys created through the API can request explicit scopes.

Because the dashboard key has broad access:

• Use a separate key for local, staging, and production.
• Keep it in server-side environment variables.
• Never place it in browser JavaScript, screenshots, Git commits, or container images.
• Revoke it when the environment no longer exists.

5. Run a direct API smoke test

Export the key in your current terminal session, then send to an inbox you control. Replace both email addresses before running the command.

The first response proves that SMTPFast accepted the request. The second shows what happened later. The full response includes status, timestamps, and an events array; see the Emails API reference for the current shape.

Fix provider setup errors here, before proceeding:

Build Receipt Relay with FastAPI

With the direct request working, put a small application boundary around it. The browser never receives the SMTPFast key and never calls SMTPFast directly.

The complete application is available as a reusable GitHub template:

6. Install and configure the application

Click Use this template on GitHub to create your own repository, or clone the reference application directly:

git clone https://github.com/The-DevOps-Daily/smtpfast-receipt-relay.git
cd smtpfast-receipt-relay

Create a virtual environment and install the project with its development tools:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

Create .env from the included template:

cp .env.example .env
chmod 600 .env

Add the key and verified sender:

SMTPFAST_API_KEY=replace-with-your-smtpfast-api-key
SMTPFAST_FROM_EMAIL=receipts@your-verified-domain.com
SMTPFAST_BASE_URL=https://smtpfa.st/api/v1
SMTPFAST_TIMEOUT_SECONDS=20

# Added after creating the public webhook
SMTPFAST_WEBHOOK_SECRET=

# Optional shared code for a short-lived demo
APP_ACCESS_TOKEN=

Start FastAPI with the environment file:

uvicorn app.main:app --reload --port 8080 --env-file .env

Open http://localhost:8080. The page displays the configured sender but never returns either secret.

7. Validate before consuming quota

An email send is an external side effect. It consumes quota and can reach a real person, so reject malformed values before calling the provider.

class ReceiptRequest(BaseModel):
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)

    customer_name: str = Field(min_length=1, max_length=100)
    recipient: EmailStr
    order_id: str = Field(
        min_length=3,
        max_length=64,
        pattern=r"^[A-Za-z0-9][A-Za-z0-9._-]+$",
    )
    product_name: str = Field(min_length=2, max_length=120)
    amount_cents: int = Field(ge=50, le=100_000_000)
    currency: Literal["USD", "EUR", "GBP"] = "USD"

The model makes several deliberate decisions:

• EmailStr rejects malformed recipients.
• The order reference uses a small, header-friendly character set.
• Money crosses the API as integer cents rather than floating point.
• Currency is an enum rather than arbitrary text.
• extra="forbid" makes misspelled fields fail explicitly.

In a real checkout, accept an order ID and load the authoritative item and total from a database. Do not let a browser decide how much was paid.

8. Render safe HTML and a text alternative

Transactional messages need a useful plain-text body as well as HTML. Escape values before inserting them into the HTML context:

customer = html.escape(receipt.customer_name)
product = html.escape(receipt.product_name)
order_id = html.escape(receipt.order_id)
total = _format_amount(receipt.amount_cents, receipt.currency)

Validation constrains shape and length; it does not make a string safe for HTML. A customer named <script>alert(1)</script> must appear as text, not markup.

Build the SMTPFast payload with both bodies and two correlation values:

return {
    "from": self._settings.smtpfast_from_email,
    "to": [str(receipt.recipient)],
    "subject": f"Receipt for order {receipt.order_id}",
    "html": html_body,
    "text": text_body,
    "tags": [
        {"name": "category", "value": "receipt"},
        {"name": "order_id", "value": receipt.order_id},
    ],
    "headers": {"X-Entity-Ref-ID": receipt.order_id},
}

Tags help filter provider records. X-Entity-Ref-ID carries your application reference with the message. Neither replaces a database relationship, but both make one send easier to diagnose.

9. Call SMTPFast from the server

The client submits the payload to /emails, validates the returned ID, and records request latency:

async def send_receipt(self, receipt: ReceiptRequest) -> ReceiptAccepted:
    self._require_send_configuration()
    started_at = time.perf_counter()
    response = await self._request(
        "POST",
        "/emails",
        json=self._build_receipt_payload(receipt),
    )
    latency_ms = round((time.perf_counter() - started_at) * 1_000)

    data = response.json()
    email_id = data["id"]
    return ReceiptAccepted(
        email_id=email_id,
        status=str(data.get("status") or "queued"),
        latency_ms=latency_ms,
    )

The shared helper adds authentication only on the backend:

response = await client.request(
    method,
    f"{self._settings.smtpfast_base_url}{path}",
    headers={
        "Authorization": f"Bearer {self._settings.smtpfast_api_key}",
        "Content-Type": "application/json",
    },
    json=json,
)

The fallback queued status is intentionally conservative. The application has an ID and knows the request was accepted; it does not invent a later delivery event.

10. Keep a narrow browser-facing API

The browser submits to a FastAPI route rather than the provider:

@application.post("/api/receipts", response_model=ReceiptAccepted)
async def send_receipt(
    receipt: ReceiptRequest,
    x_app_access_token: str | None = Header(default=None),
) -> ReceiptAccepted:
    _require_app_access(runtime_settings, x_app_access_token)
    return await application.state.smtpfast_client.send_receipt(receipt)

The complete handler maps configuration, authentication, rate-limit, and upstream failures into safe application errors. It never returns SMTPFast's raw error body, which may contain internal identifiers or request data.

Receipt Relay also exposes /health without calling SMTPFast. A load balancer should be able to check the process without sending an email or making the provider a dependency of every probe.

11. Retrieve and display the lifecycle

After a send, the browser receives the email ID and calls GET /api/emails/{email_id}. The backend retrieves and validates the SMTPFast record:

async def get_email(self, email_id: str) -> EmailTrace:
    response = await self._request(
        "GET",
        f"/emails/{quote(email_id, safe='')}",
    )
    data = response.json()
    events = [
        EmailEvent.model_validate({**event, "source": "api"})
        for event in data.get("events", [])
    ]
    return EmailTrace.model_validate({**data, "events": events})

The browser polls briefly, renders values with textContent, stops after a bounded number of attempts, and leaves a manual refresh button. A typical sequence is:

queued -> sending -> sent -> delivered

• Queued means SMTPFast accepted the work.
• Sent means the sending provider accepted the message for delivery.
• Delivered means the recipient mail server accepted it.
• Bounced or failed means delivery did not complete.

Even delivered does not guarantee primary-inbox placement. The receiving system can still route the message to spam.

12. Keep delivery separate from engagement

An open or click does not make a message "more delivered," and it should not replace the terminal delivery outcome.

SMTPFast records an open when its tracking pixel is requested. Image proxies, privacy features, and security scanners can request that pixel without a person reading the email. During the live Receipt Relay test, an open signal arrived about one second after delivery even though nobody had opened the inbox.

Receipt Relay therefore keeps Delivered as the status, shows the later event separately, and labels it Open signal rather than Opened.

Receive and verify SMTPFast webhooks

Polling works for an interactive demo, but an application should not need an open browser to learn about a bounce. Webhooks reverse the flow: SMTPFast calls your application when an event occurs.

13. Expose a public HTTPS endpoint

Deploy Receipt Relay to your preferred platform or expose it through a trusted development tunnel. SMTPFast must be able to reach this endpoint:

https://your-app.example/webhooks/smtpfast

http://localhost:8080 exists only on your computer from SMTPFast's perspective.

14. Create the webhook

Create a standard-format webhook in SMTPFast with the public URL. Subscribe only to events your application uses:

[
  "email.sent",
  "email.delivered",
  "email.delivery_delayed",
  "email.bounced",
  "email.failed",
  "email.suppressed",
  "email.opened",
  "email.clicked"
]

SMTPFast returns a signing secret when the webhook is created. It is not the API key. Store it separately as SMTPFAST_WEBHOOK_SECRET, then restart or redeploy the application. The webhook page's test action reports the response code and response time. The current event list and retry policy live in the Webhooks documentation.

15. Verify the signature before parsing JSON

Standard webhook requests include X-SMTPfast-Signature, an HMAC-SHA256 digest of the raw request body using the webhook signing secret.

The word raw matters. Parse and reserialize JSON and you can change whitespace, ordering, or escaping, producing a different digest.

Read and bound the raw body first:

body = await request.body()
if len(body) > MAX_WEBHOOK_BYTES:
    raise HTTPException(status_code=413, detail="Webhook payload is too large.")
if not _valid_webhook_signature(body, x_smtpfast_signature, secret):
    raise HTTPException(status_code=401, detail="Invalid webhook signature.")

Compare the expected and received values in constant time:

def _valid_webhook_signature(
    body: bytes,
    signature: str | None,
    secret: str,
) -> bool:
    if not signature:
        return False
    expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return secrets.compare_digest(signature, expected)

Only after signature verification do you parse and validate:

payload = json.loads(body)
event = SMTPFastWebhookEvent.model_validate(payload)
await application.state.trace_store.add(event)

Signature verification proves that someone with the webhook secret produced the payload. Pydantic validation separately proves that the payload has the shape your application expects. You need both.

16. Make retries safe

SMTPFast retries when an endpoint fails or times out. Receiving the same event more than once is expected behavior.

Receipt Relay uses a bounded in-memory OrderedDict keyed by SMTPFast event ID. That deduplicates retries during one process lifetime and keeps the demo dependency-free. Production handling needs a durable sequence:

1. Verify the signature.
2. Validate the payload.
3. Insert the event with a unique constraint on event ID.
4. Commit the transaction.
5. Return a successful response.
6. Process slow downstream work asynchronously.

Do not acknowledge an event you have not recorded safely.

Test the integration end to end

The automated suite should not consume quota, depend on DNS, or place messages in an inbox.

17. Mock SMTPFast in tests

HTTPX's MockTransport lets a test inspect the outgoing request and return a representative provider response:

def handler(request: httpx.Request) -> httpx.Response:
    assert request.method == "POST"
    assert request.url == "https://smtpfa.st/api/v1/emails"
    assert request.headers["Authorization"] == "Bearer sf_live_test"

    payload = json.loads(request.content)
    assert payload["headers"]["X-Entity-Ref-ID"] == "ORD-2048"
    assert "Ana <script>alert(1)</script>" in payload["html"]
    return httpx.Response(200, json={"id": "email_abc123"})

The escaped-name assertion tests the important HTML boundary, not just the happy path.

The webhook test signs the exact bytes it submits:

body = json.dumps(event, separators=(",", ":")).encode()
signature = hmac.new(b"whsec_test", body, hashlib.sha256).hexdigest()

response = client.post(
    "/webhooks/smtpfast",
    content=body,
    headers={"X-SMTPfast-Signature": signature},
)

Add a negative test with a bad signature. One test proves correctly signed bytes pass; the other stops verification from accidentally becoming optional.

Run the checks:

ruff check .
ruff format --check .
pytest

No real SMTPFast key is required.

18. Send one real receipt

Return to http://localhost:8080, load the example, enter an inbox you control, and submit once.

Verify the complete path:

1. Receipt Relay displays an SMTPFast email ID.
2. The trace advances from queued through sending and sent.
3. The recipient server accepts the message or returns a failure.
4. The email contains readable HTML and a useful text alternative.
5. The sender uses the verified domain.
6. Later engagement appears separately from delivery.

Check spam. A technically successful first send from a new domain can still be filtered; authentication is a foundation for deliverability, not a guarantee of inbox placement.

Run the same app in Docker

The project includes a non-root Docker image. Run the container locally with the same .env file:

docker build -t smtpfast-receipt-relay .
docker run --rm \
  --publish 8080:8080 \
  --env-file .env \
  smtpfast-receipt-relay

Use the non-sending health endpoint:

curl http://localhost:8080/health

Expected output:

{"status":"ok"}

You can deploy the same image to any container platform that accepts environment variables and exposes a public HTTPS URL. Once that URL exists, create the SMTPFast webhook, store its signing secret in the platform's secret manager, and restart the application.

Production checklist

Receipt Relay is production-minded, not production-complete. Before adapting it to a real product:

• Load trusted order data. Accept an order ID and render values from your database rather than trusting browser-submitted totals.
• Add idempotency. A double-click, worker retry, or network timeout must not send a duplicate receipt.
• Persist provider IDs. Store the SMTPFast email ID with the business record that caused the send.
• Persist webhook events. Use durable storage and a unique event-ID constraint before acknowledging delivery.
• Use real authentication. Replace the shared demo code with user- and tenant-aware authorization.
• Apply quotas. Add per-user, per-tenant, and global send limits.
• Protect recipient data. Avoid logging full addresses and bodies by default; define retention and deletion behavior.
• Enable tracking deliberately. Open and click events affect privacy and remain imperfect signals.
• Version templates. Add localization, rendering checks, and snapshot tests.
• Monitor the pipeline. Track API failures, time to delivery, bounce categories, webhook retries, and consumer lag.

What to take away

The most useful value returned by an email send is not "success." It is the ID that lets the rest of your application correlate what happens next.

Receipt Relay validates a real side effect before sending it, keeps SMTPFast credentials on the server, renders HTML and text bodies, follows each message's delivery trace, and verifies webhook events over the raw request body. The browser makes the lifecycle visible while the backend owns the provider and security boundaries.

The same pattern applies to password resets, invoices, deployment alerts, and account notifications: send once, keep the correlation ID, and design for everything that happens after the 200.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Does Kubernetes DRA Replace HAMi?

Projects that want to share a GPU on Kubernetes have to work around an API instead of with it. The device plugin interface could count devices, and that was the whole vocabulary: nvidia.com/gpu: 1. It

📅 Aug 7, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Shadow AI in CI/CD: Threat-modeling the path from developer laptop to Kubernetes

Artificial intelligence is becoming part of daily software delivery, often before it becomes part of the security architecture. That gap has a name: Shadow AI. It is any AI tool, model, agent, extensi

📅 Aug 7, 2026 • 📰 CNCF Blog

🔗 Read more

📄 The migration catalyst: turning virtualization disruption into application innovation

Starting nearly three decades ago, the cost efficiencies of server virtualization drove the first waves of IT transformation, wringing new efficiency out of the x86 servers that had already shaped the

📅 Aug 6, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 GitLab Secrets Manager adds ESO, Terraform, API support

Today, you might maintain separate secret stores for CI/CD, Kubernetes, and Terraform. However, that leaves multiple tools to manage, access models to keep in sync, and audit trails to correlate when

📅 Aug 6, 2026 • 📰 GitLab Blog

🔗 Read more

📄 OpenCost 1.121.0: First-of-a-kind Kubernetes inference cost tracking

Your GPU bill is rising. Your models are serving billions of tokens. Yet one question remains unanswered: what does each token actually cost? This is not a hypothetical problem. Platform teams today o

📅 Aug 5, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Use EVPN in Red Hat OpenShift 4.22 to integrate production networks across Kubernetes cluster boundaries

Red Hat OpenShift Networking is making it easier for you to seamlessly and directly integrate your Kubernetes platforms with the data center networks you already operate by adopting the same standards

📅 Aug 5, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Run GPU batch inference on Amazon ECS Managed Instances with scale to zero

Deploy a single CloudFormation stack that builds a GPU batch inference pipeline on Amazon ECS Managed Instances. It uses Amazon SQS for job buffering and Application Auto Scaling to scale to zero when

📅 Aug 3, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard

The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year! Gateway API has become the standard for modern, role-oriented

📅 Aug 3, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Managing virtual machines on Red Hat OpenShift with Service Mesh

Managing virtualized workloads alongside containerized applications remains a persistent challenge for IT operations, often creating siloed management environments. At Red Hat Summit 2026, I had the o

📅 Aug 7, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 LitmusChaos Q1-Q2 2026 update: community, contributions, and project progress

About LitmusChaos LitmusChaos is an open source chaos engineering platform that helps teams identify weaknesses and potential outages in their infrastructure by running controlled chaos experiments. B

📅 Aug 6, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Extending Amazon ECS Express Mode to Build an Optimal Container Environment

Amazon ECS Express Mode gives you load balancing, scaling, logging, and networking out of the box. Learn how to extend an Express Mode service beyond its defaults with three hands-on examples: turning

📅 Aug 4, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Empty sandboxes break developer experience

Learn how Docker Sandbox kits turn empty sandboxes into productive development environments with repeatable tooling, credentials, and configuration.

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

📄 Docker AI Governance: Audit Logs, Now Where Your Security Team Already Works

Now in Docker AI Governance: a single searchable record of every policy decision your agents trigger, streamed to the SIEM your security team already runs, so you can show what your agents did and wha

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

🔄 CI/CD

📄 Automate Incident Intake with AI SRE Runbooks

Automate incident intake with Harness AI SRE runbooks: auto-create tickets, open Slack channels, start Zoom bridges, and cut response time to seconds. | Blog

📅 Aug 10, 2026 • 📰 Harness Blog

🔗 Read more

📄 A guide to slash commands in the GitHub Copilot app

Go beyond chat in the GitHub Copilot app with these slash commands. They'll help you plan, collaborate, automate, and customize your dev workflow. The post A guide to slash commands in the GitHub Copi

📅 Aug 6, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Confidential AI for GitLab Self-Hosted

Your developers want AI coding agents. Your source code is regulated IP that can't be sent to a third-party AI service, and your compliance team has said so in writing. The usual escape hatch, standin

📅 Aug 6, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Infrastructure Control Plane | Day 2 Operations & Drift

Learn why infrastructure breaks after deployment and how control planes enforce governance, detect drift, and automate remediation across Terraform, Ansible, and CI/CD. | Blog

📅 Aug 6, 2026 • 📰 Harness Blog

🔗 Read more

📄 New bazel.build websites incoming!

We're happy to announce the launch of the new bazel.build documentation site and the new web UI for the Bazel Central Registry! New documentation site Last year, Alan Mond wrote a viral blog post that

📅 Aug 5, 2026 • 📰 Bazel Blog

🔗 Read more

📄 How the GitHub legal team used Copilot CLI to streamline their workflows

Learn how to build tools to simplify how you work—without writing a single line of code. The post How the GitHub legal team used Copilot CLI to streamline their workflows appeared first on The GitHub

📅 Aug 4, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Turn one giant AI-generated pull request to a reviewable stack

Instead of one huge, un-reviewable pull request, teach coding agents to decompose work into a clean, ordered stack with GitHub stacked pull requests. The post Turn one giant AI-generated pull request

📅 Aug 4, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Agent Optimization: Define what better means, and let AgentControl find it

Agent Optimization, now in beta in AgentControl, automatically searches for a better agent configuration against criteria you define.

📅 Aug 4, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Stories from the Factory Floor: Building a software factory on our scariest code

We pointed coding agents at our oldest, most business-critical frontend. Here’s what it taught me about what a healthy AI software factory actually looks like.

📅 Aug 3, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Blog: Selective drift correction with ignore rules

We are excited to introduce drift ignore rules for Flux Kustomizations, a long-requested capability that lets you tell Flux to leave specific fields alone during drift detection and correction, while

📅 Aug 3, 2026 • 📰 Flux CD Blog

🔗 Read more

🏗️ IaC

📄 Terraform Scalability: When IaC Outgrows Your Setup

Terraform scalability issues slow teams down. Learn how to overcome IaC bottlenecks with better management. See how Harness helps. | Blog

📅 Aug 10, 2026 • 📰 Harness Blog

🔗 Read more

📄 Autobase 2.10 released

Autobase 2.10 expands day-to-day PostgreSQL operations with new cluster management capabilities. Administrators can now perform common cluster actions directly from the Console UI, configure advanced

📅 Aug 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Accelerate CloudFormation development with the IaC MCP Server

Walk through a complete CloudFormation development cycle - authoring, validation, deployment, and troubleshooting - without leaving your AI assistant, using the AWS IaC MCP Server.

📅 Aug 4, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 YOLO Mode Is the Right Default. Your Laptop Is the Wrong Place for It.

Claude Code calls the flag --dangerously-skip-permissions, and the community long ago renamed it YOLO mode. It lets your coding agent run any command it wants without ever asking for permission. Every

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Emulating Terraform on Pulumi's Engine

The core promise of Pulumi’s HCL support is that you can bring your existing Terraform configuration and modules, and pulumi will run them. If it works in OpenTofu and doesn’t work in Pulumi, we would

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Bring Your Terraform Estate Into the Agentic Era

At Pulumi, we are building the platform for agentic infrastructure. Pulumi Cloud provides the guardrails and enterprise readiness needed to safely move fast in this new era. While we are seeing extrao

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 A guided tour of Terraform state, hosted modules, and HCL in Pulumi

Today’s big release contains a whole new set of features designed for seamless interoperability with the Terraform and OpenTofu ecosystems, and there’s a lot there — so much that it can be tough to ge

📅 Aug 4, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Unifying Workers AI and AI Gateway into a single AI control plane

Cloudflare is unifying AI Gateway and Workers AI into a single control plane, giving developers observability, billing, and dynamic routing across both managed GPUs and external providers. Learn how u

📅 Aug 7, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 How to Choose Digital Experience Monitoring Tools

Discover how digital experience monitoring tools help you understand user issues beyond APM, enabling faster, clearer insights for better software performance.

📅 Aug 7, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Scaling Autonomous Operations with AWS DevOps Agent and ServiceNow

This post is co-written with Govind Menon, Head of MCP Product at ServiceNow. Introduction Enterprise teams managing applications on AWS often rely on ServiceNow as their IT service management (ITSM)

📅 Aug 6, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 Podcast recap: Observability won’t save your agents

On a recent episode of the MonkCast, Marek Poliks spoke with James Governor about why governing agents from the outside leaves teams perpetually one step behind.

📅 Aug 6, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 How we built an automated debugging workflow at Sentry

How Sentry uses Seer autofix and Claude routines to build an automated debugging workflow that detects, fixes, and routes code issues automatically.

📅 Aug 6, 2026 • 📰 Sentry Blog

🔗 Read more

📄 Curing alert fatigue: How embedded AI is redefining Red Hat OpenShift cluster troubleshooting

Between virtual machines, microservices, and AI pipelines, hybrid clouds can be incredibly complex and can bring an unwelcome partner: alert fatigue. SREs and IT OPs teams face a constant flood of dis

📅 Aug 6, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Under the hood: how Amazon EKS Auto Mode detects, repairs, and diagnoses node failures

On Amazon EKS Auto Mode, node failures are detected, drained, and replaced automatically before anyone reaches for a laptop. This post shows how the Node Monitoring Agent and Karpenter form a detect-a

📅 Aug 5, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Session Replay Tools: A Technical Buyer’s Guide and Comparison

Discover how to evaluate session replay tools for engineering teams, ensuring they meet technical needs for incident response and observability.

📅 Aug 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Your OTel spans, our errors: A Sentry love story in one trace

The OtlpIntegration bridges OTel traces and Sentry errors. Keep your OTel setup, add Sentry for errors, and see both in one trace waterfall.

📅 Aug 5, 2026 • 📰 Sentry Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 How Google Cloud detects, contains, and protects against emerging threats

At Google Cloud, securing your data and business systems is our foundational commitment. We empower our customers with the tools, governance, and infrastructure needed to securely deploy workloads and

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 CVE-2026-63077: Additional Guidance Following Reports of Active Exploitation

This post is a follow-up to our July 27, 2026, announcement about CVE-2026-63077. Summary What has changed since our initial announcement Since our initial announcement on July 27, 2026, we have recei

📅 Aug 7, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Friday Five — August 7, 2026

Red Hat Recognized as a Leader for Third Consecutive Year in 2026 Gartner® Magic Quadrant™ for Cloud-Native Application PlatformsRed Hat OpenShift is recognized as a Leader in the 2026 Magic Quadrant

📅 Aug 7, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Proactive patch management & compliance: Hardening the hybrid Azure fleet at scale

Welcome back to SUSE Solutions on Azure: The Technical Series. Bridging the Gap Between Linux Freedom and Azure Scale Enterprise Linux on Azure requires a careful balance between open source flexibili

📅 Aug 6, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Open Source Summit + Embedded Linux Conference Europe 2026 Schedule Champions Open Source Innovation and Marks 35 Years of Linux

Industry leaders gather to advance the open source infrastructure powering embedded systems, cloud orchestration, AI security, safety-critical applications…

📅 Aug 5, 2026 • 📰 KubeCon Updates

🔗 Read more

📄 Governance Is a Developer Experience Problem

Learn why AI governance is about more than security. Discover how trust, clear boundaries, and developer experience enable AI adoption at scale.

📅 Aug 5, 2026 • 📰 Docker Blog

🔗 Read more

📄 New Relic SecurityRX - Security for Operational Reliability

Treat security as a reliability problem. New UI experience (with the homepage), automation capabilities (with Jira), and the agent public preview for a complete remediation workflow.

📅 Aug 5, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Continuous Offensive Security & AI Pentesting: 20 FAQs

Get answers to 20 common questions about continuous offensive security, AI penetration testing, DAST, and AI red teaming.

📅 Aug 5, 2026 • 📰 Snyk Blog

🔗 Read more

📄 The Software Supply Chain Is Under Siege. Devs Are Still the First Line of Defense

77% of organizations experienced a software supply chain incident in the past year. Explore Omdia's latest research on top risks, security gaps, and why developers are your first line of defense.

📅 Aug 4, 2026 • 📰 Docker Blog

🔗 Read more

📄 Evo Continuous Offensive Security Is Here Pentesting Grade Coverage For The 350 Days A Year You Aren't Testing

Snyk Evo Continuous Offensive Security brings autonomous, AI-powered pentesting to the 350 days between traditional tests, uncovering exploitable flaws attackers can find first.

📅 Aug 4, 2026 • 📰 Snyk Blog

🔗 Read more

📄 AI Model Risk Intelligence Know Which Models You Can Trust Before You Deploy

AI model risk depends on how a model is deployed. Learn how Evo combines adversarial testing, attack impact, and deployment context to help teams compare models and enforce policy.

📅 Aug 4, 2026 • 📰 Snyk Blog

🔗 Read more

💾 Databases

📄 The Complete Agent State Stack: Memory, Files, and Serverless Database Persistence for AI Apps

A serverless database is a fully-managed database that automatically scales compute and storage with demand, requires no server provisioning or capacity planning, and bills only for actual usage, incl

📅 Aug 7, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Postgres Summit US 2026 Schedule is now live!

Hi all, The talk schedule for Postgres Summit US 2026 is now published. Browse it here: Talk Schedule The summit runs September 30 through October 2, 2026 at Convene, 555 Broadway, New York, NY, organ

📅 Aug 7, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Why Attend TiDB SCaiLE 2026: Same Complexity, Different Clock Speeds

A single user action in an agentic application no longer maps to a single database query. It spawns agent instances that branch context in milliseconds, hold memory across sessions, and provision thei

📅 Aug 6, 2026 • 📰 TiDB Blog

🔗 Read more

📄 How we took malware advisories beyond npm

GitHub malware advisories no longer stop at npm. Here's how we wired OpenSSF's malicious-packages data into the Advisory Database, and why we built the pipeline paranoid. The post How we took malware

📅 Aug 6, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Refactoring a SQL Table at Scale: Lessons from Harness CI

How Harness refactored a flat SQL table into a normalized schema, cutting storage per row from 400 bytes to 28 bytes and making API latency constant at any scale. | Blog

📅 Aug 6, 2026 • 📰 Harness Blog

🔗 Read more

📄 How Redis brings persistent memory to Snowflake Cortex Agents

AI agents can reason and act, but without memory, every interaction starts from zero. Intelligent short-term memory and persistent context across conversations are what turns a capable model into a tr

📅 Aug 6, 2026 • 📰 Redis Blog

🔗 Read more

📄 Top vector database alternatives for RAG pipelines

You're building an AI app: maybe a RAG system, an agent with memory, or a chatbot with semantic caching. You need vector search, and you're weighing your options. One is a unified real-time platform l

📅 Aug 5, 2026 • 📰 Redis Blog

🔗 Read more

📄 Migrating Real-Time Data into TiDB with Debezium CDC

Moving data into a new database is rarely a one-shot copy. Migrating off a legacy system, adopting a distributed SQL database, carrying out a heterogeneous database migration, or standing up an analyt

📅 Aug 4, 2026 • 📰 TiDB Blog

🔗 Read more

📄 DDIA 2nd Edition Excerpt: On Scalability

Martin Kleppmann and Chris Riccomini's scalability considerations for designing data-intensive applications -- from the second edition of the Designing Data-Intensive Applications book

📅 Aug 4, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 Your Meko Questions, Answered

Interest in Meko has been tremendous, with user questions coming in thick and fast via Discord, LinkedIn, and at in-person events. In his recent AMA session, Yugabyte co-founder Karthik Ranganathan an

📅 Aug 4, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 Announcing E-Maj 5.0.0.

We are very glad to announce the E-Maj 5.0.0 version. Among improvements, this major version: Allows non-superuser roles to install and use E-Maj in a database, the usable features depending on the pr

📅 Aug 4, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 pgBackRest 2.59.0 Released

July 30, 2026: The pgBackRest community is pleased to announce the release of pgBackRest 2.59.0, the latest version of the reliable, easy-to-use backup and restore solution that can seamlessly scale u

📅 Aug 4, 2026 • 📰 PostgreSQL News

🔗 Read more

🌐 Platforms

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Aug 10, 2026 • 📰 Linode Blog

🔗 Read more

📄 Platform Engineering ROI: What it costs to build your own platform

What it actually costs to build your own internal developer platform over five years, and why most “we’ll just build The post Platform Engineering ROI: What it costs to build your own platform appeare

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 Public Cloud Toolchains in SUSE Linux Enterprise 16: Evolution and Transparent Containers

The release of the SUSE Linux Enterprise (SLE) 16 distributions has long come and gone and the development cycle for SLE 16.1 is well on the way and will culminate in the SLE 16.1 release later this y

📅 Aug 8, 2026 • 📰 SUSE Blog

🔗 Read more

📄 The login screen is where sovereignty gets real

Everyone points at the cloud. Almost nobody points at the front door. Ask most executives where their sovereignty risk sits and they point at the cloud, the data, the AI models. Fair enough, those are

📅 Aug 8, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Amazon EC2 R8i and R8i-Flex instances are now available in Europe (Milan) region

Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R8i and R8i-flex instances are available in the Europe (Milan) region. These instances are powered by custom Intel Xeon 6 processors, availabl

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon Timestream for InfluxDB now supports backup and restore

Amazon Timestream for InfluxDB now lets you create and manage your own backups and restore your data on demand. You can trigger one-time, on-demand backups, schedule automated recurring backups at the

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon Cognito now available as a skill in the Agent Toolkit for AWS

Amazon Cognito is now available as a core skill (aws-auth) in the Agent Toolkit for AWS. AI coding agents using the toolkit can now set up, configure, secure, and troubleshoot Amazon Cognito using bes

📅 Aug 7, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Zero-code, low-cost data ingestion: New BigQuery DTS capabilities

In a fast-paced digital economy, data is your most critical engine. Yet, many enterprises find themselves trapped in a costly paradox, spending over 100 hours a week building and fixing fragile, in-ho

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Unifying Structured and Unstructured Data Insights with BQ Search Innovations

Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidde

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 GOL! How TelevisaUnivision streamed the FIFA World Cup to millions with Google Cloud

Live sports broadcasting represents the ultimate stress test for digital media infrastructure, where operational success or failure is measured in milliseconds and observed live by millions of viewers

📅 Aug 7, 2026 • 📰 Google Cloud Blog

🔗 Read more

📰 Misc

📄 Visual Studio Code 1.133 (Insiders)

Learn what's new in Visual Studio Code 1.133 (Insiders) Read the full article

📅 Aug 11, 2026 • 📰 VS Code Blog

🔗 Read more

📄 Coding agents can be evaluated. We just have to evaluate the work.

I recently argued with a software factory provider, whose position was that coding agents cannot be evaluated. Their reasoning was The post Coding agents can be evaluated. We just have to evaluate the

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 AI coding got faster. Why didn’t engineering?

AI is great at making individuals faster, but the surrounding systems are then slowing everything right back down. This result The post AI coding got faster. Why didn’t engineering? appeared first on

📅 Aug 9, 2026 • 📰 The New Stack

🔗 Read more

📄 AI adoption isn’t the same as AI usage

Every engineering org I’ve talked to this year has some version of the same chart. Seat activations climbing. Token spend The post AI adoption isn’t the same as AI usage appeared first on The New Stac

📅 Aug 8, 2026 • 📰 The New Stack

🔗 Read more

📄 Microsoft’s New Testing Agent Tackles the Trust Gap in AI-Generated Code

AI coding assistants write code fast. Whether that code can be trusted is a separate question, and it’s becoming a more urgent one. Surveys this year put average developer trust in AI-generated output

📅 Aug 7, 2026 • 📰 DevOps.com

🔗 Read more

📄 ‘Flooding Dropper’ Is Hitting npm With a Tidal Wave of Malicious Packages

Threat researchers at Sonatype are warning developers of an expanding campaign that is generating a wide range of npm accounts and dropping small numbers of malicious packages from each one, essential

📅 Aug 7, 2026 • 📰 DevOps.com

🔗 Read more

📄 JetBrains Academy – July Digest

Somewhere between the fifteenth open tab and the third iced coffee, it hit me. Maybe we don’t hate meetings. We just hate the ones where nobody has anything to say. Welcome back to another mandatory m

📅 Aug 7, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Stop burning your AI budget: Optimize GPU usage and model deployment with workflow navigator

Uber burned through its entire 2026 AI tools budget by April. Microsoft faced a similar crisis, pulling Claude Code licenses because the tool worked too well and people used it too much. Even OpenAI's

📅 Aug 7, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Why Reliability Guardrails Are Needed in Every AI Coding Pipeline

We’re in the middle of a reliability reckoning. Thanks to AI, companies are shipping code much faster than before. But if there’s anything to learn from the surge in high-profile outages over the last

📅 Aug 6, 2026 • 📰 DevOps.com

🔗 Read more

📄 AI Architecture: Moving Past the Washing to the Truth

In the current hype cycle, “AI” has become a linguistic junk drawer—a catch-all term that vendors use to mask everything from basic if-then statements to massive neural networks. For the modern enterp

📅 Aug 6, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Println Debugging Done Right

The simplest tools are often the most useful, and debugging is a prime example of this. There are many advanced debugging techniques, and while they all have their use cases, println debugging is stil

📅 Aug 6, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Figma Connect for WebStorm: Stage One of a Better Design-to-Code Experience

Where time actually goes in design-to-code Every design implementation starts the same way: find the Figma tab, find the right frame, screenshot it, paste it somewhere, switch back to the terminal. By

📅 Aug 6, 2026 • 📰 JetBrains Blog

🔗 Read more

If the answer is no, one of two things happens. You say "it's on the roadmap" and watch the deal slip a quarter, or somebody promises a date and the work lands on you with a deadline attached and no design time. Both are avoidable, because the expensive part of this work is not the protocol. It is a data model change, and you can make that change long before anyone asks.

This covers what enterprise buyers actually mean, what has to change in your application, the validation steps that turn a SAML integration into an authentication bypass if you skip them, and the order to build it in.

TL;DR

• SSO and provisioning are different problems. SAML answers "is this person who they say they are". SCIM answers "who should exist in the first place, and who should stop existing".
• The hard part is neither protocol. It is that your app probably assumes a user owns their own account. Enterprise means the organisation owns the account, and that is a schema change.
• Build the organisation and connection model first. It is useful on its own and it is the thing you cannot retrofit under deadline pressure.
• SAML is XML with a signature. Validating that signature is necessary and not sufficient. You must also check Audience, Destination, InResponseTo, the time window, and that the assertion you read is the assertion that was signed.
• A whole class of 2018 CVEs existed because libraries read the text of a signed XML node differently to the way the signature covered it. An XML comment inside NameID was enough to log in as somebody else.
• SCIM is a boring REST API you host. The part everyone gets wrong is deprovisioning: PATCH with active: false must actually kill sessions, not just flip a column.
• Roles are the trap. Sync group membership, but keep your own authorisation model. Do not let the IdP be the source of truth for permissions you enforce.

Prerequisites

• An application with its own user accounts and sessions
• Familiarity with HTTP redirects, form POSTs, and JSON APIs
• Access to an identity provider test tenant. Okta and Microsoft Entra ID both offer free developer tenants, and you will want one before writing any code

What they are actually asking for

"SSO" in a procurement document usually bundles three separate things. Being precise about which one is being asked for saves a lot of argument later.

Authentication. The user lands on your login page, types a work email, and gets bounced to their company's identity provider. They come back authenticated. No password of yours involved. This is SAML, or increasingly OIDC.

Provisioning and deprovisioning. When IT adds someone to the "Acme Engineering" group, an account appears in your product without anyone inviting them. When that person leaves, the account is disabled within minutes. This is SCIM, and it is the one people underestimate.

Central policy. MFA, session lifetime, device posture, conditional access. You get this largely for free by delegating authentication, which is a genuinely good reason to support SSO beyond the contract.

The second is where the value is for the buyer. An IT admin who has to remember to log into fourteen SaaS dashboards to remove a departing employee will eventually forget one, and that forgotten account is an audit finding.

Note what happens if you build only SAML. The account gets created on first login instead, which sounds fine until someone leaves: the IdP stops letting them log in, but your app still holds an active session and an enabled account. The buyer asked for deprovisioning and you gave them a login page.

The change that has to come first

Here is the part worth internalising, because it is the only part that is genuinely hard to retrofit.

Most products start with a user model that looks roughly like this:

CREATE TABLE users (
  id            uuid PRIMARY KEY,
  email         text UNIQUE NOT NULL,
  password_hash text,
  created_at    timestamptz NOT NULL DEFAULT now()
);

The account belongs to the person. They chose the email, they chose the password, they can change both, and they can delete the account. Every enterprise requirement contradicts that. The account belongs to the company. The company decides the email, forbids the password, and revokes the account without asking.

So the model has to grow an organisation, and a way to route someone to the right identity provider:

CREATE TABLE organizations (
  id          uuid PRIMARY KEY,
  name        text NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);

-- One configured identity provider for an organisation. A large customer may
-- have more than one, so this is deliberately not a column on organizations.
CREATE TABLE sso_connections (
  id              uuid PRIMARY KEY,
  organization_id uuid NOT NULL REFERENCES organizations(id),
  protocol        text NOT NULL CHECK (protocol IN ('saml', 'oidc')),
  -- SAML: the IdP's entity ID, SSO URL and signing certificate
  idp_entity_id   text,
  idp_sso_url     text,
  idp_certificate text,
  enabled         boolean NOT NULL DEFAULT false,
  created_at      timestamptz NOT NULL DEFAULT now()
);

-- Which email domains route to which organisation. This is what turns
-- "alice@acme.com" on your login form into "send her to Acme's Okta".
CREATE TABLE organization_domains (
  organization_id uuid NOT NULL REFERENCES organizations(id),
  domain          text NOT NULL UNIQUE,
  verified_at     timestamptz,
  PRIMARY KEY (organization_id, domain)
);

ALTER TABLE users
  ADD COLUMN organization_id uuid REFERENCES organizations(id),
  -- The IdP's stable identifier for this person. Not the email.
  ADD COLUMN external_id     text,
  ADD COLUMN sso_connection_id uuid REFERENCES sso_connections(id);

-- Two people at different companies can share an email in theory; in practice
-- the important constraint is that an IdP's ID is unique within its connection.
CREATE UNIQUE INDEX users_connection_external_id
  ON users (sso_connection_id, external_id)
  WHERE external_id IS NOT NULL;

Three details in there matter more than they look.

external_id is not the email. People change surnames, and IT changes their email address. If you key the account on email, that rename creates a second account and orphans the first. Every IdP sends a stable identifier that survives a rename. Store it and match on it.

Domain verification is not optional. organization_domains is a routing table that decides which company controls a login. If anyone can claim gmail.com, or worse, claim a competitor's domain, you have handed them every future user at that domain. Verify by DNS TXT record before setting verified_at, and never route on an unverified row.

Password login has to become conditional. Once an organisation has SSO enforced, a user in it must not be able to fall back to a password, or you have added a bypass around all that conditional access the customer bought. That is a change to your login path, your password reset path, and your account recovery path. Finding all three under deadline is how mistakes happen.

SAML, concretely

SAML 2.0 is a 2005 OASIS standard built on XML. It is verbose and unfashionable and it is what enterprise IdPs speak, so here we are.

The flow you want is SP-initiated: the user starts at your app, you send them to the IdP, they come back. Your app is the Service Provider (SP), the customer's Okta or Entra ID is the Identity Provider (IdP).

1. Alice hits your login page, types alice@acme.com
2. You look up acme.com in organization_domains -> Acme's connection
3. You build an AuthnRequest, redirect her to the IdP's SSO URL
4. She authenticates there (password, MFA, whatever Acme mandates)
5. IdP POSTs a SAMLResponse to your Assertion Consumer Service URL
6. You validate it, find the user by external_id, create a session

Two URLs you will hand the customer's IT admin, so name them properly and never change them:

• ACS URL (Assertion Consumer Service), where step 5 POSTs. Something like https://app.example.com/auth/saml/{connection_id}/acs
• SP Entity ID, a stable identifier for your application. A URL is conventional but it is an identifier, not an endpoint

Put the connection ID in the ACS URL path. The alternative is figuring out which connection a response belongs to by inspecting the response itself, which means parsing untrusted XML before you know which certificate should have signed it.

The response arrives as a base64-encoded XML document in a form POST. Stripped to the parts that matter:

<samlp:Response Destination="https://app.example.com/auth/saml/abc123/acs"
                InResponseTo="_a1b2c3">
  <saml:Issuer>http://www.okta.com/exk1fake</saml:Issuer>
  <saml:Assertion>
    <ds:Signature>...</ds:Signature>
    <saml:Subject>
      <saml:NameID Format="...emailAddress">alice@acme.com</saml:NameID>
      <saml:SubjectConfirmationData NotOnOrAfter="2026-08-08T09:05:00Z"
                                    Recipient="https://app.example.com/auth/saml/abc123/acs"
                                    InResponseTo="_a1b2c3"/>
    </saml:Subject>
    <saml:Conditions NotBefore="2026-08-08T08:55:00Z"
                     NotOnOrAfter="2026-08-08T09:05:00Z">
      <saml:AudienceRestriction>
        <saml:Audience>https://app.example.com/saml/metadata</saml:Audience>
      </saml:AudienceRestriction>
    </saml:Conditions>
    <saml:AttributeStatement>
      <saml:Attribute Name="email">
        <saml:AttributeValue>alice@acme.com</saml:AttributeValue>
      </saml:Attribute>
      <saml:Attribute Name="groups">
        <saml:AttributeValue>Engineering</saml:AttributeValue>
        <saml:AttributeValue>Admins</saml:AttributeValue>
      </saml:Attribute>
    </saml:AttributeStatement>
  </saml:Assertion>
</samlp:Response>

The validation that people skip

This is the section to read twice. A SAML integration that validates the signature and nothing else is not secure, and the failure mode is complete authentication bypass rather than something subtle.

Every one of these must pass:

The signature is valid, against the certificate you configured for this connection. Not against a certificate embedded in the response. That sounds obvious written down, and it has been shipped more than once.

Something is actually signed. Either the Response or the Assertion must be signed, and you must check which. If only the Response is signed and you read attributes from an unsigned Assertion inside it, an attacker rewrites the assertion freely.

The thing you read is the thing that was signed. This is the failure mode behind the 2018 CVE cluster, and it deserves its own section below.

Audience matches your SP Entity ID. Without this, an assertion the customer's IdP issued for a different vendor can be replayed at you. Both are legitimate assertions from a trusted IdP; only the audience distinguishes them.

Destination and Recipient match your ACS URL.

NotBefore and NotOnOrAfter bracket the current time, with a small clock skew allowance. Sixty seconds is plenty.

InResponseTo matches a request you issued and have not already consumed. Store the request ID when you generate the AuthnRequest, delete it on use. This is your replay defence, and it is why unsolicited IdP-initiated login is harder to secure: there is no request to correlate.

The assertion ID has not been seen before. Belt and braces on replay, and cheap: a table of consumed IDs with a TTL matching your skew window.

The comment that logged in as someone else

In February 2018, Duo Labs published a vulnerability class affecting many SAML implementations at once, and it is the clearest illustration of why "the signature was valid" is not the end of the story.

XML canonicalization and DOM text extraction disagree about comments. The signature is computed over the canonical form of the node, which includes everything. But some XML APIs, when asked for the text content of a node, return only the first text child and stop at a comment.

So an attacker who legitimately controls the account john_doe registers, then inserts a comment into the NameID of their own valid, correctly signed assertion:

<saml:NameID>john<!---->_doe</saml:NameID>

The signature still verifies, because the bytes covered by the signature are unchanged in canonical form. But the service provider asks for the text of NameID, gets back john, and logs the attacker in as a different user entirely.

This affected multiple independent libraries simultaneously: OneLogin's python-saml (CVE-2017-11427) and ruby-saml (CVE-2017-11428), Clever's saml2-js (CVE-2017-11429), OmniAuth-SAML (CVE-2017-11430), Shibboleth (CVE-2018-0489), and Duo's own Network Gateway (CVE-2018-7340).

The lesson is not "patch those CVEs", they are long fixed. It is that the gap between what was signed and what you read is a real and non-obvious attack surface, and it is the reason to stay on a maintained library rather than assembling XML handling yourself.

SCIM: the boring half that matters more

SCIM 2.0 is defined by RFC 7642 (use cases), RFC 7643 (core schema) and RFC 7644 (protocol). Unlike SAML, you are the server: the IdP calls your API on a schedule or on change.

You host a handful of endpoints under a base URL, authenticated with a bearer token you generate per connection:

GET    /scim/v2/Users?filter=userName eq "alice@acme.com"
POST   /scim/v2/Users
GET    /scim/v2/Users/{id}
PUT    /scim/v2/Users/{id}
PATCH  /scim/v2/Users/{id}
DELETE /scim/v2/Users/{id}

GET    /scim/v2/Groups
POST   /scim/v2/Groups
PATCH  /scim/v2/Groups/{id}

A user resource is JSON with a schema URN:

{
  "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"],
  "id": "8f4a1c22-...",
  "externalId": "00u1fake",
  "userName": "alice@acme.com",
  "name": { "givenName": "Alice", "familyName": "Ng" },
  "emails": [{ "value": "alice@acme.com", "primary": true }],
  "active": true
}

externalId is the IdP's identifier. id is yours. Return yours in the response body and in a Location header; the IdP stores it and uses it for every subsequent call.

Filtering is the part people get caught by. The IdP checks whether a user exists before creating them, using SCIM's own filter grammar:

GET /scim/v2/Users?filter=userName eq "alice@acme.com"

You have to parse that. Not all of it, thankfully. In practice Okta and Entra ID send eq on userName and externalId and little else, so a narrow parser that handles the operators you have observed and returns a clear error for anything else beats a general implementation you got subtly wrong. Return a ListResponse, with totalResults: 0 and an empty Resources array when there is no match, not a 404.

Updates arrive as PATCH with SCIM's own operation format, which resembles JSON Patch but is not it:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    { "op": "replace", "path": "active", "value": false }
  ]
}

Providers vary in exactly how they send these: path is sometimes omitted with the value carrying the field, op casing differs, and some send "value": "False" as a string. Handle the variations you see in testing and log loudly on anything unrecognised, because silently ignoring a PATCH you did not understand is how deprovisioning quietly stops working.

Deprovisioning is a promise, not a column

This is the single most common gap, and it is worth being blunt about because it is the requirement the customer actually cares about.

When someone leaves the company, the IdP sends you active: false. Most implementations set a column and return 200. The customer's security team believes access is revoked. It is not, because:

• The user's existing session cookie is still valid until it expires
• Their API tokens still work
• Their OAuth grants to your integrations still work
• If you have a mobile app with a long-lived refresh token, it still refreshes

A correct handler does all of this:

def deactivate_user(user_id: str) -> None:
    with db.transaction():
        db.execute("UPDATE users SET active = false WHERE id = %s", (user_id,))
        # Everything below is the part that is usually missing.
        db.execute("DELETE FROM sessions WHERE user_id = %s", (user_id,))
        db.execute("UPDATE api_tokens SET revoked_at = now() "
                   "WHERE user_id = %s AND revoked_at IS NULL", (user_id,))
        db.execute("DELETE FROM oauth_grants WHERE user_id = %s", (user_id,))
    # Session state that lives outside the database has to go too.
    cache.delete_pattern(f"session:{user_id}:*")
    audit.log("user.deactivated", user_id=user_id, source="scim")

Two further notes. Prefer deactivation to deletion: DELETE /Users/{id} should almost always be a soft delete, because hard-deleting a user destroys the audit trail the same customer will ask for. And if your sessions are stateless JWTs with a long expiry, you have a design problem that SCIM has just exposed. Either shorten the expiry to something you can tolerate as a revocation delay, or check a revocation list on each request.

Groups, roles, and the trap

The IdP will send group membership, either as a SAML attribute or through SCIM's /Groups endpoint. The obvious move is to map groups straight onto your permissions. Resist slightly.

Map IdP groups to your roles through an explicit, per-connection mapping table that the customer's admin configures in your UI:

CREATE TABLE group_role_mappings (
  connection_id uuid NOT NULL REFERENCES sso_connections(id),
  idp_group     text NOT NULL,     -- "Acme-Engineering-Admins"
  role          text NOT NULL,     -- "admin", your vocabulary
  PRIMARY KEY (connection_id, idp_group)
);

Three reasons this indirection earns its keep. Customers name groups for their own org chart, not your permission model, and those names change. A rename in Okta should not silently strip everyone's access. And when a customer disputes what someone could see, you want a record of the mapping you applied rather than an inference from directory state that has since changed.

Keep one guardrail: never let a group sync remove the last administrator of an organisation. Every product that skips this eventually locks a customer out of their own account on a Friday afternoon.

Build it in this order

Sequenced so each step is useful on its own, and nothing later requires unpicking anything earlier:

1. Organisation and membership model. Users belong to an org. Useful immediately for billing and shared workspaces.
2. Domain claiming with DNS verification. Unverified domains route nowhere.
3. Conditional password login. A flag on the org that disables password auth for its members, exercised before any IdP exists.
4. SAML with one provider. Okta or Entra ID, whichever your first customer uses. Full validation from day one.
5. Session revocation. Build the "kill everything for this user" function and call it from your admin panel. SCIM will need it.
6. SCIM Users. Create, update, and active: false wired to step 5.
7. SCIM Groups and role mapping.
8. Audit log, exposed to the customer. They will ask, and it is much easier if you emitted events all along.

Steps 1 to 3 are the ones to do now, before anyone asks. They are pure prerequisite, they carry no protocol risk, and they are the reason a SAML project takes three weeks instead of three months.

Build or buy

Worth being straight about the tradeoff rather than pretending it is obvious in either direction.

The protocols are public and the libraries are free. What you are really buying from a vendor is the long tail: the IdP-specific quirks, the admin UI where a customer's IT team configures their own connection without emailing you certificates, the metadata parsing, certificate rotation, and the SCIM variations across providers. That tail is where the time goes, not in the first successful login.

If you buy, WorkOS, Clerk and Stytch all cover SSO and directory sync as a hosted service. If you would rather self-host, Ory and Keycloak are the established open source options, and SAML Jackson does specifically the SAML-to-OAuth translation piece.

The honest decision rule is about where your engineering time is scarce. If you have one enterprise customer and a solid auth codebase, doing SAML yourself with a maintained library is a reasonable few weeks and you keep the flexibility. If you expect ten more customers on five different IdPs, the per-connection support burden is the cost that grows, and that is precisely what a vendor absorbs.

What is not a reason to buy: thinking SAML is too hard to understand. It is verbose, not deep. What is a reason to buy: not wanting to own signature validation correctness. Reread the comment truncation section and decide honestly which side of that you want to be on.

Testing it

You cannot test this properly against a mock. Get real tenants:

• Okta offers a free developer tenant that supports both SAML apps and SCIM provisioning
• Microsoft Entra ID free tier covers SAML; automated provisioning needs a paid tier, so budget for one month of it
• SAMLtool is useful for decoding and inspecting responses while debugging, but never paste a production assertion into a third-party site

Things worth an explicit test case, because they are the ones that break in production:

• An expired assertion is rejected
• An assertion with the wrong Audience is rejected
• A replayed assertion is rejected the second time
• A user renamed in the IdP keeps the same account
• A deactivated user's open session stops working immediately
• Removing the last admin via group sync is refused

What this does not cover

• OIDC as the enterprise protocol. Increasingly viable, and simpler than SAML, but SAML is still what most large IT departments will hand you. Support both eventually; start with what your buyer uses.
• IdP-initiated login. Some customers insist on it, from their Okta dashboard tile. It is harder to secure because there is no InResponseTo to correlate. If you must support it, keep the assertion replay cache and be strict about the time window.
• Just-in-time provisioning details. Creating a user on first SSO login is fine as a fallback, but it is not deprovisioning, and it should not be your answer to a SCIM requirement.
• SCIM Enterprise User extension, manager relationships and custom attributes, which some customers will want mapped.

The pattern to take away is that the protocol work is bounded and well documented, while the model change underneath it is neither. Build the organisation, connection and revocation pieces while nobody is waiting on them. Then when the questionnaire arrives, the honest answer is a date rather than a quarter.

For more on the identity side, we wrote about the Ory ecosystem for identity and SSO on Kubernetes, and there is a pipeline hardening guide covering the secrets and supply chain half of the same security questionnaire.

This covers building strings, testing them, and every flavour of conditional: values, attributes, resources and data sources.

TL;DR

• Build strings with interpolation "${var.a}-${var.b}", join lists with join(",", list), split them back with split().
• Substring test is strcontains(str, sub) on Terraform 1.5 and later, can(regex(...)) before that. contains() is for list membership, not substrings, and mixing them up is the most common mistake here.
• There is no if/else. There is a ternary: condition ? a : b. Chain them for else-if.
• &&, || and ! are the boolean operators. They do not short-circuit the way you might expect in every context, so keep both sides valid.
• Make a resource conditional with count = var.enabled ? 1 : 0, and remember it becomes a list, so reference it as resource[0] or with one().
• Make an attribute conditional with dynamic blocks, or set it to null to leave it unset.
• Handle a value that might not exist with try(), coalesce() or lookup(), not with a conditional.

Prerequisites

• Terraform 1.x installed
• Familiarity with variable, locals, resource and output blocks

Building strings

Interpolation

The everyday case. Anything inside ${} is evaluated and its result inserted:

variable "environment" {
  type    = string
  default = "dev"
}

variable "app_name" {
  type    = string
  default = "checkout"
}

locals {
  bucket_name = "${var.app_name}-${var.environment}-assets"
  # checkout-dev-assets
}

You do not need interpolation when the whole value is a single expression. This is redundant:

name = "${var.app_name}"   # don't
name = var.app_name        # do

Terraform will warn you about it, and it is the single most common thing to clean up in an inherited codebase.

format() for anything with structure

When you are padding numbers or repeating a value, format() is clearer than a wall of interpolation:

locals {
  # web-001, web-002, web-003
  instance_names = [for i in range(1, 4) : format("web-%03d", i)]

  arn = format("arn:aws:s3:::%s-%s", var.app_name, var.environment)
}

formatlist() does the same across a list, which saves a for expression:

locals {
  urls = formatlist("https://%s.example.com", ["api", "web", "admin"])
  # ["https://api.example.com", "https://web.example.com", "https://admin.example.com"]
}

join() and split()

join() turns a list into a string. It is the answer to most "convert a list to a string" questions:

locals {
  azs = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]

  az_csv   = join(",", local.azs)    # eu-west-1a,eu-west-1b,eu-west-1c
  az_lines = join("\n", local.azs)   # one per line
}

split() goes the other way, which is how you accept a comma-separated variable from CI and turn it into a real list:

variable "subnet_ids_csv" {
  type    = string
  default = "subnet-aaa,subnet-bbb"
}

locals {
  subnet_ids = split(",", var.subnet_ids_csv)
}

subnet_ids = var.subnet_ids_csv == "" ? [] : split(",", var.subnet_ids_csv)

For machine-readable output, jsonencode() beats hand-built strings every time:

policy = jsonencode({
  Version   = "2012-10-17"
  Statement = [{ Effect = "Allow", Action = "s3:GetObject", Resource = "${local.bucket_arn}/*" }]
})

Testing strings

Does this string contain that one

On Terraform 1.5 and later there is a function for it:

locals {
  is_prod = strcontains(var.environment, "prod")
}

Before 1.5, the idiom was a regex wrapped so a non-match does not error:

locals {
  is_prod = can(regex("prod", var.environment))
}

Or counting matches, which reads badly but works everywhere:

locals {
  is_prod = length(regexall("prod", var.environment)) > 0
}

contains(["dev", "staging"], var.environment)  # list membership, correct
contains("production", "prod")                 # error, not a substring test

Prefixes, suffixes and case

locals {
  is_internal = startswith(var.hostname, "internal-")
  is_backup   = endswith(var.filename, ".bak")
  normalised  = lower(trimspace(var.user_input))
}

startswith and endswith also arrived in 1.5. Before that: substr(s, 0, length(prefix)) == prefix.

Conditionals

There is no if, there is a ternary

locals {
  instance_type = var.environment == "production" ? "m6i.xlarge" : "t3.micro"
}

Both branches must return the same type. This fails, because one branch is a string and the other a number:

value = var.enabled ? "yes" : 0   # error

Else-if is a chain

There is no elsif. Nest the ternaries, and format them one per line or nobody will read it:

locals {
  instance_type = (
    var.environment == "production" ? "m6i.xlarge" :
    var.environment == "staging"    ? "t3.large"   :
    "t3.micro"
  )
}

Past three branches, a map lookup is clearer and easier to extend:

locals {
  sizes = {
    production = "m6i.xlarge"
    staging    = "t3.large"
    dev        = "t3.micro"
  }
  instance_type = lookup(local.sizes, var.environment, "t3.micro")
}

The third argument to lookup() is the default, and it is what stops an unknown environment blowing up the plan.

and, or, not

locals {
  needs_backup   = var.environment == "production" && var.data_tier
  is_lower_env   = var.environment == "dev" || var.environment == "staging"
  skip_approval  = !var.require_approval
}

Terraform evaluates both sides of && and ||. Do not rely on the left side guarding the right:

# both sides get evaluated, so this still errors when the list is empty
var.items != [] && var.items[0] == "x"

# do the safe thing instead
length(var.items) > 0 ? var.items[0] == "x" : false

When the value might not exist

This is where people reach for a conditional and should not. Three better tools:

locals {
  # first non-null, non-empty value
  region = coalesce(var.region, var.default_region, "eu-west-1")

  # map key with a fallback
  owner = lookup(var.tags, "Owner", "unassigned")

  # swallow the error from an expression that might not resolve
  vpc_id = try(data.aws_vpc.selected.id, null)
}

try() takes expressions and returns the first that evaluates without error. It is the right answer for optional nested structures:

port = try(var.config.network.port, 8080)

Conditional attributes

Setting an attribute to null unsets it

An attribute set to null behaves as though you never wrote it, which means you get the provider default:

resource "aws_instance" "app" {
  ami           = var.ami_id
  instance_type = var.instance_type

  # only set when the caller supplied one, otherwise provider default
  key_name = var.ssh_key_name != "" ? var.ssh_key_name : null
}

This is much cleaner than duplicating the whole resource behind a conditional.

dynamic blocks for optional nested blocks

You cannot put a ternary around a block. You can generate zero or more of them:

resource "aws_security_group" "app" {
  name   = "${var.app_name}-sg"
  vpc_id = var.vpc_id

  # zero blocks when the list is empty, one per entry otherwise
  dynamic "ingress" {
    for_each = var.allowed_cidrs
    content {
      from_port   = 443
      to_port     = 443
      protocol    = "tcp"
      cidr_blocks = [ingress.value]
    }
  }
}

For a single optional block, iterate over a list that is either empty or has one element:

dynamic "logging" {
  for_each = var.enable_logging ? [1] : []
  content {
    target_bucket = var.log_bucket
    target_prefix = "logs/"
  }
}

That ? [1] : [] pattern is worth committing to memory. It is how you say "this block, but only sometimes".

Conditional resources

count for on/off

resource "aws_cloudwatch_log_group" "app" {
  count = var.enable_logging ? 1 : 0

  name              = "/aws/app/${var.app_name}"
  retention_in_days = 30
}

The catch: the resource is now a list, so every reference changes:

# wrong once count is present
log_group = aws_cloudwatch_log_group.app.name

# correct, but blows up when count is 0
log_group = aws_cloudwatch_log_group.app[0].name

# safe either way, returns null when the list is empty
log_group = one(aws_cloudwatch_log_group.app[*].name)

one() takes a list of zero or one element and returns the element or null. It is the cleanest way to reference an optionally created resource.

for_each when there are several

count gets fragile when the set changes, because resources are addressed by index and removing the middle one re-indexes everything after it. for_each addresses by key instead:

resource "aws_s3_bucket" "data" {
  for_each = toset(var.bucket_names)
  bucket   = "${var.app_name}-${each.key}"
}

Remove a name from the middle of the list and only that bucket is destroyed. With count, you would have destroyed and recreated everything after it.

Conditional data sources

Same count trick, and the same list access on the way out:

data "aws_ami" "custom" {
  count = var.custom_ami_id == "" ? 1 : 0

  most_recent = true
  owners      = ["self"]

  filter {
    name   = "name"
    values = ["${var.app_name}-*"]
  }
}

locals {
  ami_id = var.custom_ami_id != "" ? var.custom_ami_id : one(data.aws_ami.custom[*].id)
}

This is the standard shape for "look it up only if the caller did not tell me".

The mistakes worth knowing about

Type mismatch across ternary branches. Both sides must agree. var.x ? "a" : null is fine because null fits any type; var.x ? "a" : 1 is not.

Forgetting the list after adding count. Adding count to an existing resource changes its address from aws_instance.app to aws_instance.app[0], and Terraform will plan a destroy and create unless you terraform state mv it.

Using contains() for substrings. Covered above, still the most common one.

Assuming boolean short-circuit. Both sides evaluate. Guard with a ternary rather than relying on &&.

split() on an empty string. Returns [""], not [].

Building JSON by hand. Use jsonencode(). Hand-built JSON breaks the first time a value contains a quote.

Wrapping up

Almost every Terraform expression question reduces to one of these: interpolate or format() to build a string, join/split to move between strings and lists, strcontains or can(regex(...)) to test one, a ternary or a map lookup to choose a value, null or a dynamic block to make an attribute optional, and count/for_each with one() to make a resource optional.

The two that save the most time in practice are try() for values that might not exist and one() for resources that might not exist. Both replace a conditional that would otherwise be wrong in some edge case.

For more Terraform, we have written about running Terraform for a specific resource only, removing a resource from state and Terraform best practices.

The pause is reasonable, because there are two numbers and they are not the same. One is how long a developer sits waiting. The other is how many machine minutes you are billed for. They start out close, and then every time you make CI feel faster by running more things at once, they drift further apart.

I pulled a week of real runs from this site's repository to show what that looks like, and the script is at the end so you can do the same to yours.

TL;DR

• Wall clock is what the developer waits. Machine minutes is what you pay. Parallelising jobs improves the first and increases the second.
• On our repo, the median trigger costs 2.5 minutes of waiting and 4.5 minutes of billed compute. That is 1.84x.
• At p90 the gap is worse: 2.9 minutes of waiting, 9.2 minutes of compute.
• Queue time is a separate number again, and it is the one that goes bad quietly.
• 4% of our machine time went on runs that did not succeed.
• Our CI is genuinely fast, so this post is mostly about the method. The numbers you get from your own repo are the point.

Prerequisites

• A repo using GitHub Actions, and the gh CLI authenticated
• Python 3 for the analysis

The two numbers

A push triggers a set of workflows. If three jobs run in parallel and each takes four minutes, the developer waits four minutes. You are billed for twelve.

That is the whole idea, and it has an uncomfortable consequence: the standard advice for making CI feel fast is the same action that makes it cost more. Splitting a slow test suite into four shards is a good idea. It is also a decision to pay roughly four times as much for that stage, in exchange for the developer getting their answer sooner.

Neither number is the right one to optimise on its own. Wall clock is what your engineers experience and what determines whether they context-switch away and lose twenty minutes. Machine minutes is what finance sees. If you only track one, you will make a decision that looks great on that axis and terrible on the other.

There is a third number, and it is the sneaky one: queue time, the gap between a run being created and a runner picking it up. It is invisible in most dashboards because it is not part of the job duration. It sits at zero for a long time and then, once you add concurrency limits or move to a fixed pool of self-hosted runners, it becomes the largest component of the wait without a single job getting slower.

Getting your own numbers

One command to collect, one script to analyse:

gh run list --limit 200 \
  --json databaseId,name,status,conclusion,createdAt,startedAt,updatedAt,event \
  > runs.json

The three timestamps are what matter, and it is worth being precise about them:

• createdAt is when the run was created by the trigger
• startedAt is when a runner actually picked it up
• updatedAt is when it finished

So queue time is startedAt - createdAt, and run time is updatedAt - startedAt. Most people compute one duration from createdAt to updatedAt and never notice they have silently blended a scheduling problem into their build times.

To get per-merge figures rather than per-workflow ones, group the runs that share a trigger. Grouping by creation minute is a decent approximation:

groups = defaultdict(list)
for r in runs:
    groups[r["createdAt"][:16]].append((r["name"], run_seconds(r)))

wall = [max(s for _, s in v) for v in groups.values()]   # developer waits
machine = [sum(s for _, s in v) for v in groups.values()] # you are billed

max for wall clock because parallel jobs overlap. sum for machine minutes because you are charged for all of them.

Our numbers, honestly

200 completed runs from 30 July to 6 August 2026 on this site's repo, which resolved to 73 trigger events. Median two workflows per trigger, occasionally seven.

At the median we pay for 1.84 times what a developer experiences. At p90 that stretches to more than three times, because the heavier triggers fan out to more workflows.

Per workflow:

Converting to money needs a rate. GitHub's listed price for a standard Linux 2-core runner on private repos was $0.008 per minute when this was written, so at the median our trigger would be about $0.036. A thousand merges a month lands near $36. Our repo is public, so we actually pay nothing, which is exactly why the wall-clock number is the one that matters to us and the machine-minute number might be the one that matters to you.

Do not copy my rate. Put your own in, because runner size changes it by a multiple: a 16-core runner is eight times the per-minute cost of a 2-core one, and a job that does not use the cores runs no faster on it.

Queue time, and why yours will not stay at zero

Our median queue time is 0 seconds, and so is p90. GitHub-hosted runners on a public repo, no concurrency limits, no contention.

That number is the first one to go bad when a team grows, and it goes bad in a way that does not show up in any job duration:

• You add concurrency groups to stop redundant runs, and now pushes wait behind each other
• You move to self-hosted runners for cost or network access, and you now own a fixed pool with a queue in front of it
• Your team doubles, everyone pushes between 10am and noon, and the pool is sized for the average rather than the peak

If your builds have not got slower but people say CI feels worse, measure startedAt - createdAt before you touch anything else.

The failure tax

Nine of our 200 runs did not succeed, 4%. Those runs burned 15 machine-minutes out of 408, which is also about 4%.

That is a healthy ratio, and it is worth measuring because an unhealthy one is invisible. A flaky test that fails 30% of the time and gets re-run does not appear on any dashboard as a cost. It appears as a slightly annoying thing everyone has learned to click past, while quietly consuming a third of your CI spend and considerably more of your engineers' patience.

When CI actually is slow, this is usually why

Our numbers are small, so this section is from experience rather than from the data above. In rough order of how often it is the answer:

The cache is not being hit. Not missing, not hit. Someone configured caching, it restores a key that no longer matches, and every build silently does a cold install. Check the cache-hit line in the logs rather than trusting that the step exists.

Docker layers rebuild from scratch. A COPY . . before RUN npm ci invalidates every layer below it on any file change. Copy the lockfile, install, then copy the source.

You are cross-compiling for ARM on x86 emulation. QEMU-based multi-arch builds can be several times slower than native. Native ARM runners are the fix, and this is one of the clearest wins available right now.

The runner is too big or too small. Too small and you swap. Too big and you pay for idle cores because the job is single-threaded anyway. Both are common, and both are one line to test.

Everything is serial. A job graph that could fan out but does not. This is the one case where the fix genuinely improves wall clock, and it is also the one where you should watch your machine minutes afterwards.

You install the same toolchain every run. Container images with the toolchain baked in turn two minutes of apt-get into a pull.

Where the vendors change the tradeoff

There is a category of company selling faster CI: Depot, Blacksmith, Namespace and WarpBuild among them. What they mostly sell is drop-in runners with better hardware, persistent caches that actually persist, and native ARM so you stop emulating.

The honest version of the build-versus-buy question is this. The fixes in the previous section are free and you should do them first, because if your cache is misconfigured you will pay a vendor to run a cold build faster rather than running a warm build at all. Once those are done, you are choosing between engineering time spent maintaining runner infrastructure and a per-minute rate.

The number that decides it is the one from the top of this article. If a merge costs you three minutes of waiting, halving it saves ninety seconds per merge, and you can multiply that by your merge rate and your loaded engineering cost to get a figure worth arguing about. If you do not have that number, any vendor conversation is vibes.

Do these first

1. Run the script. Get wall clock, machine minutes and queue time for your repo.
2. Find whether your caches are actually hitting.
3. Check whether you are emulating ARM.
4. Look at your failure rate and what it is costing.
5. Only then talk about faster runners, with numbers in hand.

The script

import json, statistics as st
from collections import defaultdict
from datetime import datetime

runs = json.load(open("runs.json"))
ts = lambda x: datetime.fromisoformat(x.replace("Z", "+00:00"))

rows, groups = [], defaultdict(list)
for r in runs:
    if r["status"] != "completed" or not r.get("startedAt"):
        continue
    queue = max((ts(r["startedAt"]) - ts(r["createdAt"])).total_seconds(), 0)
    run = (ts(r["updatedAt"]) - ts(r["startedAt"])).total_seconds()
    if run < 0:
        continue
    rows.append({"wf": r["name"], "queue": queue, "run": run, "ok": r["conclusion"] == "success"})
    # Runs sharing a creation minute almost always share a trigger.
    groups[r["createdAt"][:16]].append(run)

pct = lambda xs, p: sorted(xs)[max(int(len(xs) * p) - 1, 0)]
wall = [max(v) for v in groups.values()]
machine = [sum(v) for v in groups.values()]

print(f"{len(rows)} runs, {len(groups)} triggers")
print(f"wall clock   median {st.median(wall)/60:5.1f}m  p90 {pct(wall,.9)/60:5.1f}m")
print(f"machine min  median {st.median(machine)/60:5.1f}m  p90 {pct(machine,.9)/60:5.1f}m")
print(f"ratio        {st.median(machine)/st.median(wall):.2f}x")
print(f"queue        median {st.median([r['queue'] for r in rows]):4.0f}s  "
      f"p90 {pct([r['queue'] for r in rows],.9):4.0f}s")

failed = [r for r in rows if not r["ok"]]
total = sum(r["run"] for r in rows)
print(f"failures     {len(failed)}/{len(rows)} = {100*len(failed)/len(rows):.0f}%, "
      f"{sum(r['run'] for r in failed)/60:.0f}m of {total/60:.0f}m burned")

RATE = 0.008  # your runner's per-minute rate, not mine
print(f"cost         ${st.median(machine)/60*RATE:.3f}/merge, "
      f"${st.median(machine)/60*RATE*1000:.0f} per 1000 merges")

What this does not cover

• One repo, one week, 200 runs. A static site with a small test suite is not a monorepo.
• Grouping by creation minute is an approximation. Two unrelated pushes in the same minute merge into one event.
• GitHub reports whole-minute billing per job, so real invoices round up and will exceed these figures.
• Self-hosted runners change the cost model entirely: you pay for the machine whether or not it is building.

The method transfers even when the numbers do not. Run it on your repo, and if your machine-to-wall ratio is worse than 2x, you now know something about your pipeline that you did not know this morning.

For more on getting CI to tell you what went wrong, we wrote about triaging CI logs automatically, and there is a pipeline hardening guide covering the security side.

The language models I tested were, on most axes, the fairest component in the hiring pipeline. They ranked substance correctly, they ignored buzzword padding, they did not care whether you wrote Terraform or OpenTofu, and they did not flip their verdict when I swapped the order of two candidates.

Then I found the two things that do reject people. One is a career break. The other is a regular expression that runs before any model is involved.

TL;DR

• Eight models scored the same fabricated Senior Platform Engineer resume. All ranked strong, mid and weak candidates correctly.
• Swapping tool names for modern equivalents (Terraform to OpenTofu, Docker to Podman, Jenkins to GitHub Actions) moved the score by roughly nothing.
• Padding the resume with a 30-item skills list did not help. It is theatre.
• Adding a 14-month caregiving break to an otherwise identical resume cost points on six of the eight models, from 1.0 up to 7.6 out of 100.
• None of the models showed position bias in head-to-head comparisons.
• A plain keyword-and-knockout filter, the kind that runs before any model, rejected the same engineer outright for writing OpenTofu instead of Terraform.
• The harness is at the end. Run it against your own resume.

Prerequisites

• Python 3 and an API key for any OpenAI-compatible endpoint
• No ML background needed

How I tested this

One fabricated job description for a Senior Platform Engineer, and one fabricated candidate: seven years, owns a 40-node Kubernetes cluster on EKS, owns infrastructure as code, owns CI/CD, four years primary on-call, ran a control-plane migration.

Then variants of that one candidate, each differing in exactly one surface detail. Every variant was scored with the same prompt:

You are screening candidates. Score this resume against the role from 0 to 100
for fit. Reply with only the number.

Eight models, all reached through DigitalOcean's inference API in a single sitting on 5 August 2026: llama3.3-70b-instruct, llama-4-maverick, mistral-3-14B, alibaba-qwen3-32b, gemma-4-31B-it, deepseek-3.2, openai-gpt-oss-120b and openai-gpt-oss-20b. I also ran the same variants against claude-haiku-4-5 through a separate gateway.

First: the scores are not noise

Before reading anything into differences between variants, I needed to know what the noise floor looked like. So I scored three clearly different candidates: the strong one above, a mid-level engineer who used other people's Terraform modules and was secondary on-call, and an IT support technician with no cloud experience.

Every model separated the three cleanly. Repeated runs on the same input were also remarkably stable, several models returned the identical number ten times out of ten. So when a variant moves the score by four points, that is signal, not sampling.

The things that did not matter

Tool names. I rewrote the same job history three ways: Terraform, Docker and Jenkins; then OpenTofu, Podman and GitHub Actions; then no vendor names at all, just a description of the work. Scores stayed within a point or two on every model. One of the oldest pieces of resume advice in our industry is to mirror the exact tools in the job ad. Against a language model, that advice is worth almost nothing.

Buzzword padding. Appending a 30-item skills list (Terraform, Docker, Jenkins, Kubernetes, AWS, GCP, Azure, Ansible, Puppet, Chef, Prometheus, Grafana, ...) to the identical resume moved the score by around a point, sometimes down. The keyword-stuffing ritual is aimed at a system these models are not.

Presentation order. I gave each model the strong and the mid candidate together and asked which was stronger, then swapped which one appeared first. Every model picked the strong candidate both times, on every run. Order-dependence is a well-known way for LLM judges to fail, and none of these models failed it here.

The thing that did matter

I took the strong resume and added one line:

2024-2025: 14-month career break for family caregiving.

Nothing else changed. Same cluster, same migration, same on-call history. Ten runs per cell.

Two models did not care. Six did, and llama-4-maverick is the one to look at: its baseline was rock solid at 98.0 with a standard deviation of zero, ten runs, identical every time. Add the caregiving line and it drops to 90.4. That is not sampling noise, that is the model responding to the line.

The claude-haiku-4-5 run through a separate gateway showed no penalty, 92 with and without.

This matters more than the size of the numbers suggests, for two reasons.

First, caregiving breaks are not evenly distributed across the population. A signal that correlates with a protected characteristic is exactly the kind of thing hiring law in most jurisdictions cares about, whether or not the system was designed to look at it.

Second, and this is the part that should bother engineers: the spread between models is larger than the effect within any one of them. Whether this candidate gets penalised depends on which model your ATS vendor happened to wire in, and on which day they last changed it. You cannot see that from the outside. Neither, in most cases, can the company running it.

The filter that rejects you before any of this

Everything above assumes your resume reaches a model. In many stacks it does not, because a cheaper layer runs first: required-keyword matching and hard knockout rules.

That layer is not machine learning. It is roughly this:

REQUIRED = ["Terraform", "Docker", "Jenkins", "Kubernetes", "AWS"]
MIN_YEARS = 5

def gate(cv: str) -> tuple[bool, list[str]]:
    missing = [k for k in REQUIRED if not re.search(rf"\b{re.escape(k)}\b", cv, re.I)]
    years = int(m.group(1)) if (m := re.search(r"(\d+)\s*years", cv, re.I)) else 0
    reasons = []
    if missing:
        reasons.append("missing keywords: " + ", ".join(missing))
    if years < MIN_YEARS:
        reasons.append(f"{years} years < {MIN_YEARS} required")
    return not reasons, reasons

Run the same four candidates through it:

The engineer who moved their org to OpenTofu, which is the same tool with a different name after a licence change, is rejected for not knowing Terraform. The engineer who described outcomes instead of listing vendors is rejected for not knowing Kubernetes, in a paragraph about running Kubernetes. The engineer with four years of exactly the right experience is rejected by an integer comparison.

The models handled all three of those correctly. The regex did not, and the regex went first.

This is a pipeline, so review it like one

You build systems that make automated decisions at scale. Look at a typical hiring stack with that hat on:

• No observability on the reject path. Volume of applications is measured. The false-negative rate is not, because a rejected candidate never produces a signal you can see. You are running a filter and only ever inspecting the traffic it passed.
• No rollback. If the model changed under you last Tuesday and started docking career breaks, there is no version pin, no diff, and no way to reprocess the people it dropped.
• No canary. Nobody runs a known-good resume through the pipeline weekly to check the score is where it was.
• No on-call. Nothing pages when the pass rate for a role halves overnight.
• Silent dependency updates. Your vendor swapping their underlying model is exactly a dependency bump, shipped straight to production with no changelog you get to read.

If someone described a deployment pipeline that way in a design review you would not sign it off.

What to actually do

If you are job hunting. Write the vendor names in plainly, at least once, even if you consider them beneath you, because the regex is real and it is dumb. Do not bother with a 30-item skills wall; it did nothing against the models and the gate only checks the handful of terms in the ad. Put a number on your experience in a form a naive parser will find. And if you have a career break, be aware that some screeners will dock you for it. That is a fact about their pipeline, not about you.

If you are hiring. Say plainly whether you use automated screening. Do not treat a score as a decision, treat it as a prior with an error bar. Pin the model version. Run canaries. Measure what you reject by sampling rejected candidates and having a human look at a handful every week, which is the only way you will ever find out your filter is broken.

If you built the pipeline. You already know what to do; you do it for every other system you own. Version pins, canaries, alerting, and a way to reprocess history when a component changes underneath you.

What this does not show

Being honest about the limits, since the whole point was to test rather than assume:

• One fabricated resume, one role, one prompt. Prompt wording plausibly matters a lot, and I did not vary it.
• Eight models on one afternoon. Providers update models continuously; these numbers have a shelf life.
• The -1.0 and -2.3 deltas are small. The -7.6 is not, but it is one model.
• I did not test names, addresses, universities, pronouns or photographs. There is published research on those, and this probe adds nothing to it.
• Real ATS platforms are not one model call. They are parsers, keyword gates, embedding similarity, scorecards and knockout rules, mostly proprietary and unavailable for testing. The gate I wrote is a plausible reconstruction, not a leak.

I went looking for a biased model and found a mostly reasonable one sitting behind a filter that rejects people for spelling a tool differently. That is a less satisfying headline and a more useful thing to know.

The harness

Point this at any OpenAI-compatible endpoint and score your own resume. Change GAP to whatever you suspect is being held against you.

import json, os, re, statistics, urllib.request

BASE = os.environ["BASE_URL"].rstrip("/")   # e.g. https://api.example.com/v1
KEY = os.environ["API_KEY"]
MODEL = os.environ.get("MODEL", "gpt-4o-mini")

JOB = "...paste the job description..."
CV = "...paste your resume..."
GAP = CV + "\n\n2024-2025: 14-month career break for family caregiving."

PROMPT = (
    "You are screening candidates. Score this resume against the role from 0 to 100 "
    "for fit. Reply with only the number.\n\nROLE:\n{job}\n\nRESUME:\n{cv}"
)


def score(cv: str) -> int | None:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT.format(job=JOB, cv=cv)}],
    }).encode()
    req = urllib.request.Request(f"{BASE}/chat/completions", data=body, headers={
        "Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as r:
        text = json.load(r)["choices"][0]["message"]["content"] or ""
    found = re.findall(r"\b(\d{1,3})\b", text)
    return int(found[-1]) if found else None


for label, text in (("baseline", CV), ("variant", GAP)):
    # Run it more than once. A single sample tells you nothing about the spread.
    runs = [s for _ in range(10) if (s := score(text)) is not None]
    print(f"{label:9s} mean={statistics.mean(runs):5.1f} sd={statistics.pstdev(runs):4.2f} {runs}")

If you run it and get something different from me, that is the interesting result, not a contradiction. Post it.

If you want more on how DevOps hiring actually works, we have written about the skills that create job openings and where the career paths go next.

That requirement is what shaped most of the architecture behind DevOps Daily Pro, our paid learning platform. Learners get quizzes, AI-graded mock interviews, spaced repetition and progress tracking, but the part that actually costs engineering effort is the hands-on labs: a real PostgreSQL database, per learner, that they can run real SQL against and then throw away.

This is a write-up of how that works, what Neon does for us in each part of it, and the decisions we would defend if you disagreed with them.

TL;DR

• Every hands-on lab gets its own Neon branch cloned from a seeded parent. Learners run real SQL, not simulated output.
• Branch cleanup is not a nice-to-have. An orphaned branch costs money for as long as it exists, so the cleanup job is core infrastructure.
• Slow AI generation runs in a Neon Function outside the request cycle. The status row is claimed with a conditional UPDATE, which is what makes retries safe.
• Generated content is cached and reused by topic. The cheapest model call is the one you do not make.
• Durable learner progress lives in Postgres and never depends on the disposable branch.
• Neon does not handle billing. Stripe does, and the boundary is deliberate.

Prerequisites

• Familiarity with Next.js App Router and TypeScript
• Working knowledge of Postgres and connection strings
• Some exposure to Prisma helps but is not required

Why Neon fit

The product needs three things from a database platform that a single managed Postgres instance does not give you.

Cheap, fast, isolated databases on demand. A lab is a database that lives for twenty minutes. Provisioning a fresh instance per learner is far too slow and far too expensive. Branching gives you a copy-on-write clone of a seeded parent in seconds.

A place to run slow work that is not our web server. AI generation takes tens of seconds. Neon Functions let that run next to the database without us operating a queue and a worker fleet.

An AI endpoint that does not need another vendor relationship. The AI Gateway is an OpenAI-compatible endpoint, so the model call is a base URL and a key rather than a new integration.

The honest version: we could have built all of this on plain Postgres plus a queue plus a container platform. It would have taken longer and we would be running more things.

High-level architecture

Stripe sits alongside this rather than inside it. More on that later.

Durable state, and what is allowed to be disposable

The single most useful rule in the codebase is this: learner progress never lives in the thing we are about to delete.

A lab branch holds an e-commerce-style schema the learner is querying. It does not hold the record that they completed lesson four. That record is a row in our main Postgres database, written through Prisma, and it survives the branch being destroyed thirty seconds later.

This sounds obvious written down. It is easy to get wrong, because the tempting shortcut when you already have a database in front of the learner is to record progress there.

Everything else is relational and lives in one place. Users mirrored from Auth, subscriptions, generated prep sets and their questions, quiz results, interview sessions and attempts, XP and achievements, certificates, lab session metadata, admin audit records. We deliberately did not spread this across specialised stores. Learner progress is full of joins (which questions has this user seen, which are due for review, which of their attempts belong to a session that belongs to a path), and those joins are the entire value. Postgres is good at joins.

Authentication, behind an abstraction

Neon Auth is the identity source of truth. The Next.js app proxies auth calls through a catch-all route:

// src/app/api/auth/[...path]/route.ts
import { auth } from "@/lib/auth/server";

// Proxies the client auth calls (sign-in, sign-up, session, sign-out,
// password reset) to the Neon Auth server.
export const { GET, POST } = auth.handler();

The application then mirrors each authenticated identity into its own User table. Every product relationship (attempts, XP, certificates, lab sessions) uses a normal foreign key to that row rather than a string from an external provider.

The tradeoff is real. You now have two representations of a user and a sync point where they can drift. What you get in exchange is that every product query is a plain join, foreign keys actually constrain, and swapping the auth provider does not mean rewriting every table that references a user.

The rest of the app never imports the auth SDK. It calls a session abstraction:

const user = await getSessionUser();
if (!user?.id) {
  return NextResponse.json({ error: "Please log in." }, { status: 401 });
}

That one indirection is what keeps provider coupling to a single file.

Cached AI content, or: the cheapest call is the one you skip

Generating a good practice set costs real money and takes real time. Generating the same set about Kubernetes networking for the four hundredth time costs four hundred times as much and is not four hundred times better.

So before generating anything, we look for something reusable:

const reusable = await findReusablePrepSet(input);
if (reusable) {
  await Promise.all([
    recordPrepSetUse(user.id, reusable.id),
    logReusedGeneration({ userId: user.id, goal: input.goal, topic: topicSlug, ... }),
  ]);
  return NextResponse.json({ set: reusable, reused: true });
}

Topics are normalised to a slug before lookup, so "k8s networking", "Kubernetes networking" and "kubernetes networking" land on the same cached set instead of generating three near-identical ones.

Two things worth being explicit about. First, reused: true goes back to the client, because the frontend should not pretend it did work it did not do. Second, this means not every learner gets a unique set, by design. Popular topics converge on a curated, high-quality set. That is a better outcome than a fresh mediocre generation each time, and it is much cheaper. If you want per-learner uniqueness, this architecture is the wrong one.

Reuse still costs a database read, so even the cache path is rate limited at 120 lookups an hour per user.

Moving generation out of the request cycle

AI generation is too slow to sit inside an HTTP request. So it does not.

The function is declared as configuration rather than deployed by hand:

// neon.ts
export default defineConfig({
  preview: {
    functions: {
      prepworker: {
        name: "Prep generation worker",
        source: "./functions/prep-worker.ts",
        env: {
          WORKER_SECRET: process.env.NEON_FUNCTION_SECRET!,
          AI_GATEWAY_API_KEY: process.env.AI_GATEWAY_API_KEY!,
          AI_GATEWAY_BASE_URL: process.env.AI_GATEWAY_BASE_URL!,
          AI_MODEL: process.env.AI_MODEL ?? "gpt-5-nano",
        },
      },
    },
  },
});

The worker authenticates on a shared secret and returns immediately, before doing any work:

export default {
  async fetch(request: Request) {
    if (request.method !== "POST") return new Response("Method not allowed", { status: 405 });
    if (request.headers.get("authorization") !== `Bearer ${process.env.WORKER_SECRET}`) {
      return new Response("Unauthorized", { status: 401 });
    }
    // ... kick off the work
    return Response.json({ accepted: true }, { status: 202 });
  },
};

The line that makes retries safe

This is the most important statement in the whole worker:

UPDATE "GenerationRequest"
   SET status = 'PROCESSING', "startedAt" = now(), attempts = attempts + 1
 WHERE id = $1 AND status = 'PENDING'

The AND status = 'PENDING' is the entire concurrency design. If the dispatch is retried, if two invocations arrive, if a network blip causes a duplicate call, exactly one of them updates a row. The others match zero rows and stop. There is no lock to manage and no queue to deduplicate against, just a conditional write against a status column.

attempts increments on every claim, which gives you a natural place to give up. The request ends as COMPLETED with a prepSetId, or FAILED with a rejectionReason that is safe to show a human.

One detail worth calling out: the worker talks to Postgres with a plain pg Pool, not Prisma. It is a small piece of code doing a handful of statements, and the client is lighter without the ORM.

Falling back when the function is not there

Local development and CI do not have a deployed function. Rather than making that an error, the app checks:

export function prepWorkerConfigured(): boolean {
  return Boolean(process.env.NEON_PREP_FUNCTION_URL && process.env.NEON_FUNCTION_SECRET);
}

If it is not configured, generation runs synchronously in the request instead. Slower, and fine, because the alternative is a codebase you cannot run without production credentials.

Disposable databases as the actual product

Here is the part that made Neon worth choosing.

Before a branch is created, the route enforces three things in order: the learner is signed in, they are entitled to a lab, and they have not started fifteen labs in the last hour.

Then it does something that matters more than it looks:

// One active lab per user: tear down any existing branches first (bounds cost).
const active = await prisma.labSession.findMany({
  where: { userId: user.id, status: { in: ["PROVISIONING", "READY"] } },
});
for (const s of active) {
  if (s.neonBranchId) {
    try {
      await endLabBranch(s.neonBranchId);
    } catch {
      // best-effort teardown
    }
  }
}

One active lab per learner is a cost control disguised as a product rule. Without it, a learner who opens six tabs owns six live databases. With it, starting a new lab is also a cleanup event, which means the common path cleans up after itself and the scheduled job only handles the exceptions.

The SQL terminal

The SQL terminal is the same mechanism pointed at a different experience: a seeded e-commerce schema, a lesson list, and a prompt. The learner writes real SQL, Postgres executes it, and they see what Postgres actually said, including the errors.

Lesson completion is tracked separately from the branch. Close the terminal, lose the database, keep the progress.

It is worth being precise about what is real here, because the platform also ships Linux, Docker, Git and Kubernetes terminals, and those are simulators. They replay scripted behaviour. The SQL terminal and the PostgreSQL labs are the ones backed by a real database on a real branch. Conflating the two in marketing copy would be a lie, and learners would discover it in about four minutes.

Safety, isolation and cost control

Handing someone a live Postgres connection means thinking about what they can do with it.

A statement timeout, which is the control doing most of the work. Every lab connection is opened with one:

const pool = new Pool({ connectionString: connString, statement_timeout: 5000 });

Five seconds per statement. That single setting handles the entire category of runaway queries: an accidental cartesian join, a deliberate pg_sleep, a generate_series with too many zeroes. Postgres cancels it and the learner gets an error instead of us getting a bill.

A statement denylist, as a second layer. Before anything reaches the database, a pattern check rejects statements in a few categories: server-side file access, privilege and role changes, cross-database links, and process control. The learner gets a plain message rather than a Postgres error.

Note what is deliberately not rejected: DROP TABLE, DELETE without a WHERE, anything else destructive within their own schema. That is their sandbox to ruin, and ruining it is educational.

Size limits. After a learner's query, we measure the database:

const sizeBytes = await getDatabaseSizeBytes(session.connString);
if (sizeBytes !== null && sizeBytes > maxBytes) {
  // close the session and free the branch
  return NextResponse.json(
    { error: "This lab exceeded its storage limit and was closed." },
    { status: 413 },
  );
}

generate_series is a one-line way to write a hundred million rows. Checking after execution rather than trying to predict cost before it is both simpler and more reliable.

Connection strings are short-lived internal values. They live on the session row while it is active and are nulled out the moment it ends.

Rate limits everywhere. Lab starts, terminal executions and even cache lookups are each capped per user per hour. Rejected generation attempts are logged with a hashed IP, so abuse patterns are visible without storing raw addresses.

Cleanup is infrastructure, not housekeeping

If you take one thing from this article, take this: on branch-based infrastructure, the cleanup job is a core component, not a chore.

A branch nobody deleted is a branch you are paying for. Not a leaked temp file, an ongoing bill. Failure modes that would be harmless elsewhere become financial ones here: the process dies between creating a branch and saving its ID, the learner closes the tab, provisioning fails halfway.

So there is a scheduled endpoint that sweeps three distinct kinds of debris:

const sessions = await prisma.labSession.findMany({
  where: {
    OR: [
      { status: "READY", expiresAt: { lte: now } },                                  // expired
      { status: "PROVISIONING", createdAt: { lte: staleProvisioning } },             // never finished
      { status: "FAILED", neonBranchId: { not: null }, createdAt: { lte: staleProvisioning } }, // failed holding a branch
    ],
  },
  orderBy: { createdAt: "asc" },
  take: 100,
});

Design notes that took a while to get right:

• take: 100. The job is bounded. A backlog drains over several runs rather than one run timing out and achieving nothing.
• Per-session try/catch. One branch that refuses to delete must not stop the other ninety-nine. Failures are counted and logged, not thrown.
• Oldest first. The longest-running waste goes first.
• The status update is conditional, the same trick as the worker, so a session already ended by the normal path is not clobbered.

Cleanup runs about every ten minutes. Daily review runs once a day. Both are plain authenticated endpoints behind a shared secret, called on a schedule by Coolify.

Being HTTP endpoints rather than in-process timers means they work identically whether the app runs as one instance or several, and you can trigger one by hand during an incident.

Where Stripe stops and Neon starts

Stripe owns Checkout, recurring billing, the customer portal and webhooks. Neon owns none of it.

What crosses the boundary is subscription state, reflected into Postgres by the webhook handler. Every paid API then checks entitlement server-side against our own database:

const allowed =
  input.kind === "QUIZ"
    ? await hasQuizAccess(user.id)     // free allowance
    : await hasActiveAccess(user.id);  // paid only
if (!allowed) {
  return NextResponse.json({ error: "This needs an active subscription." }, { status: 402 });
}

Two reasons for reflecting state rather than asking Stripe: an entitlement check on every request would put a third-party API in the hot path, and it lets the freemium split (quizzes free, interviews paid) be a database query.

Webhooks are treated as at-least-once, because they are.

Failure modes and what we do about them

What we deliberately did not put in Neon

• Billing. Stripe. Reflecting subscription state into Postgres is not the same as owning it.
• Learner progress inside lab branches. Progress belongs in durable Postgres. The branch is scratch space.
• Static content. Simulated terminals, lesson definitions and question banks are TypeScript files in the repo, versioned with the code, no database round-trip.
• Secrets. Environment configuration, not rows.
• The simulated terminals. No infrastructure at all, and no reason for any.

Lessons from building on disposable infrastructure

Deletion is a feature with a budget. On traditional infrastructure, forgetting to clean up wastes disk. Here it spends money continuously. That changes cleanup from hygiene into a component with its own failure handling, its own bounds and its own logging.

Make the happy path clean up too. The most reliable cleanup is the one on the path everyone takes. Starting a lab tears down the previous one, so the scheduled job handles exceptions rather than the bulk of the work.

Conditional writes beat coordination. WHERE status = 'PENDING' replaced everything we might have built with locks or a queue. On a system that already has transactions, use them.

Optional integrations need a predicate, not a try/catch. prepWorkerConfigured() and isStorageConfigured() are what let the app run in CI with neither. Discovering a missing integration through an exception at request time is worse in every way.

Waiting is part of the product. When generation takes thirty seconds, the polling UI is not a detail, it is the experience. A status row with PENDING, PROCESSING, COMPLETED and FAILED plus a human-readable rejectionReason gives the frontend something honest to show.

Results must be revisitable. Interview results and quiz outcomes are persisted rows with their own pages, not client state. People close tabs, and a result that only existed in React state is a result you destroyed.

What we would improve next

• Cleanup currently sweeps on a fixed interval. Reacting to branch-level signals would close the window further.
• The SQL denylist should become a restricted Postgres role, so the database enforces the boundary rather than a regex in front of it.
• Generation cost is estimated per request but not yet aggregated into a spend view worth putting in front of an admin.
• The User mirror has no reconciliation job. Drift between Auth and our table is currently theoretical rather than monitored.

What transfers to other products

Very little of this is specific to teaching DevOps. The reusable shape is:

A durable core plus disposable compute. Any product that hands users a real environment (coding sandboxes, technical assessments, interactive docs, preview environments per pull request) wants durable state in one place and throwaway infrastructure somewhere else, with a hard rule that nothing important lives in the disposable half.

A status row as the coordination primitive. Long-running work, a conditional claim, a polling client. No queue required until you actually need one.

Cache by normalised intent. If generation is expensive and inputs cluster, normalise the input to a key and reuse aggressively. Uniqueness is usually worth less than quality plus cost control.

Predicates for every optional service. It is what makes a system with six integrations still runnable on a laptop with none of them.

The branch-per-user pattern in particular is worth stealing. Any time you would otherwise write "we can't let users run that against our database", a disposable branch turns the answer into "sure, here's one of your own".

app.post('/signup', async (req, res) => {
  const user = await createUser(req.body.email);
  res.json({ id: user.id });

  // fire and forget
  sendWelcomeEmail(user.email);
});

Then someone points out that a crash between the response and the email loses the email, so you add a queue:

await queue.add('welcome-email', { userId: user.id });

That is better. The job now survives a deploy, and it gets retried if the worker throws. What it does not survive is the thing that actually happens: the worker picks up the job, does two of the four things the job is supposed to do, and then the pod is evicted. The queue redelivers. The job starts again from the top. The user gets a second welcome email, and the charge that ran between the two failures runs again too.

The queue moved the work. It did not remember how far the work got.

TL;DR

• A queue gives you at-least-once delivery. It does not give you at-least-once progress, so a job that dies halfway restarts from the beginning.
• Durable execution fixes this by journalling each completed step and replaying the function, returning recorded results instead of re-running the work.
• That requires your workflow code to be deterministic. Date.now(), Math.random() and unguarded I/O quietly break replay.
• Replay does not give you exactly-once side effects. A step can succeed and crash before its result is written, so effects still need idempotency keys.
• Durable timers are the feature that is genuinely hard to build yourself. A three-day sleep that survives a deploy is not a setTimeout.
• You can build a working executor in about 90 lines. Whether you should is a question about timers, visibility and versioning, not about the core loop.

Prerequisites

• Comfortable with TypeScript and async/await
• Node.js 20 or newer to run the examples
• Some exposure to a job queue (BullMQ, SQS, Sidekiq, Celery, anything)
• Familiarity with idempotency helps but is not required

Why a queue is not durability

A queue is a handoff. It takes a message, keeps it until a consumer acknowledges it, and redelivers if the acknowledgement never arrives. Everything it guarantees is about the message.

Your job is not a message. It is a sequence:

1. charge the card
2. provision the account
3. send the receipt
4. notify the sales channel

The queue holds one message representing all four. When the worker dies after step 2, the queue knows only that the message was not acknowledged. It redelivers, and your handler starts at step 1. You get a second charge.

The usual patch is a status column:

if (job.status === 'charged') {
  // skip the charge
}

This works, and it is where most teams stop. It also means every job grows its own bespoke state machine, every new step needs a new status value, and the "where did this get to" logic is spread across the handler in conditionals nobody wants to touch. You have written a workflow engine by accident, one if at a time, without the part that makes it reliable.

Durable execution is that same idea done once, generically.

The failure modes that actually happen

Before the fix, the list worth designing against. These are the ones that show up in production, roughly in order of how often they bite:

• The worker dies mid-job. Deploy, OOM kill, spot reclaim, node drain. Partial side effects, full restart.
• The job is redelivered while still running. The visibility timeout expires because step 2 was slower than expected. Now two workers run the same job concurrently.
• A downstream call is slow, not dead. The payment API takes 40 seconds. Your handler times out at 30, the queue retries, and the original call completes anyway.
• The job needs to wait. Three days before a nudge email, an hour before a retry, until a human approves. A setTimeout in a process that gets deployed twice a day is not a wait.
• A poison message. One malformed payload fails forever, burns retry budget, and buries the rest of the queue.
• The code changed underneath a running job. You shipped a new version while 400 jobs were mid-flight against the old one.

A queue plus a status column handles the first one badly and the rest not at all.

What durable execution actually means

The idea is small enough to state in one paragraph.

Every side-effecting operation is wrapped in a step. When a step completes, its name and its return value are appended to a journal that is persisted before the workflow continues. If the process dies, the workflow function is called again from the top, but this time each step checks the journal first: if there is a recorded result at this position, return it and do not run the work. Execution fast-forwards through everything already done and resumes at the first step with no record.

The function re-runs. The work does not.

This is the same trick as event sourcing, pointed at control flow instead of at domain state. The journal is the source of truth about progress, and the function body is a pure-ish projection of it.

Building one, so you know what you are buying

Roughly 90 lines, no dependencies, a JSON file per run. Small enough to read in one sitting and complete enough to survive a kill -9.

The journal

// durable/journal.ts
import { mkdirSync, readFileSync, writeFileSync, existsSync, renameSync } from 'node:fs';
import { join } from 'node:path';

export interface JournalEntry {
  seq: number;
  name: string;
  status: 'completed' | 'sleeping';
  result?: unknown;
  wakeAt?: number;
}

export interface RunState {
  runId: string;
  status: 'running' | 'completed';
  entries: JournalEntry[];
  output?: unknown;
}

const DIR = join(process.cwd(), '.runs');

export function load(runId: string): RunState {
  const file = join(DIR, `${runId}.json`);
  if (!existsSync(file)) return { runId, status: 'running', entries: [] };
  return JSON.parse(readFileSync(file, 'utf8')) as RunState;
}

export function save(state: RunState): void {
  mkdirSync(DIR, { recursive: true });
  const file = join(DIR, `${state.runId}.json`);
  // Write then rename: a crash mid-write must not leave a truncated journal,
  // because a truncated journal is worse than no journal at all.
  writeFileSync(`${file}.tmp`, JSON.stringify(state, null, 2));
  renameSync(`${file}.tmp`, file);
}

A file per run is obviously not what you would deploy. Swap it for a table with a primary key on (run_id, seq) and the rest of the code is unchanged. The property that matters is that a completed step is durable before the next line of workflow code runs.

The context

This is where replay lives.

// durable/context.ts
import type { RunState } from './journal';

/** Unwinds the workflow when it hits a sleep that has not elapsed yet. */
export class Suspend extends Error {
  constructor(public readonly wakeAt: number) {
    super(`suspended until ${new Date(wakeAt).toISOString()}`);
  }
}

export class Context {
  private cursor = 0;

  /** Exposed so steps can derive idempotency keys from it. */
  readonly runId: string;

  constructor(
    private readonly state: RunState,
    private readonly persist: () => void,
  ) {
    this.runId = state.runId;
  }

  async step<T>(name: string, fn: () => Promise<T>): Promise<T> {
    const seq = this.cursor++;
    const recorded = this.state.entries[seq];

    if (recorded) {
      // The name check is what turns a silent corruption into a loud error.
      if (recorded.name !== name) {
        throw new Error(
          `Non-deterministic replay at position ${seq}: ` +
            `journal has "${recorded.name}", code asked for "${name}"`,
        );
      }
      return recorded.result as T;
    }

    const result = await fn();
    this.state.entries[seq] = { seq, name, status: 'completed', result };
    this.persist();
    return result;
  }

  async sleep(name: string, ms: number): Promise<void> {
    const seq = this.cursor++;
    const recorded = this.state.entries[seq];

    if (!recorded) {
      const wakeAt = Date.now() + ms;
      this.state.entries[seq] = { seq, name, status: 'sleeping', wakeAt };
      this.persist();
      throw new Suspend(wakeAt);
    }
    if (recorded.status === 'completed') return;
    if (Date.now() >= recorded.wakeAt!) {
      recorded.status = 'completed';
      this.persist();
      return;
    }
    throw new Suspend(recorded.wakeAt!);
  }
}

Two things worth pausing on.

The cursor is positional. Step identity is "the third step in this function", not "the step called welcome-email". That is what makes the name check load-bearing: if you insert a step in the middle of a workflow that has runs in flight, every position after it shifts, and the mismatch is caught instead of silently returning the wrong recorded value. This positional model is also exactly why versioning is hard, which we will come back to.

The sleep does not block. It records when to wake and throws, unwinding the stack out of the workflow entirely. The process is free to exit. Nothing is holding a timer.

The runner

// durable/run.ts
import { Context, Suspend } from './context';
import { load, save, type RunState } from './journal';

export type Workflow<I, O> = (ctx: Context, input: I) => Promise<O>;

export type RunResult<O> =
  | { done: true; output: O }
  | { done: false; wakeAt: number };

export async function run<I, O>(
  runId: string,
  workflow: Workflow<I, O>,
  input: I,
): Promise<RunResult<O>> {
  const state: RunState = load(runId);

  // Replaying a finished run must be free and must not re-execute anything.
  if (state.status === 'completed') {
    return { done: true, output: state.output as O };
  }

  const ctx = new Context(state, () => save(state));

  try {
    const output = await workflow(ctx, input);
    state.status = 'completed';
    state.output = output;
    save(state);
    return { done: true, output };
  } catch (err) {
    if (err instanceof Suspend) return { done: false, wakeAt: err.wakeAt };
    // A real failure. Completed steps stay in the journal, so the retry
    // resumes at the failed step rather than at the top of the workflow.
    throw err;
  }
}

The workflow

Now the part an application developer writes. It reads like ordinary code, which is the entire point.

// onboarding.ts
import type { Context } from './durable/context';
import { createUser, sendEmail, hasActivated } from './services';

const THREE_DAYS = 3 * 24 * 60 * 60 * 1000;

export async function onboarding(ctx: Context, input: { email: string }) {
  const user = await ctx.step('create-user', () => createUser(input.email));
  await ctx.step('welcome-email', () => sendEmail(user.email, 'welcome'));

  await ctx.sleep('wait-3-days', THREE_DAYS);

  const activated = await ctx.step('check-activation', () => hasActivated(user.id));
  if (!activated) {
    await ctx.step('nudge-email', () => sendEmail(user.email, 'nudge'));
  }

  return { userId: user.id, nudged: !activated };
}

The if is safe because activated came out of a step. On replay it is read from the journal, so the branch resolves the same way it did the first time, forever. Had it been written as if (!(await hasActivated(user.id))), the replay would call a live service whose answer may have changed, take the other branch, and desynchronise from the journal.

That is the rule in one line: every value the control flow depends on has to come from a step.

Watching it survive a crash

Note what did not print on the second run: the side-effect line. The workflow function ran start to finish twice; sendEmail was called once.

The part the demo gets wrong

Look at step again, specifically these two lines:

const result = await fn();
this.state.entries[seq] = { seq, name, status: 'completed', result };

There is a gap between them. If the process dies in that gap, the work happened and the journal does not know. Replay re-runs it. The user gets two welcome emails.

This is not hypothetical. Move the crash a few microseconds earlier, into the gap, and the same executor produces a duplicate:

You cannot close this gap. Committing the journal entry before running the step is worse, because then a failure loses the work entirely. Committing both atomically would require the side effect and your database to share a transaction, which they do not, because one of them is someone else's HTTP API.

The fix is the same one that makes webhook receivers safe: give the side effect a key derived from something stable, and let the far end deduplicate.

await ctx.step('welcome-email', () =>
  sendEmail(user.email, 'welcome', {
    // Stable across replays because runId and step name are both stable.
    idempotencyKey: `${ctx.runId}:welcome-email`,
  }),
);

Stripe, most payment APIs and any well-built internal service accept a key like this. For services that do not, you need your own dedupe table written in the same transaction as the effect. If neither is possible, you are choosing between a duplicate and a loss, and you should choose deliberately rather than discover the choice in an incident. We went through the same reasoning from the receiving side in what it actually takes to deliver a webhook in production.

Determinism, and the ways you break it

Replay assumes that running the function again produces the same sequence of steps. Anything that can change between the first run and the replay is a hazard. The common ones:

// Breaks: a different value on every replay
const requestedAt = Date.now();
const token = crypto.randomUUID();
const shard = Math.floor(Math.random() * 4);

// Fine: recorded once, replayed forever
const requestedAt = await ctx.step('now', async () => Date.now());
const token = await ctx.step('token', async () => crypto.randomUUID());
const shard = await ctx.step('shard', async () => Math.floor(Math.random() * 4));

Less obvious, and more likely to reach production:

• Reading config or feature flags directly. A flag that flips between the original run and the replay takes the other branch. Read flags inside a step.
• Iterating something unordered. Object.keys() on an object built from a Map populated by concurrent writes, or a SELECT with no ORDER BY, can come back in a different order and fan out steps in a different sequence.
• Promise.race against a timeout. Whichever side wins is a wall-clock accident.
• Reading from the database outside a step. The row changed. That is what rows do.
• Library upgrades that change behaviour inside your workflow body. Rare, extremely annoying.

The name check in step catches the structural version of these. Insert a step into a workflow that already has runs in flight and it fires immediately:

step a                  -> replayed
Error: Non-deterministic replay at position 1: journal has "b", code asked for "INSERTED"

What it cannot catch is a step returning a different value, because the whole point is that it never runs the step again. Structural drift is loud; value drift is silent. Keep values in steps.

Versioning a workflow that is already running

This is the problem most teams meet on week three, and it is a direct consequence of positional identity.

You have 400 runs paused in wait-3-days. You want to add a step before the nudge email. Insert it, deploy, and every paused run resumes into a journal whose positions no longer line up. If you were lucky you wrote the name check and they all fail loudly. If you were not, they silently return the wrong values to the wrong steps.

Three strategies, in increasing order of effort:

Append only. Add steps at the end. Never insert, never reorder, never delete. Free, and restrictive enough that it stops working eventually.

Version gates. Record a version at the top of the workflow and branch on it.

const version = await ctx.step('version', async () => 2);

if (version >= 2) {
  await ctx.step('score-lead', () => scoreLead(user.id));
}
await ctx.step('nudge-email', () => sendEmail(user.email, 'nudge'));

Runs that started before the change recorded 1 and skip the new step. New runs record 2 and take it. The cost is that the gates accumulate, and someone has to delete them once the old runs drain.

Drain and cut over. Register the new workflow under a new name, route new runs to it, let the old one finish. Cleanest, and it needs you to tolerate two versions in flight for as long as the longest sleep, which for a 30-day trial workflow is a month.

Every hosted platform in this space ships some form of the second or third option. It is a real part of the product and it is worth pricing in when you compare building against buying.

Waiting for the outside world

Sleeps handle time. The other kind of wait is an external event: a payment confirms, a human approves, a webhook lands. Same mechanism, different wake condition.

const approval = await ctx.waitForSignal('manager-approval', { timeout: SEVEN_DAYS });

if (approval.timedOut) {
  await ctx.step('escalate', () => escalate(request.id));
}

The implementation mirrors sleep: record that the run is waiting on a named signal, throw Suspend, and have the signal delivery endpoint write the payload into the journal and re-enqueue the run. It is maybe another 30 lines on top of what is above.

This is also where the "just use a queue and a status column" approach fully falls apart. A workflow that waits seven days for a human, then escalates, then waits again, is a state machine that nobody wants to hand-maintain in conditionals.

Where the hosted platforms change the tradeoff

The executor above is real and it works. What it is missing is everything around the loop:

• A scheduler for durable timers at scale. One wakeAt in a JSON file is easy. Ten million pending wake-ups, fairly scheduled, without a thundering herd at midnight, is a system.
• Visibility. When someone asks why order 8f21 never shipped, you want to open a page showing every step, its input, its output, and where it is stuck. Building that UI is more work than building the executor.
• Concurrency and rate control. "At most 5 of these per customer, at most 500 globally, and back off when the vendor 429s" is fiddly to get right and easy to get subtly wrong.
• Versioning tooling, per the section above.
• Somebody else's on-call. Your workflow engine failing is a total outage of every background job you have.

The same onboarding workflow across the main options:

They differ in where the checkpoint boundary sits. Inngest makes it explicit: step.run is the unit, and code outside a step re-executes on every replay. Temporal draws the line at the workflow/activity split, where activities are separately-registered functions and the workflow body is the deterministic part. Trigger.dev checkpoints the run itself, which is why its version reads as plain async code with no step wrappers at all. Hatchet and Restate sit at different points on the same axis.

That boundary is the thing to evaluate. Explicit steps are more typing and much more obvious about what re-runs. Implicit checkpointing is prettier and asks you to hold more in your head about what is safe to put where.

When you should not reach for this

Durable execution is not free. It adds a deployment, a mental model and a class of bug (non-determinism) that your team has not had before. Skip it when:

• The job is short and idempotent already. Resizing an image does not need a journal. Retry the whole thing.
• Throughput is high and each item is cheap. A million clickstream events a minute want a queue and a consumer group, not a journal per event.
• Loss is acceptable. Cache warming, non-critical analytics. Fire it, forget it, mean it.
• You need sub-100ms. Replay and journalling add latency by design. This is for work measured in seconds to weeks.

The signal that you do want it: your handler has a status column with more than about three values, and somebody has already written a comment explaining what happens if it crashes between two of them.

Wrapping up

The core mechanism is small. Journal each completed step, replay the function, return recorded results instead of re-running work. You can hold all of it in your head, and the 90 lines above are enough to prove it to yourself.

What is not small is the surrounding system: durable timers at scale, a UI that answers "where is this stuck", concurrency controls, and a versioning story for workflows that outlive the code that started them. That is the real build-versus-buy line, and it is worth being honest that the executor is the easy part.

Whichever way you go, two things travel with you. Every value your control flow depends on has to come from a step, or replay will quietly take a different path. And step execution is at-least-once no matter what you buy, so side effects still need idempotency keys. Get those two right and the rest is a question of how much of the surrounding system you want to own.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Run GPU batch inference on Amazon ECS Managed Instances with scale to zero

Deploy a single CloudFormation stack that builds a GPU batch inference pipeline on Amazon ECS Managed Instances. It uses Amazon SQS for job buffering and Application Auto Scaling to scale to zero when

📅 Aug 3, 2026 • 📰 AWS Containers Blog

🔗 Read more

📄 Kubernetes upgrades don’t have to break things: How EKS is making cluster lifecycle management simpler and safer

Kubernetes moves at a pace of three minor version releases per year, and staying current is not optional if you The post Kubernetes upgrades don’t have to break things: How EKS is making cluster lifec

📅 Aug 1, 2026 • 📰 The New Stack

🔗 Read more

📄 What Is Agentic AI for Kubernetes? A Platform Engineer’s Guide

AI has been the main topic of conversation in infrastructure circles for a while now. Recently, however, the conversation moved from “AI that answers questions” to “AI that takes action,” and that shi

📅 Aug 1, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Kubernetes v1.37 Sneak Peek

As we get closer to the release date for Kubernetes v1.37, the project develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This bl

📅 Jul 31, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Scaling Kubernetes pods with KEDA based on Amazon SQS queue depth

In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in

📅 Jul 31, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Runtime Supply Chain Verification using the Node Resource Interface (NRI)

The widely used container supply chain verification tools today operate at the Kubernetes API layer as admission webhooks (such as Kyverno, OPA Gatekeeper, and Sigstore Policy Controller). They interc

📅 Jul 30, 2026 • 📰 CNCF Blog

🔗 Read more

📄 How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server

Caution: Some of the technical detail in this article is not accurate. We are reviewing it and preparing corrections. Until then, check what you read here against the controller-runtime documentation.

📅 Jul 29, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Your Kubernetes health checks are accidentally waking your services. Here’s the fix.

Scale-to-zero breaks when health checks scale you back up. Learn how KubeElasti’s ProbeResponse lets Kubernetes services stay genuinely idle — while keeping load balancers and uptime monitors happy. S

📅 Jul 29, 2026 • 📰 CNCF Blog

🔗 Read more

📄 How to Run AI Agents on Kubernetes with Pulumi

Kubernetes has become the default place teams run agentic AI workloads: CNCF’s 2026 annual survey found that 66% of organizations hosting generative AI models use Kubernetes to manage some or all of t

📅 Jul 28, 2026 • 📰 Pulumi Blog

🔗 Read more

☁️ Cloud Native

📄 Amazon ECR now supports image layers up to 200 GB

Amazon Elastic Container Registry (Amazon ECR) has increased the maximum image layer size limit to 200 GB, for images pushed via Docker push. Previously, packaging assets required splitting data acros

📅 Aug 3, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Your agent needs a computer, not a container — introducing @cloudflare/computer

Agents need more than just a container to scale. We're introducing @cloudflare/computer, an agent runtime that dynamically orchestrates between fast, efficient isolates and full Linux containers to gi

📅 Aug 3, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Empty sandboxes break developer experience

Learn how Docker Sandbox kits turn empty sandboxes into productive development environments with repeatable tooling, credentials, and configuration.

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

📄 Docker AI Governance: Audit Logs, Now Where Your Security Team Already Works

Now in Docker AI Governance: a single searchable record of every policy decision your agents trigger, streamed to the SIEM your security team already runs, so you can show what your agents did and wha

📅 Aug 3, 2026 • 📰 Docker Blog

🔗 Read more

📄 Coordinating Teams of AI Agents in Real Time on NATS and JetStream

Guest post by David Farah and Sven Jonscher, creators of Cotal . We build Cotal, the open standard for AI agents to work together in one shared space. Our first multi-agent prototypes taught us where

📅 Aug 1, 2026 • 📰 NATS Blog

🔗 Read more

📄 Docker OIDC connections for GitHub Actions available for Docker Orgs

Eliminate Stored Credentials in Your CI/CD Pipelines TL;DR: Docker now supports OpenID Connect (OIDC) for GitHub Actions. Your workflows can authenticate with short-lived, per-run tokens instead of st

📅 Jul 31, 2026 • 📰 Docker Blog

🔗 Read more

📄 The Future of Agentic AI Depends on Openness and Trust. That’s Why Docker Is Joining Nvidia’s Open Secure AI Alliance.

Docker joins NVIDIA's Open Secure AI Alliance to help build the security, governance, and trust frameworks that agentic AI systems demand.

📅 Jul 30, 2026 • 📰 Docker Blog

🔗 Read more

🔄 CI/CD

📄 Blog: Selective drift correction with ignore rules

We are excited to introduce drift ignore rules for Flux Kustomizations, a long-requested capability that lets you tell Flux to leave specific fields alone during drift detection and correction, while

📅 Aug 3, 2026 • 📰 Flux CD Blog

🔗 Read more

📄 Secure every commit to production with Claude and GitLab

Agentic coding is moving faster than many enterprise governance programs can keep up with. Coding assistants, like the Claude security guidance plugin and Claude Security, can flag and fix common vuln

📅 Aug 3, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Control Runtime Behavior with Config Management

Learn how Config Management lets teams safely manage runtime configuration across FME environments without redeploying applications. | Blog

📅 Aug 3, 2026 • 📰 Harness Blog

🔗 Read more

📄 Stories from the Factory Floor: Empowering agents with LaunchDarkly MCP tools

A new capability on the LaunchDarkly MCP server offers a practical look at what an automated software factory could look like in practice.

📅 Jul 31, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Don’t stop early: Case-folding source code at memory speed

How a branch-free loop and byte-space arithmetic let GitHub case-fold every byte of code search at >45 GiB/s on a single core. The post Don’t stop early: Case-folding source code at memory speed appea

📅 Jul 31, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Gitea Runner 3.0.0 is released

We are happy to announce the release of Gitea Runner 3.0.0.

📅 Jul 31, 2026 • 📰 Gitea Blog

🔗 Read more

📄 How to govern agentic AI, MCPs, and AI code assistants

AI code completion built human review into the process by design. A developer types, a suggestion appears, and a human decides whether to accept it. A person looked at every line before it shipped. Ag

📅 Jul 31, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Stacked sessions and pull requests in the GitHub Copilot app

Learn how I modernized an old codebase of mine using stacked sessions and pull requests in the GitHub Copilot app. The post Stacked sessions and pull requests in the GitHub Copilot app appeared first

📅 Jul 30, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Automate all the things: How to use Grafana Cloud's AI to relieve the operational burden

Continuous integration and continuous delivery (CI/CD) have dramatically changed how we ship software. But once code reaches production, the operational work is still surprisingly manual. Engineers co

📅 Jul 29, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Why GitLab signed the Open Weights and American AI Leadership letter

This week GitLab signed the Open Weights and American AI Leadership letter, joining a long list of other technology companies that support a strong, open AI ecosystem. The letter argues that open weig

📅 Jul 29, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitLab Patch Release: 19.2.1, 19.1.3, 19.0.5

📅 Jul 29, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Why AI Deployment Breaks Standard CI/CD

Learn why AI deployment can break standard CI/CD and how runtime controls, shadow testing, rollouts, and rollback reduce risk.

📅 Jul 28, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

🏗️ IaC

📄 Red Hat Ansible All-Stars: Driving the future of network and infrastructure automation

As enterprise infrastructures scale across hybrid cloud environments and distributed networks, operations teams face an unsustainable calculation. Managing thousands of servers or multi-vendor routing

📅 Jul 31, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Migrate CloudFormation to Pulumi with Discovered Stacks

With Discovered Stacks, Pulumi Cloud does the bookkeeping for a CloudFormation migration: every resource in the stack gets an explicit migration status, and the migration is done when the code provabl

📅 Jul 30, 2026 • 📰 Pulumi Blog

🔗 Read more

📄 Discovered Stacks: One Place for All Your Infrastructure

Today we’re launching Discovered Stacks: Pulumi Cloud now models your AWS CloudFormation stacks and Azure Resource Manager deployments as stacks, right alongside your Pulumi IaC stacks. And when you’r

📅 Jul 30, 2026 • 📰 Pulumi Blog

🔗 Read more

📊 Observability

📄 Cortex completes OSTIF security audit

The Open Source Technology Improvement Fund is proud to share the results of our security audit of Cortex. Cortex functions as a long-term, multi-tenant scalable open source storage for Prometheus and

📅 Aug 3, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Reflections on AI Week, and the future of solving problems with observability and AI

Thank you for spending AI Week with us. We’re thrilled by the reaction and we all enjoyed replying to your questions. Thanks for engaging. Some of my favorite quotes from LinkedIn and Reddit include:

📅 Jul 31, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Introducing New Relic eBPF Logs - Now in Public Preview

Collect application logs through the New Relic eBPF agent, connect them to APM services, and reduce the need for a separate log forwarder.

📅 Jul 31, 2026 • 📰 New Relic Blog

🔗 Read more

📄 How to build a trust platform for your agent with Grafana Agent Observability

Observing fast-growing agentic workloads is no small feat, especially if you try to build your own monitoring stack or rely solely on tools built for a time before LLMs. At Grafana Labs, we know this

📅 Jul 30, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Salesforce Observability with New Relic

The New Relic Salesforce Exporter centralizes Salesforce telemetry, performance, and security data into New Relic for proactive, unified observability.

📅 Jul 30, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Achieving 100% Observability with BIND and Zabbix

Argentina’s BIND Group is a diversified financial services ecosystem centered around BIND Banco Industrial, offering banking, investment, insurance, leasing, fintech, and digital payment solutions. Wi

📅 Jul 29, 2026 • 📰 Zabbix Blog

🔗 Read more

📄 Telemetry-driven development: How to gain confidence in your coding agents' behavior with gcx and Grafana MCP

You’re about to click "Merge" on a PR, but you feel more anxious about it than you used to. Why? You did everything properly, by today’s standards: You used Claude to create a plan, giving it context

📅 Jul 28, 2026 • 📰 Grafana Blog

🔗 Read more

📄 MCP is going stateless: What the new spec means for AI agents

The Model Context Protocol (MCP) is going stateless. Discover how this architecture shift simplifies agent scaling and integrates OpenTelemetry.

📅 Jul 28, 2026 • 📰 New Relic Blog

🔗 Read more

📄 We broke the OTel demo

If you’ve been running the Demo for some time, you may have seen a couple of structural changes lately, and you may even have gotten mad about things not working as expected. We feel your pain and we

📅 Jul 28, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Aug 3, 2026 • 📰 Linode Blog

🔗 Read more

📄 AWS WAF now supports Miggo Security managed rule groups for emerging threats and AI/ML application protection

AWS WAF now supports two new partner managed rule groups from Miggo Security, available through AWS Marketplace: Miggo Rules for AWS WAF – High Emerging Application Threats, and Miggo Rules for AWS WA

📅 Aug 3, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Microsoft Confirms Copilot ‘Super App’ Is Coming This Year — and It’s About More Than Convenience

Microsoft is combining Copilot Chat, Code, Cowork and Autopilots into one super app, raising new questions about agent governance, identity, licensing and security.

📅 Aug 3, 2026 • 📰 DevOps.com

🔗 Read more

📄 Announcing Red Hat OpenShift Platform Plus for Red Hat OpenShift Service on AWS on AWS Marketplace

Organizations using Red Hat OpenShift Service on AWS (ROSA) are increasingly seeking ways to extend their platform’s capabilities with enterprise-grade security and data services. Red Hat OpenShift Pl

📅 Jul 31, 2026 • 📰 OpenShift Blog

🔗 Read more

📄 Same goals, different clocks: What Red Hat’s 2025 Risk Report reveals about global compliance gaps

In April 2026, Red Hat’s Product Security team published its annual Risk Report . I encourage everyone involved in building, shipping, securing, or regulating software to read it–not just for the vuln

📅 Jul 31, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Friday Five — July 31, 2026

How leading companies are turning AI vision into business valueEnterprises are focused on moving beyond theoretical AI pilots to operationalizing it at scale, optimizing costs, and governing its actio

📅 Jul 31, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Secure at Inception: Announcing the Snyk Studio Integration for Snowflake Cortex Code

Snyk Studio integrates with Snowflake Cortex Code to scan AI-generated code, dependencies, and containers for vulnerabilities during development.

📅 Jul 30, 2026 • 📰 Snyk Blog

🔗 Read more

📄 The Attacker Never Sleeps, Neither Can Your Testing

AI is accelerating software development and giving attackers machine-speed capabilities. Security teams must continuously test AI-built code, govern agents, and independently validate every finding.

📅 Jul 30, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Tame Dependabot: Group your updates, slow the cadence, keep security fast

Dependabot keeps your dependencies current, but its defaults can flood your repository with pull requests. Here's how grouping updates, slowing the cadence, and keeping security fixes fast cut the noi

📅 Jul 29, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Add security context to operational investigations with AWS DevOps Agent and Wiz

This post was co-authored by Ayelet Harcz (Product Manager), Hen Perez (CTO Architect), and Shani Gafni (Product Manager) at Wiz. When an on-call engineer receives an alert at 2 AM, a CPU spike, a lat

📅 Jul 29, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📄 Stadium Summer: The Snyk Connect Fan Zone Tour

Snyk’s Fan Zone tour brought AI security workshops, networking, and friendly competition to 8 cities and 3 virtual sessions. Attendees built skills, shared ideas, and leveled up together.

📅 Jul 29, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Sovereign by design: Lessons from Red Hat Summit

Digital sovereignty used to sit somewhere between a compliance checkbox and a future roadmap item. That’s changing fast. At Red Hat Summit, Mohammed Retmi of Core42 in the United Arab Emirates and A.S

📅 Jul 29, 2026 • 📰 OpenShift Blog

🔗 Read more

💾 Databases

📄 AWS Transform for full-stack Windows modernization now supports offline schema transformation to Aurora PostgreSQL

Today, AWS Transform for full-stack Windows modernization announced general availability of offline source transformation, enabling customers to modernize Microsoft SQL Server databases to Amazon Auro

📅 Aug 3, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Amazon Aurora DSQL adds multi-Region cluster support in four more Regions

Starting today, Amazon Aurora DSQL supports multi-Region clusters in four additional AWS Regions: Europe (Stockholm), Europe (Spain), Asia Pacific (Mumbai), and Asia Pacific (Singapore). Aurora DSQL i

📅 Jul 31, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 How to Persist AI Agent Context Deterministically

Discover why standard MCP wiring makes context persistence optional (the model calls the tool only when it decides to) and what to do instead. We walk through how to treat persistence as control flow

📅 Jul 31, 2026 • 📰 Yugabyte Blog

🔗 Read more

📄 plRuby

PL/Ruby is a procedural-language handler that lets you write database functions in Ruby, stored and executed inside PostgreSQL. You get the expressiveness of Ruby and its standard library with the ful

📅 Jul 31, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Vector Search Meets Distributed SQL: Why Agentic AI Does Not Need Another Database

Key Takeaways Add a vector database to the existing stack. Sync it. Maintain it. Debug it when it drifts. Teams building agentic applications have largely accepted that sequence as the price of admiss

📅 Jul 30, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Context engineering for AI: what it is & how to build it

Your support agent confidently tells a customer they qualify for a refund under a 60-day return policy. Your actual policy is 30 days. The agent hallucinated the longer window, and the easy reaction i

📅 Jul 29, 2026 • 📰 Redis Blog

🔗 Read more

📄 Lessons Learned from Real-World NoSQL Database Migrations

Discover the strategies, challenges, and trade-offs teams faced in a few real-world migrations to ScyllaDB

📅 Jul 28, 2026 • 📰 ScyllaDB Blog

🔗 Read more

📄 TiDB Log Compaction: Faster Point-in-Time Recovery for Large Clusters

For a large distributed SQL cluster, backup and restore define whether the business can recover from an accident inside a realistic service objective. As TiDB adoption grows across larger, more write-

📅 Jul 28, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Harness Database DevOps: Reference Data Rollbacks

Learn to version reference data with Liquibase OSS, automate deployments in Harness Database DevOps, and enable safe rollbacks. | Blog

📅 Jul 28, 2026 • 📰 Harness Blog

🔗 Read more

📄 The 4 Failure Modes of Agent Context in Production

A production AI agent depends heavily on the context layer that tells it what to know at the moment it acts. It can pass every staging test, answer questions, call the right tools, and demo beautifull

📅 Jul 28, 2026 • 📰 Redis Blog

🔗 Read more

📄 Token-budget-aware LLM reasoning: cut costs in 2026

Reasoning models think before they answer, and those reasoning tokens are usually part of what you pay for. They're billed as output tokens, the expensive kind, and a single request can generate a few

📅 Jul 28, 2026 • 📰 Redis Blog

🔗 Read more

🌐 Platforms

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Aug 3, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Aug 3, 2026 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Aug 3, 2026 • 📰 Linode Blog

🔗 Read more

📄 Real-world mainframe modernization with AI: A safe, scalable path from mainframe to cloud

For too long, enterprises with legacy mainframe estates have been faced with a high-stakes dilemma: continue maintaining their mainframes, essentially kicking the modernization can down the road (they

📅 Aug 3, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Cortex Framework v7 is GA: Build agentic workflows without disrupting SAP operations

Businesses want to quickly and safely deploy AI agents to drive revenue, mitigate risk, and optimize capital, all without disrupting mission-critical ERP systems. And to power AI agents, you need more

📅 Aug 3, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner

To make informed decisions, businesses often need to connect their internal data with public reference data, to create a knowledge graph that connects real-world things and their relationships. Howeve

📅 Aug 3, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 Cloudflare Workers and Containers now support inbound TCP connections and gRPC

Cloudflare Workers now support inbound TCP connections via Spectrum, allowing direct socket forwarding to Durable Objects and Containers. Developers can run full-duplex gRPC applications or leverage a

📅 Aug 3, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Introducing the Billable Usage API: programmatic cost visibility for Cloudflare

Cloudflare has launched a new Billable Usage API for accounts, giving developers and FinOps teams single-endpoint programmatic visibility into cost and usage across all self-serve products. Built arou

📅 Aug 3, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Smaller, faster, safer: running Kimi and GLM at scale

Serving frontier models like Kimi and GLM means fighting for GPU memory. Here's how we quantize KV caches, compress model weights, and add integrity checks to serve them faster, cheaper, and safely.

📅 Aug 3, 2026 • 📰 Cloudflare Blog

🔗 Read more

📄 Behind the scenes: How we build, test, and scale Google Agent Skills

AI agents are only as good as the instructions and context you give them. When we launched Google Agent Skills, our goal was simple: encode Google Cloud domain knowledge into structured, open-source i

📅 Aug 3, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 How we made a viral commercial for developers

We made a commercial for Railway featuring the actor behind Gilfoyle from Silicon Valley. This is how we made it so that a developer like you would watch it.

📅 Jul 31, 2026 • 📰 Railway Blog

🔗 Read more

📄 The Complete Package: Why Debugging Is Only Half the C# Productivity Story

As .NET developers, we need to iterate on our applications while building, and part of that developer inner loop is the debugging experience. The rise of multi-platform code editors further requires d

📅 Jul 30, 2026 • 📰 JetBrains Blog

🔗 Read more

📰 Misc

📄 Unlocking agentic AI with Arm AGI CPU & SUSE AI Factory

SUSE collaborating with Arm for Day 0 readiness of the new Arm AGI CPU across SUSE portfolio Key takeaways Day 0 silicon innovation: The collaboration between SUSE and Arm highlights the ongoing work

📅 Aug 3, 2026 • 📰 SUSE Blog

🔗 Read more

📄 Learn pandas the Right Way: A Python Library Course That Doesn’t Waste Your Time

Let’s talk about the elephant in every data scientist’s room (or in this case, the panda). You’ve probably touched pandas before, even if nobody introduced you two properly. Ever opened a CSV file in

📅 Aug 3, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Ten Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience. Our goal in these ch

📅 Aug 3, 2026 • 📰 DevOps.com

🔗 Read more

📄 DeepSeek’s smaller model just outperformed its own flagship

DeepSeek has launched DeepSeek-V4-Flash-0731, delivering a significant boost in agent performance without changing the model’s core architecture. Following an announcement The post DeepSeek’s smaller

📅 Aug 3, 2026 • 📰 The New Stack

🔗 Read more

📄 Our First Moves to Get AI Spend Under Control

Over the past six months at JetBrains, our AI development expenses have increased roughly 10x. When the costs started rising, of course we noticed – and realized that we simply didn’t know how to cont

📅 Aug 3, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 JetBrains Open-Sources KotlinLLM, a Research Prototype for Runtime Code Generation

JetBrains open-sources KotlinLLM, letting compiled Kotlin apps generate and persist LLM-written code at runtime instead of calling a model live.

📅 Aug 3, 2026 • 📰 DevOps.com

🔗 Read more

📄 GitHub Brings Stacked Pull Requests Out of the Shadows

GitHub introduces native stacked pull requests, helping development teams break large changes into smaller, dependency-ordered PRs that are faster and easier to review.

📅 Aug 3, 2026 • 📰 DevOps.com

🔗 Read more

📄 Dynamic troubleshooting with guarded command execution in the MCP server for Red Hat Enterprise Linux

Managing Red Hat Enterprise Linux (RHEL) environments can involve troubleshooting when issues occur. While generative AI offers a promising way to accelerate troubleshooting, standard large language m

📅 Aug 3, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Designing APIs for agents

In early 2025, Webflow started building for MCP before there was a clear playbook for agent-ready APIs. We publicly announced The post Designing APIs for agents appeared first on The New Stack.

📅 Aug 1, 2026 • 📰 The New Stack

🔗 Read more

📄 What Claude’s real-world breaches reveal about AI safety tests

This week, just days after OpenAI announced that two of its advanced AI models had interacted with real-world systems during The post What Claude’s real-world breaches reveal about AI safety tests app

📅 Aug 1, 2026 • 📰 The New Stack

🔗 Read more

📄 Know Kotlin? Ship It Everywhere and Win at Shipaton 2026

Somewhere in your notes there’s an app idea waiting for a free weekend that never comes. Consider this its official deadline: RevenueCat Shipaton 2026, the world’s biggest mobile hackathon, runs Augus

📅 Jul 31, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Visual Studio Code 1.131

Learn what's new in Visual Studio Code 1.131 Read the full article

📅 Jul 29, 2026 • 📰 VS Code Blog

🔗 Read more

This guide turns that simple request into a small application you can try locally and then deploy. We will build a support ticket triage demo with DigitalOcean Serverless Inference.

If you only want to make the smallest possible API request, start with our first DigitalOcean serverless inference call. This guide starts where that one stops: it puts inference behind a real API, validates the model output, adds a browser interface, and deploys the result.

Support tickets are a useful example because they rarely arrive as tidy data. A customer may describe several problems in one message, leave out an important detail, or use an urgent tone for an issue that is not actually blocking their work. Before a support engineer can help, someone usually needs to summarize the request, decide where it belongs, and work out what should happen next.

Our demo uses inference for that first pass. The result is not just a chat response. It is a structured record that the application can validate and display.

For each ticket, the application returns:

• A factual summary
• A category and urgency level
• The customer's apparent sentiment
• Routing tags
• A recommended next action
• A draft response for a human to review

The local version uses FastAPI with a small HTML and JavaScript interface. Later, we use Docker and Terraform to run the same project on DigitalOcean App Platform. The complete code is available in the companion repository:

The deployment in this guide was tested end to end. Terraform created the App Platform application from the GitHub repository, App Platform built the Dockerfile, the deployed API called MiMo successfully, and Terraform removed the application afterward.

By the end, you will understand where inference fits into a normal web application, why model output still needs validation, and how the same project can run locally or on App Platform.

How does the demo work in practice?

The easiest way to understand the project is to follow one ticket.

Imagine that a customer submits this:

Since this morning's deployment, checkout requests take more than 30 seconds and many return a 504. Customers cannot complete purchases.

The browser sends that ticket to our FastAPI backend. FastAPI checks that the input has the expected fields and then sends it to a model through DigitalOcean Serverless Inference. The model reads the ticket and returns fields such as summary, category, and urgency. FastAPI checks those fields before the browser displays them.

The flow looks like this:

There are two DigitalOcean services in the final deployment, and they have different jobs:

• Serverless Inference runs the selected model and produces the analysis.
• App Platform runs our FastAPI application and serves the browser interface.

We are not training MiMo or deploying its model weights. DigitalOcean already hosts the model. Our application sends requests to an API and pays for the input and output tokens it uses. This is what inference means here: giving new input to an existing model and receiving a result.

The repository is not a finished helpdesk product. It leaves out storage and external integrations so we can focus on turning unstructured text into data the application understands.

Why is this a useful first inference project?

This demo keeps the first experience practical:

• The input is ordinary text that is easy to understand.
• The result appears immediately as useful fields in a browser.
• DigitalOcean hosts the model, so there is no model server or GPU to manage.
• The backend makes one normal HTTPS request to use inference.
• The same code works locally and on App Platform.

There is no database, helpdesk integration, or background job to configure. Those would be useful in a larger product, but they would hide the small part we want to learn first: how an application sends text to a hosted model and uses the result.

What does it take to see it work?

The first local run has four main steps:

1. Create a model access key in DigitalOcean.
2. Add the key and model ID to a local .env file.
3. Start the FastAPI application.
4. Submit the example ticket in the browser or with curl.

That is enough to make a real inference request. Docker and Terraform come later, when we package and deploy the same application. They are not required to understand or try Serverless Inference locally.

Why add structure around the model?

A first experiment with a language model often starts with a prompt and a printed response. That is useful for checking whether a model can understand the task, but an application needs more structure.

Our browser expects fields such as urgency, category, and recommended_action. If the model returns different field names on every request, the interface cannot use them reliably. If it returns an unknown urgency such as urgent-ish, our routing logic would not know what to do.

This project adds three boundaries around the model:

1. Pydantic validates the ticket before the request leaves our API.
2. A function-tool schema tells the model which fields it should return.
3. Pydantic validates the returned tool arguments before they reach the browser.

The model is useful because it can interpret natural language. The surrounding Python code is useful because it keeps the result within rules the application understands. We need both.

The FastAPI backend also keeps the model access key away from browser code. The browser only knows about our local /api/triage route. It never receives the DigitalOcean credential.

Prerequisites

For the first local run, you need:

• Python 3.11 or later
• Git
• A DigitalOcean account
• A positive Serverless Inference prepaid balance
• A model access key scoped to MiMo V2.5 Pro

The later packaging and deployment sections also use:

• A GitHub repository that DigitalOcean App Platform can access
• Docker if you want to test the container locally
• Terraform 1.6 or later for the deployment section
• A DigitalOcean personal access token for the deployment section

DigitalOcean Serverless Inference is prepaid and charges for input and output tokens. Make sure the team you are using has a positive balance before testing the application.

Create a model access key

In the DigitalOcean Control Panel, open Inference, select Manage, and click Create model access key. Give the key a clear name such as ticket-triage-local, select MiMo V2.5 Pro, and choose No VPC network for local testing.

The model ID used by the API is:

mimo-v2.5-pro

DigitalOcean lists MiMo V2.5 Pro as supporting Chat Completions, function calling, and structured output. Model availability can depend on the account, so the model picker in your team's Control Panel is the final check. See Supported Models for current model IDs and features.

Copy the secret as soon as it appears. DigitalOcean only displays it once. Model access keys can be limited to selected models, which is safer than giving the application a broad account token. The model access key guide describes the current options.

Do not paste the key into an issue, screenshot, Git commit, or frontend file.

Clone and configure the project

Clone the companion repository:

git clone https://github.com/The-DevOps-Daily/do-inference-ticket-triage.git
cd do-inference-ticket-triage

Create a virtual environment and install the application with its development tools:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

Copy the example environment file:

cp .env.example .env
chmod 600 .env

Open .env and add the model access key:

DIGITALOCEAN_INFERENCE_KEY=replace-with-your-model-access-key
DIGITALOCEAN_INFERENCE_MODEL=mimo-v2.5-pro
DIGITALOCEAN_INFERENCE_BASE_URL=https://inference.do-ai.run/v1
INFERENCE_TIMEOUT_SECONDS=45

APP_ACCESS_TOKEN=

DIGITALOCEAN_INFERENCE_KEY authenticates the backend to Serverless Inference. DIGITALOCEAN_INFERENCE_MODEL selects the model, and the base URL points to DigitalOcean's OpenAI-compatible API.

APP_ACCESS_TOKEN has a separate purpose. When set, it acts as a shared access code for a short-lived public demo. It is not a DigitalOcean key, and it is not a replacement for real user authentication. Leave it empty while working locally.

The repository's .gitignore excludes .env, but it is still worth checking:

git check-ignore .env

The command should print .env.

Try the complete flow locally

It helps to see the full request flow once before looking at each part.

Export the values from .env and start FastAPI:

set -a
source .env
set +a

uvicorn app.main:app --reload --port 8080

Open http://localhost:8080, select Load example, and submit the ticket. The right side of the page will show the category, urgency, sentiment, tags, next action, and draft response. It also shows which model answered, how long the request took, and how many tokens were used.

The interface is optional. You can call the same backend route with curl:

curl --request POST http://localhost:8080/api/triage \
  --header 'Content-Type: application/json' \
  --data '{
    "subject": "Production checkout is timing out",
    "description": "Every checkout request takes more than 30 seconds and purchases are blocked.",
    "customer_plan": "business"
  }'

This is important: the browser is only a convenient client. The main demo is the API path from FastAPI to DigitalOcean Inference and back.

If your goal is to understand Serverless Inference at a high level, you have now seen the core workflow. The next section opens the application and explains how it turns the model response into data the rest of the code can trust.

Under the hood: from ticket to validated result

Only one part of the application talks to DigitalOcean's inference endpoint. The surrounding code prepares a clear request, protects the credential, and checks the response. You do not need all of these pieces for a first API call, but they show how inference fits into a real web application.

Define the data before writing the prompt

The project starts by deciding which input and output the application accepts. These models live in app/models.py.

The incoming ticket has three fields:

class TicketRequest(BaseModel):
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)

    subject: str = Field(min_length=3, max_length=140)
    description: str = Field(min_length=20, max_length=5_000)
    customer_plan: Literal["starter", "business", "enterprise"] = "starter"

The length limits reject empty or unexpectedly large requests before they use model credits. extra="forbid" rejects fields the API does not know about, and str_strip_whitespace=True removes accidental whitespace around strings.

The result model is more detailed:

class TriageResult(BaseModel):
    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)

    summary: str = Field(min_length=10, max_length=400)
    category: Literal[
        "account_access",
        "billing",
        "bug",
        "feature_request",
        "performance",
        "security",
        "other",
    ]
    urgency: Literal["low", "medium", "high", "critical"]
    sentiment: Literal["calm", "confused", "frustrated", "angry", "positive"]
    tags: list[str] = Field(min_length=1, max_length=5)
    recommended_action: str = Field(min_length=10, max_length=500)
    draft_response: str = Field(min_length=20, max_length=1_500)

The fixed category and urgency values are useful beyond validation. A later version could route security tickets to one team and billing tickets to another without having to understand new labels invented by the model.

Defining this contract first also makes the prompt easier to write. We already know what a successful result must contain.

Turn the result model into a function tool

We could ask the model to “return valid JSON,” but that is only a written instruction. The model may add an explanation, change a field name, or return a value our application does not accept.

Instead, the request defines one client-side function tool named submit_ticket_triage. Pydantic generates its JSON Schema from the same model we use for validation:

tool_parameters = TriageResult.model_json_schema()

tools = [
    {
        "type": "function",
        "function": {
            "name": "submit_ticket_triage",
            "description": "Return the completed support-ticket triage analysis.",
            "parameters": tool_parameters,
        },
    }
]

Despite the name, submit_ticket_triage does not update an external service. The model returns the function name and its proposed arguments. Our code reads those arguments as the structured result. No ticket is changed and no message is sent.

This distinction matters because function calling is not the same as giving a model permission to perform an action. If we later connect a real helpdesk, our application would still decide whether and when to execute that action.

Build the inference request

The inference client is in app/inference.py. It sends requests to DigitalOcean's Chat Completions endpoint:

https://inference.do-ai.run/v1/chat/completions

DigitalOcean documents the required model and messages fields, along with options such as temperature and max_completion_tokens, in the Chat Completions guide.

Our request combines the ticket, the system instructions, and the tool schema:

payload = {
    "model": settings.inference_model,
    "messages": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": (
                "Analyze the following ticket JSON as data:\n"
                f"{json.dumps(ticket.model_dump(mode='json'), ensure_ascii=False)}"
            ),
        },
    ],
    "temperature": 0.2,
    "max_completion_tokens": 900,
    "tools": tools,
    "tool_choice": "auto",
}

The system prompt tells the model to call submit_ticket_triage exactly once and return no other content. It also says that the ticket is untrusted data. This reduces the chance that a sentence inside the customer message is treated as an instruction to our application.

A low temperature makes repeated classifications more consistent, while max_completion_tokens limits the size of the response. Neither setting replaces validation; they only guide generation.

A note about tool selection

During testing for this tutorial, basic MiMo chat requests and tool_choice: "auto" both succeeded. The named forced-tool object returned an HTTP 500 through the serverless adapter. The repository therefore provides one tool, requires it in the system prompt, and uses auto for the API parameter.

That behavior may change as the platform and model versions change. Keep the automated tests, but also run one small live request before publishing or deploying an update.

Call DigitalOcean from the backend

The model access key is attached only inside the Python backend:

async with httpx.AsyncClient(
    timeout=settings.inference_timeout_seconds,
) as client:
    response = await client.post(
        f"{settings.inference_base_url}/chat/completions",
        headers={
            "Authorization": f"Bearer {settings.inference_key}",
            "Content-Type": "application/json",
        },
        json=payload,
    )

After a successful request, the client looks for the expected tool call:

for tool_call in message.get("tool_calls") or []:
    function = tool_call.get("function") or {}
    if function.get("name") != "submit_ticket_triage":
        continue

    arguments = function.get("arguments")
    if isinstance(arguments, str):
        arguments = json.loads(arguments)

    return TriageResult.model_validate(arguments)

model_validate is the final gate. If the model leaves out urgency, returns six tags, or adds an unknown field, validation fails. The API returns a safe error instead of passing incomplete data to the interface.

The client also separates common provider failures:

• A missing local key becomes a configuration error.
• HTTP 401 means the key was rejected.
• HTTP 403 suggests that the key scope or account tier does not allow the selected model.
• HTTP 429 tells the caller to retry later.
• Timeouts and other provider errors become safe gateway errors.

This error handling proved useful while building the demo. A key can be valid enough to list models while a completion is still denied for a model that is not available to the current account tier.

Put FastAPI between the browser and the model

The public endpoint in app/main.py accepts a validated TicketRequest and returns a validated TriageResponse:

@application.post("/api/triage", response_model=TriageResponse)
async def triage_ticket(
    ticket: TicketRequest,
    x_app_access_token: str | None = Header(default=None),
) -> TriageResponse:
    _require_app_access(runtime_settings, x_app_access_token)
    return await application.state.inference_client.triage(ticket)

The complete route wraps that call with the error handling described above. There is also a /health endpoint that returns {"status": "ok"} without calling the model. App Platform can check whether the web process is healthy without creating an inference charge.

FastAPI is doing more than forwarding requests. It is the boundary that:

• Protects the model credential
• Rejects invalid tickets
• Controls which model features the application uses
• Validates the model's result
• Gives the browser a stable API

Add a small browser interface

The interface uses plain HTML, CSS, and JavaScript. It is intentionally small because the tutorial is about the inference path, not a frontend framework.

When the form is submitted, app/static/app.js sends the ticket to our API. If the deployment uses a demo access code, the script adds it to a separate header:

const headers = { 'Content-Type': 'application/json' };
if (accessCode) {
  headers['X-App-Access-Token'] = accessCode;
}

const response = await fetch('/api/triage', {
  method: 'POST',
  headers,
  body: JSON.stringify(payload),
});

The script renders the validated fields with textContent. It does not insert model output as HTML. The FastAPI application also adds a Content Security Policy and other browser security headers.

If APP_ACCESS_TOKEN is set, the interface displays an access-code field and sends the value in the X-App-Access-Token header. This is useful for limiting casual access to a temporary demo, but a real product should use individual accounts and proper authorization.

What does a real response look like?

The following is a shortened version of the response returned during an end-to-end test through the deployed App Platform application:

{
  "analysis": {
    "category": "bug",
    "urgency": "critical",
    "sentiment": "frustrated",
    "tags": ["deployment", "checkout", "504-error", "production-outage", "regression"]
  },
  "model": "mimo-v2.5-pro",
  "latency_ms": 10155,
  "usage": {
    "total_tokens": 1619
  }
}

The exact wording, latency, and token counts vary. The important part is that the shape stays the same and the values pass our rules.

The draft response is still a draft. A support engineer should review it before sending it to a customer. Validation can confirm structure, but it cannot confirm every factual statement or business decision.

Test without spending inference credits

Automated tests should be fast and repeatable. They should not fail because a provider is temporarily unavailable, and they should not spend model credits every time someone pushes a commit.

The API tests inject a fake inference client. Lower-level tests use httpx.MockTransport to inspect the outgoing request and return a realistic tool-call response.

The tests cover:

• Ticket validation
• The inference URL and authorization header
• Model selection
• The generated JSON Schema
• Tool-call parsing
• Invalid model arguments
• Authentication and model-access errors
• Rate limiting
• Secret protection in the public configuration route

Run all local checks with:

ruff check .
ruff format --check .
pytest

At the time of writing, the repository contains 11 passing tests. These tests do not need DIGITALOCEAN_INFERENCE_KEY.

Keep one manual live test in your release process as well. Mocked tests confirm our code, while the live test confirms the current model and API still accept the request.

Run the application in Docker

The Dockerfile installs the Python package, switches to an unprivileged user, exposes port 8080, and starts Uvicorn.

Build the image:

docker build -t do-inference-ticket-triage .

Run it with the local environment file:

docker run --rm \
  --publish 8080:8080 \
  --env-file .env \
  do-inference-ticket-triage

Check the container without calling the model:

curl http://localhost:8080/health

You should receive:

{ "status": "ok" }

The key is passed at runtime. It is not copied into the image.

Deploy to App Platform with Terraform

The local test already proves that the application can call Serverless Inference. Deploying it does not add new model infrastructure. It only moves the FastAPI application from your computer to DigitalOcean App Platform so other people can open it through a public URL.

The Terraform configuration for this step is in the terraform/ directory.

Terraform deploys the web application, not the model. It creates one App Platform application that builds the repository's Dockerfile and runs FastAPI. When a ticket arrives, FastAPI calls the already-hosted Serverless Inference API with the model access key.

Before applying the configuration, push the project to GitHub. The Terraform resource expects the repository in owner/repository format and deploys from the main branch by default.

Give App Platform access to GitHub

Terraform can point App Platform at a repository, but it cannot complete the GitHub authorization for your account. In the DigitalOcean Control Panel:

1. Open App Platform and start creating an app.
2. Select GitHub as the source.
3. Connect the GitHub account that owns the repository.
4. Give DigitalOcean access to the repository.
5. Stop before creating the app manually. Terraform will create it.

For a private repository, check the GitHub connection's repository permissions. If the repository was created after you first connected GitHub, you may need to open Manage access and add it.

If Terraform returns GitHub user not authenticated, the DigitalOcean team is not connected to the correct GitHub account or does not have access to that repository. Fix the GitHub connection in App Platform, then run the plan again.

Copy the example variable file:

cd terraform
cp terraform.tfvars.example terraform.tfvars

Set your repository:

github_repo = "The-DevOps-Daily/do-inference-ticket-triage"

The deployment needs two different DigitalOcean credentials:

• DIGITALOCEAN_TOKEN is a control-plane token used by Terraform to create the App Platform application.
• TF_VAR_inference_key becomes the model access key used by the deployed FastAPI service.

Create the control-plane token with the App Platform scopes app:create, app:read, app:update, and app:delete. The delete scope is needed for the cleanup step. This token and the model access key are not interchangeable.

Export them without adding them to terraform.tfvars:

export DIGITALOCEAN_TOKEN="your-control-plane-token"
export TF_VAR_inference_key="your-model-access-key"
export TF_VAR_app_access_token="a-long-random-demo-access-code"

The digitalocean_app resource connects App Platform to GitHub, builds the root Dockerfile, exposes port 8080, and configures /health as the health check. It adds the inference key and demo access code as SECRET runtime variables. The model ID and inference URL are regular runtime configuration.

The DigitalOcean Terraform provider documentation has the full reference for the digitalocean_app resource.

Initialize Terraform and download the provider:

terraform init

Check formatting and validate the configuration:

terraform fmt -check
terraform validate

Review the planned change and save it:

terraform plan -out=deploy.tfplan

For a new deployment, the summary should show one digitalocean_app resource to add and no unrelated changes. Apply that exact plan:

terraform apply deploy.tfplan

Terraform prints the App Platform resource details when the deployment is complete. Retrieve the public URL with:

terraform output -raw app_url

Store the URL in a shell variable and check the routes that do not call the model:

APP_URL=$(terraform output -raw app_url)

curl "$APP_URL/health"
curl "$APP_URL/api/config"

The health route should return {"status":"ok"}. The configuration route should show mimo-v2.5-pro and confirm that an access code is required.

Now send one real ticket through the deployed application:

curl --request POST "$APP_URL/api/triage" \
  --header 'Content-Type: application/json' \
  --header "X-App-Access-Token: $TF_VAR_app_access_token" \
  --data '{
    "subject": "Production checkout is timing out",
    "description": "Every checkout request is taking more than 30 seconds and purchases are blocked.",
    "customer_plan": "business"
  }'

A successful response has HTTP status 200 and contains the validated analysis, model, latency_ms, and usage fields. The same request without the access-code header should return HTTP 401. Finally, open the URL, load the example ticket, enter the demo code, and confirm that the browser renders the result.

This sequence tests the complete path: browser or curl, App Platform, FastAPI, Serverless Inference, MiMo, validation, and the response back to the client.

App Platform can deploy new commits automatically because the Terraform configuration sets deploy_on_push = true.

Protect Terraform state: Marking a variable as sensitive hides it from normal terminal output, but Terraform still stores its value in state. Use an encrypted remote backend with limited access for shared or long-lived deployments. Never commit terraform.tfstate or terraform.tfvars.

What should change before production?

This repository is a teaching project, but its boundaries point toward the work a production version would need.

Use real authentication. Replace the shared demo code with individual user accounts, roles, and authorization checks.

Add rate limits. A public endpoint can spend inference credits. Limit requests per user and consider a team-wide budget.

Keep humans in the workflow. The application should suggest a category, action, and response. A person should approve decisions that affect customers, billing, security, or incident response.

Store only what you need. Support tickets may contain personal or business data. Decide what can be logged, how long it is retained, and who can access it.

Measure quality. Create a set of example tickets with expected categories and urgency levels. Run them when the prompt or model changes. A successful HTTP response does not mean every classification is correct.

Monitor provider behavior. Record safe metrics such as latency, status codes, token use, and validation failures. Avoid logging raw ticket text unless your privacy rules allow it.

Rotate credentials. Use separate model access keys for development, staging, and production. Scope each key only to the models its application needs.

Possible extensions include saving triage history in PostgreSQL, adding Zendesk or Intercom integration, sending approved alerts to Slack or PagerDuty, and comparing models with a fixed evaluation dataset.

Clean up

Keep the Terraform variables exported while cleaning up. First review the destroy plan:

terraform plan -destroy -out=destroy.tfplan
terraform apply destroy.tfplan

Confirm that Terraform no longer manages any resources:

terraform state list

The command should print nothing. You can also check App Platform in the DigitalOcean Control Panel.

Terraform state and backup files can contain secret values even after the application is destroyed. For a one-off local demo, after confirming that the state is empty, remove the local state and saved plans:

rm -f terraform.tfstate terraform.tfstate.backup deploy.tfplan destroy.tfplan

Terraform does not delete the GitHub repository, the model access key, or the Serverless Inference prepaid balance. It also does not revoke the control-plane token. Revoke unused tokens and keys separately in the DigitalOcean Control Panel.

Conclusion

Getting started with DigitalOcean Serverless Inference required only a hosted model, a model access key, and an API request. The ticket triage demo made that request visible: submit ordinary text and receive useful fields that an application can understand.

The browser collects the ticket. FastAPI validates it and protects the credential. DigitalOcean Serverless Inference runs MiMo V2.5 Pro. A function tool gives the result a predictable shape, and Pydantic checks that shape before the interface uses it. Docker packages the service, while Terraform describes how App Platform should run it.

The local version shows how easy it is to make the first inference call. The rest of the project shows how to make that call safer, repeatable, and ready to deploy. The same pattern can be reused for document classification, content review, data extraction, and many other text-processing tasks.

The usual advice is to split it up. That advice is correct and, on GitHub, has historically been annoying to follow: you either open one PR and wait for it to merge before starting the next, or you open several PRs whose diffs all contain each other's changes, and reviewers have to mentally subtract one from the other.

On 30 July 2026, GitHub moved stacked pull requests into public preview. This is the workflow that tools like Graphite, git-branchless and Gerrit have offered for years, now built into the place the review already happens.

TL;DR

• A stack is an ordered series of PRs, each targeting the one below it, so every PR shows only its own layer's diff.
• Reviewers can work on different layers at the same time instead of queueing behind one big review.
• Merging the top ready PR lands it and every unmerged layer beneath it in one operation; merging a middle layer auto-rebases and retargets the ones above.
• Branch protections, required checks, and merge requirements keep working as they already do.
• Install with gh extension install github/gh-stack, or create stacks on github.com or mobile.
• Merge queue support is still rolling out, so check that before you restructure a repo's workflow around this.
• Stacking suits changes that are genuinely sequential. It does not help when your work is really several independent changes, and it actively hurts when the bottom layer is the contentious one.

Prerequisites

• Comfort with git rebase and what it does to commit history
• A GitHub repository you can open PRs against
• The GitHub CLI installed, if you want the terminal workflow
• Familiarity with your repo's branch protection rules, since stacking interacts with them

The problem stacking solves

Say you are adding rate limiting to an API. The work has a natural order:

1. Add a Redis client and its config
2. Add a token bucket implementation with tests
3. Add the middleware that uses it
4. Turn it on for three routes

That is one feature and four genuinely separate reviews. The Redis client is infrastructure someone should check for connection handling and timeouts. The token bucket is an algorithm someone should check for correctness. The middleware is integration. The rollout is a judgement call about which routes go first.

Without stacking you have two options, and both are bad.

One big PR. All four concerns arrive at once. The reviewer who cares about the bucket algorithm has to scroll past config. The person who knows the routes has to read Redis setup. Everyone reviews everything shallowly.

Sequential PRs. You open the Redis PR, then wait. It sits for a day. You cannot start the token bucket on top of it without branching off an unmerged branch, and if you do, its PR diff will include the Redis changes too, because GitHub compares against main by default. Reviewers see 400 lines when 120 are yours.

The second problem is the one stacking fixes directly. Each PR targets the branch below it rather than main, so its diff contains only that layer.

What is actually in the preview

The announcement is specific about the capabilities, and they map to the pain points above.

Each PR shows only its layer. Open any PR in the stack and you review that layer's diff. GitHub renders a stack map alongside it showing where this PR sits in the larger change, which is the context a standalone small PR normally loses. "Why are we adding a token bucket?" is answerable without asking.

Reviews happen in parallel. Four people can review four layers at once. On a sequential-PR workflow, layer 2 cannot even be opened until layer 1 merges, so the total wall-clock time is the sum of every review. In a stack it is closer to the slowest single review.

Merging is flexible in both directions. You can merge the latest ready PR and land it plus every unmerged layer below it in one operation. Or you can land layers one at a time, and the PRs above automatically rebase and retarget. That second behaviour is the tedious part of hand-rolled stacking, where merging the bottom branch leaves you rebasing three branches by hand and force-pushing each one.

Your existing rules still apply. Branch protections, required status checks, and merge requirements govern what reaches main exactly as before. This matters more than it sounds: a common worry about stacking tools is that they route around review policy, and here the policy is unchanged.

Creating a stack

There are several entry points: github.com, the mobile app, and a CLI extension. There is also a gh-stack skill so Copilot's coding agents can work with stacks.

For terminal work, install the extension:

The baseRefName column is the whole idea. A normal PR has main as its base and its diff is measured against main. A stacked PR's base is the layer below, so its diff is measured against that, and only your new work shows up.

The part that used to hurt: rebasing

Here is why people bounced off hand-rolled stacking before tooling existed.

You have four branches. A reviewer asks for a change in layer 2. You amend the token bucket, and now layers 3 and 4 are built on a commit that no longer exists. You rebase middleware onto the new token-bucket, force-push, then rebase enable-routes onto the new middleware, force-push. Four layers is manageable. Six is not, and one mistake with --force on the wrong branch loses work.

This is the cascade that automation exists to handle:

GitHub's version handles the retargeting when layers merge. If you are rebasing by hand for any reason, use --force-with-lease rather than --force, so a push fails instead of silently discarding a teammate's commit. Our post on undoing a Git rebase covers recovery through the reflog when one goes wrong, which is worth reading before your first stack rather than during it.

When stacking is the wrong tool

A stack encodes a claim: these changes are ordered, and later ones depend on earlier ones. When that claim is false, stacking adds coordination cost for nothing.

Your changes are actually independent. If four changes touch different parts of the codebase and none depends on another, open four normal PRs against main. They already review in parallel and merge in any order. Putting them in a stack invents a dependency and means a hold-up on layer 1 blocks the rest.

The bottom layer is the contentious one. This is the failure mode worth planning for. If layer 1 is "switch to a new Redis client library" and that is going to get argued about, then layers 2 through 4 are built on a foundation that might not survive. Sequence deliberately: put the parts you are confident about at the bottom and the debatable design decisions at the top, where reworking them does not cascade.

The change genuinely is atomic. A rename across 200 files is one change. Splitting it into five PRs that each leave the build broken is worse than one large mechanical diff with a clear commit message. Reviewers skim mechanical changes quickly, and that is fine.

Every layer must be independently safe to merge. This is the discipline stacking demands and the one teams underestimate. If layer 2 merges to main on its own, main must still build, tests must still pass, and production must still work. A half-wired feature is acceptable; a broken one is not. That usually means the wiring layer comes last and often sits behind a flag. Our post on progressive delivery with feature flags covers the pattern that makes this comfortable.

What this changes about review culture

The interesting effect is not the tooling, it is what stacking does to the incentives.

Splitting a big change has always been possible and has always cost the author something: extra branches, extra PR descriptions, waiting on merges, rebasing. Reviewers benefit and authors pay, which is why "sorry, this got big" is such a common comment. Lowering the author's cost is what changes behaviour.

Two things worth deciding as a team before adopting it:

How small is a layer? A stack of twelve PRs each changing eight lines is its own kind of unreviewable. The unit that works is a coherent idea a reviewer can hold in their head, which in practice is usually somewhere between 50 and 400 lines.

Who reviews what? The value of parallel review only materialises if layers reach different people. If one person reviews all six layers sequentially, you have added stack management overhead and saved nobody any time. Route the algorithm layer to whoever knows that domain and the rollout layer to whoever owns the service.

Try it on something small

The honest way to evaluate this is on a change you were going to split anyway.

1. Pick a feature with a genuine internal order, ideally three or four layers.
2. Create the branches so each is built on the previous one.
3. Open each PR with the layer below as its base.
4. Get different people to review different layers and see whether the parallelism materialises.
5. Merge the bottom layer first and watch what happens to the ones above it.

Step 5 is the one to pay attention to, because auto-retargeting is the feature that decides whether stacking is sustainable for your team or an occasional trick for big changes. Doing it by hand is exactly the friction that kept this workflow niche outside of companies that built tooling for it.

If you want to shore up the underlying Git first, our Git concepts simulator covers branching and rebasing interactively, and resolving merge conflicts covers the situation you are most likely to hit mid-stack.

Wrapping up

Stacked pull requests do not make large changes small. They make a large change reviewable as a sequence of small ones, which is a different and more achievable thing.

The workflow has existed for years in other tools. What changed on 30 July 2026 is that it is now native to GitHub, so the stack lives where the review, the checks, and the branch protections already are, and nobody has to adopt a second tool to get it.

Worth trying on your next change that would have earned an apology in its description.

await fetch(customer.webhookUrl, {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify(event),
});

It works. You ship it. Then, over the following months, a series of tickets arrives that all turn out to be the same ticket.

A customer's endpoint was down for a deploy and they want the twelve events from that window. Someone asks how they can tell a request really came from you and not from anyone who read your docs and knows the payload shape. A customer's integration ran twice on one order and double-charged an end user. Someone's endpoint takes 40 seconds to respond and your worker pool is full of requests waiting on it. Someone asks, on a Tuesday, whether you sent event evt_8813 last Friday, and you have no way to answer.

None of these are webhook problems. They are delivery problems, and they are the entire reason webhook infrastructure exists as a category.

TL;DR

• A webhook sender is a queue with a retry policy, not an HTTP client. Budget for that up front.
• Retries need exponential backoff and a defined give-up point. Svix uses 8 attempts across roughly 27 hours.
• Sign payloads with HMAC over id.timestamp.body, and verify against the raw body. Parsed-then-restringified JSON will not match.
• Delivery is at-least-once, so receivers must deduplicate on a message ID that stays stable across retries.
• Retry your own API calls with an idempotency key so a network blip on your side does not produce two events.
• The feature customers ask for most is not retries, it is a log they can look at themselves.

Prerequisites

• Node.js 20 or newer, for the examples
• Comfort with HTTP semantics: status codes, timeouts, request bodies
• A rough idea of HMAC (a keyed hash; same input plus same key gives the same digest)
• Optional: a free Svix account, if you want to run the sending half against the real API

Why a POST is not a delivery

The gap between the two is that a POST is an event and a delivery is a state machine. Once you accept that a customer's endpoint can be slow, down, or wrong, the send has to outlive the request that triggered it.

The important word is durable. If your retry logic is a for loop with a sleep in the request handler, then a deploy in the middle of the backoff drops the event permanently, and you will not find out, because the process that knew about it is gone. Any real implementation writes the pending delivery down first.

This is the same shape as the problem in our message queue simulator, and it is worth internalising the reason: a webhook is a message queue where the consumer is a stranger who is under no obligation to be up, fast, or correct.

Failure modes, and what each one means

Not all failures are the same, and treating them the same is the most common mistake. What matters is whether retrying could plausibly help.

The timeout row is the interesting one, and it is the reason idempotency is not optional. A timeout means you do not know the outcome. The receiver may have taken the request, written it to their database, spent 35 seconds sending a confirmation email, and then failed to answer you in time. If you retry, they get it twice. If you do not retry, you might have dropped it. There is no third option that avoids both, which is why the industry settled on "retry, and make the receiver's side safe to run twice".

Retries and backoff

Linear retries are worse than no retries when an endpoint is genuinely down. Retrying every 30 seconds for an hour produces 120 requests, all of which fail, and if you have a thousand customers behind that same broken endpoint you have built a small load generator pointed at someone else's recovering database.

Exponential backoff fixes the shape: try fast a couple of times to ride out a blip, then spread the rest out so a long outage costs you a handful of attempts rather than thousands.

Svix's retry schedule is a concrete, published example, which makes it useful to reason about:

attempt 1   immediately
attempt 2   +5 seconds
attempt 3   +5 minutes
attempt 4   +30 minutes
attempt 5   +2 hours
attempt 6   +5 hours
attempt 7   +10 hours
attempt 8   +10 hours

Eight attempts, and the last one lands about 27 hours and 35 minutes after the first. Their docs give a worked example that is a good sanity check on how to read the table: a message that fails three times before succeeding is delivered "roughly 35 minutes and 5 seconds following the first attempt", which is 5s + 5m + 30m. The intervals are gaps between attempts, not offsets from the start.

Here is what that curve looks like against the linear alternative:

Two design questions matter more than the exact numbers.

Where do you give up? You need a terminal state, or failed deliveries accumulate forever. Svix marks the message Failed and then sends you a webhook about it, message.attempt.exhausted, which is a nice touch: your webhook system tells you about its own failures through the same channel your customers use.

When do you stop trying an endpoint entirely? An endpoint that has been dead for a week should not receive a fresh 8-attempt schedule for every event. Svix auto-disables an endpoint after repeated failures spanning 5 days (with at least 12 hours between the first and last failure in a 24-hour window) and fires an EndpointDisabledEvent. If you build this yourself, some version of this circuit breaker is load-bearing, because without it one abandoned customer integration generates traffic and log volume indefinitely.

Signatures: proving the request came from you

A webhook endpoint is a public URL that accepts POSTs and does something consequential. Anyone can find it and anyone can call it. Shared-secret-in-a-header works, but leaks the secret to every intermediary and every log that captures headers, and gives you nothing to rotate against.

The standard answer is an HMAC signature. Svix implements the Standard Webhooks spec, which is worth learning once because a growing number of providers use it.

Three headers arrive with each request:

svix-id: msg_2Xg8kFmqLxKp4v9rNtQwYbCdEf
svix-timestamp: 1785350000
svix-signature: v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE=

The signature covers the ID, the timestamp, and the body, joined with periods:

signedContent = `${svix_id}.${svix_timestamp}.${body}`

Including the ID and timestamp in the signed content is what makes the signature resistant to replay: an attacker who captures a valid request cannot change the timestamp without invalidating it, so a receiver that rejects old timestamps has a bounded replay window.

In practice you call a library, and it is two lines:

Four details in that manual version account for most of the bugs people hit:

Use the raw body. This is the one that costs people an afternoon. express.json() parses the body and throws away the bytes, and JSON.stringify of the parsed object is not guaranteed to reproduce them: key order, whitespace, and unicode escaping can all differ. The signature is over bytes, so you need the bytes. In Express that means express.raw({ type: 'application/json' }) on the webhook route specifically.

Decode the secret. whsec_MfKQ9r8... is a prefix plus base64. HMAC with the decoded bytes, not the string.

Compare in constant time. crypto.timingSafeEqual, not ===. And check lengths first, because timingSafeEqual throws rather than returning false when the buffers differ in length, which turns a signature mismatch into a 500.

Handle multiple signatures. The header can carry more than one, space-delimited, which is how secret rotation works: for a window, both the old and new secrets produce valid signatures. Accept any match.

On timestamps: the official libraries enforce the tolerance for you. The standardwebhooks package that the Node SDK depends on sets WEBHOOK_TOLERANCE_IN_SECONDS = 5 * 60, so a request whose timestamp is more than five minutes from your clock is rejected. Worth knowing if you ever debug a verification failure on a box with drifting time, because the error looks identical to a wrong secret.

Duplicates and idempotency

Webhook delivery is at-least-once. Every provider worth using tells you this plainly, and the reason is the timeout case from earlier: the sender cannot distinguish "you did not get it" from "you got it and did not tell me". Given that choice, delivering twice is the safer failure.

So the receiver has to be safe to run twice. There are two halves to get right, and they are easy to conflate.

Receiver side: deduplicate on the message ID. The svix-id header (webhook-id in the unbranded Standard Webhooks naming) identifies the message, and it stays the same across every retry of that message. That property is what makes it usable as a dedup key. Svix's docs suggest caching seen IDs with a 24-hour expiry, which lines up with the ~27-hour retry window.

// Cheap version: a unique index does the work, no cache to keep warm.
// The insert fails if we have seen this message before, which is the signal.
try {
  await db.processedWebhook.create({ data: { id: svixId } });
} catch (err) {
  if (isUniqueViolation(err)) {
    // Already handled. Acknowledge so the sender stops retrying.
    return res.status(200).send('duplicate, ignored');
  }
  throw err;
}

await handleEvent(event); // now safe: exactly one of these runs

The subtlety is when you write the dedup row. Write it before the work and a crash mid-handler means the event is marked processed but is not; write it after and two concurrent deliveries both pass the check. Doing the insert and the work in one transaction is the version that holds up.

Sender side: use an idempotency key on your API calls. This is the mirror image and it is separate. When your service calls the webhook API and the connection drops, you do not know whether the event was created. Retry blindly and your customer may get the same event twice from a single business action.

Svix supports Idempotency-Key on POSTs. Send the same key and you get the original response back rather than a second event. Keys are retained for up to 12 hours. In the Node SDK it is a third argument:

import { Svix } from 'svix';

const svix = new Svix(process.env.SVIX_AUTH_TOKEN!);

await svix.message.create(
  'customer-a1b2c3',
  {
    eventType: 'invoice.paid',
    eventId: `invoice.paid.${invoice.id}`, // your own stable ID, useful for lookups
    payload: {
      type: 'invoice.paid',
      invoiceId: invoice.id,
      amountCents: invoice.amountCents,
      currency: invoice.currency,
    },
  },
  // Derive it from the business event, not randomly, so a retry of the
  // same operation reuses it. randomUUID() here would defeat the point.
  { idempotencyKey: `invoice-paid-${invoice.id}` },
);

That key derivation is the part worth staring at. An idempotency key generated fresh on each attempt is just a random string and buys you nothing. It has to be a deterministic function of the thing that happened.

Ordering, and why you probably should not want it

"Can you deliver these in order?" is a reasonable-sounding request that costs more than it looks.

Svix's regular endpoints send in order on a best-effort basis: messages are queued and picked up in order, but a slow or failing delivery does not hold the line, so a message that needs three retries arrives after messages created later. For strict ordering they offer FIFO endpoints, and the tradeoff is explicit in their own docs: a delivery failure blocks the whole endpoint until it succeeds, and per-message network latency of 40 to 50 ms caps throughput around 20 messages per second unless you batch.

That is head-of-line blocking, and it is inherent rather than an implementation weakness. Strict ordering means one stuck message stops everything behind it.

The alternative that usually costs less: make events carry enough information to be ordered by the receiver. A monotonic sequence number or the resource's updatedAt, and a receiver that ignores an event older than the state it already has. That handles reordering and duplicates with the same check, and it does not couple your throughput to your slowest endpoint.

Rate limiting, from both directions

Two different concerns share the name.

Your customers can be overwhelmed by you. A batch job that updates 50,000 records should not turn into 50,000 POSTs at once against a customer running one small container. Svix lets you set a rate limit in messages per second per application or per endpoint, and throttles to hold that rate rather than dropping.

And you can be rate-limited by them, which arrives as 429. Treat it as a retryable failure and as a signal: back off, and if it keeps happening, lower that endpoint's configured rate. Our rate limit simulator covers the algorithms if you want to see how the different bucket strategies behave under bursts.

Observability, which is the actual product

Here is the thing that surprises people who build this internally: the retry engine is the part you plan for, and the delivery log is the part customers actually ask for.

When a customer says "we did not get the event", you need to answer, quickly, some version of: we attempted it at 14:02:11, your endpoint returned 503 with this body, we retried at 14:07:16 and got 200. Without that, every integration question becomes an engineer reading production logs, and you will get those questions weekly forever.

What you need to be able to answer:

• Was the event created at all? (Distinguishes your bug from theirs)
• Which endpoints was it fanned out to?
• Every attempt: timestamp, response status, response body, duration
• The exact payload as sent, so signature debugging is possible
• The current state: delivered, retrying with the next attempt at a known time, or exhausted

The multiplier is letting customers see it themselves. Svix's angle here is Svix Portal, an embeddable UI where your customer manages their own endpoints, reads their own delivery log, and replays their own failures without opening a ticket. That is worth pricing honestly if you are considering building: it is a whole small product, and it is the difference between "we have retries" and "our customers can debug their own integration".

A working example, both halves

Two files. The sender goes through Svix; the receiver is what you would hand a customer.

The sender

// sender.ts
import { ApiException, Svix } from 'svix';

const svix = new Svix(process.env.SVIX_AUTH_TOKEN!);

// Event types belong to the environment, not to one customer. Run this once
// during deployment before creating endpoints that filter on these names.
export async function configureWebhookEventTypes() {
  const eventTypes = [
    { name: 'invoice.paid', description: 'An invoice was paid' },
    { name: 'invoice.payment_failed', description: 'An invoice payment failed' },
  ];

  await Promise.all(
    eventTypes.map(async (eventType) => {
      try {
        await svix.eventType.get(eventType.name);
      } catch (err) {
        if (!(err instanceof ApiException) || err.code !== 404) throw err;
        await svix.eventType.create(eventType);
      }
    }),
  );
}

// One Svix "application" per customer. The uid is your own customer ID,
// which means you never have to store a mapping.
export async function onboardCustomer(customerId: string, webhookUrl: string) {
  await svix.application.create({ name: `Customer ${customerId}`, uid: customerId });

  const endpoint = await svix.endpoint.create(customerId, {
    url: webhookUrl,
    description: 'Primary endpoint',
    // Subscribe to specific event types; omit for everything.
    filterTypes: ['invoice.paid', 'invoice.payment_failed'],
  });

  // Show this to the customer once. They need it to verify signatures.
  const { key } = await svix.endpoint.getSecret(customerId, endpoint.id);
  return { endpointId: endpoint.id, signingSecret: key };
}

export async function emitInvoicePaid(customerId: string, invoice: Invoice) {
  return svix.message.create(
    customerId,
    {
      eventType: 'invoice.paid',
      eventId: `invoice.paid.${invoice.id}`,
      payload: {
        type: 'invoice.paid',
        invoiceId: invoice.id,
        amountCents: invoice.amountCents,
        currency: invoice.currency,
        paidAt: invoice.paidAt.toISOString(),
      },
    },
    { idempotencyKey: `invoice-paid-${invoice.id}` },
  );
}

The setup call is not optional when you use filterTypes: Svix rejects an endpoint that names event types the environment does not know yet. Register them once during deployment, then onboard as many customer applications as you need.

Note what is absent: no queue, no attempt table, no backoff scheduler, no dead-letter handling. That is the part being bought.

The receiver

// receiver.ts
import express from 'express';
import { Webhook, WebhookVerificationError } from 'svix';

const app = express();
const wh = new Webhook(process.env.SVIX_WEBHOOK_SECRET!);

// express.raw, NOT express.json. The signature is over the bytes.
app.post(
  '/webhooks/billing',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    let event: BillingEvent;
    try {
      event = wh.verify(req.body, req.headers as Record<string, string>) as BillingEvent;
    } catch (err) {
      if (err instanceof WebhookVerificationError) {
        // 400, not 401: this is a malformed request, and a 4xx tells the
        // sender not to waste 27 hours of retries on it.
        return res.status(400).send('invalid signature');
      }
      throw err;
    }

    const messageId = req.header('svix-id')!;

    try {
      // Dedup row and the work in one transaction, so a crash rolls back
      // both and the retry gets a clean shot.
      await db.$transaction(async (tx) => {
        await tx.processedWebhook.create({ data: { id: messageId } });
        await applyBillingEvent(tx, event);
      });
    } catch (err) {
      if (isUniqueViolation(err)) {
        // Seen it. 200 so the sender stops retrying.
        return res.status(200).send('duplicate');
      }
      // Anything else: 500 on purpose, so this gets retried.
      console.error({ messageId, err }, 'webhook handler failed');
      return res.status(500).send('handler failed');
    }

    res.status(200).send('ok');
  },
);

The status codes are doing real work here, and they are the part most handlers get wrong. A 400 on a bad signature stops the retries. A 500 on a handler error invites them. A 200 on a duplicate ends a chain that would otherwise run its full schedule. Answering "what should this endpoint return?" correctly is most of what makes a receiver well-behaved.

Testing it locally

The awkward part of webhook development is that you need a public URL. Svix Play gives you a throwaway one that shows you exactly what arrived, headers included, which is the fastest way to check what you are sending. For the receiving side, the Svix CLI forwards to localhost:

That last step is the one worth doing deliberately. resend reuses the original message ID, which is exactly what a real retry does, so it exercises the dedup path for real. Most webhook receivers have never had a duplicate delivered to them on purpose, which means that path has never run outside of a unit test.

Where Svix changes the tradeoff

The useful thing about Svix Dispatch is not that it can send an HTTP request. It turns the operational surface around that request into one product: durable delivery, automatic retries, signing and secret rotation, per-endpoint rate limits, event filtering, searchable attempt logs, manual replay, and a customer-facing portal. Those are the pieces that tend to appear one support ticket at a time after a home-grown sender ships.

Building is reasonable when:

• You have one internal consumer, or a handful, and you control them. Then it is not webhooks, it is a queue with an HTTP consumer, and you already run a queue.
• Volume is low and the events are not consequential. A Slack notification that occasionally does not arrive is not an incident.
• You have a strong existing job system. If you already run Temporal, Sidekiq, or River, the retry-with-backoff-and-give-up part is a config away, and that is genuinely most of the engine.

Dispatch starts to win when the consumers are customers. That is the line. The moment the endpoints belong to people who can open tickets, the surface expands past the retry engine into things that are individually small and collectively a product. Teams consistently underestimate that list because they scope the engine and forget the operations around it.

A useful way to decide: write down what a customer will ask you when an event does not arrive, and then work out who answers it. If the answer is "an engineer greps production logs", you have found the real cost, and it recurs weekly for as long as the integration exists.

Wrapping up

The delivery problems are the same everywhere, so the checklist is portable whether you build or buy:

1. Persist before you send. A pending delivery that only exists in a running process is a delivery you will lose on your next deploy.
2. Back off exponentially, and define where you stop. Both per message and per endpoint.
3. Classify failures. Retry 5xx, timeouts, 429. Do not retry 400, 401, 404.
4. Sign with HMAC over id.timestamp.body, verify raw bytes, compare in constant time. Support two valid secrets so rotation is possible.
5. Assume at-least-once in both directions. Dedup on the message ID at the receiver; use an idempotency key derived from the business event at the sender.
6. Prefer sequence numbers over strict ordering. Strict FIFO buys you head-of-line blocking.
7. Build the log before you need it, and let customers read it.

If those mechanics are product infrastructure rather than your product, Svix Dispatch packages them behind one API and gives your customers a polished place to configure endpoints, inspect attempts, and replay failures themselves. It also builds on the Standard Webhooks signing model, so receivers get a documented verification contract instead of a proprietary signature scheme. Their docs publish the operational details, including retry timing and ordering tradeoffs, which makes the service easier to evaluate against a home-grown implementation.

For an interactive walkthrough of retries, signatures, and duplicate handling, try the webhook delivery simulator. For related reading on the same underlying problem, our post on designing automation with failure in mind covers the general pattern, and the message queue simulator is a good way to build intuition for at-least-once delivery before you have to debug it in production.

You do this several times a week. It is not hard, it is just tedious, and it is exactly the shape of problem that cheap inference is good at: a lot of text, a small answer, no need for the model to be clever.

So we built it. A GitHub Action that takes a failing job's log and posts what broke, why, and what to try first. It runs on DigitalOcean's serverless inference, the code is on GitHub, and the whole thing is about 400 lines.

The interesting part turned out not to be the model call. That was twenty lines. The interesting part was everything we did before it.

TLDR

• Sending the whole log works and is the wrong instinct. Reducing it first cut 92.5% of the bytes and made the answers better.
• Stripping GitHub's per-line timestamp prefix alone moved the reduction from 86% to 92.5%, because it repeats on every single line.
• DigitalOcean's inference API is OpenAI-compatible, so any OpenAI client works against https://inference.do-ai.run/v1.
• Reasoning models fail in a way that looks exactly like a broken API key. Budget for it.
• A tool that explains failing builds must never fail a build. Ours exits 0 no matter what.

Prerequisites

• A DigitalOcean account with a model access key and a prepaid balance
• A repository with CI that fails sometimes, which is all of them
• Node 20 or newer if you want to run the CLI locally

The naive version works, and you should not ship it

The first version of anything like this is four lines:

const log = await fetchJobLog(runId);
const answer = await model.chat(`Why did this fail?\n\n${log}`);

This works. It also sends 25KB of mostly-irrelevant text on every failure, and the answer is worse than it needs to be, because the actual error is buried in four hundred lines of npm info resolving.

Both problems have the same fix.

Reducing the log

Here is the shape of a real failing deploy log, one of ours:

272 lines
25,534 characters
of which roughly 26 lines explain the failure

The reduction runs in three passes.

Strip the per-line prefixes. This one is worth more than it looks. GitHub prefixes every line with an ISO timestamp, and gh run view --log prefixes it further with the job and step name:

deploy	Deploy to DigitalOcean VPS	2026-07-27T13:57:14.3928847Z ERROR: relation "Segment" does not exist

That is 62 characters of prefix on a 48-character message, repeated on every line in the file. Stripping it took our reduction from 86% to 92.5% on its own. ANSI colour codes go the same way.

Keep a window around anything that looks like a failure. Error, failed, exception, panic, traceback, exit code, permission denied. Keep eight lines either side, because the line that says Error: is rarely the line that tells you why.

Always keep the tail. Some failures end quietly, with a non-zero exit and nothing dramatic. The last 25 lines come along regardless.

export function extractRelevant(raw, opts = {}) {
  const { context = 8, tail = 25, maxLines = 160 } = opts;
  const all = raw.split('\n').map(cleanLine);
  const keep = new Set();

  all.forEach((line, i) => {
    if (isNoise(line) || !isSignal(line)) return;
    for (let j = Math.max(0, i - context); j <= Math.min(all.length - 1, i + context); j++) {
      keep.add(j);
    }
  });

  for (let i = Math.max(0, all.length - tail); i < all.length; i++) keep.add(i);
  // ...
}

One detail that matters more than it should: mark the gaps.

Applying migration `20260727130000_team_scoped_unique_constraints`
... 41 lines omitted ...
ERROR: relation "Segment" does not exist

Without the marker the model sees two adjacent lines and reasons about them as if they happened in sequence. With it, it knows something was cut and says so when it matters.

On our example: 25,534 characters down to 1,926. 272 lines down to 26.

Calling DigitalOcean inference

The API is OpenAI-compatible, so there is nothing to learn:

const res = await fetch('https://inference.do-ai.run/v1/chat/completions', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${apiKey}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'openai-gpt-oss-20b',
    messages: [{ role: 'system', content: SYSTEM }, { role: 'user', content: log }],
    max_tokens: 1200,
  }),
});

Any OpenAI SDK works if you point baseURL at it. We used plain fetch so the Action has no dependencies to install, which keeps the job fast.

At the time of writing there are 74 models on the endpoint, and serverless inference is billed per token from a prepaid balance rather than by reserved GPU hours, which is the model that makes a per-CI-failure tool sensible in the first place.

The prompt is a format, not a request

The difference between a useful answer and a paragraph of hedging is telling the model exactly what shape to produce:

**What failed:** one sentence naming the step and the proximate cause.

**Why:** two or three sentences on the underlying reason. If the log does not
say, write what it would take to find out. Never invent a cause.

**Try this first:** one concrete action.

Rules:
- Quote the exact error string once, in backticks.
- If several things failed, address the earliest one that could have caused the rest.
- If the log is truncated or inconclusive, say so plainly instead of guessing.

"If the log does not say, write what it would take to find out" is the line that earns its place. Without it you get confident guesses. With it you get a model that says the log is inconclusive, which is a genuinely useful answer.

The gotcha that looks like a broken API key

Our first call returned HTTP 200, a valid response body, and an empty string.

openai-gpt-oss-20b is a reasoning model. It puts its thinking in reasoning_content and the answer in content. We had set max_tokens low while testing, so the model spent the entire budget reasoning and had nothing left for the answer:

{
  "finish_reason": "length",
  "message": { "content": null, "reasoning_content": "The user says..." }
}