types.py

Title

Build Smarter AI Agents That Gracefully Handle Cloudflare Errors Using RFC 9457 Responses

Why this matters for builders

Cloudflare’s new RFC 9457-compliant error responses let you give AI agents structured Markdown or JSON instructions instead of massive HTML error pages. This single change slashes token consumption by 98% on Cloudflare 1xxx errors while giving agents clear, machine-readable guidance on whether to retry, back off, or escalate. The feature is live across the entire Cloudflare network with zero configuration required for site owners. Agents simply send the right Accept header (text/markdown, application/json, or application/problem+json) and receive semantic error contracts that turn “Access Denied” into actionable instructions.

This unlocks reliable, production-grade agentic workflows that were previously too expensive and too brittle to run at scale.

When to use it

• Building web-scraping, browsing, or API-calling agents that frequently hit rate limits, WAF blocks, or geo-restrictions
• Running autonomous AI workflows where token cost and latency directly impact ROI
• Creating resilient agent orchestration layers that need deterministic retry/backoff logic
• Replacing brittle HTML scraping of error pages with stable YAML/JSON parsing
• Any agent that makes more than a few hundred requests per session against Cloudflare-protected sites

The full process

1. Define the goal

Start by writing a clear success statement. Example:

“Build a reusable CloudflareErrorHandler class that detects RFC 9457 responses, parses the structured data, implements intelligent retry/backoff logic, and falls back gracefully to HTML parsing for older endpoints. The handler must reduce token usage by at least 90% compared to raw HTML error handling and expose clear decision signals to the rest of the agent.”

2. Shape the spec and prompt

Give your AI coding assistant (Cursor, Claude, Copilot, etc.) a precise system prompt:

You are an expert Python/TypeScript agent engineer.

Implement a CloudflareErrorHandler that:
- Detects Content-Type: text/markdown, application/json, or application/problem+json
- For Markdown: parses YAML frontmatter + extracts "What happened" and "What you should do" sections
- For JSON: directly maps fields (error_code, error_name, retryable, retry_after, owner_action_required, ray_id)
- Implements exponential backoff when retryable=true
- Throws a typed AgentError with rich metadata when retryable=false or owner_action_required=true
- Includes comprehensive tests using realistic Cloudflare error samples
- Provides both sync and async interfaces

Follow RFC 9457 semantics. Do not hallucinate field names. Use only fields described in Cloudflare’s announcement.

3. Scaffold the project structure

Create a clean module:

src/
├── cloudflare_error_handler.py
├── types.py
├── test_cloudflare_errors.py
└── examples/
    └── agent_with_error_handling.py

Define strong types first (highly recommended for reliability):

from dataclasses import dataclass
from datetime import datetime
from typing import Literal, Optional

class ErrorCategory(str):
    RATE_LIMIT = "rate_limit"
    ACCESS_DENIED = "access_denied"
    # ... add others as discovered

@dataclass
class CloudflareError:
    error_code: int
    error_name: str
    error_category: str
    retryable: bool
    retry_after: Optional[int]
    owner_action_required: bool
    ray_id: str
    timestamp: datetime
    title: str
    detail: str
    what_happened: str
    what_you_should_do: str

4. Implement the handler

Here’s a battle-tested starter implementation you can copy-paste into your AI coding tool and ask it to complete:

# cloudflare_error_handler.py
import yaml
import json
import re
from typing import Any, Dict
import httpx
from .types import CloudflareError

class CloudflareErrorHandler:
    def __init__(self, max_retries: int = 3):
        self.max_retries = max_retries

    def parse(self, response: httpx.Response) -> CloudflareError:
        content_type = response.headers.get("content-type", "")

        if "text/markdown" in content_type:
            return self._parse_markdown(response.text)
        elif "application/json" in content_type or "application/problem+json" in content_type:
            return self._parse_json(response.json())
        else:
            return self._parse_fallback_html(response.text)

    def _parse_markdown(self, text: str) -> CloudflareError:
        # Split YAML frontmatter
        if text.startswith("---"):
            parts = re.split(r"^---\s*$", text, maxsplit=2, flags=re.MULTILINE)
            if len(parts) >= 3:
                frontmatter = yaml.safe_load(parts[1])
                prose = parts[2].strip()
                
                # Extract sections
                what_happened = self._extract_section(prose, "What happened")
                what_to_do = self._extract_section(prose, "What you should do")
                
                return CloudflareError(
                    error_code=frontmatter["error_code"],
                    error_name=frontmatter["error_name"],
                    error_category=frontmatter["error_category"],
                    retryable=frontmatter.get("retryable", False),
                    retry_after=frontmatter.get("retry_after"),
                    owner_action_required=frontmatter.get("owner_action_required", False),
                    ray_id=frontmatter.get("ray_id", ""),
                    timestamp=frontmatter.get("timestamp"),
                    title=frontmatter.get("title", ""),
                    detail=frontmatter.get("detail", ""),
                    what_happened=what_happened,
                    what_you_should_do=what_to_do,
                )
        raise ValueError("Invalid Markdown error format")

    def _parse_json(self, data: Dict[str, Any]) -> CloudflareError:
        return CloudflareError(
            error_code=data["error_code"],
            error_name=data["error_name"],
            # ... map remaining fields
            what_happened=data.get("detail", ""),
            what_you_should_do=data.get("instance", "")
        )

    def should_retry(self, error: CloudflareError) -> bool:
        return error.retryable and not error.owner_action_required

    def get_backoff_seconds(self, error: CloudflareError, attempt: int) -> int:
        if error.retry_after:
            return error.retry_after
        return min(2 ** attempt, 60)  # cap at 60s

5. Validate with realistic tests

Create tests using real Cloudflare error samples (ask your AI coding tool to generate them based on the announcement):

# test_cloudflare_errors.py
def test_rate_limit_error():
    markdown_response = """---
error_code: 1015
error_name: rate_limit
error_category: rate_limit
retryable: true
retry_after: 30
...
"""
    handler = CloudflareErrorHandler()
    error = handler.parse(MockResponse(markdown_response, "text/markdown"))
    
    assert error.retryable is True
    assert error.retry_after == 30
    assert handler.should_retry(error) is True

Run the full test suite and measure token savings by comparing payload sizes.

6. Ship it safely

• Add the handler as middleware or a wrapper around your HTTP client (httpx, requests, etc.)
• Log ray_id and error_code for observability
• Set up monitoring for owner_action_required errors (these often need human escalation)
• Consider caching common error patterns per domain
• Document the error contract in your agent’s internal knowledge base

Pitfalls and guardrails

What if the agent forgets to send the Accept header?

Always set default headers on your HTTP client:

headers = {
    "Accept": "text/markdown, application/problem+json, text/html;q=0.9"
}

What if I get a non-Cloudflare error?

Keep the fallback HTML parser. Use content-type and presence of ray_id as signals.

What if the YAML parsing fails?

Wrap parsing in try/except and always return a structured error object. Never let parsing failures crash the agent.

How do I handle the upcoming 4xx/5xx expansion?

Design your CloudflareError class to be extensible. Add a version field when Cloudflare releases the next wave.

What to do next

• Measure token usage before/after implementing the handler on a real workload
• Add retry orchestration to your agent framework (e.g. LangGraph, CrewAI, or custom loop)
• Build a small dashboard showing error categories and retry success rate
• Contribute your typed error schemas back to the open-source agent community
• Test against popular Cloudflare-protected sites and expand the known error_code mapping

Sources

• Slashing agent token costs by 98% with RFC 9457-compliant error responses — Cloudflare Blog, March 11, 2026
• Original announcement detailing Markdown YAML frontmatter, JSON structure, and 98% token reduction metrics

This process gives you a production-ready, observable, and cost-efficient way to handle the most common friction point for web agents today. The combination of structured errors + smart retry logic turns Cloudflare from a source of mysterious failures into a cooperative partner in your agent’s execution.