Inbound Email Processing for AI Agents: Classification, Routing, Actions

Most agent email work focuses on sending — formatting a message, threading a reply, hitting deliverability targets. The harder problem is the inbound side: a raw email arrives, and your agent has to decide what it is, who should handle it, and what to do next. This post covers the full pipeline from MIME parsing to triggered action, with concrete implementation patterns.

What makes inbound email hard for agents

Inbound email is hard because the input is radically underspecified. Unlike a structured API call, an email can be plain text, HTML, multipart MIME with attachments, a forwarded thread with quoted headers, an auto-reply, a bounce notification, or a calendar invite — all arriving at the same address with no schema contract. Your agent has to extract signal from that noise before it can do anything useful.

Four specific challenges compound this:

• Format ambiguity: A single message might have a text/plain part, a text/html part, and an application/pdf attachment. The "real" content your agent needs might be in any of them.
• Thread pollution: Quoted replies include previous messages. Naively feeding the full raw body to a classifier or LLM inflates token counts and introduces stale context.
• Operational noise: Roughly 5-15% of inbound email in automated systems is operational traffic — bounces, OOO auto-replies, DMARC reports, spam — not user-initiated messages. Misclassifying these as real intent is a common failure mode.
• Idempotency: Email delivery isn't exactly-once. Your webhook endpoint may receive the same message twice. Every action triggered by an inbound email must be idempotent, keyed on the message's Message-ID header.

The inbound processing pipeline

A solid inbound pipeline has five distinct stages. Each stage has a clear input/output contract, which makes the whole system testable in isolation.

flowchart LR
  A[Raw MIME Message] --> B[Parse and Normalize]
  B --> C[Deduplicate by Message-ID]
  C --> D[Classify Intent]
  D --> E[Route to Handler]
  E --> F[Execute Action]

Stage 1: Parse and normalize

Before classification, you need a clean representation of the message. Parse the raw MIME tree and extract:

• Envelope fields: From, To, Cc, Reply-To, Message-ID, In-Reply-To, References
• Body text: Prefer text/plain if present; strip HTML tags from text/html as fallback. Strip quoted reply blocks (lines starting with >, or content following On [date] [name] wrote: patterns).
• Attachments: Record filename, MIME type, and size. Decide per-handler whether to fetch content.
• Headers of interest: Auto-Submitted, X-Auto-Response-Suppress, Precedence, List-Unsubscribe

Stripping quoted reply blocks is worth implementing carefully. A regex like /^>.*$/m catches simple quoting but misses Outlook's -----Original Message----- delimiter or Gmail's On Mon, Jan 6 wrote: prefix. Build a small corpus of real quoted reply formats and test against it.

The output of Stage 1 is a normalized struct — call it ParsedMessage — that every downstream stage consumes. Never pass raw MIME further down the pipeline.

Stage 2: Deduplicate by Message-ID

Every RFC 5322-compliant email has a Message-ID header that uniquely identifies it — something like <018f2c3d.abc123@mail.example.com>. Store a processed-message log (Redis, Postgres, whatever your stack uses) and check this ID before proceeding. If you've seen it, acknowledge the webhook and return 200 without processing.

This is not optional. Webhook delivery systems retry on timeout or 5xx. Email relay infrastructure occasionally delivers duplicates. Without deduplication, your agent will execute the same action twice, and many actions — sending a reply, creating a ticket, charging a card — are not naturally idempotent.

Stage 3: Classify intent

Classification is where you determine what kind of message this is and what the sender wants. These are two separate axes.

Message type (structural classification):

• Human-authored message
• Auto-reply (OOO, vacation responder)
• Delivery Status Notification (DSN/bounce)
• Spam or phishing
• DMARC aggregate/forensic report
• Mailing list digest
• System-generated notification

User intent (semantic classification, applies only to human-authored messages):

• Support request
• Purchase inquiry
• Cancellation
• Data request (GDPR/CCPA)
• Complaint
• General question
• Reply to agent-initiated outreach

Do structural classification first, rule-based. It's fast, cheap, and catches the 10-15% of operational traffic before it hits your LLM. Check Auto-Submitted header (any value other than no indicates automated mail), Precedence: bulk/list/junk, and whether From is a known bounce address pattern (e.g., MAILER-DAEMON@, postmaster@). For DSNs, look for Content-Type: multipart/report; report-type=delivery-status.

Only pass messages that survive structural classification to semantic classification. For semantic classification, two practical options:

Embedding + nearest-neighbor: Embed the normalized body text and compare cosine similarity against labeled examples. Fast, deterministic, works well when intent categories are stable and you have training examples. Classify in under 50ms.

LLM with structured output: Send the normalized body to an LLM with a classification prompt and require JSON output with intent, confidence, and entities. More flexible for open-ended intent detection, but 3-10x slower and costs tokens per message.

A reasonable hybrid: use embedding classification as the primary path, fall back to LLM when max similarity is below a threshold (say, 0.75). Log every LLM classification and use those outputs to expand your embedding training set over time.

For systems handling significant volume, Mails.ai takes part of this off your handler: bounces of your sends arrive as their own message.bounced events rather than as inbound mail, and built-in email classification adds an intent to first-contact mail before your webhook fires. Auto-replies and DMARC reports are not filtered out, so keep the header checks from Stage 1.

Stage 4: Route to handler

Routing maps (intent, metadata) to a specific handler function. Keep routing logic explicit and declarative — a routing table beats a long if/else chain because it's inspectable and testable.

ROUTING_TABLE = [
    # (predicate, handler)
    (lambda m: m.type == "bounce", handle_bounce),
    (lambda m: m.type == "auto_reply", handle_auto_reply),
    (lambda m: m.intent == "cancellation", handle_cancellation),
    (lambda m: m.intent == "data_request", handle_gdpr_request),
    (lambda m: m.intent == "support" and m.in_reply_to is not None, handle_support_reply),
    (lambda m: m.intent == "support", handle_new_support_ticket),
    (lambda m: True, handle_default),  # catch-all
]

def route(message: ParsedMessage):
    for predicate, handler in ROUTING_TABLE:
        if predicate(message):
            return handler
    raise ValueError("No handler matched — check catch-all")

Predicate order matters. More specific rules go first. The catch-all at the end ensures nothing falls through silently.

Note the distinction between handle_support_reply and handle_new_support_ticket. A reply to an existing thread should be appended to a ticket, not create a new one. Use In-Reply-To and References headers to detect thread membership.

Stage 5: Execute action

Actions are the side effects your agent takes in response to a classified, routed message. Common actions include:

• Send a reply: Generate a response and send it, preserving Message-ID → In-Reply-To threading.
• Create a record: Open a support ticket, log a lead, create a task.
• Update state: Mark a subscription cancelled, flag an account for review.
• Trigger a workflow: Kick off a multi-step agent process.
• Escalate to human: Route to a support queue when confidence is low or intent is high-stakes.

Every action must be idempotent. The Message-ID is your natural idempotency key. Before executing an action, check whether you've already executed it for this message ID. Store (message_id, action_type) → result in durable storage.

For actions that trigger external systems, use a job queue (Celery, BullMQ, Inngest) rather than executing inline in the webhook handler. This decouples receipt acknowledgment from processing time, lets you retry failed actions independently, and prevents webhook timeouts from causing duplicate deliveries.

Handling low-confidence classifications

Your classifier will be wrong. The question is what happens when it is.

Define confidence thresholds per intent category, not a single global threshold. A cancellation request acted on incorrectly has far higher cost than a misrouted general question. For high-stakes intents, set a higher confidence floor and route anything below it to a human review queue.

Build an explicit uncertain routing path. When confidence is below threshold:

1. Store the full parsed message in a review queue.
2. Optionally send an acknowledgment to the sender ("We received your message and will respond within 24 hours").
3. Alert a human reviewer.
4. Log the classification result and reviewer's final decision — this becomes training data.

Over time, your low-confidence cases should decrease as you feed corrections back into the classifier.

Thread context and conversation state

For agents that engage in multi-turn email conversations, classification alone isn't enough. You need to reconstruct conversation state from the thread.

Email threads are identified by References and In-Reply-To headers. A well-formed thread has a root message ID in References that all subsequent messages share. Use the root Message-ID as your conversation ID and store all messages under it.

When processing a reply, fetch the full thread history keyed on that conversation ID. Pass the thread history — not just the latest message — to your agent so it has context. Watch your token budgets. A long thread can run to thousands of tokens. Summarize older turns or truncate with a clear [earlier messages truncated] marker rather than silently dropping context.

One practical pattern: store a conversation_summary that updates after each exchange, alongside the raw thread. Your agent receives the summary plus the last 2-3 raw messages, keeping context tight without losing continuity.

Operational signals: bounces and auto-replies

Bounces and auto-replies are operationally important but easy to mishandle.

Bounces (DSNs) tell you a message wasn't delivered. Two kinds:

• Hard bounces (5xx SMTP status): Permanent failures — invalid address, domain doesn't exist. Remove from send lists immediately.
• Soft bounces (4xx SMTP status): Transient failures — mailbox full, server temporarily unavailable. Retry with backoff; convert to hard bounce after 3-5 consecutive failures over several days.

Parse the attached message/delivery-status part to extract the original recipient, final status code, and diagnostic message. Don't pattern-match the subject line — that's fragile. The structured delivery-status attachment is reliable.

Auto-replies (OOO messages) should be detected and suppressed from triggering agent responses. An agent that replies to an OOO message creates an email loop. Check Auto-Submitted: auto-replied header, and additionally check whether the same address has sent you an auto-reply in the last 24 hours — loop detection by recency.

Testing your classification pipeline

A classification pipeline without a test suite will drift. Build a labeled test corpus of at least 200-300 real messages across all intent categories, including edge cases and adversarial inputs. Run classification against this corpus on every deploy and alert on accuracy regression.

Specifically test:

• Messages with no body (common with mobile clients)
• Messages with only an attachment and no text
• Forwarded messages (the body starts with ---------- Forwarded message ---------)
• Messages in non-English languages if your system is international
• Very long messages (> 10,000 characters) that may truncate
• Replies where the quoted body is longer than the new content

For full inbound email parsing infrastructure that handles MIME normalization and delivers clean structured payloads to your webhook, that's a separate concern from classification logic — but the two need to be designed in concert. Your parsing layer determines what fields are available; your classification layer must be designed around what it actually receives.

For the full operational playbook — routing inbound to the right agent, classifying by intent, monitoring per-agent reputation — see the AI email management solutions guide.

Frequently Asked Questions

How do I prevent my agent from replying to auto-replies and creating email loops?

Check the Auto-Submitted header first — any value other than no or absent indicates automated mail. Also check X-Auto-Response-Suppress (common in Microsoft Exchange environments). As a second defense, track reply attempts per address per time window: if the same address triggers more than 2-3 inbound messages in an hour with no human-initiated message in between, suppress further responses and alert for review.

Should I use an LLM for classification or a traditional ML classifier?

Use both in a hybrid. Rule-based filters handle structural classification (bounces, auto-replies, DMARC reports) at zero cost. Embedding-based nearest-neighbor handles stable, well-defined intent categories fast and cheaply. Reserve LLM classification for ambiguous cases or open-ended intents that don't fit a fixed taxonomy. The LLM is 10-50x more expensive per message than embedding classification — use it surgically.

What's the right way to store conversation state for email threads?

Use the root Message-ID of the thread as a stable conversation key. Store all messages under that key with timestamps and sender metadata. Maintain a rolling summary for long threads. For agents that need to take actions based on conversation history, store the action log alongside the message log — you need to know what your agent already did, not just what was said.

How do I handle attachments in classification?

For classification purposes, extract text content from common attachment types (PDF, DOCX, plain text) and append it to the body text before classifying, but weight it lower than the body. Don't block classification on attachment extraction — if extraction fails or the attachment is binary (image, spreadsheet with no extractable text), classify on the body alone. Log attachment metadata (filename, type, size) regardless; some routing rules may trigger on attachment presence even without reading content.

How should I handle messages where confidence is below my threshold?

Route low-confidence messages to a human review queue rather than guessing. Send the sender an acknowledgment so they know their message was received. Store the full parsed message and your classifier's output (top-N intents with scores) alongside the queued message so the human reviewer has context. Feed the reviewer's decision back as a labeled example — over time this builds your training set and reduces the low-confidence rate.

What Message-ID format should I use for replies my agent sends?

Generate RFC 5322-compliant Message-IDs in the format <timestamp.random@yourdomain.com>. The domain part should be a domain you control. When sending a reply, set In-Reply-To to the Message-ID of the message you're replying to, and set References to the full chain (the original References value plus the message you're replying to). Correct threading headers ensure your replies appear as part of the thread in all major email clients, not as disconnected new messages.