Handling LLM Errors, Timeouts, and Rate Limits in Python Applications

Introduction

You've written your first LLM-powered Python application. It works beautifully in testing — you ask it to summarize customer support tickets, it returns crisp, useful summaries every time. Then you deploy it. Real users start hitting it, and suddenly things get messy. Sometimes the API returns a cryptic 429 error. Sometimes it just... hangs. Sometimes you get back a response but it's truncated halfway through a sentence. Your application crashes, users get blank screens, and you're staring at logs trying to figure out what went wrong.

This is the part of building with LLMs that tutorials rarely cover, and it's the part that separates a demo from a production-ready application. LLM APIs — whether you're using OpenAI, Anthropic, Google, or any other provider — are network services. That means they're subject to all the same failure modes as any other network call: timeouts, rate limits, temporary outages, and malformed responses. The difference is that LLM calls are often slower and more expensive than a typical API call, which makes handling failures correctly even more important.

By the end of this lesson, you'll know how to build Python applications that handle LLM failures gracefully, retry intelligently, and give users meaningful feedback even when things go wrong.

What you'll learn:

• What the most common LLM API errors are and why they happen
• How to implement retry logic with exponential backoff
• How to set and handle timeouts so your app never hangs indefinitely
• How to respect rate limits without breaking your application's flow
• How to structure error handling so failures degrade gracefully instead of crashing

---

Prerequisites

• Basic Python familiarity (functions, try/except blocks, loops)
• You've made at least one API call to an LLM provider (OpenAI, Anthropic, etc.)
• A working Python environment with openai installed (pip install openai)

You don't need to be an expert. If you've followed any "getting started with GPT" tutorial, you have enough background.

---

The Failure Landscape: What Can Go Wrong

Before writing a single line of error handling, you need to understand what you're handling. There are four main categories of failure when calling LLM APIs.

Rate limit errors (HTTP 429) happen when you send too many requests in a given time window. Every API provider enforces limits — for example, you might be allowed 60 requests per minute or 90,000 tokens per minute. When you exceed those limits, the API doesn't just slow down; it refuses your request entirely and tells you to back off. Think of it like a highway on-ramp with a traffic light — when traffic is heavy, the light makes you wait before merging.

Timeout errors happen when a request takes too long and your code (or the server) gives up waiting. LLM calls are slow compared to most API calls. A complex generation might take 20-30 seconds. If you don't set an explicit timeout, your code can hang indefinitely waiting for a response that may never come. And if you set the timeout too short, you'll abort requests that were actually going to succeed.

Server errors (HTTP 500, 502, 503) happen when something goes wrong on the provider's end — a server is overloaded, a deployment is in progress, or there's a transient infrastructure hiccup. These are usually temporary and safe to retry.

Client errors (HTTP 400, 401, 422) happen when your request is the problem. A malformed payload, an expired API key, or a request that exceeds the model's context window. These are not safe to retry automatically — you'd just fail again. You need to fix the root cause.

Understanding which category an error falls into tells you what to do about it.

---

Setting Up: A Realistic Example

Throughout this lesson, we'll work with a realistic scenario: a Python function that takes a customer support ticket and generates a suggested response. This is the kind of task you'd actually build in production.

Here's the naive version — the kind you'd write when things are going well in development:

from openai import OpenAI

client = OpenAI()  # picks up OPENAI_API_KEY from environment

def generate_support_reply(ticket_text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful customer support agent. Write a professional, empathetic reply to the customer's ticket."
            },
            {
                "role": "user",
                "content": ticket_text
            }
        ]
    )
    return response.choices[0].message.content

This works fine in isolation. But call it from 20 concurrent users, run it at 3am when the API has a hiccup, or accidentally pass in a ticket that's 50,000 words long, and you'll have problems. Let's harden it.

---

Catching the Right Exceptions

The first step is understanding what exceptions the OpenAI library actually raises. The modern openai Python library (version 1.x) has a clean exception hierarchy you can import directly.

from openai import (
    OpenAI,
    RateLimitError,
    APITimeoutError,
    APIConnectionError,
    APIStatusError,
    AuthenticationError,
    BadRequestError,
)

Here's what each one means in practice:

• RateLimitError — You hit the 429 limit. Safe to retry after waiting.
• APITimeoutError — The request timed out. Safe to retry.
• APIConnectionError — Network issue between your machine and the API. Safe to retry.
• APIStatusError — A catch-all for other HTTP errors. Check the status code before retrying.
• AuthenticationError — Bad API key. Do NOT retry; fix the key.
• BadRequestError — Your request was malformed (invalid parameter, context too long, etc.). Do NOT retry; fix the request.

Let's write a first version with basic exception handling:

from openai import (
    OpenAI,
    RateLimitError,
    APITimeoutError,
    APIConnectionError,
    AuthenticationError,
    BadRequestError,
    APIStatusError,
)

client = OpenAI()

def generate_support_reply(ticket_text: str) -> str:
    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": "You are a helpful customer support agent. Write a professional, empathetic reply to the customer's ticket."
                },
                {
                    "role": "user",
                    "content": ticket_text
                }
            ]
        )
        return response.choices[0].message.content

    except AuthenticationError:
        raise RuntimeError("Invalid API key. Check your OPENAI_API_KEY environment variable.")

    except BadRequestError as e:
        raise ValueError(f"Request was invalid: {e}. Check the ticket text and model parameters.")

    except RateLimitError:
        raise RuntimeError("Rate limit hit. The application needs retry logic.")

    except (APITimeoutError, APIConnectionError) as e:
        raise RuntimeError(f"Connection problem: {e}. The application needs retry logic.")

    except APIStatusError as e:
        raise RuntimeError(f"API returned status {e.status_code}: {e.message}")

This is already better — you're catching specific exceptions instead of letting Python crash with a raw traceback. But notice that for rate limits and timeouts, we're still just raising a RuntimeError. That's not good enough for production. We need to actually retry.

---

Retry Logic with Exponential Backoff

Retrying is the correct response to transient failures. But how you retry matters enormously. The worst thing you can do is retry immediately in a tight loop:

# DON'T DO THIS
while True:
    try:
        response = client.chat.completions.create(...)
        break
    except RateLimitError:
        pass  # try again instantly

This is called a "thundering herd" — if you and 100 other users all hit the rate limit simultaneously and all immediately retry, you just hammer the API even harder and make the problem worse.

The correct approach is exponential backoff: wait a bit, then retry. If that fails, wait twice as long. Then four times as long. Each successive retry doubles the wait, giving the server time to recover and spreading out the retry attempts over time.

There's a wonderful Python library called tenacity that handles all of this for you. Install it with pip install tenacity.

import time
import random
from openai import OpenAI, RateLimitError, APITimeoutError, APIConnectionError
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type,
    before_sleep_log,
)
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

client = OpenAI()

@retry(
    retry=retry_if_exception_type((RateLimitError, APITimeoutError, APIConnectionError)),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    stop=stop_after_attempt(5),
    before_sleep=before_sleep_log(logger, logging.WARNING),
)
def _call_openai_with_retry(messages: list) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        timeout=30,  # we'll explain this below
    )
    return response.choices[0].message.content


def generate_support_reply(ticket_text: str) -> str:
    messages = [
        {
            "role": "system",
            "content": "You are a helpful customer support agent. Write a professional, empathetic reply to the customer's ticket."
        },
        {
            "role": "user",
            "content": ticket_text
        }
    ]
    return _call_openai_with_retry(messages)

Let's unpack the @retry decorator parameters:

• retry=retry_if_exception_type(...) — Only retry on these specific exception types. Rate limits and connection issues are retryable; bad API keys and malformed requests are not.
• wait=wait_exponential(multiplier=1, min=2, max=60) — Start waiting 2 seconds, double each time, cap at 60 seconds. So the sequence is roughly: 2s, 4s, 8s, 16s, 32s, 60s.
• stop=stop_after_attempt(5) — Give up after 5 total attempts. Without this, tenacity will retry forever.
• before_sleep=before_sleep_log(...) — Log a warning before each retry so you can see what's happening in your logs.

Tip: Add a small random component to your wait times if you have many concurrent workers. This is called "jitter" and prevents all your workers from retrying at exactly the same moment. wait_exponential combined with wait_random(0, 1) using wait_exponential_jitter achieves this automatically.

---

Setting Timeouts Correctly

Notice the timeout=30 parameter in the API call above. This is critically important and often overlooked.

Without a timeout, your application will wait as long as the server takes. LLM APIs occasionally get very slow — especially when generating long completions or during high-traffic periods. A request that normally takes 3 seconds might take 90 seconds, or it might never respond at all. Every hanging request ties up a thread or async worker in your application.

The OpenAI Python client accepts a timeout parameter measured in seconds:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    timeout=30,  # give up after 30 seconds
)

You can also set a default timeout for all calls when you create the client:

client = OpenAI(timeout=30.0)

Choosing the right timeout value requires some judgment. A few guidelines:

• For short completions (a sentence or two), 15-20 seconds is usually generous.
• For longer generations (a full document, complex analysis), 60-90 seconds may be appropriate.
• For streaming responses, the timeout applies to connection establishment, not the full stream.

When a timeout fires, the OpenAI library raises APITimeoutError, which our retry logic above already handles. The request will be retried with the same timeout value.

Warning: If you're generating very long outputs and your timeout keeps firing, the solution is to increase the timeout — not to remove it entirely. A timeout of 120 seconds is still much better than no timeout at all.

---

Handling Rate Limits Proactively with Token Counting

Retry logic handles rate limits reactively — you hit the limit, then back off. But you can also be proactive by tracking how many tokens you're using and throttling yourself before you hit the limit.

The tiktoken library (from OpenAI) lets you count tokens before you send a request:

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    """Count the number of tokens in a string for a given model."""
    encoder = tiktoken.encoding_for_model(model)
    return len(encoder.encode(text))

# Check before sending
ticket_text = "I've been waiting 3 weeks for my order and nobody has responded to my emails..."
token_count = count_tokens(ticket_text)

if token_count > 100_000:
    raise ValueError(f"Ticket text is too long ({token_count} tokens). Please truncate before processing.")

print(f"Token count: {token_count}")  # Helps you understand your usage

Install tiktoken with pip install tiktoken.

For batch processing — say, you're processing 500 support tickets overnight — you can build a simple rate-limiting wrapper that tracks your per-minute token usage:

import time
from collections import deque

class TokenBucketRateLimiter:
    """Simple token-per-minute rate limiter using a sliding window."""
    
    def __init__(self, tokens_per_minute: int = 80_000):
        self.tokens_per_minute = tokens_per_minute
        self.usage_log = deque()  # stores (timestamp, token_count) pairs
    
    def wait_if_needed(self, estimated_tokens: int):
        """Block if sending estimated_tokens would exceed the rate limit."""
        now = time.time()
        window_start = now - 60  # 60-second sliding window
        
        # Remove entries older than 60 seconds
        while self.usage_log and self.usage_log[0][0] < window_start:
            self.usage_log.popleft()
        
        # Sum tokens used in the current window
        tokens_used = sum(tokens for _, tokens in self.usage_log)
        
        if tokens_used + estimated_tokens > self.tokens_per_minute:
            wait_time = 60 - (now - self.usage_log[0][0]) + 1
            logger.info(f"Rate limit approaching. Waiting {wait_time:.1f} seconds.")
            time.sleep(wait_time)
        
        self.usage_log.append((time.time(), estimated_tokens))


limiter = TokenBucketRateLimiter(tokens_per_minute=80_000)

def process_ticket_batch(tickets: list[str]) -> list[str]:
    results = []
    for ticket in tickets:
        estimated_tokens = count_tokens(ticket) + 200  # +200 for system prompt + response buffer
        limiter.wait_if_needed(estimated_tokens)
        reply = generate_support_reply(ticket)
        results.append(reply)
    return results

This is more sophisticated than it looks. By tracking usage over a rolling 60-second window, you smooth out your request rate and avoid the burst-then-429 pattern that naive batch code falls into.

---

Graceful Degradation: What to Return When Everything Fails

After all retries are exhausted, your code needs to decide what to do. "Crash with an exception" is rarely the right answer in a user-facing application. Instead, think about graceful degradation — what's the least-bad outcome for the user?

For a customer support reply generator, a few options make sense:

from tenacity import RetryError

DEFAULT_REPLY = (
    "Thank you for reaching out to us. We've received your message and a member of "
    "our support team will get back to you within 24 hours. We apologize for any "
    "inconvenience this may have caused."
)

def generate_support_reply(ticket_text: str) -> dict:
    """
    Returns a dict with 'reply', 'success', and 'error' keys.
    On failure, returns a safe fallback reply instead of crashing.
    """
    try:
        messages = [
            {
                "role": "system",
                "content": "You are a helpful customer support agent. Write a professional, empathetic reply to the customer's ticket."
            },
            {
                "role": "user",
                "content": ticket_text
            }
        ]
        reply = _call_openai_with_retry(messages)
        return {"reply": reply, "success": True, "error": None}

    except RetryError as e:
        # All retries exhausted
        logger.error(f"All retries failed for support ticket: {e}")
        return {
            "reply": DEFAULT_REPLY,
            "success": False,
            "error": "ai_unavailable"
        }

    except (AuthenticationError, ValueError) as e:
        # Configuration error — not retryable, needs human attention
        logger.critical(f"Non-retryable error: {e}")
        return {
            "reply": DEFAULT_REPLY,
            "success": False,
            "error": "configuration_error"
        }

By returning a structured dict instead of raising an exception, the calling code can log the failure, alert your on-call engineer, queue the ticket for manual review, and still show the user something useful — all without crashing.

Tip: Always log the full error with enough context to debug it later. Include the ticket ID, the error type, and the timestamp. "An error occurred" logs are useless at 2am when you're trying to understand what happened.

---

Hands-On Exercise

Here's a practice exercise that will solidify everything in this lesson.

Scenario: You're building a Python script that reads customer feedback from a CSV file and generates a one-sentence summary for each entry using an LLM. The CSV has two columns: feedback_id and feedback_text.

Your task: Build a process_feedback_file(filepath: str) function that:

1. Reads the CSV file using Python's built-in csv module
2. For each row, calls the LLM to generate a one-sentence summary
3. Handles rate limits with retry and exponential backoff (use tenacity)
4. Sets a 25-second timeout on each API call
5. If a single row fails after all retries, logs the error and stores "[Summary unavailable]" for that row instead of crashing
6. Returns a list of dicts: [{"feedback_id": "...", "summary": "...", "success": True/False}]

Starter code:

import csv
from openai import OpenAI, RateLimitError, APITimeoutError, APIConnectionError
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type, RetryError
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

client = OpenAI()

# Step 1: Write your @retry-decorated function here
# Step 2: Write the CSV reading and processing logic
# Step 3: Handle per-row failures gracefully

To test without burning API credits: Create a mock CSV file with 5 rows of fake customer feedback. You can verify your error handling logic by temporarily replacing the OpenAI call with raise RateLimitError(...) in your inner function.

---

Common Mistakes & Troubleshooting

Retrying on non-retryable errors. If you catch the generic Exception and retry everything, you'll waste time and credits retrying requests that will never succeed (like a bad API key or a context window overflow). Always check whether an error is retryable before retrying.

Setting the timeout too low. A 5-second timeout sounds reasonable until you realize GPT-4 generating a detailed analysis can legitimately take 15-20 seconds. Measure your actual p95 latency in production before choosing a timeout value.

Not logging retries. Without logging, a 4-retry sequence that eventually succeeds looks invisible. But that invisible sequence represents 90 seconds of latency for your user. The before_sleep_log in tenacity is your friend.

Silently swallowing errors. Returning None or "" on failure without logging anything is worse than crashing — at least a crash is visible. Always log failures, even when you gracefully degrade.

Forgetting about RetryError. Tenacity raises RetryError when all attempts are exhausted. If you don't catch RetryError outside your @retry-decorated function, it will bubble up uncaught. Always catch it at the boundary where you want to handle failure.

Using time.sleep() for rate limiting in async code. If you're using asyncio, time.sleep() blocks the entire event loop. Use asyncio.sleep() instead, and use tenacity's async retry support with AsyncRetrying.

---

Summary & Next Steps

You started with a fragile, one-liner API call. You now have the mental model and the tools to build LLM-powered Python applications that behave reliably in the real world.

Here's what you've built understanding of:

• Error taxonomy — knowing the difference between retryable transient errors (429, timeouts, connection issues) and non-retryable client errors (bad API key, malformed request) determines your entire error-handling strategy.
• Exponential backoff with tenacity — a declarative, readable way to add retry logic without manual sleep loops.
• Timeouts — every API call should have one. Set it at the client level for a default, and override per-call for long-running generations.
• Proactive rate limiting — counting tokens and tracking usage in a sliding window prevents you from hitting limits in the first place during batch processing.
• Graceful degradation — returning structured results with success flags instead of crashing gives calling code the information it needs to handle failures intelligently.

Where to go next:

• Streaming responses — if you're building chat interfaces, learn to handle streaming API responses, which have their own timeout and error patterns.
• Async LLM calls — when processing many items concurrently, move to asyncio and the async OpenAI client for much better throughput.
• Observability — look into structured logging and tools like Langfuse or LangSmith to trace LLM calls in production and spot failure patterns before they become outages.
• Circuit breakers — the next level beyond retry logic; if the API is clearly down, a circuit breaker stops all calls and fails fast instead of hammering a broken service.

The gap between an LLM demo and a production application is mostly made up of exactly this kind of unglamorous, essential reliability engineering. Now you know how to close that gap.