Data Pipeline Design Patterns: Fan-Out, Fan-In, and Branching Workflows Explained

Introduction

Imagine you work at an e-commerce company. Every night, a batch of the day's transaction data lands in your data warehouse. From that single source, your analytics team needs a sales summary report, your finance team needs a revenue reconciliation file, your marketing team wants customer segmentation data, and your fraud detection system needs to flag suspicious transactions. All of that comes from one stream of raw data — and it all needs to happen in a coordinated way.

Most data engineers, when they start building pipelines, think in straight lines: data comes in, gets transformed, goes out. That works fine when you have one data source and one destination. Real-world data engineering rarely works that way. Data flows in from multiple sources, needs to be processed in parallel, gets combined with other data, and splits off into different directions based on conditions. The patterns that govern how those flows are structured are called pipeline design patterns, and the three most important ones for any working data engineer are fan-out, fan-in, and branching.

By the end of this lesson, you'll understand these three patterns well enough to recognize when to use each one, implement them in Python using realistic examples, and avoid the design mistakes that trip up even experienced engineers.

What you'll learn:

• What fan-out, fan-in, and branching patterns are and the problems they solve
• How to implement each pattern in Python with realistic data processing scenarios
• How to combine patterns to build more sophisticated workflows
• The tradeoffs and failure modes each pattern introduces
• How orchestration tools like Airflow represent these patterns as Directed Acyclic Graphs (DAGs)

---

Prerequisites

You should be comfortable with:

• Basic Python (functions, lists, dictionaries)
• A general understanding of what a data pipeline does (read data, transform it, load it somewhere)
• Familiarity with the concept of a function that takes input and returns output

No prior data engineering experience is required.

---

What Is a Pipeline Design Pattern?

Before we dive into the specific patterns, let's establish what we mean by a design pattern in the context of data pipelines.

A design pattern is a reusable solution to a commonly recurring problem. In software engineering broadly, patterns like "factory" or "observer" describe how to structure code to solve known categories of problems. In data engineering, pipeline design patterns describe how to structure the flow of data between processing steps.

Think of a data pipeline as a series of tasks (also called nodes or steps), where each task takes data as input, does something to it, and produces output. The pattern describes how those tasks connect to each other.

A Directed Acyclic Graph, or DAG, is the formal structure used to represent these connections. "Directed" means data flows in one direction. "Acyclic" means there are no loops — data doesn't flow in circles. Tools like Apache Airflow, Prefect, and dbt all use DAGs to represent pipelines, and the patterns we're about to learn are really just recurring shapes that appear in those graphs.

---

The Fan-Out Pattern

The Core Idea

Fan-out means one task produces output that feeds into multiple downstream tasks simultaneously. The name comes from the visual shape: one node at the top, multiple arrows spreading outward like a hand-held fan.

This is exactly the scenario from our introduction. You have one source — the day's transaction data — and from that single source, multiple independent processes need to run.

Here's a simple diagram described in words: imagine a single box labeled "Load Raw Transactions" with arrows pointing to four separate boxes: "Generate Sales Report," "Build Finance Reconciliation," "Segment Customers," and "Flag Fraud Candidates." Those four tasks don't depend on each other — they only depend on the first task finishing.

Why Fan-Out Matters

The key word there is independent. Fan-out is valuable when downstream tasks can run in parallel without needing each other's results. Instead of processing the sales report, waiting for it to finish, then processing the finance file, waiting for that, and so on — all four tasks can run at the same time. On a dataset with millions of rows, this can reduce your pipeline runtime from an hour to fifteen minutes.

Implementing Fan-Out in Python

Let's build a concrete example. We'll simulate loading transaction data and then fanning it out to three different processing functions.

import concurrent.futures
import pandas as pd
from datetime import datetime

def load_transactions(filepath: str) -> pd.DataFrame:
    """Simulates loading a day's transaction data."""
    # In production, this might read from S3, a database, or an API
    data = {
        'transaction_id': [1001, 1002, 1003, 1004, 1005],
        'customer_id': ['C001', 'C002', 'C001', 'C003', 'C002'],
        'amount': [250.00, 89.99, 4500.00, 30.00, 15000.00],
        'category': ['electronics', 'clothing', 'electronics', 'food', 'jewelry'],
        'timestamp': pd.to_datetime(['2024-01-15 09:00', '2024-01-15 10:30',
                                     '2024-01-15 11:00', '2024-01-15 13:00',
                                     '2024-01-15 14:45'])
    }
    return pd.DataFrame(data)

def generate_sales_report(transactions: pd.DataFrame) -> dict:
    """Summarizes sales by category."""
    summary = transactions.groupby('category')['amount'].agg(['sum', 'count'])
    return summary.rename(columns={'sum': 'total_revenue', 'count': 'num_transactions'}).to_dict()

def build_finance_reconciliation(transactions: pd.DataFrame) -> dict:
    """Calculates totals needed for finance team."""
    return {
        'total_transactions': len(transactions),
        'gross_revenue': transactions['amount'].sum(),
        'avg_transaction_value': transactions['amount'].mean(),
        'report_date': datetime.today().strftime('%Y-%m-%d')
    }

def flag_fraud_candidates(transactions: pd.DataFrame) -> pd.DataFrame:
    """Flags transactions over $5,000 for fraud review."""
    HIGH_VALUE_THRESHOLD = 5000.00
    flagged = transactions[transactions['amount'] > HIGH_VALUE_THRESHOLD].copy()
    flagged['flag_reason'] = 'high_value_transaction'
    return flagged

# --- Fan-Out Execution ---

def run_fan_out_pipeline(filepath: str):
    # Step 1: Single upstream task
    print("Loading transactions...")
    transactions = load_transactions(filepath)
    print(f"Loaded {len(transactions)} transactions.\n")

    # Step 2: Fan-out — run all three downstream tasks in parallel
    downstream_tasks = {
        'sales_report': generate_sales_report,
        'finance_reconciliation': build_finance_reconciliation,
        'fraud_flags': flag_fraud_candidates,
    }

    results = {}
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        futures = {
            task_name: executor.submit(task_fn, transactions)
            for task_name, task_fn in downstream_tasks.items()
        }
        for task_name, future in futures.items():
            results[task_name] = future.result()
            print(f"✓ {task_name} completed")

    return results

results = run_fan_out_pipeline("transactions.csv")

Notice a few things here. We use concurrent.futures.ThreadPoolExecutor to actually run those three functions at the same time rather than one after another. The transactions DataFrame is passed to each function independently — each function gets its own view of the data and doesn't interfere with the others. If generate_sales_report crashes, it doesn't mean flag_fraud_candidates can't run.

Important: In this example we're sharing a Pandas DataFrame across threads. Pandas DataFrames are not thread-safe for writes, but since each downstream function only reads the data and creates its own copy, this is safe. If your downstream tasks mutated the shared DataFrame, you'd want to pass transactions.copy() to each one.

---

The Fan-In Pattern

The Core Idea

Fan-in is the mirror image of fan-out. Multiple upstream tasks produce outputs that converge into a single downstream task. Instead of one spreading into many, many combine into one.

A practical example: your data team collects customer data from three different sources — a CRM system, a web analytics platform, and a loyalty program database. Each of those sources needs to be extracted and cleaned separately. Once they're all clean, a single task joins them together into a unified customer profile. That join task is your fan-in point.

Why Fan-In Introduces Complexity

Fan-in requires careful coordination. The downstream task can't start until all upstream tasks have finished. If your CRM extraction takes 10 minutes but your web analytics extraction takes 45 minutes, your join task has to wait 45 minutes regardless. This makes fan-in a natural point for bottleneck analysis in your pipeline.

Fan-in also introduces a question of failure handling: what happens if one upstream task fails? Does the entire pipeline stop, or do you proceed with partial data? This is an architectural decision, not a technical one, and different business contexts call for different answers.

Implementing Fan-In in Python

import pandas as pd
from typing import Optional

# --- Three independent upstream extraction tasks ---

def extract_crm_customers() -> pd.DataFrame:
    """Pulls customer records from the CRM system."""
    return pd.DataFrame({
        'customer_id': ['C001', 'C002', 'C003', 'C004'],
        'email': ['alice@example.com', 'bob@example.com',
                  'carol@example.com', 'dave@example.com'],
        'signup_date': pd.to_datetime(['2022-01-10', '2022-03-15',
                                       '2023-06-01', '2021-11-20']),
        'lifetime_value': [3200.00, 450.00, 890.00, 7800.00]
    })

def extract_web_analytics() -> pd.DataFrame:
    """Pulls session data from the analytics platform."""
    return pd.DataFrame({
        'customer_id': ['C001', 'C002', 'C003', 'C005'],  # Note: C005 is new
        'sessions_last_30_days': [12, 3, 8, 1],
        'last_page_viewed': ['checkout', 'homepage', 'product', 'about'],
        'avg_session_duration_min': [4.2, 1.1, 3.8, 0.5]
    })

def extract_loyalty_program() -> pd.DataFrame:
    """Pulls loyalty point balances from the rewards system."""
    return pd.DataFrame({
        'customer_id': ['C001', 'C003', 'C004'],  # C002 not enrolled
        'loyalty_tier': ['Gold', 'Silver', 'Platinum'],
        'points_balance': [1500, 400, 9200]
    })

# --- Single downstream fan-in task ---

def build_unified_customer_profiles(
    crm_data: pd.DataFrame,
    web_data: pd.DataFrame,
    loyalty_data: pd.DataFrame
) -> pd.DataFrame:
    """
    Fan-in task: merges three upstream datasets into a single
    unified customer profile. Uses left joins so CRM is the
    source of truth — customers must exist in CRM to appear.
    """
    profile = crm_data.merge(web_data, on='customer_id', how='left')
    profile = profile.merge(loyalty_data, on='customer_id', how='left')

    # Fill in sensible defaults for customers missing from some sources
    profile['sessions_last_30_days'] = profile['sessions_last_30_days'].fillna(0)
    profile['loyalty_tier'] = profile['loyalty_tier'].fillna('Not Enrolled')
    profile['points_balance'] = profile['points_balance'].fillna(0)

    return profile

# --- Fan-In Pipeline ---

def run_fan_in_pipeline():
    import concurrent.futures

    print("Running upstream extractions in parallel...")
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        crm_future = executor.submit(extract_crm_customers)
        web_future = executor.submit(extract_web_analytics)
        loyalty_future = executor.submit(extract_loyalty_program)

        # All three must complete before we proceed
        crm_data = crm_future.result()
        web_data = web_future.result()
        loyalty_data = loyalty_future.result()

    print("All upstream extractions complete. Building unified profiles...")
    unified_profiles = build_unified_customer_profiles(crm_data, web_data, loyalty_data)

    print(f"\nUnified customer profiles ({len(unified_profiles)} records):")
    print(unified_profiles.to_string(index=False))
    return unified_profiles

unified = run_fan_in_pipeline()

The fan-in task build_unified_customer_profiles takes all three DataFrames as explicit parameters. This makes the dependency clear: you can't call this function until you have all three results. The pipeline structure enforces the coordination.

Tip: Notice we use a left join with crm_data as the base. This is a deliberate business logic decision — we only want customers who exist in the CRM. A full outer join would include C005 from web analytics even though we have no CRM record for them. Always ask "what is the source of truth?" before writing your fan-in merge logic.

---

The Branching Pattern

The Core Idea

Branching means the path your data takes through the pipeline depends on a condition evaluated at runtime. Unlike fan-out (where every downstream task always runs), branching routes data to different tasks based on logic.

Think about processing incoming files from a retail partner. Some files contain sales data, some contain inventory updates, and some contain returns. Each type needs a completely different transformation. You check the file type at the start and route accordingly — that's branching.

Branching is also called conditional routing in some frameworks.

Branching vs. Fan-Out: The Key Distinction

This is a common point of confusion. In fan-out, all downstream tasks run — they all get the data. In branching, only one downstream path runs based on a condition. They look superficially similar (one task pointing to multiple downstream tasks), but the execution behavior is fundamentally different.

Implementing Branching in Python

import pandas as pd
from enum import Enum
from typing import Callable

class FileType(Enum):
    SALES = "sales"
    INVENTORY = "inventory"
    RETURNS = "returns"
    UNKNOWN = "unknown"

def detect_file_type(filepath: str) -> FileType:
    """
    Inspects the incoming file to determine its type.
    In production, this might check a filename convention,
    a header row, or a manifest file.
    """
    if "sales" in filepath.lower():
        return FileType.SALES
    elif "inventory" in filepath.lower():
        return FileType.INVENTORY
    elif "returns" in filepath.lower():
        return FileType.RETURNS
    else:
        return FileType.UNKNOWN

def process_sales_file(filepath: str) -> pd.DataFrame:
    """Handles sales-specific transformations."""
    print(f"  Processing as SALES file: {filepath}")
    # Simulate reading and transforming sales data
    return pd.DataFrame({
        'type': ['sale', 'sale'],
        'sku': ['SKU-001', 'SKU-002'],
        'units_sold': [10, 25],
        'revenue': [199.90, 749.75]
    })

def process_inventory_file(filepath: str) -> pd.DataFrame:
    """Handles inventory-specific transformations."""
    print(f"  Processing as INVENTORY file: {filepath}")
    return pd.DataFrame({
        'type': ['inventory', 'inventory'],
        'sku': ['SKU-001', 'SKU-003'],
        'quantity_on_hand': [140, 55],
        'reorder_point': [50, 20]
    })

def process_returns_file(filepath: str) -> pd.DataFrame:
    """Handles returns-specific transformations."""
    print(f"  Processing as RETURNS file: {filepath}")
    return pd.DataFrame({
        'type': ['return', 'return'],
        'sku': ['SKU-002', 'SKU-001'],
        'units_returned': [2, 1],
        'return_reason': ['defective', 'wrong_size']
    })

def handle_unknown_file(filepath: str) -> None:
    """Raises an alert for files we don't recognize."""
    print(f"  WARNING: Unrecognized file format: {filepath}")
    # In production, this would send a Slack alert or page on-call
    raise ValueError(f"Cannot process file with unknown format: {filepath}")

# The branch router: this is the heart of the branching pattern
BRANCH_MAP: dict[FileType, Callable] = {
    FileType.SALES: process_sales_file,
    FileType.INVENTORY: process_inventory_file,
    FileType.RETURNS: process_returns_file,
    FileType.UNKNOWN: handle_unknown_file,
}

def run_branching_pipeline(filepath: str):
    print(f"\nReceived file: {filepath}")

    # Step 1: Evaluate the condition
    file_type = detect_file_type(filepath)
    print(f"Detected type: {file_type.value}")

    # Step 2: Route to the correct branch
    handler = BRANCH_MAP.get(file_type, handle_unknown_file)
    result = handler(filepath)

    return result

# Simulate processing different incoming files
test_files = [
    "partner_data/2024-01-15_sales_report.csv",
    "partner_data/2024-01-15_inventory_update.csv",
    "partner_data/2024-01-15_returns_log.csv",
]

for filepath in test_files:
    result = run_branching_pipeline(filepath)
    if result is not None:
        print(f"  Result preview: {len(result)} rows processed\n")

Notice how the BRANCH_MAP dictionary keeps the routing logic clean and extensible. If the retail partner adds a new file type next quarter — say, promotional pricing — you add a new handler function and one entry to the dictionary. You don't touch the routing logic itself. This is sometimes called the open/closed principle: open for extension, closed for modification.

---

Combining Patterns: Building a Real-World Pipeline

In practice, these patterns don't appear in isolation. A real pipeline might fan-in multiple sources, then branch based on data quality, then fan-out to multiple destinations.

Here's how a combined pipeline for our e-commerce scenario might look described as a flow:

1. Fan-In: Collect transaction data from three regional databases (North America, Europe, Asia-Pacific) and merge them into one global dataset.
2. Branch: Evaluate data quality. If the error rate is under 1%, proceed to processing. If it's over 1%, route to a data quality remediation workflow and halt the main pipeline.
3. Fan-Out: From the cleaned global dataset, simultaneously produce the sales report, the finance reconciliation, and the fraud detection feed.

Each of those stages maps directly to a pattern you now understand. The skill of pipeline design is recognizing which pattern fits which stage.

---

How Orchestration Tools Represent These Patterns

Tools like Apache Airflow let you define these patterns as explicit DAGs in Python. Each task becomes an @task decorated function, and you connect them with >> operators.

A fan-out in Airflow looks like this conceptually:

# Airflow DAG pseudocode — illustrates pattern structure
load_task >> [sales_task, finance_task, fraud_task]

A fan-in:

[crm_task, web_task, loyalty_task] >> build_profiles_task

A branch:

detect_type_task >> BranchPythonOperator(task_id='router', ...) 
>> [sales_branch, inventory_branch, returns_branch]

The visual DAG graph in the Airflow UI literally shows the shapes described by these patterns. When you look at a pipeline and see arrows spreading outward from one node, you're looking at a fan-out. When you see arrows converging to one node, that's fan-in. When you see a decision diamond with multiple paths where only one will execute, that's branching.

---

Hands-On Exercise

Build a pipeline that combines all three patterns. Here's the scenario:

You receive order data from two different warehouse systems (Warehouse A and Warehouse B) each night. Your task is to:

1. Fan-In: Extract data from both warehouses and merge them into a combined orders dataset.
2. Branch: Check if the total order count is greater than 1,000. If yes, process in "high volume" mode (which just prints a different message for now). If no, process in "standard" mode.
3. Fan-Out: From the merged dataset, simultaneously produce:
   • A summary of orders by product category
   • A list of orders with a total value over $500 (high-value orders)
   • A count of orders per warehouse for operational tracking

Start by creating two small mock DataFrames to represent each warehouse's data. Each should have columns: order_id, product_category, total_value, warehouse.

Try writing each function independently, then connect them with the fan-in, branch, and fan-out structure shown in the examples above.

---

Common Mistakes & Troubleshooting

Mistake 1: Treating fan-out and branching as the same thing. The error usually shows up when someone writes a fan-out but wraps each downstream task in an if condition. If only some downstream tasks should run, you have a branch. If all of them always run, you have a fan-out. Check your execution model before you write any code.

Mistake 2: Not handling partial failure in fan-in. If three upstream tasks run in parallel and one fails, what happens? With concurrent.futures, calling .result() on a failed future raises the exception immediately and your other results may be lost. Wrap individual future.result() calls in try/except blocks and decide explicitly what "partial upstream failure" means for your business case.

Mistake 3: Mutating shared data in fan-out tasks. If your fan-out passes the same object to multiple threads and any of them writes to it, you'll get unpredictable results. Always have downstream tasks treat their input as read-only, or pass dataframe.copy() to each task explicitly.

Mistake 4: Hard-coding branch conditions. Putting your routing logic inside a chain of if/elif statements works, but it becomes fragile fast. Prefer a dispatch dictionary (like the BRANCH_MAP example above) that maps conditions to handlers. Adding a new branch then requires zero changes to existing logic.

Mistake 5: Ignoring the bottleneck introduced by fan-in. Fan-in forces your pipeline to wait for the slowest upstream task. Profile each upstream task individually before assuming parallelism gives you full benefits. If one task takes ten times longer than the others, optimizing the fast tasks gives you almost nothing — fix the slow one first.

---

Summary & Next Steps

Let's bring it together. You now understand three fundamental patterns that appear in virtually every real-world data pipeline:

• Fan-out distributes one upstream result to multiple downstream tasks that run independently and in parallel. Use it when you have one data source and multiple consumers.
• Fan-in collects results from multiple upstream tasks into a single downstream task. Use it when you're consolidating data from multiple sources. Watch out for partial failures and the bottleneck of the slowest upstream task.
• Branching routes data to different downstream paths based on a runtime condition. Use it when the correct processing logic depends on something you can only know when the data arrives.

These patterns combine to form the backbone of sophisticated pipelines. Once you can name what you're looking at, you can reason about it, discuss it with colleagues, and debug it systematically.

Where to go next:

• Error handling and retry strategies: What happens when a single task in a fan-out fails? How do you retry only the failed branch?
• Idempotency in pipelines: How do you design tasks so that running them twice doesn't corrupt your data?
• Airflow DAGs in depth: Build your first real orchestrated pipeline using the patterns you learned here.
• Backfill and reprocessing strategies: How do you rerun parts of a pipeline without rerunning the whole thing?

Understanding these patterns is the foundation. Everything else in data engineering — orchestration, monitoring, testing, optimization — is built on top of knowing how data flows.