The Batch, Streaming, and Orchestration Mental Model
Every data pipeline, whatever tool builds it, is answering the same three questions.
How much data does one run of this pipeline process - a bounded chunk, or an unbounded, ongoing stream?
What decides when and in what order each step of this pipeline runs?
And what happens the second time a step runs on the same input, whether because of a retry, a backfill, or a redelivered message?
Data Engineering Basics shows the concrete API for building batch and streaming jobs; this page is about the three concepts underneath every one of them - batch versus streaming as a trade-off, the DAG as a dependency graph rather than a schedule, and idempotency as the property that makes failure recovery safe.
Summary
- A data pipeline is defined by how it bounds its input (batch vs. streaming), how it sequences dependent work (a DAG of tasks), and how safely it can be re-run (idempotency) - every orchestration tool is really just automating these three concerns.
- Insight: Pipelines fail - jobs crash, workers die, messages get redelivered - and a pipeline that isn't idempotent turns every failure into a data-quality incident instead of a routine retry.
- Key Concepts: batch vs. streaming, DAG (directed acyclic graph), orchestration vs. scheduling, idempotency, watermark, delivery guarantee.
- When to Use: Batch for bounded, periodic reporting and backfills; streaming for low-latency, event-driven reactions; a DAG-based orchestrator whenever tasks have real dependencies between them, not just a time to run.
- Limitations/Trade-offs: Streaming buys latency at the cost of coordination complexity (watermarks, state, ordering); batch buys simplicity and completeness at the cost of staleness; idempotency has to be designed into every sink, it isn't a setting you enable.
- Related Topics: workflow orchestration (Airflow, Prefect, Dagster), stream processing (Kafka, Faust), distributed compute (PySpark), transformation layers (dbt).
Foundations
A batch pipeline processes a bounded, known-size set of data - "yesterday's orders," "this month's log files" - and finishes; running it again later processes the next bounded chunk.
A streaming pipeline processes an unbounded sequence of events as they arrive, with no natural "end" to wait for, which means it has to produce useful output continuously rather than after a full pass over the data.
This is a genuine trade-off, not just two implementations of the same idea: batch can look at all the data before deciding an answer (a full day's total, a complete join), while streaming has to answer with only what's arrived so far, accepting that late-arriving data may need to be reconciled afterward.
A DAG (directed acyclic graph) is the second core idea: it's a graph of tasks and dependencies - "task B needs task A's output" - with no cycles, and critically, a DAG defines what depends on what, not when things run.
A common misreading is to think of a DAG as a fixed sequence, like a numbered list of steps; in reality, a scheduler is free to run any two tasks with no dependency between them in parallel, or in either order, as long as every task's declared dependencies finish first.
Orchestration is the layer that turns a DAG definition into actual execution: it decides which tasks are ready to run (their dependencies are satisfied), retries failed tasks, and tracks each run's state - this is distinct from plain scheduling, which just decides when a job starts, with no notion of dependency between jobs.
Idempotency is the property that running an operation twice with the same input produces the same result as running it once - and it is the property that makes every retry, backfill, and redelivered message safe rather than dangerous.
Mechanics & Interactions
The batch/streaming choice and the idempotency requirement are more connected than they first appear: any streaming system that promises reliable delivery under failure has to redeliver messages sometimes, and only idempotent processing makes that redelivery harmless instead of duplicating data.
Streaming systems mostly offer at-least-once delivery by default - a message might be processed more than once after a crash and restart, because acknowledging "I processed this" and "commit the result" cannot be made perfectly atomic across two different systems (the message broker and the output sink) without extra coordination.
Exactly-once processing, where it's claimed, is usually achieved by combining at-least-once delivery with an idempotent sink - for example, writing results keyed by a message ID so a duplicate write simply overwrites the same row instead of adding a second one - not by some lower-level guarantee that duplicates never occur.
A watermark is how streaming systems reason about "done enough": since a stream never truly ends, a watermark is a heuristic claim that all data up to some point in event-time has probably arrived, letting windowed aggregations (like "events per 5-minute window") finalize a result while accepting that a small fraction of very late data may be missed or handled separately.
Batch orchestrators face a parallel version of the same idempotency problem: if a nightly job crashes halfway through and is retried, appending its partial output a second time would double-count rows - the standard fix is to make each run's output keyed by a stable partition (a date, a batch ID) and have the job overwrite that partition rather than append blindly.
DAG-based orchestrators (Airflow, Prefect, Dagster) also encode how much history a task run "belongs to" via a data interval - the logical time range a run represents - which is what lets a backfill re-run last month's DAG runs and get last month's data back, rather than accidentally processing today's data under yesterday's label.
# a DAG expresses dependency, not execution order:
# extract_orders and extract_customers have no dependency on each other,
# so an orchestrator is free to run them concurrently
extract_orders >> transform_orders
extract_customers >> transform_orders # transform waits on BOTH upstream tasks
transform_orders >> load_warehouse # load only starts once transform finishesAdvanced Considerations & Applications
At scale, the batch-versus-streaming choice stops being binary and becomes a spectrum: micro-batch systems (structured streaming in PySpark, for instance) process small, frequent bounded chunks, trading some of streaming's latency for batch's simpler bounded-data reasoning.
Choosing between Airflow, Prefect, and Dagster is less about which DAG engine is "better" and more about which unit of abstraction matches the team's mental model - Airflow centers on tasks and operators, Prefect centers on plain Python functions with light annotation, and Dagster centers on data assets (the outputs a pipeline produces) rather than the tasks that produce them.
Data validation belongs as a first-class node in the DAG, not an afterthought: a pipeline that loads bad data on schedule is often worse than one that fails loudly, because downstream consumers (dashboards, ML features, other pipelines) may trust the output without knowing it's wrong.
Observability for pipelines has to answer questions batch monitoring and streaming monitoring ask differently: batch asks "did this run finish, and how did its row counts compare to history," while streaming asks "how far behind is this consumer" (consumer lag) and "how many messages are stuck in a dead-letter queue."
File format choice interacts with all of this: columnar formats like Parquet support partition pruning and predicate pushdown that make idempotent partition-overwrite patterns cheap, while row-oriented formats make selective reprocessing of just one partition far more expensive.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
| Batch | Simple reasoning, can see all data before answering, easy backfills | Latency measured in minutes-to-hours; stale by definition between runs | Scheduled reporting, historical backfills, non-time-critical aggregation |
| Micro-batch | Bridges batch simplicity with near-real-time delivery | Still adds a scheduling/trigger latency floor; not truly event-driven | Near-real-time dashboards where seconds-to-minutes latency is acceptable |
| Streaming | Lowest latency, reacts to individual events | Requires watermarks/windowing, harder to reason about "done," state management overhead | Fraud detection, real-time alerting, event-driven architectures |
| DAG orchestration (Airflow/Prefect/Dagster) | Explicit dependency graph, retries, backfills, scheduling in one system | Adds an operational system to run and monitor; overkill for a single linear script | Multi-step pipelines with real task dependencies across sources |
Common Misconceptions
- "A DAG is just a fancy word for a sequence of steps." - A DAG only encodes dependencies; tasks with no dependency between them can run in parallel or in any order, which is exactly what lets orchestrators speed up pipelines without changing the DAG definition.
- "Exactly-once delivery means duplicates literally cannot happen." - In distributed systems, "exactly-once" almost always means at-least-once delivery paired with an idempotent sink that absorbs duplicates, not a guarantee that a message is ever sent or processed only a single time.
- "Streaming is strictly better than batch because it's faster." - Streaming trades away the ability to look at all the data before answering, forces watermark/late-data trade-offs, and adds real operational complexity that batch simply doesn't have to deal with.
- "Idempotency is a library feature you turn on." - Idempotency has to be designed into how a task writes its output (keyed overwrites, upserts by natural key, partition replacement) - no orchestrator or framework can make an inherently append-only, non-keyed write idempotent for you.
- "A backfill is just running the pipeline again for old dates." - A correct backfill relies on the DAG's data interval concept so each historical run is scoped to the right logical time range; running "today's" pipeline code against old data without that scoping easily mislabels or duplicates results.
FAQs
What's the actual difference between batch and streaming, beyond "batch is slower"?
- Batch processes a bounded, known-size input and can see all of it before producing an answer.
- Streaming processes an unbounded sequence and must produce useful output continuously, without ever seeing "all" the data.
- This changes what kinds of answers are even possible - a batch total is exact, a streaming total up to now is provisional until late data is reconciled.
Does a DAG define the order tasks run in?
Not directly - a DAG defines dependencies between tasks, and an orchestrator is free to run any two tasks with no dependency between them concurrently or in either order, as long as every task's declared upstream dependencies have finished first.
What's the difference between scheduling and orchestration?
Scheduling decides when a job starts (a cron-like trigger), while orchestration additionally tracks dependencies between tasks, retries failures, and manages the state of a whole DAG's execution - a scheduler with no dependency model is not an orchestrator.
Why does idempotency matter so much for pipeline reliability?
Failures are routine in distributed pipelines - workers crash, networks partition, messages get redelivered - and idempotency is what turns "this task ran twice" from a data-corrupting event into a harmless no-op, which is what makes automatic retries and backfills safe to run without manual cleanup.
How is "exactly-once" processing actually achieved in practice?
- Delivery itself is almost always at-least-once: a message can be redelivered after a crash.
- The illusion of "exactly-once" comes from writing to an idempotent sink - for example, an upsert keyed by message ID.
- A duplicate delivery then just overwrites the same record instead of creating a second one.
What is a watermark in streaming, and why is it necessary?
A watermark is a heuristic claim that all events up to a certain point in event-time have probably arrived, which lets a windowed aggregation (like a 5-minute count) finalize and emit a result instead of waiting forever for data that might theoretically still arrive late.
How does a batch job stay idempotent across retries?
By writing output scoped to a stable partition key - a date, a run ID, a data interval - and overwriting that partition rather than appending to it, so a retried run replaces its own prior (possibly partial) output instead of duplicating rows next to it.
What's the difference between Airflow, Prefect, and Dagster at a mental-model level?
Airflow centers its model on tasks and operators connected in a DAG; Prefect centers on ordinary Python functions with light decoration for dependency tracking; Dagster centers on the data assets a pipeline produces rather than the tasks that produce them - all three still orchestrate DAGs of dependent work underneath.
When does micro-batch make more sense than either pure batch or pure streaming?
Micro-batch fits when near-real-time output matters (seconds-to-minutes, not hours) but the team wants to keep reasoning about small bounded chunks rather than building full streaming infrastructure with watermarks and long-lived stateful consumers.
Why do dashboards or downstream pipelines need to know if a data-validation step failed, not just if the load succeeded?
A pipeline that "succeeds" while loading invalid, incomplete, or duplicated data is often more damaging than one that fails loudly, because downstream consumers have no signal to distrust the output - which is why data validation is treated as a DAG node with its own pass/fail state, not a side check.
What does "consumer lag" mean in a streaming context, and why watch it?
Consumer lag is the gap between the latest message produced to a stream and the latest message a given consumer has processed; a growing lag means the consumer can't keep up with incoming volume, which is an early warning sign before a pipeline visibly falls behind or drops data.
Is a single Python script that calls functions in sequence "a DAG"?
Only in the trivial sense that a strict sequence is technically a (linear) dependency graph - the practical value of an actual DAG orchestrator shows up once there are genuinely independent branches that could run in parallel, or tasks that need retries, backfills, and dependency-aware scheduling that a plain script has no framework for.
How does file format choice (Parquet vs. CSV/JSON) interact with idempotent reprocessing?
Columnar formats like Parquet support partition pruning, so overwriting or reprocessing just one partition (one date, one key range) touches only the relevant files; row-oriented formats like CSV usually require rewriting or scanning a whole file even to fix one partition, making idempotent partial reprocessing much more expensive.
Related
- Data Engineering Basics - the concrete batch/streaming/pipeline API this page's model sits underneath.
- Airflow - DAGs, data intervals, and backfills in practice.
- Prefect and Dagster - alternative orchestration abstractions (functions and data assets).
- Streaming (Kafka / Faust) - delivery guarantees, offsets, and windowed aggregation in practice.
- Workflow Reliability - the idempotency, retry, and observability decision workflow referenced above.
- PySpark - distributed batch and micro-batch (structured streaming) execution.
Stack versions: This page was written for Python 3.14 (stable) / 3.13 (maintenance); the concepts (batch/streaming, DAGs, idempotency) are tool-agnostic and not tied to a specific orchestrator or stream-processing library version.