Orchestration Limitations

What Orchestration Does

Modern orchestration tools excel at "run Task B after Task A completes." They manage dependencies, retries, scheduling. But they can't answer "should we run Task B given the current state of the data?" This is the key distinction between orchestration and decisioning.

# Orchestration defines the sequence
@dag
def customer_pipeline():
    raw = extract_customers()
    clean = clean_data(raw)
    enrich = add_segments(clean)
    load = write_to_warehouse(enrich)

This runs every time on schedule. It doesn't check: Is this data still valid? Has the segmentation logic changed? Should test customers be excluded today?

What's Missing

Decision logic that validates whether a task should execute, not just when. Questions like:

Example Failure

Your pipeline classifies data quality for incoming records. The business changes the quality scoring rules to require additional validation checks. The orchestrator keeps running the old classification logic because it doesn't know the rule changed. The pipeline completes successfully but produces incorrect quality scores. The underlying issue is that business logic is embedded in SQL rather than being queryable context.