Orchestrating Matia with Prefect

Matia handles ingestion and reverse ETL well on its own. dbt handles the transformation layer in between. The interesting engineering problem starts when those systems have to behave like one pipeline: ingestion must finish before tests run, tests must pass before transformations are built, and reverse ETL must not fire until the transformed data is ready.

At that point, scheduling each tool independently is not enough. Something has to own the dependencies, wait states, retries, and failure path across the entire sequence. In this example, that something is Prefect.

I built a small reference pipeline to experiment with how you might orchestrate Matia with Prefect. The pipeline looks something like this:

1. Trigger a Matia ingestion integration through the API.
2. Resolve the Matia integration-run ID and attach it to the current Prefect flow run as a tag.
3. Suspend the Prefect flow while the ingestion job runs.
4. Resume the exact waiting flow when a correlation-bearing completion callback arrives.
5. Run dbt test, run dbt run, and then trigger a Matia reverse ETL integration.

Why scheduling is not enough

The dependency between ingestion and dbt is simple to describe: do not transform data until the load has actually finished. It is also exactly the kind of dependency that becomes awkward when each system owns its own schedule. But it’s not just about scheduling transformation: you might need to ensure several ingestion jobs are completed before the data can be utilized, or you might have workflows to trigger in other connected services as soon as the data is available.

You can estimate how long ingestion usually takes and delay follow up tasks accordingly, but that is still a guess. A fast load leaves the pipeline idle for no reason. A slow load allows those other tasks to start against incomplete data. Polling Matia from a worker is more accurate, but it keeps execution infrastructure occupied, produces repetitive status checks, and forces you to choose a polling interval that balances latency against API traffic. Instead, Matia supports triggering a webhook after a job has run, allowing for a near-instant notification that the pipeline is complete.

An orchestrator gives the sequence one place to live. Prefect can trigger the external job, preserve the state of the workflow while it waits, continue with dbt only after the gate is cleared and the webhook call is received, and show the trigger, tests, transformations, and downstream submission as one connected run.

Correlating each Matia run to the correct Prefect flow

Suppose flow A triggers Matia run A, then flow B triggers Matia run B. When a completion webhook arrives, “resume the most recent paused run” cannot reliably determine which workflow owns that event. The receiver could resume the wrong flow without raising an obvious error, allowing the pipeline to continue against the wrong ingestion run.

The fix is deterministic correlation. Once the ingestion job has been triggered and its Matia integration-run ID is available, the Prefect flow adds that ID to its own tags as matia-run:<id> before suspending. Matia then includes the same integration-run ID when it calls the webhook endpoint.

The receiver uses that ID to construct the corresponding Prefect tag and search for the exact flow run waiting for it. If one matching paused flow exists, Prefect resumes it. If no flow matches, or more than one flow somehow has the same tag, the receiver returns an explicit error rather than guessing. Duplicate webhook deliveries are also safe: if the matching flow is no longer paused, the receiver simply reports that no action is needed.

That gives the pipeline a clean one-to-one relationship:

Start with the orchestration graph

The top-level flow is intentionally small. It shows the dependency graph without burying it under the client, tagging, or receiver details:

@flow(name="matia-ingestion-pipeline", persist_result=True, log_prints=True)
def matia_ingestion_pipeline(
    ingestion_job_id: str = "demo-ingestion-job",
    reverse_etl_job_id: str = "demo-reverse-etl-job",
    dbt_project_dir: str = os.getenv("DBT_PROJECT_DIR", "./dbt_project"),
) -> None:
    matia_run_id = trigger_matia_ingestion_job(ingestion_job_id)

    tag_current_flow_run_with_matia_run(matia_run_id)
    print(f"Tagged this flow run with matia-run:{matia_run_id}")

    print("Suspending until Matia posts to the webhook receiver...")
    suspend_flow_run(timeout=3600)

    run_dbt_test(dbt_project_dir)
    run_dbt_run(dbt_project_dir)

    trigger_matia_reverse_etl_job(reverse_etl_job_id)

The flow waits for the ingestion side because dbt depends on the landed data. After the flow resumes, it runs the data tests, builds the transformation, and submits the reverse ETL integration.

Attach correlation after Matia starts the run

The Matia run ID is not known when the Prefect flow is created. It only becomes available after the integration has been triggered. That does not prevent the flow from using it for correlation because Prefect tags can be updated while the flow run exists.

This example around this helper uses prefect.runtime.flow_run.id to identify the currently executing flow run. That avoids passing the Prefect flow-run ID through every function call. It also keeps the helper straightforward to test because Prefect's runtime value can be supplied through PREFECT__RUNTIME__FLOW_RUN__ID in a test environment.

The result is a searchable association in Prefect's state store:

matia-run:<matia_run_id>

Once that tag exists, a receiver with the same Matia run ID can locate the exact flow that is waiting for it.

The flow uses suspend_flow_run() rather than a sleep loop. Suspension allows the running process to exit while Prefect retains the workflow state needed to continue later. The flow can remain suspended for minutes or hours without keeping the original worker process alive.

The body of the webhook call from Matia contains both a status and the integration-run ID. If the status comes back as anything other than “success,” we can fail the suspended flow; and if it is a successful run, we can resume the suspended flow using the matia-run:<matia_run_id> tag.

Validate the data before building on it

Once ingestion is complete, the next question should not be, “Can we start transforming this data?” It should be, “Did we receive data we can trust?” A successful load confirms that records moved between systems, but it does not guarantee that the dataset is complete, unique, current, or structurally valid. This pipeline therefore validates the newly landed data before running transformations or triggering reverse ETL.

That matters because many data-quality problems do not cause ingestion itself to fail. Duplicate records, missing identifiers, unexpected nulls, or incomplete loads can all reach the destination successfully. If transformation begins immediately, joins, aggregations, and default values may hide or amplify those problems while still producing models that appear valid.

Testing the landed data first creates a clear quality gate. When a fundamental expectation fails, the pipeline stops while the issue is still close to its source, making it easier to diagnose and preventing compute from being spent on models that should not be built.

It also protects operational systems downstream. Reverse ETL may send customer attributes, account classifications, or campaign audiences back into business applications. Catching bad data before transformation is inconvenient; catching it after it has triggered workflows or changed customer-facing behavior is an incident. Validation keeps that failure contained before the data can spread.

Why put Matia inside the Prefect graph at all?

Matia natively supports scheduling integrations on an interval or cron schedule. This works for a lot of pipelines, where data is aggregated an analyzed at a later time, and the next steps aren’t expected to run in near-real-time. For more complicated needs, orchestration provides several benefits:

Dependencies are expressed as code. dbt does not run because a timer suggests ingestion is probably finished. It runs after the flow's completion gate has been cleared. Reverse ETL is submitted only after both dbt steps succeed.

Retry policies live with the workflow. The API calls can retry transient client or network failures, while dbt validation can fail immediately because a broken data contract is not normally fixed by repeating the same test. Those choices are visible alongside the rest of the pipeline.

The ingestion-to-transformation failure path is connected. If ingestion has completed but dbt test fails, the Prefect run shows the failed task and its logs rather than leaving the transformation failure disconnected from the job that supplied the data.

Concluding the experiment

Matia’s API and webhook capabilities make its integrations composable parts of a larger data platform rather than isolated scheduled jobs. Teams can place validation, transformation, approvals, or any other required logic between ingestion and reverse ETL while retaining the speed and reliability of Matia’s managed data movement. The orchestration layer may change from team to team, but the core pattern remains the same: move the data with Matia, verify it before building on it, and only push it downstream when the full pipeline has succeeded.

‍