Beyond the App: A Software Engineer’s Guide to How Data Engineering Really Works

My first post at Matia was “What I Learned Moving from Software Engineering to Data Engineering”. Since then, I’ve had more conversations with software developers who are interested in data engineering but are concerned about how foreign the data world will feel. I took all the questions I was hearing and tried to create a conference talk that answered all of their questions.

I recently gave the talk at KCDC, one of the largest independent software conferences in the Midwest. You can watch the recording, check out the slides, or read my summary of the talk below. 

My first impression of data engineering was not great

The first big data environment I spent time around, 20 some odd years ago, was built heavily on SQL Server Integration Services, or SSIS. If you've never had the pleasure, SSIS is a visual programming environment where you drag boxes onto a canvas, connect them with arrows, dig through layers of properties, and eventually produce something that looks like the conspiracy wall from a detective show.

The weird thing is that I never thought the actual work was boring. Moving data between systems is interesting. Reconstructing the history of a business is interesting. Figuring out why two departments have different numbers for the same supposedly simple metric is very interesting. What bothered me was that the engineering practices around the work felt years behind the rest of software development.

I wanted the “DevOps” stuff. Put the pipeline in source control. Give me a diff. Let somebody review it. Run tests before the change reaches production. Give me an environment I can throw away and rebuild. If a deploy fails, make it obvious. If someone changes something, give me a history.

I wanted data engineering to feel like software engineering.

The funny part is that my wish list turned out to be a pretty good description of modern data engineering.

Not everywhere, obviously. Somewhere out there a 4,000-line stored procedure is still quietly holding together a Fortune 500 company, and everybody is afraid to touch it. But the tools and practices around data engineering have changed a lot, and the modern version looks much more familiar to a software engineer than the version I first encountered.

So if you're a software engineer who has spent years vaguely aware that “data engineering” exists but has never really understood what those people are doing all day, here's the mental model I wish I'd had earlier.

Orchestration: it’s not just cron

A data platform is usually made up of a lot of jobs that depend on other jobs. Maybe you need to pull yesterday's orders from a production database, load them into a warehouse, transform them, run some tests, publish the finished tables, and only then refresh the dashboards that depend on them.

The order of operations matters, otherwise you get unpredictable results. You definitely do not want the executive revenue dashboard refreshing from half-updated tables because one job finished while three upstream dependencies were still broken or hadn’t finished.

This is where orchestration comes in. One common term you’ll encounter is “DAG,” or directed acyclic graph, which is a wonderfully computer-science-y way of saying “a dependency graph where the arrows point forward.”

Cron can run something at 2:00 AM, scheduling was never the problem. The problem is that if run tests fails, publish should never start, which means the dashboard refresh doesn't start either. The failure stays contained instead of quietly contaminating everything downstream.

Tools like Airflow and Prefect are popular orchestration tools today. They understand dependencies, retries, failures, reruns, and all the annoying real-world things that happen around schedules. Modern orchestration tools also increasingly define this stuff in code. Your pipeline can live in a repository. You can review it, diff it, revert it, and use git blame to participate in one of software engineering's oldest traditions.

That was one of the first things that made modern data engineering click for me. The weird visual pipeline hidden on a server was disappearing. The dependency graph had become code.

First you have to get the data

Before you can transform or model anything, you have to get the data out of wherever it currently lives. This is ingestion, and the main challenge is that data lives basically everywhere, from internal databases, to third party API’s, to files and events, and everything in between.

An enterprise data platform usually has to deal with many, many of these at the same time, and eventually all of that data needs somewhere to land.

This is where you run into terms like data warehouse and data lakehouse. A warehouse is generally the more structured, SQL-first side of the world: Snowflake, BigQuery, Redshift, and similar systems. They're optimized for analytical queries that scan large amounts of data rather than application-style queries that fetch a few records by primary key.

A lakehouse starts from cheap object storage and layers table-like behavior on top of it. You can keep enormous quantities of raw or semi-structured data in formats like Parquet and still query it in useful ways. Databricks, Delta Lake, and Apache Iceberg all live somewhere in this neighborhood.

The boundary between warehouse and lakehouse has gotten pretty blurry, and unless you're building one of these systems yourself, you probably don't need to develop a strong tribal identity around either camp. Most companies have an analytical home for data that is separate from the transactional database your application is using.

The much more interesting change, at least to me, is the move from ETL to ELT.

Old-school ETL means Extract, Transform, Load. You pull data out of a source system, transform it while it's moving, and then store the finished result.

ELT changes the order: Extract, Load, Transform. You pull the source data out, land the raw version first, and do the transformation later inside the warehouse or lakehouse.

Changing the order of two letters doesn't sound revolutionary, but it represents a huge change in how these systems are designed. Storage got cheap enough that you don't need to immediately throw the source away, and compute got elastic enough that rebuilding large amounts of historical data is much less frightening than it used to be.

That means you can keep an immutable copy of what the source system actually said. If you discover six months later that your revenue logic was wrong, you fix the transformation and rerun history from the original input. You don't have to reverse-engineer what you think the source probably contained before an old pipeline destructively changed it.

Modeling is where this starts looking a lot like software

Raw data is useful, but raw data is shaped around the applications that created it, and businesses generally don't think in application schemas.

Your product database might have six tables spread across three services, internal IDs that mean nothing outside the application, status codes that made perfect sense to the team in 2021, and four different timestamps that are all named some variation of created_at.

Nobody in Finance wants to understand any of that. They want customers, orders, subscriptions, revenue, churn, inventory, and whatever other concepts the business actually uses to think about itself.

Data modeling is the layer where you turn source-shaped data into business-shaped data.

This is where a tool like dbt comes in, and dbt was probably the thing that made modern data engineering feel most familiar to me. A dbt model can be nothing more than a SQL file containing a SELECT statement:

-- models/marts/customers.sql
with orders as (
    select
        customer_id,
        count(*) as order_count,
        sum(amount) as lifetime_value
    from {{ ref('stg_orders') }}
    group by 1
)

select
    c.customer_id,
    c.email,
    coalesce(o.order_count, 0) as order_count,
    coalesce(o.lifetime_value, 0) as lifetime_value
from {{ ref('stg_customers') }} c
left join orders o using (customer_id)

The double braces are Jinja templating, but the more interesting piece is ref(). Instead of hard-coding the physical location of another table, the model declares that it depends on another model. dbt can use those references to work out build order, point the same code at different environments, and construct lineage between models.

The file lives in Git. You change it in a branch. Someone reviews it. It gets built and tested before merge. The final output might be a table or a view, but the thing you actually maintain is code.

Most dbt projects also develop a layered structure. Staging models do the boring cleanup: rename columns, normalize types, standardize timestamps. Intermediate models handle joins, deduplication, and reusable business logic. Marts expose finished concepts that people and applications actually want to consume.

That sounds pretty straightforward until somebody asks a seemingly innocent question like, “What's our revenue?”

Then you discover that “revenue” is not really a column, it’s a definition.

Does revenue mean paid orders, shipped orders, or completed orders? Gross revenue, or net of refunds? Before tax or after tax? Do discounts count before or after the number you're reporting? If you operate internationally, which exchange rate applies? Which timezone determines whether something happened yesterday or today?

You can write several perfectly reasonable SQL queries and get several different numbers. And none of them are wrong. They are just using different definitions of “revenue.”

This is the part of data engineering I underestimated the most. I assumed most of the complexity would be plumbing, but a lot of it is semantic. Somebody has to decide what the company means by “revenue,” turn that definition into code, and make sure everybody else builds from the same definition.

If Product and Finance are using two different definitions, there isn't an infrastructure purchase on earth that's going to make those numbers agree.

Yes, you can test data

Testing was probably the thing I wanted most when I was working around older data systems. I hated the idea that validation could amount to “run it and look at the result for a while.”

Modern data tooling is much better about this.

One of the useful concepts is a data contract, which is an explicit promise about the shape and meaning of a dataset. Software engineers already live with this idea. OpenAPI schemas, protobuf definitions, typed interfaces, and JSON Schema are all ways of saying, “If you depend on this thing, here's what you can assume about it.”

A data contract applies that same thinking to a table. Maybe order_id must always be present and unique. Maybe status is only allowed to contain paid, refunded, or canceled. Maybe every customer_id in the orders table has to correspond to an actual customer.

With dbt, those sorts of assertions can live directly beside the model:

models:
  - name: customers
    columns:
      - name: customer_id
        tests: [not_null, unique]
  - name: orders
    columns:
      - name: status
        tests:
          - accepted_values:
              values: [paid, refunded, canceled]

You can also go beyond simple assertions and write tests with SQL for business rules. The convention is delightfully backwards: write a query that returns the rows that violate the rule. Zero rows means the test passes. If rows come back, those are the failures.

So you can write a test that says modeled daily revenue must stay within one percent of the raw payment total. You can test cross-table relationships, tolerances, reconciliations, or pretty much anything else SQL can express.

Then you put the whole thing in CI.

That feels so normal from a software engineering perspective that it's almost boring, which is exactly why I like it. It should be boring. “We automatically test changes before deploying them” is not supposed to be an exotic idea.

But data can be wrong while everything is green

Tests help a lot, but data systems have a failure mode that takes a little while to get used to if most of your background is in application development. Software tends to fail in ways we're good at noticing. A process dies, latency spikes, requests start returning 500s, a queue backs up, or PagerDuty finds a creative new way to ruin dinner.

Data often fails without any of that.

Maybe yesterday's row count dropped by 90 percent because an upstream export was incomplete. Maybe a source column suddenly became null. Maybe a webhook was delivered twice and duplicated thousands of records. Maybe a vendor simply didn’t update their database.

The computation can be completely correct given the input it received. The problem is that the input itself is bad or incomplete.

You can't solve that entirely with unit tests because the thing you're processing changes every day. Instead, data platforms also monitor the behavior of the data. This is where data observability comes in.

The most common signals are things like freshness (did today's data arrive?), volume (is the amount of data roughly what we expect?), schema changes (did a column disappear or a type change?), and distribution (did the null rate on an important field suddenly jump from almost nothing to 40 percent?).

This is one place where application engineering instincts need a slight adjustment. A successful application operation is usually decent evidence that the operation worked. In a data pipeline, successful execution mostly proves that the computation completed. It doesn't prove the result is sensible.

Once you have enough data, nobody knows where anything is

There's another problem that shows up as a data platform gets larger: eventually nobody knows what tables exist, and what fields came from what sources.

This is what a data catalog is for. A catalog is essentially a searchable index of the datasets in the organization. You search for “revenue” or “customer” and see what's available, who owns it, when it was last updated, what the columns mean, whether it contains sensitive data, and what downstream systems are using it.

Closely related to cataloging is lineage, which is probably my favorite concept in this whole stack because it makes the invisible dependencies around data visible.

Lineage is the family tree of a number.

Suppose an executive dashboard says yesterday's revenue was $1.4 million. With good lineage, you should be able to trace that value backward through the model that produced it, through intermediate and staging models, all the way back to the application database and payment API that supplied the original records.

That's obviously useful when you're debugging something. If the dashboard looks wrong, walk upstream until you find where reality diverged from expectations.

But lineage is just as useful in the other direction. Suppose you're about to change stg_payments. Before you break the downstream executive report, you can ask what depends on it and ensure that the data is migrated fully.

When shared data breaks, figuring out what failed is only half the incident. You also need to know who consumed the bad result.

Put it together and you get something that looks suspiciously familiar

At this point we have version control, automated tests, CI/CD, reproducible environments, dependency management, monitoring, metadata, and safer deployment practices.

We've seen this movie before.

The broad term you'll hear for applying these ideas to data is DataOps, and the DevOps comparison is pretty obvious. Like DevOps, it's more useful as a set of practices than as a rigid job title. Not every data engineer owns every part of the system, just like plenty of excellent application engineers would prefer never to learn what a Helm chart is.

The cultural move is the same, though. Take a fragile manual process and make it reproducible. Put important artifacts in source control. Automate the boring parts. Make failures visible. Shorten the distance between introducing a problem and finding out that you introduced it.

Once I started looking at data engineering this way, the field became much less mysterious. It wasn't an alien branch of computing with its own incomprehensible rules. It was engineering applied to a different kind of system, where the thing you're managing is not just code or infrastructure but the historical record the business uses to understand itself.

The stakes are higher because data isn't just feeding reports anymore

It's easy to think of data engineering as a back-office reporting function because, for a long time, that's how a lot of companies treated it. Run jobs overnight, refresh dashboards, produce the monthly report, make sure Finance gets the spreadsheet.

Those things still matter, but data pipelines increasingly feed actual product behavior: pricing, recommendations, fraud detection, inventory systems, experimentation platforms, personalization, and machine learning.

If those pipelines break, the product also breaks. AI makes this even more obvious, hallucinating answers and giving demonstrably false answers.

For decades, a human was usually somewhere near the end of an analytical pipeline. Humans are pretty good at noticing when a result is obviously ridiculous. If the dashboard says the company made fourteen dollars last quarter, somebody will probably ask a question.

A model is less likely to complain.

A training pipeline will happily ingest bad data, optimize against it, and hand you a worse model. The problem might be subtle enough that you spend days tuning prompts, changing model parameters, or blaming the algorithm when the real bug happened three transformations upstream.

This is because AI products are data products.

This is why you need a DataOps platform

I work at Matia, which you can probably tell from where this post is published, and a lot of the questions above are a big part of why. ETL, observability, catalog, and lineage are often separate tools with separate views of the same data. You spend far more time than you might expect connecting separate products, ensuring that everything is communicating and has the right level of visibility. Is the lineage platform connected to everything? Can the observability platform inspect all your data sources and destinations? It’s a lot of overhead.

Matia brings those pieces into one platform built around shared metadata, giving you a single place to handle all of your DataOps needs. A unified platform means an ingestion issue, an observability alert, the upstream source, and the downstream models or dashboards are already connected. At Matia, we don’t want to replace your data architecture; we want to help you move your data from the sources to where you need it, and give you all the operational details around your data to ensure it’s accurate and timely.

None of the ideas in this post depend on using Matia. You can build a great data platform from separate tools. I just find the unified approach compelling because all of these systems are ultimately trying to answer the same questions: where did this data come from, is it healthy, and what depends on it? If you’re interested in learning more, click here to schedule a short demo!

The wish list wasn't a fantasy

For a long time, I assumed data engineering was inherently messy because the version I had seen was messy: no source control, manual deployments, giant stored procedures, visual pipelines nobody could meaningfully diff, and testing that involved staring at the output until everyone felt brave enough to ship.

What I found is that none of those things were fundamental properties of data engineering. They were engineering problems the industry hadn't solved yet, or at least hadn't solved in a way that was widely adopted.

Today, if I walk into an unfamiliar data platform, I don't care that much about the logos on the architecture diagram. I care about the shape of the system.

How does work get orchestrated? Where does the source data come from, and do we preserve the original? How do raw source schemas become shared business concepts? What gets tested before a change ships? How do we detect bad data when every job still reports success? Can I take an important number and trace it backward to its sources and forward to the systems that depend on it?

If you can answer those questions, you already understand a lot of what matters.

Application engineering makes data true in the moment. Data engineering takes those events and makes them useful over time. It preserves them, combines them, gives them meaning, and moves them into the places where people and systems can actually use them.

Every number on a dashboard has a supply chain. So does every metric in a quarterly report, every feature in a machine learning model, and every AI system whose behavior depends on the data we fed it.

Your application writes a row somewhere. That's not the end of the story. It's where the next system starts.