AI is accelerating. Data Engineering needs to keep pace.
Because scaling AI takes more than good models
There is a lot of discussion today about Artificial Intelligence, new models and the possibilities these technologies are creating.
But between experimenting with AI and creating value with it at scale, there is a less visible layer of work: making sure the data is ready to support what we want to build.
Even the most advanced model is of little value if the information it receives cannot be trusted. Duplicate data, inconsistent definitions across systems, unclear business rules or fragile integrations can compromise results before AI even comes into play.
The faster AI advances, the more important the engineering behind the data becomes.
What does a Data Engineer do in an increasingly AI-driven environment?
Much of our day-to-day work involves integrating different sources, developing pipelines, transforming data and ensuring that information is organized and ready to be used.
Working with technologies such as Microsoft Fabric and Power BI, we quickly realize that building a data solution involves much more than simply getting data from one place to another.
We need to understand where the data comes from, what it represents, how it has been transformed and which business rules should be applied.
Medallion architecture is a good example. Separating data into bronze, silver and gold layers allows us to organize it from a form close to its original source through to information that is ready for consumption and analysis.
But creating these layers is not enough. We need to define rules, ensure consistency and, above all, understand whether what we are building accurately represents the reality of the business.
A technically sound architecture is not enough if the data does not make sense to the people who need it to make decisions.
When AI provides an answer, we need to trust the data behind it
This becomes even more important when we talk about Artificial Intelligence.
Imagine a solution that allows users to ask questions about a company's sales or revenue. If Sales and Finance interpret the same metric differently, or if data is handled differently across systems, AI may produce an answer that sounds perfectly plausible and still fails to represent reality.
That is why preparing data for AI is not simply about making it available. It means ensuring that the data is reliable, consistent and understood within the context in which it will be used.
The quality of an answer starts long before the prompt. It starts with the data.
AI is also changing how we work
Naturally, this transformation is also changing the work of Data Engineers.
In my case, AI tools such as Claude are already part of my day-to-day work. I use them to develop code, analyse errors, explore solutions and discuss different technical approaches.
But what I find most interesting is not simply being able to write code faster. It is the ability to challenge an approach, explore alternatives and find different ways of solving a problem more quickly.
That does not mean automatically accepting what AI produces. Code can work and still fail to respect business rules, be the wrong solution for a particular problem or create issues when applied in a different context.
AI can accelerate execution. We still need to validate whether what we are building makes sense.
From a pilot that works to a solution that can scale
The difference becomes particularly important when we move from experimentation to real-world use.
In a pilot, we often work with a limited set of data, users and systems. Some exceptions can be handled manually, and certain limitations are acceptable while we are validating a hypothesis.
Scaling changes the problem. The solution needs to continue working as data volumes increase, new sources are introduced, more systems need to be integrated and more people begin to depend on the information it provides.
We need to ensure that pipelines remain reliable, transformations stay consistent, we can trace where the data comes from and a failure does not compromise the entire process.
A pilot proves that an idea can work. Scaling means ensuring that it can keep working.
This is also where Data Engineering takes on a new dimension. It is no longer simply about preparing data for an AI use case. It is about creating the conditions that allow that use case to grow without losing trust as it scales.
AI does not make Data Engineering less important. It raises the bar.
I believe AI will continue to transform the way we work as Data Engineers.
Some tasks will become faster. Others may be partially automated. And we will have new tools to help us develop, test and solve problems.
But that does not make Data Engineering less important. Quite the opposite.
The more organizations rely on AI to analyse information, support decisions or execute actions, the greater the need to ensure that the data behind those capabilities is reliable, consistent and ready to scale.
We can have increasingly capable models and increasingly advanced tools. But moving from a successful demonstration to a solution that creates sustainable value still depends on what sits behind it.
AI can accelerate innovation. Data Engineering helps ensure that innovation can scale.