In a data project with dozens or hundreds of tables, manually writing every CREATE TABLE, transformation and data quality rule can quickly become a bottleneck for Data Engineering teams.
This is where metadata-driven architectures can make a significant difference. Instead of developing and maintaining each table independently, the system’s behavior is defined through metadata and a generic engine is used to execute that definition.
The result is a more automated, consistent and scalable data architecture, particularly useful when working with lakehouse platforms such as Databricks.
The Challenge: Scaling a Medallion Architecture
The Medallion Architecture is a common pattern for organizing data in a lakehouse through different processing layers. In its most common implementation, data moves from Bronze, where it is stored in its raw form, to Silver, where it is validated and transformed, and finally to Gold, where it is prepared for business and analytical consumption.
Databricks recommends this approach to progressively improve data quality and structure as data moves through the different layers. You can learn more about the approach in Databricks’ documentation on Medallion Architecture.
However, the challenge becomes increasingly significant as the number of tables and data sources grows.
-
Repetitive development
Every new table may require its own DDL, transformation logic, ingestion process and data quality rules. In many cases, these developments are based on existing scripts that are copied and adapted.
-
Lack of consistency
Consistency across tables depends on every team member applying the same conventions, patterns and best practices.
The more people and tables involved in the project, the harder it becomes to guarantee that every implementation follows exactly the same criteria.
-
Changes are difficult to scale
If a transformation pattern, data quality rule or layer structure changes, dozens or even hundreds of processes may need to be modified.
This increases maintenance effort and the risk of introducing errors.
-
Outdated documentation
Documentation describing what data exists, where it comes from and why it has been designed in a particular way can quickly become outdated when it lives separately from the system itself.
This is where a metadata-driven architecture can make a significant difference.
The Approach: A Metamodel That Describes the System
A metadata-driven architecture separates two concepts that are often mixed together:
- Definition: what tables exist, which fields they contain, where they come from and which rules they must follow.
- Execution: how that definition is applied to the data engine.
The definition is stored in a data metamodel: a structure that describes the source, schema, transformations and data quality rules for each table or entity.
Ideally, this metamodel should have version control, validation and governance, just like any other critical component of a production data architecture.
A generic engine then interprets this metadata. This engine can run through Databricks notebooks and Python scripts and can automatically generate or execute different components of the data platform.
These may include:
Structure creation
The engine can generate the structures required for the different layers of the Bronze, Silver and Gold model, with additional Raw or Staging layers when required by the architecture.
Data ingestion
The definition can specify how data should be ingested from different sources, including relational and non-relational databases, files, APIs and other systems.
Data quality
Data quality rules can be defined declaratively and applied consistently across different datasets.
This approach fits particularly well with Databricks capabilities such as Lakeflow, where expectations can define data quality constraints and determine what happens when a record does not meet them, such as warning, dropping the record or failing the update. Databricks provides further information on how to manage data quality with pipeline expectations.
Transformations and KPIs
The metamodel can also describe the transformations and calculations required to build analytical datasets and business KPIs, avoiding the need to replicate the same logic across multiple processes.
Orchestration
Finally, an orchestrator coordinates the execution of the different stages and their dependencies, automating processing according to the needs of the project.
The Main Advantage: Automation and Scalability
The most tangible benefit of a metadata-driven architecture becomes apparent when new data sources need to be added or existing schemas need to be modified.
Instead of developing, testing and maintaining specific code for every new table, the team can add or modify its definition in the metamodel.
The engine then applies the corresponding pattern.
This can help organizations:
- Reduce repetitive development.
- Standardize transformations and data quality rules.
- Simplify the integration of new data sources.
- Reduce the maintenance effort associated with hundreds of tables.
- Apply changes more consistently.
- Improve the traceability of architectural decisions.
- Make the data architecture easier to evolve as data volumes grow.
However, automation also introduces a new critical point: the metamodel itself.
If an incorrect definition can automatically propagate to hundreds of tables, metadata governance becomes just as important as the code executing it.
7 Critical Considerations for a Metadata-Driven Architecture
Before implementing this type of architecture, several aspects should be carefully evaluated.
-
Metamodel governance
An unmanaged metamodel can become a critical point of failure for the entire system.
There should be version control, change validation and permission management, particularly when multiple teams can modify definitions.
The goal is to prevent an incorrect definition from automatically propagating throughout the environment.
-
Schema change management
It is essential to define what happens when the schema of a table or one of its fields changes.
Performing an ALTER on an existing structure is not the same as executing a CREATE OR REPLACE, particularly when considering existing data and historical records.
The engine must therefore clearly define which strategy should be applied to each type of schema change.
-
Reprocessing and idempotency
What happens if an execution fails halfway through?
Can the process be safely restarted, or is there a risk of duplicating data?
The architecture should address reprocessing, failures and idempotency from the beginning, particularly for pipelines processing large volumes of data.
-
Lineage and traceability
The metamodel should not be the only source of information about data lineage.
In Databricks, Unity Catalog provides governance capabilities and automatically captures data lineage, showing relationships between tables, queries, jobs, notebooks and downstream assets.
Unity Catalog’s data lineage capabilities can help teams understand where data comes from, how it is transformed and which assets could be affected by a change.
Unity Catalog can also extend lineage beyond Databricks workloads. External lineage can be used to connect external upstream sources and downstream tools, helping organizations build a more complete view of the data journey.
-
Active alerting
Defining data quality rules does not necessarily mean that someone will act when those rules are violated.
There is an important difference between logging a data quality failure in a table —which requires someone to actively query it— and triggering an alert that reaches the appropriate person or team.
A mature architecture should therefore define not only how quality issues are detected, but also how they are communicated and managed.
-
Automated testing
A generic framework is only as reliable as the tests supporting it.
Without a strong automated testing strategy, any change to the engine can have unexpected effects on the tables and processes that depend on it.
Tests should cover both the engine’s core logic and the different patterns and scenarios it can generate.
-
Evaluate Databricks native capabilities
Not everything needs to be built from scratch.
Before developing a fully custom orchestration and transformation engine, it is worth evaluating which requirements can be addressed through Databricks’ native capabilities.
Lakeflow Spark Declarative Pipelines, based on Apache Spark Declarative Pipelines, allows teams to define pipelines using SQL and Python while the platform manages aspects such as dependencies and pipeline orchestration. Databricks provides further information on Apache Spark Declarative Pipelines and its core concepts.
This does not mean that a custom metadata-driven architecture is no longer useful. On the contrary, in certain scenarios it can complement these capabilities by addressing specific requirements around generation, parameterization and governance.
The key is to avoid rebuilding functionality that the platform already provides natively.
Metadata-Driven Does Not Mean Governance-Free
A well-designed metadata-driven architecture can significantly reduce the effort required to onboard new data sources and maintain large numbers of tables.
But its main strength —the ability to automatically generate and apply the same logic at scale— is also its greatest risk.
That is why the metamodel should be treated with the same rigor as any other production component:
- Version control
- Change management
- Validation
- Permissions
- Testing
- Traceability
- Monitoring
The real question when evaluating a metadata-driven architecture is not simply:
Can it generate the tables automatically?
The question is:
What happens when the metamodel contains an error?
The answer determines whether you have a truly scalable data architecture or simply a system capable of automatically propagating mistakes.
Need to Scale Your Data Architecture in Databricks?
As a data platform grows, the challenge is no longer simply how to create new tables. It becomes how to maintain hundreds of processes in a consistent, governed and sustainable way.
At Bravent, we help organizations design, audit and evolve modern data architectures on Databricks and Microsoft Fabric, from Medallion architectures and pipeline automation to data governance, quality and lineage.
If your team is spending too much time maintaining pipelines, duplicating code or solving consistency and data quality issues, our experts can help you identify what can be automated, what needs to be governed and how to build a more scalable architecture around it.
Contact us at info@bravent.net and let our data experts help you turn a complex data platform into a more scalable, automated and governed architecture.
