{"id":59669,"date":"2026-10-13T11:02:17","date_gmt":"2026-10-13T11:02:17","guid":{"rendered":"https:\/\/www.bravent.net\/?p=59669"},"modified":"2026-10-13T11:02:17","modified_gmt":"2026-10-13T11:02:17","slug":"metadata-driven-architecture-in-databricks-how-to-scale-the-medallion-model","status":"publish","type":"post","link":"https:\/\/www.bravent.net\/en\/news\/metadata-driven-architecture-in-databricks-how-to-scale-the-medallion-model","title":{"rendered":"Metadata-Driven Architecture in Databricks: How to Scale the Medallion Model"},"content":{"rendered":"

In a data project with dozens or hundreds of tables, manually writing every CREATE TABLE, transformation and data quality rule can quickly become a bottleneck for Data Engineering teams<\/strong>.<\/p>\n

This is where metadata-driven architectures<\/strong> can make a significant difference. Instead of developing and maintaining each table independently, the system’s behavior is defined through metadata and a generic engine is used to execute that definition.<\/p>\n

The result is a more automated, consistent and scalable data architecture<\/strong>, particularly useful when working with lakehouse platforms such as Databricks.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

The Challenge: Scaling a Medallion Architecture<\/strong><\/h2>\n

The Medallion Architecture<\/strong> is a common pattern for organizing data in a lakehouse through different processing layers. In its most common implementation, data moves from Bronze<\/strong>, where it is stored in its raw form, to Silver<\/strong>, where it is validated and transformed, and finally to Gold<\/strong>, where it is prepared for business and analytical consumption.<\/p>\n

Databricks recommends this approach to progressively improve data quality and structure as data moves through the different layers. You can learn more about the approach in Databricks’ documentation on Medallion Architecture<\/a>.<\/p>\n

However, the challenge becomes increasingly significant as the number of tables and data sources grows.<\/p>\n

    \n
  1. \n

    Repetitive development<\/strong><\/h3>\n<\/li>\n<\/ol>\n

    Every new table may require its own DDL, transformation logic, ingestion process and data quality rules. In many cases, these developments are based on existing scripts that are copied and adapted.<\/p>\n

      \n
    1. \n

      Lack of consistency<\/strong><\/h3>\n<\/li>\n<\/ol>\n

      Consistency across tables depends on every team member applying the same conventions, patterns and best practices<\/strong>.<\/p>\n

      The more people and tables involved in the project, the harder it becomes to guarantee that every implementation follows exactly the same criteria.<\/p>\n

        \n
      1. \n

        Changes are difficult to scale<\/strong><\/h3>\n<\/li>\n<\/ol>\n

        If a transformation pattern, data quality rule or layer structure changes, dozens or even hundreds of processes may need to be modified.<\/p>\n

        This increases maintenance effort and the risk of introducing errors.<\/p>\n

          \n
        1. \n

          Outdated documentation<\/strong><\/h3>\n<\/li>\n<\/ol>\n

          Documentation describing what data exists, where it comes from and why it has been designed in a particular way can quickly become outdated when it lives separately from the system itself.<\/p>\n

          This is where a metadata-driven architecture can make a significant difference.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

          The Approach: A Metamodel That Describes the System<\/strong><\/h2>\n

          A metadata-driven architecture<\/strong> separates two concepts that are often mixed together:<\/p>\n

            \n
          1. Definition:<\/strong> what tables exist, which fields they contain, where they come from and which rules they must follow.<\/li>\n
          2. Execution:<\/strong> how that definition is applied to the data engine.<\/li>\n<\/ol>\n

            The definition is stored in a data metamodel<\/strong>: a structure that describes the source, schema, transformations and data quality rules for each table or entity.<\/p>\n

            Ideally, this metamodel should have version control, validation and governance<\/strong>, just like any other critical component of a production data architecture.<\/p>\n

            A generic engine<\/strong> then interprets this metadata. This engine can run through Databricks notebooks and Python scripts and can automatically generate or execute different components of the data platform.<\/p>\n

            These may include:<\/p>\n

            Structure creation<\/strong><\/h3>\n

            The engine can generate the structures required for the different layers of the Bronze, Silver and Gold<\/strong> model, with additional Raw or Staging layers when required by the architecture.<\/p>\n

            Data ingestion<\/strong><\/h3>\n

            The definition can specify how data should be ingested from different sources, including relational and non-relational databases, files, APIs and other systems.<\/p>\n

            Data quality<\/strong><\/h3>\n

            Data quality rules can be defined declaratively and applied consistently across different datasets.<\/p>\n

            This approach fits particularly well with Databricks capabilities such as Lakeflow<\/strong>, where expectations can define data quality constraints and determine what happens when a record does not meet them, such as warning, dropping the record or failing the update. Databricks provides further information on how to manage data quality with pipeline expectations<\/a>.<\/p>\n

            Transformations and KPIs<\/strong><\/h3>\n

            The metamodel can also describe the transformations and calculations required to build analytical datasets and business KPIs<\/strong>, avoiding the need to replicate the same logic across multiple processes.<\/p>\n

            Orchestration<\/strong><\/h3>\n

            Finally, an orchestrator coordinates the execution of the different stages and their dependencies, automating processing according to the needs of the project.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

            The Main Advantage: Automation and Scalability<\/strong><\/h2>\n

            The most tangible benefit of a metadata-driven architecture becomes apparent when new data sources need to be added or existing schemas need to be modified<\/strong>.<\/p>\n

            Instead of developing, testing and maintaining specific code for every new table, the team can add or modify its definition in the metamodel.<\/p>\n

            The engine then applies the corresponding pattern.<\/p>\n

            This can help organizations:<\/p>\n