The average data platform project starts with genuine ambition and ends with a refactoring budget nobody planned for. In our experience, the architectural decisions that cause the most damage aren’t the ones that look obviously wrong at the time. They’re the ones that look like reasonable shortcuts.

What we’ve learned from building and inheriting data platforms across financial services and manufacturing: the debt doesn’t accumulate gradually. It hits in waves, usually around months 14 to 20, when the second or third business unit wants access to data that was built for one team, in one context, without any of the structural assumptions that make sharing possible.

This is a practical guide for CDOs and Heads of Data Engineering who want to avoid that wave.

Three Takeaways

1. Before finalizing your data platform architecture, document the three business use cases most likely to require cross-domain data within 18 months and verify your architectural choices support them from day one, not as a retrofit.
2. Establish a Business Data Owner for each domain before the first data product goes live. Without this, your Data Catalog will be built by IT, defined without business input, and contested within six months.
3. If you’re running 20+ downstream consumers, implement a Views abstraction layer and an additive-only schema policy before the first breaking change request arrives, not after.

The Shortcuts That Become Crises

The two most common sources of architectural debt we’ve seen aren’t exotic. They’re the same two decisions, made under the same deadline pressure, on project after project.

Bypassing the integration layer. In one project integrating SAP data (products, suppliers, finance) into a cloud Data Lake alongside 40+ existing sources, the team skipped integration with the global dimension and lookup layer to cut initial delivery time by roughly 30%. For one department, this worked. The moment other business units requested access to the same data, the absence of alignment with global master data meant analysts had to build complex mapping tables from scratch. Two separate processing paths, two sets of inconsistency risks, and eventually a full-scale architectural refactoring project to unify what should have been unified at the start. The 30% time saving cost more than it saved.

Keeping the legacy orchestration tool. During a migration of a business unit into an independent cloud environment, the team retained the legacy corporate ETL tool to reduce deployment risk. A reasonable call in week one. Six months later, the licensing and infrastructure costs of that enterprise-grade tool were consuming a significant portion of the smaller organization’s IT budget. Its architecture was fundamentally incompatible with the agile Data Lake approach built around it, and release cycles stretched to multiple man-days per change. The migration to Apache Airflow that followed took months and could have been avoided entirely.

The pattern is the same: a local optimisation that ignores system-level consequences.

Warehouse or Lakehouse: How to Actually Decide

A lakehouse architecture is not a warehouse replacement. It’s an evolution, and whether it’s the right evolution depends entirely on the nature of your data and the shape of your use cases, not on what’s fashionable.

Before making this decision, two questions need honest answers. First: does your data come exclusively from relational ERP systems, or do you have (or plan to have) unstructured data, streaming data, or advanced analytics use cases? Second: what proportion of your reports will continue to rely on a classic multidimensional model?

If more than three-quarters of your reporting will stay in that classical model, building a completely new lakehouse may not be cost-effective. If you’re planning to use unstructured data, support Data Science teams, or run image processing at any point in the next two years, it’s worth building a lakehouse alongside the existing warehouse rather than trying to make the warehouse do something it wasn’t designed for.

The best solution is often hybrid. That’s not a hedge. It’s the realistic answer for most mid-size financial institutions.

Governance Decisions You Cannot Defer

In our experience, governance that gets deferred doesn’t get done. It gets retrofitted under pressure, which is more expensive and less effective than doing it before the first data product goes live.

Three decisions must be made early.

Data Ownership. Establish a Business Data Owner for each domain, someone with the primary competence to understand and use that data in business processes. Without this, you end up with the most common failure mode we see: a Data Catalog built six months after go-live, when nobody can agree on what „active customer” means, because the business wasn’t involved in defining it.

Data Catalog and Data Lineage. The catalog is not an IT deliverable. Business must play an active role at this stage. Misunderstandings between IT and business about what data means are not uncommon. They’re routine. Reliable lineage pays for itself the first time you need to trace a processing error upstream.

Security and Access Control. Role- and domain-based access control needs to be established before people start requesting it. The alternative is ticket-based access management, and in practice that means multiplying the waiting time for every access request, creating bottlenecks that slow data adoption across the organization.

Schema Evolution at Scale

Once you’re serving 20 or more downstream consumers, schema changes become the highest-risk routine operation on the platform. Four practices make this manageable.

Never give downstream consumers direct access to raw tables. Always implement a Views abstraction layer, which lets you refactor underlying storage or logic without breaking the interface your consumers depend on.

Prioritize additive changes. Add columns and tables; avoid modifying or deleting existing ones. Backward compatibility is immediate when you never take anything away.

Build a CI/CD pipeline with automated deployment tests and a dedicated pre-production environment. This is your primary defense against breaking changes reaching production.

Have a rollback strategy before you need one. Tools like Time Travel in Snowflake or Delta Lake snapshots allow near-instant data recovery. Where technology doesn’t provide this natively, enforce procedural guidelines that ensure every deployment can be safely reversed without data loss.

To understand why these four practices matter in combination, consider what breaks when even one is missing. In one project serving a financial reporting team alongside 15 analytical pipelines, a developer renamed a column in a source table to align with a new naming convention. No Views layer existed. Within hours, 8 downstream pipelines failed simultaneously. The fix took three days, not because the technical change was complex, but because identifying which pipelines were affected and in what sequence required manual investigation. A Views layer would have contained the change entirely. A CI/CD pipeline with pre-production testing would have caught it before deployment. Neither existed, because both had been deprioritized in the first six months to accelerate delivery.

What a Realistic Roadmap Looks Like

For a mid-size financial institution, the first six months should be focused on building a functional foundation, not a full solution.

The practical approach is to deliver an MVP centered on a maximum of three key business domains: typically the General Ledger (Finance), CRM (Customer), and Loans. The priority in this phase is establishing the core landing zone and automating ingestion from those sources. The goal is to move from manual data extraction to a structured environment where the first production-ready reports can reach business stakeholders. This is proof of concept and proof of trust simultaneously.

What „done” looks like at month six is specific: automated ingestion running without manual intervention, at least one domain with a defined Business Data Owner, and two or three reports that business stakeholders actually use. If you can’t demonstrate that by month six, the 18-month plan is at risk.

The 18-month horizon is where the platform transitions from a functional tool to a scalable enterprise solution. This is the critical window to actively engage the business in Data Ownership roles, to have them define the metadata and business rules for their domains. That engagement is the foundation of a fully operational Data Catalog. Without it, you’re building a technical structure that the business doesn’t understand and won’t maintain.

By month 18, the platform should be serving multiple business units, schema governance should be formalized, and the Data Catalog should reflect business definitions written by business owners, not reverse-engineered by IT. The gap between those two outcomes is almost always a people and process problem, not a technology one.

The Most Underestimated Cost

Technology is not the biggest cost in a data platform programme. Neither is process.

People are.

The primary driver of hidden cost is the absence of clear communication and a shared understanding of business definitions. When cross-functional teams operate without a unified language, the result is redundant work, misaligned data products, and extensive refactoring, all of which are expensive and none of which show up in the original project estimate.

The most effective tool we’ve seen for mitigating this cost is the Architecture Design Session, a structured process for aligning technical possibilities with business expectations before decisions are made that are difficult to reverse. In practice, this means bringing business stakeholders, data engineers, and architects into the same room before the first line of pipeline code is written, to agree on domain boundaries, ownership, and the definition of critical business terms. Organizations that run ADS sessions at the start of a programme typically spend 20 to 30 percent less time on rework in months 9 to 18. Those that skip it often find themselves running an unplanned ADS anyway, under deadline pressure, after something has already broken.

FAQ

What is the most common mistake organizations make in the first six months?

Optimizing for speed within a single business domain without accounting for how other teams will need to access the same data later. Decisions that feel efficient in month one, skipping the integration layer, deferring governance, keeping legacy tooling, compound into structural problems by month 14. The foundation built in the first six months determines whether the platform scales or gets refactored.

Should we modernize our existing data warehouse or build a lakehouse alongside it?

Start with two questions: what proportion of your reporting relies on a classic multidimensional model, and do you have plans for unstructured data or advanced analytics in the next two years? If more than three-quarters of your reports stay in the classical model, a full lakehouse migration may not be cost-effective. If Data Science use cases are on the roadmap, building a lakehouse alongside the existing warehouse is worth the investment.

When is the right time to set up a Data Catalog?

Before the first data product goes live. Business must be involved in defining the catalog from the start, not brought in after IT has already made the definitions. Organizations that build the catalog after go-live spend the next several months resolving conflicts between departments that have been using different definitions of the same terms for years.

What does schema governance look like when serving 20+ downstream consumers?

Four practices in combination: a Views abstraction layer, an additive-only modeling policy, a CI/CD pipeline with pre-production testing, and a documented rollback strategy. Any one of these in isolation reduces risk. All four together make schema evolution a manageable routine operation.

How do we budget accurately for a data platform programme?

Build in a people cost that most project estimates leave out. The primary driver of hidden expense is the absence of a shared understanding of business definitions across cross-functional teams. Running Architecture Design Sessions before technical decisions are made typically reduces rework in months 9 to 18 by 20 to 30 percent.