Recently I have had the impression that a new data architecture is forming. My instinct says that is wrong, because we already have more architectures than most organisations know what to do with, and adding another one to the pile is nobody’s idea of progress.

But the signals keep arriving:

• Machine-generated data has stopped being a side channel. IDC’s Global DataSphere has the enterprise half growing at more than twice the rate of the consumer half, and their latest forecast attributes the shift to generative AI and what they carefully call new forms of high-value data.
• Archives have become a live workload. The reason most enterprise retrieval underperforms is not chunking or embeddings, it is that the knowledge base is typically twelve to eighteen months deep and drawn from a handful of active systems, while everything older sits in a format nothing can reach. That data was archived on the assumption nobody would ever need it at speed. Somebody now does.
• And retrieval itself has turned into an access pattern nothing was designed to serve. A production retrieval deployment commonly runs seven distinct stores: source archive, parsed chunks, embeddings, vector index, keyword index and the rest, each tuned to a different access pattern, with the whole thing sitting on the hot path of every inference call. Enterprise intent to adopt hybrid retrieval tripled in a single quarter in early 2026 as the first generation of these systems hit a wall. I do not see how any of that gets absorbed by what we already have.

I do not see how that gets absorbed by what we already have. So either my instinct is wrong, or something is about to give.

Data Architecture: A definition

I went looking for a working definition and found three, held by three groups of people who mostly do not read each other’s work.

• Database researchers: A data architecture is a set of structural decisions: shared-nothing or shared-disk, rows or columns, whether the compute moves to the data or the other way round. Stonebraker’s 1986 paper arguing for shared-nothing is the genre in one document. These decisions matter, but most of them happen inside a product you have already bought, and nobody re-plans a budget because their engine switched to vectorised execution.
• Enterprise Architect: it is a governance discipline. TOGAF treats data as one of four architecture domains. DAMA-DMBOK makes it one of eleven knowledge areas sitting alongside quality, metadata and master data. Both are useful and neither has changed much in twenty years. Nobody has ever cancelled a platform because their Zachman framework became intolerable.
• Practitioners: The third definition is the one practitioners use, and it is the only one where anything actually moves. It is the level where somebody says “we are going to the lakehouse” and the consequence is a procurement cycle, a migration plan, six new tools, a hiring round, and a year of arguments. Warehouse, lake, lakehouse, mesh. These are the words that appear in board papers.

This post will focus on the altitude of practitioners. The other two are perfectly valid, they just do not produce the thing I am trying to explain. When the industry moved from warehouse to lakehouse, thousands of teams felt it in their budgets and their job descriptions. When a vendor swapped a row store for a column store underneath them, faster aggregations but business as usual.

The history of data architectures

Every architecture below fixed something real and created something new to complain about. The complaints are usually where the next architecture came from, though the order is messier than it looks from here.

1970 – Relational

Before relational, application code knew where the data physically sat and how to walk to it. Change the storage layout and you changed the programs. Codd’s paper proposed the opposite arrangement: describe the answer you want, let the database work out how to get it.

(Which, now I write it down, is roughly what we are all doing to language models. Describe the answer you want and let something statistical work out the path. We have been here before.)

Then nothing happened for fifteen years.

The maths was settled in 1970 and enterprises were still buying navigational databases in 1980. Three separate things were in the way, and they got fixed at different times by different people.

What was blocking it	What fixed it	Roughly when
No optimizer could beat a human hand-writing the access path, so giving up control of the route cost you performance you could not spare	System R proved cost-based optimization could compete	Mid 1970s
No standard interface, so committing to relational meant committing to one vendor’s dialect	SQL standardisation, which made it something a purchasing department could sign off	Early 1980s
Machine time cost more than the human time an optimizer would save	The ratio slowly inverting as machines got cheaper and skilled people got dearer	Through the 1970s

That third row is the one people skip. An optimizer spends machine cycles so a human does not have to think about access paths, and in 1970 that was the wrong way round. A mainframe cost $4.6 million for 12.5 MIPS, and Jim Gray’s rule of thumb from the period was one data administrator per gigabyte when a gigabyte cost about a million dollars. Machines were the expensive resource and people were the cheap one, and relational asked you to spend the expensive one.

So cheaper machines were necessary and nowhere near sufficient. Two of the three blockers had nothing to do with price, which is the first time I noticed the thing this whole post turned out to be about: cheap opens the door, easy decides when anyone walks through it. What it cost us was an engine complicated enough to be opaque, which is why database tuning is still a specialism fifty years later.

1990’s – Data Warehousing

Relational systems were good at recording what the business did and bad at explaining it. Run a year-long analytical scan against the database taking orders and you find out quickly that the two jobs do not share well. It is auditing the books while the shop is open.

So we made a second copy, modelled for questions rather than transactions and loaded overnight, kept deliberately separate. Kimball’s lifecycle method turned that into something a team could deliver repeatedly, and by the mid-1990s a warehouse with its own staff and its own budget line was standard equipment in large companies.

It cost us twice over, and the two costs went off in completely different directions.

1. The first was money: warehouses ran on proprietary hardware with proprietary licences, which put a hard ceiling on how much data was worth keeping at all, and somebody in every organisation was deciding what to throw away.
2. The second was the queue, because every question the business wanted answered now went through one team, and that team became the constraint on how fast anyone could learn anything. The money problem got attacked in 2006. The queue takes until 2019, and it is still not fixed.

2006 to 2012 – Hadoop, NoSQL and Data Lakes

Three things arrived close together, got told as one movement, and came from genuinely different places.

Hadoop went at the cost ceiling. Enterprise SAN storage ran around $20,000 per usable terabyte while commodity SATA drives were about $110. Cheap disks had been sitting there for years though, and nobody had built anything with them, because a rack of cheap disks is a rack of things that break.

What was blocking it	What fixed it	Roughly when
Cheap machines fail constantly, and nothing made a pile of them behave like one reliable system	Google’s GFS and MapReduce papers gave away the abstraction that handles failure for you	2003 to 2004
Even with the design published, building it in-house was out of reach for anyone who was not Google	Yahoo funding an open-source implementation and running it at scale	2006 onwards
Proprietary warehouse licensing made storing everything uneconomic regardless of hardware	Open source removed the per-terabyte licence from the equation entirely	2006 onwards

Disk had been getting cheaper every year for three decades without producing Hadoop, so cheap disk cannot be the trigger on its own. And the resource everybody assumes Hadoop was exploiting was not even available: the MapReduce paper explicitly calls network bandwidth scarce and schedules work on whichever machine already holds the data precisely to avoid using it. The bottleneck was still there. Hadoop was designed to route around it.

NoSQL belongs to a different family altogether and gets dragged into this story because the dates overlap. Amazon needed a store that stayed available when the network split because an unavailable checkout costs real money by the minute. Google needed structured data across thousands of machines. Both gave up joins, schemas and strong consistency to get there. Nothing in the analytical line caused any of it.

The lake came from a narrower argument than the one it became. Dixon’s original post was about the detail that warehouses destroy when they aggregate, and the proposal was to keep the raw thing so you could ask questions you had not thought of yet.

All three sent the bill quickly, and in each case it was the same bill: complexity moved rather than disappearing.

Hadoop’s free software needed expensive people. Cloudera’s S-1 describes a platform stitching together 26 open-source projects and states plainly that running it requires exceptional skills, which is a peculiar thing to put in a document trying to sell you the platform. NoSQL left estates with four databases where there had been one, and by 2013 Gartner was describing deployments as overwhelmingly supplemental to relational rather than replacing anything. And the lake deferred the curation work rather than removing it, which is how everyone learned the word swamp. Dixon spent 2014 pointing out that what got built was not what he had proposed.

2012 – Cloud Data Warehouses

This goes after the same warehouse cost problem Hadoop did, from the opposite end. Hadoop replaced expensive hardware with cheap hardware. The cloud warehouse stopped owning hardware.

Redshift launched in late 2012 at $999 per terabyte per year against the $19,000 to $25,000 enterprises were paying on-premises. The engine underneath was ParAccel, which was already the better part of a decade old. Almost nothing about the architecture was new. What disappeared was the procurement cycle, and for most buyers that mattered more than anything happening inside the engine.

This is also the one point in the whole history where cheap resources genuinely do explain the move, and the blockers show why.

What was blocking it	What fixed it	Roughly when
Warehouses meant capital commitment: buy for the peak, wait months, own it for five years	Rented capacity billed by the hour, sold as a service	2012
Keeping one durable copy separate from compute made no sense while storage was priced like compute	S3 falling from ~$0.095 per GB-month to about $0.03 while active compute stayed an order of magnitude dearer	2012 to 2014
Reading data over a network was far slower than reading a local disk, so compute had to sit next to storage	10GbE overtaking 1GbE in datacentre switch shipments, making remote reads roughly as fast as local ones	End of 2013

Two rows, both prices, and the same architecture Stonebraker dismissed went from a bad idea to an obvious one with nothing about the idea itself having changed.

Except the dates run the wrong way.

The story everyone tells is that coupled cloud warehouses hit a scaling wall, and that pain drove the industry to separate storage from compute. BigQuery went generally available in 2012. Redshift reached GA in February 2013. The separated architecture was on sale before the coupled one it supposedly reacted to. Whatever caused separation, it was not Redshift’s limitations, and the tidy sequence is something the industry assembled afterwards.

What the whole era cost us arrived with the convenience. When capacity takes thirty seconds to acquire, the hard question stops being whether you can get hardware and becomes who authorised the query that cost four thousand dollars. Spend governance had to be invented as a discipline, and lock-in moved from the contract to the egress bill.

2017 to 2020 – The Lakehouse

By the late 2010s plenty of organisations were running a lake and a warehouse at the same time, holding the same data twice and paying people to move it between the two. The obvious answer was to put warehouse behaviour directly onto the files already sitting in the lake. Obvious, and for years impossible.

What was blocking it	What fixed it	Roughly when
Object stores have no atomic rename, so two writers could corrupt a table and readers could see half a write	A transaction log sitting above the object store, tracking which files are actually part of the table	2017 to 2019
No open format that competing engines would all agree to trust	Delta and Iceberg, both donated to open governance rather than kept proprietary	2019 to 2020
Nobody had enough pain to pay for fixing it	Roughly half of Databricks’ support escalations between 2014 and 2016 turned out to be object-store consistency and corruption problems	2014 to 2016

Object storage had been cheap for years by this point, and cheapness is arguably what caused the problem rather than solving it: storage got cheap enough that everyone put everything in it, and only then did they find out what it could not do. This one happened because something broke often enough that people stopped putting up with it.

In return the industry took on metadata management, file compaction, catalog sprawl, and a format argument between Iceberg and Delta that ate several years of everyone’s attention.

2019 – Data mesh

Everything above is a technology change with organisational consequences. Data mesh is the reverse, and it finally goes after the warehouse’s other unpaid debt from the early 1990s: the central team everyone has to queue behind. Give the domains that generate data the ownership of it, treat what they publish as a product, federate the governance.

Nearly thirty years is a long time for a problem to sit there untouched, and mesh is the one entry on this list where I can point at the blocker and it has not moved.

What was blocking it	What fixed it	Roughly when
Domains had no way to publish data without a central team building the pipeline	Self-serve platform tooling got good enough to hand over	2018 onwards
No agreed contract for what a domain owed its consumers	Data product definitions, contracts, catalogues	2019 onwards
Domains have no budget, no headcount and no incentive to serve anyone but themselves	Nothing. This has not been fixed	Still open

That third row is why the adoption record looks the way it does. The research base is thin and Thoughtworks reports adoption inside parts of large organisations rather than whole ones. Mesh asks a company to redistribute accountability without redistributing the budget, which is close to the hardest thing you can ask an organisation to do.

2023 Onwards -Whatever we are doing for AI

The thing happening now does not have a settled name yet, but retrieval has clearly become a first-class workload. Embeddings need somewhere to live, context has to be assembled at inference time from sources that were never designed to serve it, and the evaluation problem is unsolved. The current answer is a layer of vector indexes, retrieval pipelines and orchestration bolted onto whatever platform you already run.

It is not replacing anything. The warehouse still serves BI, the lakehouse still runs batch, and this sits on top consuming from both. Every major platform is absorbing the capability rather than ceding it, which is why the dedicated vector database market has grown more slowly than the excitement suggested it would.

What it costs us is the open question. The answer to it is what illuminates what comes next.

Summary of the last 50 years

Architecture	What it fixed	What it cost us
Relational	Application code welded to physical layout	Optimizer opacity, tuning as a permanent specialism
Warehouse	Analytics competing with transactions	A cost ceiling, and a central team everyone had to queue behind
Hadoop	The cost ceiling	Free software, expensive people
NoSQL	Availability and scale for web workloads	Four databases where there had been one
Data lake	Warehouses discarding raw detail	Curation deferred rather than removed
Cloud warehouse	Procurement and operations	Spend governance, lock-in via egress
Lakehouse	Holding the same data twice	Catalog sprawl and a format war
Data mesh	The central queue	Coordination cost, and an organisational ask most firms refuse
Retrieval layer	Serving context to models	Too early to say

The first thing in that table is that nothing replaced anything. The warehouse sits next to the operational database. The lake sits next to the warehouse. The lakehouse tried to merge two of them and mostly became a third. Every large data estate I have seen is an accumulation assembled one reasonable decision at a time, and nobody ever sat down and chose the whole thing.

The right-hand column is where the next architecture usually comes from, but not tidily. The warehouse threw off two separate problems that were attacked thirteen years apart by unrelated movements. NoSQL never connects to the analytical line at all. And the cloud warehouse’s spending problem has been sitting unaddressed for over a decade.

Here is the insight: Nothing moved when it got cheap. It changed when it got easy. Relational was settled in 1970 and irrelevant until an optimizer made it usable. Separating storage from compute was proposed and rejected in 1986 because the interconnect could not carry it, and the objection held for twenty-seven years until datacentre networks caught up. Hadoop was possible in 2006 and still required exceptional operational skill a decade later, which is why Cloudera existed as a business. Meanwhile prices fell the entire time and never once set the timing. The Federal Reserve’s quality-adjusted storage index dropped roughly 800-fold between 2000 and 2023 in a smooth line, while the transitions above happened in clusters years apart. Cheapness only ever set the boundary of what could be built, and something else always decided when anybody bothered.

Specialisation runs until the coordination cost overtakes it. The warehouse specialised analytics away from transactions. NoSQL specialised by consistency model. The lake specialised for cheap raw storage. Each won by doing one thing properly and each added another system to run. Eventually the cost of running them separately exceeded the benefit of keeping them apart, and the lakehouse turned up to merge two of them. Merging was not technically superior. The split had simply got more expensive than it was worth.

That cycle has not stopped. On dbt’s numbers, 70% of data teams now run five to seven tools just to keep daily workflows moving, and 40% say integration is their single largest cost. That is what the late stage of a specialisation cycle looks like from the inside.

So what is Architecture.next ?

I started with three things that did not fit: machine-generated data outgrowing anything a person produces, archives turning into live training workloads, and retrieval patterns nothing in the stack was built to serve.

History says my instinct was half right. We probably do not need another storage architecture, because we have never actually retired one and adding a ninth to the pile would follow the pattern rather than break it. But it also says that is not how this works. Nobody chooses to add a system. They accept a specific charge they can no longer tolerate, and the accumulation happens as a side effect.

So the question is not whether a new architecture is coming. It is which line in that right-hand column has got expensive enough that somebody will pay to escape it, and what has recently become easy enough to make the escape worth the disruption.

Both halves have to be true. Expensive on its own gets you nothing, which is why the central team queue survived from the early nineties to 2019, and mostly survived that too. Easy on its own gets you a technology nobody adopts, which is most of the things I found while researching this.