The AI Problem Arrives

Now comes the next unpleasant truth.

Most organizations are not ready for AI because their data is not ready for AI.

They have dashboards. They have warehouses. They have data lakes. They have CRM exports. They have PDFs. They have application logs. They have customer messages. They have support tickets. They have SharePoint folders named after departments that stopped existing in 2019.

Somewhere inside all of that, there is useful institutional knowledge.

Good luck.

AI does not remove the need for data architecture. It makes the need more critical.

Traditional BI mostly asks:

• What happened?
• How often?
• Which customers?
• Which segment?
• Which trend?

AI-enabled systems ask harder questions:

• What does this record mean?
• Can this content be retrieved semantically?
• Can this field be trusted?
• Can this document be safely exposed to this user?
• What is the lineage of this fact?
• Is this data useful for training, retrieval, evaluation or decision support?
• Can this be joined with operational context?
• Can an agent act on it without leaking something catastrophic?

It's a different pressure profile.

From Analytics Data to AI-Ready Data

AI changes the requirements of big data because it increases the importance of context.

A lakehouse built for dashboards needs reliable tables, schemas, partitions and metrics.

A lakehouse built for AI needs all of that, plus:

• document lineage
• semantic metadata
• embeddings
• vector search integration
• access control at retrieval time
• row-level and object-level security
• prompt-safe data boundaries
• data freshness indicators
• source attribution
• quality scoring
• human feedback loops
• model evaluation datasets
• governance for training and inference use

Retrieval-augmented generation, or RAG, is one of the clearest examples of this shift. RAG systems retrieve relevant external data at query time and provide it to a language model as context, allowing responses to be more current, specific and grounded than relying only on static training data.

This means the future lakehouse is not just a reporting substrate.

It becomes the factual memory layer for intelligent systems.

That sounds grand until you realize it also means every bad schema, missing timestamp, ambiguous customer ID and undocumented transformation is now potentially part of an automated reasoning chain.

Congratulations. Your data quality issues have been promoted.

What AI Will Change

AI will transform big data systems in several obvious ways and several deeply annoying ones.

1. Metadata Becomes a First-Class Product

In the old world, metadata was what you fixed when Athena couldn't find a partition.

In the AI world, metadata is how systems decide whether data can be trusted, retrieved, joined, summarized or exposed.

Expect more investment in:

• semantic catalogs
• column-level descriptions
• lineage metadata
• transformation provenance
• freshness markers
• ownership signals
• sensitivity classification
• retrieval policies

Metadata will stop being a side effect of pipelines. It will become one of the primary outputs.

2. Schema Evolution Becomes Constant

AI workflows are going to increase the volume and variety of derived data.

Summaries. Embeddings. Classifications. Labels. Extracted entities. Confidence scores. Prompt traces. Evaluation results. Human feedback. Agent actions.

If the schema cannot evolve, the system cannot breathe.

This is where open table formats become more important over time. Today, a platform might get very far with Parquet, Glue metadata, Redshift Spectrum and careful partition management. Tomorrow, as update patterns, lineage requirements, time travel, schema evolution and multi-engine writes become more demanding, Iceberg-style table semantics become harder to ignore.

The future lakehouse is not only file-backed.

It is table-aware.

3. Raw Data Gets Valuable Again

For years, teams were told to clean data aggressively for reporting.

AI changes that calculus.

The raw text, notes, descriptions, documents and event payloads that once looked too messy for dashboards may now contain the context needed for search, summarization, classification and agentic workflows.

This does not mean dumping garbage into S3 and calling it strategy.

It means raw data should be preserved with lineage, access control and enough metadata to make later transformation possible.

AI does not want only clean rows.

It wants context.

Unfortunately, context is usually where organizations have been hiding the bodies.

4. Security Boundaries Move Closer to Retrieval

In dashboard systems, the access boundary is usually around tables, dashboards, roles and reports.

In AI systems, the access boundary has to move into retrieval itself.

If a model or agent retrieves context from the lakehouse, then permissions must be enforced before the context enters the prompt. Not after. Not in the UI. Not in the optimistic little comment that says “TODO: validate tenant.”

Before.

This will force lakehouse architectures to integrate more tightly with policy enforcement, identity, data classification and tenant-aware retrieval.

5. Data Platforms Become Agent Substrates

The next generation of enterprise AI systems will not only query data. They will act on it.

That means the lakehouse becomes part of an agentic operating environment.

Agents will need to:

• inspect schemas
• discover datasets
• retrieve context
• generate SQL
• explain anomalies
• recommend transformations
• identify data-quality issues
• propose new derived datasets
• trigger downstream workflows

This is what's useful.

It's also exactly the kind of thing that becomes dangerous if the metadata layer is sloppy, the permissions model is vague and the data catalog is an afterthought.

An agent with access to bad metadata is not intelligent. It's confidently astray.

The Practical Future

The future of big data is not “AI replaces data engineering.”

That's the sort of sentence people say right before hiring more data engineers.

The real future is that AI raises the floor for what data platforms must provide.

Big data platforms will need to become more:

• automated
• metadata-driven
• schema-aware
• policy-enforced
• lineage-rich
• retrieval-ready
• multi-modal
• compatible with AI and agentic workflows

The industry is moving from data lakes that support analytics to data platforms that support intelligence.

This is the one of the first major disrupts in data engineering.

It used to be that analytics explained the business.

Now, it's intelligence that participates in it.

Lessons From The Lakehouse

Keep the Storage Boring

S3, Parquet and partitioned object paths are not glamorous. Good.

Glamour is not an architectural requirement.

Durable, queryable, cost-effective and boring is usually the better foundation.

Make Metadata Explicit

Do not wait for crawlers to infer what your platform should already know.

If your pipeline understands the structure of incoming data, emit that structure. Persist it. Catalog it. Version it where appropriate.

Your catalog is not housekeeping.

It is the index of reality.

Normalize Early Enough

Raw data should be preserved, but analytic consumers need consistency.

The trick is not to destroy raw data in pursuit of cleanliness. The trick is to normalize into stable analytic formats while keeping enough provenance to recover how the answer was produced.

But raw data holds the gold for future intelligence to make meaningful use out of it. Keep your archives intact and the data lineage clean.

Partition With Intent

Partitions are not decoration.

They're performance boundaries. They're cost boundaries. They're sometimes tenant, time and lifecycle boundaries.

Bad partitions punish every query that comes after them.

Automate the Customer Stack

A customer-specific lakehouse deployment should be boring to create.

If onboarding a customer requires manual infrastructure work, the platform is not done. It is merely operationally expensive.

As AI demands grow, deployments need to be ever more seamless and automated. The days of manual infrastructure management are a relic of the past. The future needs automation.

Prepare for AI Now

Even if your platform starts with analytics dashboards, design it as if the next consumer is an AI system. More than likely, you already have this backlog growing. Start early. Get ahead of the storm if you can.

Because there's no doubt it's coming on fast.

1. Preserve raw data
2. Strengthening metadata
3. Track data lineage
4. Enforce access boundaries
5. Prepare for embeddings, vector indexes, semantic search and retrieval-time governance.

Closing Thoughts

While the third generation data lakehouse enhanced our data lakes by recognizing that storage, schema, metadata, governance and query access all have to cooperate, we have yet another evolution upon us.

Our warehouses were rigid.

Our lakes started to overflow.

The lakehouse raised the walls and gave us speed boats.

Now, the future agentic lakehousing gives us nets and depth charges for us to begin exposing what our data's been hiding and consume it.

The lakehouse is not the final form of big data architecture.

It is the bridge.

The next stage is AI-ready data infrastructure: systems that do not merely store and report on data, but prepare it for retrieval, reasoning, automation and agentic workflows.

That future will reward teams that treat metadata as a product, schema as evolutionary, access control as non-negotiable and automation as infrastructure discipline.

The rest will keep throwing files into buckets and wondering why their AI strategy doesn't flourish.