Introduction

We just completed the launch of a third generation Data Lakehouse. The platform hosts private data lakes and analytics dashboards to hundreds of clients around the world. Data is securely stored, segmented and delivered using AWS services such as Redshift, Glue, Lambda and S3.

This article focuses on the general architecture and how we were able to deliver individual big data solutions to customers in a deployable bundle with a fraction of the spend over other solutions.

Here's a quick breakdown of how this design compares to the standard and the self-hosted alternatives:

Comparison Overview: EMR + Glue vs. Lambda Lakehouse

Cost Breakdown Examples

AWS Managed EMR + Glue

• EMR On-Demand Pricing (m5.xlarge, 4 vCPU, 16GB RAM):
  $0.096 per core-hour × 4 cores ≈ $0.384/hour
  → For a modest 3-node cluster, running 8h/day:
  3 × $0.384 × 8 × 30 ≈ $276 per month
• Glue Crawler Pricing:
  $0.44 per DPU hour → A typical crawler job costs $0.44–$5 depending on dataset size.
  → For 2 crawls per day: $2 × 30 = $60/month
• Total Monthly (light usage): ~$400–$1000+
  Heavy usage: upwards of $10K/month when factoring persistent compute or multiple jobs.

Self-hosted EMR + Spark

• Cost depends on instance types, cluster size, and data volume.
• Lower AWS bills, but offset by:
  • Human time for ops (cluster tuning, monitoring, patching)
  • Higher risk of downtime or poor job performance without Spark experts

Lambda-Based Lakehouse

• Lambda Cost:
  $0.00001667 per GB-second + $0.20 per million invocations
  → For 100GB processed per day in 512MB memory for 5 seconds:
  100 GB × (5/2) × $0.00001667 ≈ $0.42/day = $12.60/month
• API Gateway/SQS Events/S3 Triggers: negligible if used efficiently.
• Glue Catalogs: This architecture auto-generates them — no crawler cost.

✅ Est. Monthly Cost: $15–$300, even with growing data volumes.

Overview

Our platform is unique in a few ways:

1. There is no monolithic lakehouse
2. Every customer's self-contained datastore, metastore, transform layer, and presentation layer are deployed by a single registration pipeline event
3. There are no Glue Crawlers, Spark, or EMR jobs for ETL/ELT
4. The transform layer is object event driven
5. The project integrates AWS CDK's CodePipeline strategy for CI/CD

Data Ingestion

Our ingress data policy is, generaly, "expect anything". Raw data files typically arrive in either:

• JSON
• CSV
• Avro
• Parquet

The Storage Layer

We use S3 for the data backend. Only S3.

Redshift Spectrum doesn't need to be loaded anywhere else but S3. Metadata gets stored alongside the transformation output which accounts for a large portion of what the Lakehouse serves.

The data written to S3 is:

• normalized
• partitioned
• cataloged
• mapped
• isolated
• reproducible

The Query Layer

As mentioned above, Redshift Spectrum allows Redshift to query external data stored in S3 through external schemas and external tables backed by catalogs such as Glue. This allows the analytics platform to reach into lake-backed data without requiring any dataset to be physically loaded into Redshift first.

This provides separation of concerns to the platform:

• S3 stores lake data.
• Glue stores table metadata.
• Redshift Spectrum queries S3 data as tables defined in glue.
• Redshift supports the analytic surface for external integration.

This is the key difference between a data lake and a data lakehouse. The data lake becomes an active part of the analytics system by fulfilling both the datastore and metastore on which analytics directly runs.

This and the Lambda transform layer entirely eliminates the need for batch rollups and manual data modeling. When data arrives in S3, the lambdas transfrom and store the data back to S3 and then create and update glue tables and partitions in one shot.

Ultimately, this results in data availability and readiness being achieved within seconds of a dataset being received.

The ETL Layer

The ETL layer is Lambda-driven.

Not every data transformation requires a persistent cluster. Not every ingestion path needs Spark. Not every pipeline is made better by adding a distributed compute engine and a four-hour onboarding document explaining why local testing is “non-trivial.”

For this platform, the ingestion workflows were designed as lean processing units:

1. Receive raw input.
2. Detect format.
3. Parse records.
4. Infer candidate field types.
5. Normalize into a standard schema representation.
6. Write optimized Parquet output.
7. Emit schema metadata.
8. Trigger catalog registration.
9. Make the data queryable.

This is not a universal claim that Lambda replaces Spark. It does not. Please do not build your next petabyte-scale feature engineering platform entirely out of Lambda functions.

But for a large class of customer-ingestion and normalization workloads, Lambda gives you elastic execution, operational simplicity and clean deployment boundaries. It also allows each deployment to scale independently.

The Cataloger

The cataloger is integral to what makes the this design possible.

It runs as a second lambda and watch event on the schema path generated by the ingest process. The catalog lambda generates:

• Glue database and table definitions
• storage descriptors
• partition keys
• partition locations
• Redshift external mappings

The ingestion process standardizes the data and the cataloger populates the metastore.

Multi-Tenant by Deployment

The platform is engineered to serve customers with their own pipeline and stack with dedicated resources and clearly segmented data backends. This gives us:

• cleaner security boundaries
• easier customer-specific scaling
• simpler blast-radius control
• independent deployment behavior
• clearer cost attribution
• reduced cross-customer coupling
• scalable repeatability

Deployment Automation

Everything is deployed through CDK and CodePipeline automation.

The infrastructure pipeline defines and deploys the lakehouse stack as code:

• storage resources
• Lambda ETL functions
• cataloging functions
• Glue metadata resources
• Redshift integration
• IAM boundaries
• customer-specific configuration
• deployment stages

The goal is automated, repeatable and scalable customer delivery.

Notable Issues

• Some data producers chose v1 over v2 file formats in Parquet. This resulted in unexpected behavior, particularly in compatibility issues and performance drops.

• The INT96 data type is one example of a deprecation from v1 and because we're not storing data in hadoop or using Scala we had to manually implement many of the ingestion routines that would normally convert these types automatically.

• Matching the data types when generating Glue tables required mapping and generating a compatible json table schema that reflrected the types accurately.