Ice Graph · semantic IoT lakehouse · AWS · sub-minute device-to-queryable
A portfolio of commercial buildings instrumented with IoT sensors — lighting, HVAC, occupancy. Telemetry arrived as Sparkplug B (Protobuf) messages at high volume. The requirements: time-series analytics with sub-minute latency, a semantic understanding of building topology (which sensor, which zone, which system), and governed access for analysts — without standing up a data warehouse.
| Sparkplug B / Protobuf | Device-side data contract. Typed, compact, standard for industrial MQTT — validation starts at the edge, not the warehouse. |
| Exploder Lambda | Fan-out and keying. One Sparkplug payload carries many metrics; the Lambda unpacks them and mints a stable UUIDv5 point_id per sensor metric — the single join key that lets SQL and SPARQL results compose. |
| Kinesis Firehose → S3 | The lake landing. Cheap, durable, decoupled from device traffic spikes. |
| Iceberg on S3 | The table format. Parquet files underneath; Iceberg's snapshot/manifest metadata on top gives ACID commits and time-travel queries. Time-partitioned for scan efficiency. This is what separates a lakehouse from a folder of files. |
| DynamoDB | The hot path. Sub-10ms point metadata for real-time APIs. DynamoDB Streams (CDC) feeds the graph sync — no dual writes, no batch reconcile jobs. |
| Neptune + Brick | The semantic layer. Building → floor → zone → equipment topology in the Brick ontology, synced as Turtle/N-Triples. Analysts ask "which coolers serve zone 4?" in SPARQL, then join the answer to time-series in SQL on point_id. |
| Glue Spark jobs | Lakehouse operations. Streaming ingestion produces many small files; scheduled Spark compaction merges them and expires old snapshots. Same engine materializes Gold tables — daily rollups and current-state aggregation on top of raw event data. |
| Lake Formation | Governance. Row-level, cross-account access control so each team sees only its buildings — policy lives in one place, not per tool. |
| Athena | The query surface. Analysts run SQL directly against Iceberg on S3. No warehouse to provision, no ETL to a separate system, no second copy of the data. |
1. Parquet files are immutable. You never modify a file in place, so "update" becomes "write new files, retire old ones" — a model that fits a log-based commit protocol perfectly.
2. The metadata is the transaction log. Iceberg's snapshot/manifest tree records every commit: which files are live, which are deleted, what the schema was. Readers replay it to get a consistent, versioned view — no partial writes, no stale files, ever.
3. Time travel for free. Every commit is a snapshot. Query yesterday's state, audit a bad ingestion, or replay a correction without re-pulling data.
4. Layers, not silos. Raw event-level time-series (bronze) → Gold rollups and current-state tables — same storage, same engine, governed by the same policies.
End-to-end lakehouse on AWS — ingestion, semantic layer, governed analytics — with no data warehouse in the middle.
Real-time and historical in one store, joined across SQL and graph queries on a single stable key.
CDC-driven graph synchronization — the semantic layer stays current without batch jobs.
Operational maturity: compaction, snapshot expiration, rollup materialization — the boring work that makes lakehouses survive contact with production.
Delivered at scale with AI-accelerated engineering — 150K+ lines of production code generated, reviewed, and shipped.