I build data platforms that keep moving when real data gets messy. I'm a Senior Data Engineer in London, with 10+ years across insurance, fintech and enterprise, from Deloitte consulting to insurtech scale-ups.
Real data rarely follows the diagram: rows fail, updates repeat, and schemas drift. NEXUS follows that journey from messy events to trusted business answers. The examples are synthetic; the engineering choices reflect my work. Making an answer survive a Tuesday is the interesting part.
Three records arrive: two pass, one fails. The good rows move on; the exception keeps its reason until correction and replay. No batch-wide pause for one indecisive row. Explore the architecture and working code.
One YAML declares each tenant's source, contract, checkpoint, Bronze destination and quarantine path. Aurora can pause and replay while Beacon and Cedar continue. Inspect the example YAML.
Aurora's first update applies; its replay is a no-op, and older state cannot win. A new customer_phone field holds Aurora at the contract gate while other tenants continue. A mock AI hook suggests a classification from schema metadata; a reviewer decides whether to change the contract and replay. Inspect the CDC design and runnable reference.
Beacon's file gets one approved decimal cast, one row quarantined, and two rows into Bronze. A retry is a no-op; a new field waits for review. The AI hook can suggest, not approve. Boring rules are welcome when money is involved. Inspect the design and runnable reference.
Clean rows still need shared meaning. In this design concept, Agumon answers a synthetic paid-orders question using an approved metric, tenant policy and Gold data, with definition, freshness and lineage attached. The chatbot is illustrative, not deployed. See the semantic-layer blueprint.
60 seconds. Keep Gold clean. Catch the duplicate, quarantine the corrupt row, and move the good data through before the SLA clock hits zero.
Platforms
Processing
Streaming & CDC
Orchestration & ops
Quality & governance
Modelling
- Kelas Semanggi: open source data engineering curriculum I maintain
- HACKTIV8: teaching Databricks, Spark, dbt and dimensional modelling to career switchers since 2019
Notes on data platforms and engineering on Medium.
Open to senior and lead data engineering roles in London and remote UK.