Case · Aviation · Anonymous client
From deeply nested XML events to nine analysis-ready business entities, with safe reprocessing, exact reconciliation and idempotency at every boundary.
The context
An airline continuously received ticket lifecycle events through a GDS. Issuances, exchanges, refunds, cancellations, taxes and payment methods arrived through a message queue as deeply nested XML.
Consolidation relied on a handcrafted Spark Streaming implementation. Manual writes, fixed triggers and rules distributed across notebooks made the system hard to test and reprocess. Commercial analytics received consolidated information with days of latency.
Our mandate covered a ground-up replacement from the queue integration boundary to the business model. Before designing the architecture, we asked where low latency actually changed a decision. That question kept streaming complexity out of layers that did not need it.
Real-time only pays for itself when there is a decision on the other side that cannot wait for the nightly run.
The challenge
The solution had to run continuously, handle complex XML, live with at-least-once delivery and preserve consistency in a model used for financial analysis. It also had to avoid turning every failure into manual intervention or creating an architecture that could not be promoted reliably across environments.
Our approach
We designed a serverless boundary to consume and preserve messages, a continuous pipeline for ingestion and normalization, and another pipeline to materialize the business model. Each layer received semantics suited to its problem, supported by versioned configuration and testable logic. The integration boundary became an observable contract between delivery and processing, while the original payload remained available for audit, diagnosis and replay.
Publishable architecture
A message only leaves the queue after its original payload has been preserved. From there, two declarative pipelines turn the XML into normalized data and business entities.
Key decisions
Landing every message as a file looked like an extra hop. In practice, it created an immutable record, decoupled the pipeline from queue retention and made every row traceable to its source payload.
Result: manual reprocessing disappeared.
The ledger protects against redelivery of the same message after a failure. In the lakehouse, explicit business-key ordering deterministically selects the correct version of each fact.
Result: manual duplicate correction disappeared.
When a reference needed to allocate a value across segments was missing, the alternative method was applied to the whole document. The sum of the segments therefore remained equal to the document total.
Result: manual reconciliation disappeared.
Ingestion and normalization run continuously. The business model uses incremental materializations, avoiding fragile joins and aggregations across multiple streams for a marginal latency reduction.
Result: low latency with an operation people can understand and test.
In production
The technical article explains where we stopped streaming, how we handled redelivery and five failures that changed how we build pipelines.