Real-time does not mean streaming everywhere 

Continuous processing has a permanent cost: active compute, operational complexity and a class of failures that only exists in streaming. The architecture only pays for itself when a real decision loses value by waiting until the next day.

Commercial reporting alone did not justify streaming throughout the solution. The ability to detect anomalous issuance patterns while they are still actionable changes that equation. We used this criterion to decide where low latency was necessary and, just as deliberately, where it was not.

That question became a design constraint. We measured value through the latency of the complete system, not through the number of boxes labeled streaming in a diagram.

The obvious alternative was to consume the queue directly from Spark Structured Streaming. We rejected it and materialized every message as a file in the landing zone before analytical processing began.

A queue has limited retention. The file preserves exactly what the source sent and lets us fix a parser months later without asking for another extract. It also decouples pipeline availability from queue behavior and creates a simple operational boundary: did the file arrive or not?

In a model used for financial analysis, lineage and auditability are not conveniences. Every row must trace back to the payload that produced it. Saving one network hop was worth less than guaranteed replay and diagnosis.

End-to-end delivery is not magically exactly-once. The design combines at-least-once delivery with consumer-side idempotency at two independent boundaries.

At the integration boundary, a message remains locked until the write finishes. If a batch fails halfway through, it returns to the queue. A ledger records only the items already written on that failure path and prevents duplicate files. After a successful batch, that temporary state is removed.

Inside the lakehouse, the source may send legitimate revisions of the same business fact. Deduplication therefore uses a business key and explicit ordering to select the right version. Technical redelivery and a business revision look similar but require different defenses.

Ingestion and normalization run continuously. The business model is materialized incrementally. This boundary does not make the system less ambitious. It protects correctness.

The final entities require aggregations and joins across several sources. Forcing streaming semantics into that layer would introduce watermarks across multiple flows, incompatible output modes and behavior that is difficult to explain during a failure. The marginal latency gain was not worth it.

Near real-time is an end-to-end property. The right boundary delivers the business window without importing complexity into layers that do not benefit from it.

The Bronze layer fed dozens of downstream tables. Keeping it as a view looked lighter, but every consumer would open an independent reader against the same source, with separate checkpoints and no reliable read order.

Materializing it created one ingestion checkpoint and consistent fan-out. The cost of persisting that boundary bought predictability for every transformation that followed.

We used file notifications managed by the governed storage layer. That removed an additional event service, its queue, credentials and the risk of configuration drift across four environments.

The schema was declared explicitly because inference over nested XML in streaming is unstable. Out-of-schema records are preserved in a rescue column, while lineage metadata enters at ingestion time. Bronze never discards a payload just because it cannot interpret it yet.

One entity allocated a document value across its segments proportionally. The reference used in the calculation did not cover every possible pair. Applying a fallback only to the segment with a missing reference made the final sum diverge from the document total.

We changed the unit of the rule. If one reference was missing, the entire document switched to the alternative method. Individual segment precision could decrease, but document-level reconciliation remained exact in every case.

The change removed manual reconciliation because the rule now operated at the same level where the business verifies consistency.

One entity joined documents to the source considered primary for value and currency. In production, the result contained zero rows. For that event type, the information never appeared in that source. It existed only in an alternative.

The fix was a LEFT JOIN with an explicit coalesce hierarchy across sources. The issue was not syntax. The join claimed that every document had a mandatory match, a business assumption hidden in ordinary SQL.

After the fix, we began reviewing joins as contracts for cardinality and presence. Before choosing the type, we ask what absence means in the domain and whether dropping a row is truly valid.

The first version used dropDuplicates. The operation removes duplicates but does not guarantee which row survives when two versions share a key. Reprocessing the same window could produce a different result even when no input had changed.

We replaced it with a window partitioned by business key and explicitly ordered by event time. The selected version no longer depended on the physical distribution of the DataFrame.

This detail is central to replay. A reprocessable platform cannot merely finish without an error. It must return the same answer for the same input, including after a cluster or partitioning change.

A dimension was enriched with geographic hierarchy from a reference based on an international standard. The join appeared naturally one-to-one, yet it increased the row count.

The reference preserved the history of codes assigned to different entities over time. Official did not mean unique. We filtered historical entries and validated key cardinality before the join.

We also adopted a convention: a DataFrame only earns the lookup suffix after it has been reduced and validated for the expected cardinality. The name communicates a guarantee, not merely an intention.

In batch, poor partitioning leaves one bad set of files. In continuous execution, every micro-batch repeats the problem. A table starts healthy and slowly degrades until listing and opening files dominates reads.

We tuned adaptive execution, partition coalescing, target size and automatic optimization. The goal was not a perfect number for every batch, but preventing the debt from growing by itself for weeks.

Monitoring job duration alone does not catch this pattern early. File count and size distribution are part of the operational health of a continuously updated table.

The migration did not end when the new pipeline produced the right result. Dozens of manually managed external objects had to leave the catalog before the new operation, without deleting underlying data or touching tables already promoted.

We created a versioned cleanup process with explicit targets and a protection list. Retirement became reviewable and repeatable like any other artifact instead of relying on a manual command session.

The process ended coexistence without data loss or an availability window. The new operation only became simple after the old one stopped competing for names, responsibilities and attention.

Declarative pipelines do not need to bind business logic to the framework runtime. We kept decorators in thin wrappers that only resolve sources. Transformations live in pure functions that receive and return DataFrames.

This pattern lets us test rules with synthetic data and pytest without starting the full pipeline. We do not claim complete test coverage. The demonstrable value is structural: each rule can be exercised in isolation, while the same code moves through development, quality, pre-production and production through configuration alone.

Pipelines are defined in versioned configuration, with catalogs, paths and compute policies resolved by environment. The same transformation file crosses four environments without local names or conditional branches. That narrows the distance between what was tested and what reaches production.

Observability also uses native platform artifacts. The pipeline event log is materialized in the catalog and queried through SQL. Execution time, counts, updates and lineage remain available without scattering logging calls through every transformation.

This choice does not remove alerts or operational responsibility. It creates one investigation source. When a flow is late, the on-call engineer can separate a delivery failure, ingestion delay and modeling issue without opening several systems before forming a hypothesis.

Naming conventions, reproducible deployment and a clear responsibility boundary rarely appear in a demo. Yet these decisions determine whether another team can maintain the platform after the project ends.

Complete latency has three parts: message availability to file, file to Bronze and Bronze to a materialized entity. We measured only the first in a transferable way. Across two independent development workflows, ledger lookup, writing and confirmation completed in under one second at the median.

That number demonstrates that the stateless component has a stable execution cost. It does not demonstrate end-to-end latency, production throughput or message frequency. Adding estimates for the other legs would produce a commercially attractive but technically indefensible figure.

The correct way to complete the measurement is to compare file modification time with the Bronze ingestion timestamp, then use the event log to measure entity materialization. A published result should include p50, p95, sample size, period and environment.

Until that sample exists, we describe the system as near real-time and keep the sub-second metric scoped to the component actually observed. Precise language is part of engineering, especially when technical buyers will read the page.

The system runs in production and continuously provides information that previously reached analytics with days of latency. The same codebase crosses four environments without changes to the business logic.

The immutable landing zone removed handcrafted reprocessing. Idempotency at two boundaries removed manual duplicate correction. Document-level fallback removed manual reconciliation. These are results without invented ROI and with a direct relationship between an engineering decision and the operational work it removed.

The only safe performance measurement covers the integration component: ledger lookup, write and queue confirmation complete in under one second at the median in development histories. We do not turn that number into end-to-end latency. The complete system is honestly described as near real-time.