Horses for courses: stop using the warehouse for work that has to answer the customer

“Data” is not one platform. Analytical warehouses such as Snowflake excel at curated, correct, deep analysis. Operational work that must answer a customer or a risk engine in milliseconds needs a low-latency store such as MongoDB, with the same events fanned out from Kafka to both. Stop staffing analytical tribes to build customer-facing product. _Source: Electric Fox operator article · slug `horses-for-courses-stop-using-the-warehouse-for-work-that-has-to-answer-the-customer` · readingTime 11 min_

When someone in a bank or insurer says “data”, the room reaches for big data, Snowflake and, historically, the Hadoop-shaped stack. That reflex is understandable: those platforms won the last decade of management information, risk analytics and regulated reporting. It is the wrong default for anything that must answer a user, an authorisation or a fraud engine in milliseconds. Confusing analytical headcount and tooling with operational product needs is how next-best-action arrives after the session, how fraud scores after the loss and how in-app transaction views sit behind warehouse freshness.

The organisational habit is older than any one vendor. A single “data platform” became the prestige answer. Boards funded warehouses, lakehouses and the tribes that run them. Product leaders then asked those same tribes for live upsell, cross-sell, balances and customer-facing history because the word on the slide was still “data”. Horses for courses is not a slogan here. It is the only architecture that matches how the business actually works.

The cost of doing operational work with an analytical tribe

Analytical people bring analytical SLAs and analytical technology. They are trained to reconcile, curate and defend correctness under audit. Those are essential skills for year-end, stress testing and deep risk analysis. They are a poor fit for a product backlog that measures success as latency to decision. The result is batch thinking applied to real-time problems: overnight or hourly pipelines dressed up as “near real time”, feature stores that refresh on a cadence the authorisation window cannot wait for and customer journeys that stall until the warehouse says the data is ready.

The commercial damage is concrete even when the ledger does not show a line item named “wrong store”. Next-best-action that cannot see the last few interactions is theatre. Fraud scoring that finishes after settlement is forensics, not prevention. Operators who have published on card and instant-payment flows routinely describe institutional scoring budgets inside roughly 50 to 100 milliseconds from enrichment through rules and model to decision; card authorisation validity windows are often cited in the low hundreds of milliseconds. Treat those ranges as industry guidance rather than a single regulated standard, but do not pretend a warehouse freshness cycle measured in minutes or hours lives inside them. Meanwhile the analytical team is still measured on pipeline completeness, mart coverage and query cost. Everyone is busy. The customer still waits.

Staffing makes the mismatch permanent. A Snowflake-centred group will optimise for curated models, governance and scan-friendly workloads. Ask that group to own a customer API and you get analytical patterns on an operational problem: more mediation layers, more eventual consistency excuses and more “we will land it in the warehouse first”. The cost is not only compute. It is product velocity spent explaining why the hot path cannot wait for the cold path’s calendar.

Two jobs, two shapes of data

A bright Hong Kong office with an operational team standing at live consoles on the left and an analytical team seated with charts on the right, harbour in daylight beyond the glass
Horses for courses: operational product work and curated analysis are different jobs, even when they share a floor and a budget line labelled data.

Analytical data work is curated, reconciled and maths-heavy. Correctness beats speed. The set is prepared so that MI, regulatory returns, actuarial deep dives and long-horizon risk analysis can be trusted. Latency is measured in minutes, hours or overnight runs. Schema discipline, lineage and audit evidence matter more than read-your-writes in the same request that changed the row.

Operational data work is different. Feedback loops are short. Latency is low. The path is user-interactive. Workloads are often write-heavy or demand read-your-writes next to the application: the balance just updated, the offer just accepted, the fraud decision just taken. Consistency models, indexing and concurrency behaviour must serve the app and the decisioning service, not a BI dashboard. Calling both of these “data” is accurate in English and disastrous in architecture.

Banks and insurers need both shapes. The mistake is collapsing them into one platform and one tribe because the budget line is labelled the same way. Analytical excellence does not confer operational latency. Operational stores do not replace curated warehouses for deep analysis. Pretending otherwise is how programmes burn years building the wrong thing with the right people.

Why the warehouse is the wrong tool for the hot path

Snowflake and its kin excel when the set is curated and the question can wait. Columnar storage, micro-partitions and scan-oriented query engines are built for large analytical reads and elastic warehouses that isolate BI and risk workloads from each other. That is a genuine strength. Point lookups by primary key, high-frequency small transactions and immediate visibility of the last write are not what standard warehouse tables were designed to do well. Independent comparisons of warehouse versus general-purpose operational databases still show the familiar split: warehouses dominate large aggregations; operational databases dominate single-row reads, updates and high-concurrency transactional traffic.

Snowflake has extended the surface with Hybrid Tables under its Unistore agenda, aiming at lower-latency row-oriented access alongside analytical queries. That closes a real gap for some mixed workloads inside the Snowflake estate. It does not make the warehouse the default home for a high-concurrency customer API or a fraud path that must finish inside an authorisation budget. Snowflake’s own documentation frames hybrid tables as complementary to standard tables, not as a wholesale replacement for purpose-built operational databases under sustained OLTP-style load. Fair reading: use Snowflake where curated analysis belongs. Do not staff it as the system of interaction for live upsell, cross-sell, in-session transaction views and real-time fraud scoring.

The Hadoop-shaped stack taught the same lesson earlier. Batch-oriented processing and HDFS-era gravity were excellent for cheap storage and offline jobs. They were a poor place to put anything a mobile app or a payment authorisation had to wait on. Replacing Hadoop with a cloud warehouse improved the analytical half. It did not license the warehouse to own the hot path.

Why now: the paradigm already split

Streaming is normal. Product teams ship features that assume fresh state. Customers expect balances, status and offers to reflect what just happened, not what last night’s load job managed to reconcile. Instant payments, open-banking-style journeys and always-on digital channels make warehouse-shaped freshness a product liability when it is pressed into operational service.

Keeping one “data platform” for both jobs is organisational nostalgia, not architecture. The industry already runs event buses, change-data-capture pipelines and multiple consumer groups as standard practice. The paradigm split is visible in every mature estate that fans the same business events into an analytical plane and an operational plane. What lags is governance language: strategy decks still speak of a singular data platform while the traffic diagrams show two jobs with incompatible SLAs. Name the split explicitly and the staffing argument becomes easier. Refuse to name it and every initiative becomes another warehouse ticket.

A pragmatic pattern: one stream, two stores

The workable pattern is simple. Domain events (or CDC from systems of record) land on Kafka or an equivalent event bus. Separate consumer groups fan those events into an analytical store and an operational store. The warehouse receives curated, transformed and reconciled feeds for MI, deep risk analytics and regulated reporting. The operational store receives the projections the application and decisioning services need for low-latency reads and writes. Horses for courses: deep analysis where the warehouse belongs; MongoDB (or a similar operational database) where the app and the decisioning live.

Kafka’s consumer-group model makes fan-out a first-class property rather than a heroic integration. Each downstream system tracks its own offsets and processes at its own pace. Analytical consumers can afford more transformation and slower materialisation. Operational consumers need tight lag budgets and idempotent upserts keyed on business identifiers. Avoid naive dual writes from the application into both stores without a coordination pattern; Confluent and others have documented the dual-write failure modes in detail. Prefer a single atomic write at the source (transactional outbox or CDC from the system of record) and derive both planes from the stream.

MongoDB is the concrete operational example for this brief because document models, secondary indexes and change streams map cleanly to customer profiles, session state, offer eligibility, transaction history views and event-driven fraud enrichment. MongoDB’s own financial-crime reference architectures describe change streams updating risk models and screening paths in near real time so decisions do not wait on batch refresh. The pattern generalises: other operational stores can fill the same role when latency, access patterns and team skill fit. Do not dilute the point by turning the article into a bake-off. The claim is dual-path architecture, not vendor monopoly.

Who owns what

Analytical groups own models, marts and regulated reporting. They own the quality bar for curated sets, the reconciliation discipline and the evidence trail auditors expect. Product and operational engineering own the low-latency path: APIs, decisioning services, operational schemas and the SLAs that sit next to the customer. Platform engineering may run the shared event bus, but ownership of outcomes must follow the job.

Stop staffing a Snowflake team to build a customer API. Stop measuring operational features only by whether the pipeline “completed”. Give analytical leaders clear remit and stop asking them to pretend they are product latency owners. Give product engineering the operational store and the latency budget and stop forcing every state change through a warehouse-shaped programme office. Shared event contracts and schema governance belong in the middle. Collapsed org charts that put both jobs under one “Chief Data” theatre without splitting accountability recreate the original failure mode with nicer titles.

What “good” looks like for banks and insurers

Concrete pairings beat abstract maturity models. Warehouse for management information, actuarial and deep risk analytics, stress and scenario work and regulated returns where correctness and lineage dominate. Operational database for next-best-action, live fraud scores, balances, limits, entitlements and in-app history where the user or the authorisation cannot wait. Clear SLAs for each path: warehouse freshness and completeness targets on one side; end-to-end latency to decision, p99 read latency and write visibility on the other.

Governance should reflect the split. Access control, retention and residency still apply to both planes, but change windows differ. Analytical marts can follow curated release trains. Operational projections need continuous, idempotent updates with rollback and shadow-scoring paths for model changes. Fraud and NBA teams should be able to ship rule or model updates without waiting for a warehouse release calendar, while analytical risk teams retain the curated history those models will later be judged against. Good looks like two accountable paths fed by one event spine, not one overloaded platform with contradictory KPIs.

Immediate next steps

Name every “data” initiative as operational or analytical before it takes budget. If it must answer a customer or a risk engine inside a session or an authorisation window, it is operational. If it must be curated, reconciled and defensible under deep analysis or regulatory return, it is analytical. Ambiguous programmes that claim both without dual architecture are the ones to challenge first.

Stop new hot-path features on the warehouse. Existing compromises can be retired on a plan; new ones should require an explicit exception with a named latency SLA the warehouse path demonstrably meets. Stand up dual fan-out from the event bus: one consumer path into the analytical estate, one into MongoDB or the chosen operational store, with lag and idempotency monitored as first-class controls. Measure latency to decision for NBA and fraud, not only pipeline completeness and mart coverage. Align headcount: analytical tribes on curated work, product and operational engineering on the hot path.

Clinging to a single warehouse-shaped answer for every problem labelled “data” is no longer the safe option. Streaming already split the paradigm. Banks and insurers that put Snowflake where curation and depth belong and MongoDB (or equivalent) where the customer and the decision live will stop paying analytical headcount to fail operational SLAs. Those that keep one tribe and one store for both jobs will keep discovering, after the loss and after the abandoned session, that the warehouse answered a question the customer never asked.