A payment authorisation, a claims intake or an in-play stake still has to clear a distant model hop before anyone acts. Session context leaves the region. Tokens travel across a WAN to a GPU farm sized for peak and idle most of the week. By the time the score returns, the fraud window has narrowed, the market has moved or the customer has abandoned the journey. That pattern is now the default architecture in too many regulated platforms: generative capability treated as a remote dependency rather than a local decision surface. The network that already sits in front of the application is being asked only to forward the problem.
The commercial damage is not abstract. Card authorisation chains often leave roughly 100 milliseconds for the full round trip of routing, fraud scoring and issuer response; model inference may be allotted only 10 to 50 milliseconds of that budget. A decision that arrives after settlement is documentation, not prevention. In wagering, a responsible-gambling intervention or bonus-abuse check that lands after the price has moved is theatre. In insurance, a first notification of loss that cannot triage near the customer pushes leakage and manual queue into the centre. Keeping session context in a far-away region also creates a privacy and conduct story that boards increasingly dislike: data leaves the jurisdiction by default because the model does.
The cost of treating models as a remote dependency
Distant inference taxes latency, locality and governance at once. Every extra hop compounds variance under load. Feature stores and policy engines that already sit close to the user must wait on a remote GPU cluster before they can act. Teams then paper over the gap with coarser rules, higher false declines or post-event review queues. None of those options recovers the millisecond window in which the decision actually matters.
The operational cost shows up as capacity left idle. Private GPU fleets are sized for peaks that arrive unevenly across products and geographies. Tokens sit unused while capital is locked in hardware, power and specialist ops. Model vendors multiply keys, logging formats and rate-limit surfaces. Security and risk teams inherit a mesh of control planes instead of one. When regulators ask where customer data went during an assistant call, the answer spans regions and contracts rather than a single isolate that already enforced authentication and policy.
Doing nothing therefore means accepting slower fraud and step-up decisions, assistants that export session context by design and AI programmes that look innovative in slide decks while remaining architecturally peripheral to the request path. The alternative is not another chatbot bolted onto a portal. It is putting a small, well-governed model next to every high-value request the customer already makes.
Inference that lives with the request
Cloudflare’s edge is no longer only a CDN and security mesh. Workers run application logic across a network Cloudflare reports as spanning more than 330 cities in over 125 countries, with roughly 95 per cent of the world’s Internet-connected population within about 50 milliseconds of a data centre. That same fabric now hosts models, persistent agents and vector retrieval. The claim worth taking seriously is architectural: AI stops being a distant farm you call over a WAN and becomes a capability that executes in the same isolate that already authenticates the caller, routes the request and enforces policy.
The technical path is concrete without being a product brochure. Workers compile Rust to WebAssembly and deploy it as ordinary edge code through workers-rs. Performance-critical logic (risk rules, pricing kernels, session state, deterministic checks) sits beside generative calls rather than behind another hop. Wasm gives isolation and predictable cold-start behaviour that regulated firms already understand from isolate-based multi-tenancy. On the model side, Workers AI hosts a growing catalogue of open and partner models on Cloudflare’s own GPUs. The unified AI binding and AI Gateway let teams call frontier models such as Grok from xAI through the same env.AI.run path: one control plane, Cloudflare billing, caching, rate limits and observability, without a separate key sprawl.
The quiet proof that inference itself is being rewritten for this network rather than bolted onto Python serving stacks is Infire. Cloudflare’s engine is written in Rust for its distributed hardware and security constraints. Published benchmarks against vLLM 0.10.0 on unloaded H100 NVL hardware showed Infire completing tasks up to about 7 per cent faster while using a fraction of the CPU; under constrained production-like conditions the gap widened further because Infire can run as a trusted process without the gVisor overhead that Python stacks require on the edge. That is engineering heat with a board-readable consequence: more tokens served per watt and per square metre of already-deployed network, closer to the user.
“Close to the application, wherever it is distributed” then means something precise. Inference can happen in the city that originated the request. Data need not leave the region as a default. The same Worker that fronts internet banking, a claims portal or an in-play book can score, retrieve, generate and act before the payload ever reaches a central cluster. Durable Objects keep agent state and session coordination near that path. Vectorize stores embeddings for retrieval-augmented checks without shipping every document to a distant vector farm. For banks that is faster fraud and step-up decisions, in-channel assistants that do not leak session context to a far-away region and document intake that stays near the customer. For insurers it is first notification of loss triage, policy Q&A and underwriting assist at the portal edge. For wagering operators it is responsible-gambling interventions, bonus abuse checks and personalisation inside the latency budget of a live event, not after the market has moved.
Why the edge moment is now
Three conditions have converged. First, the runtime is mature enough for regulated production: V8 isolates, Wasm and workers-rs give a path for both deterministic Rust logic and generative calls in one deployment unit. Second, the model layer is no longer a science project bolted sideways. Workers AI and AI Gateway have been unified into a single control plane, so hosted open models and partner frontier models share observability, credits and policy. Third, the inference engine underneath is being rewritten for the network’s shape. Infire exists because centralised Python serving did not match a lean, widely distributed GPU footprint. That is a signal that the platform vendor is investing in the hard part rather than renting you capacity in someone else’s region.
Policy and commercial pressure reinforce the timing. Locality expectations for privacy, conduct and operational resilience are tightening across banking, insurance and wagering. Boards are tired of AI line items that require private GPU fleets sized for peaks and idle the rest of the week. Token billing and a shared security surface are easier to govern than a mesh of vendor keys. Competitors who put scoring and retrieval on the request path will decide inside the window; those who keep calling a distant farm will keep documenting losses after the fact.
Waiting for a perfect central AI platform before moving decision logic to the edge embeds the wrong default. Every month spent treating models as remote services teaches product teams to design around round trips instead of around the isolate that already holds the request.
A pragmatic rollout that starts with high-value decisions
Start where latency is margin and governance already exists. Map the requests that already authenticate, authorise and log at the edge: payment step-up, fraud score enrichment, claims FNOL classification, policy Q&A, responsible-gambling prompts, bonus eligibility. Choose one or two where a distant model hop clearly burns the budget or forces session context out of region. Keep the deterministic core in Rust via workers-rs: rules, thresholds, pricing kernels and audit-friendly checks that must behave the same every time. Bind generative or embedding calls through env.AI.run so the same Worker owns both paths.
Instrument before you expand. Route inference through AI Gateway from day one so latency, token spend, errors and prompts sit on one logging surface. Prefer hosted open models on Workers AI for classification, summarisation and embeddings where quality is sufficient; reserve frontier calls such as Grok for tasks that need them, still via the same binding. Store retrieval corpora in Vectorize only for the documents that must stay near the journey. Put agent or session continuity on Durable Objects when the interaction spans turns, not for every stateless score.
Success criteria should be operator-readable: p95 decision latency against the existing central path, false-positive and false-negative movement on a shadow cohort, proportion of requests that never leave the region and cost per thousand decisions versus the idle GPU baseline. A compact team of edge, security and model-ops specialists can own the first path without boiling the ocean. Fall back to the central cluster remains available until the edge path earns trust under peak load.
Scaling from single decisions to agents that persist
Once a few high-value scores run next to the request, expand carefully into multi-step work. Durable Objects give each agent or customer session a durable identity and local state so an in-channel assistant, claims triage workflow or responsible-gambling coach can continue without reconstructing context from a distant database on every turn. Vectorize then supports retrieval that stays on the same network as the Worker: policy documents, product terms, historical claim patterns or market rules queried as embeddings generated through Workers AI.
Frontier models enter through the same control plane rather than a new vendor island. AI Gateway’s path to Grok from xAI, alongside other providers and Workers AI catalogue models, means product teams can switch or fail over without rewriting key management. Caching and rate limits become platform controls instead of application folklore. Security reviews a single surface for prompts, responses and token attribution. That is how you scale capability without scaling key sprawl.
Transactional expansion still demands discipline. Keep write-heavy ledgers and final settlement where they belong. Use the edge to decide, enrich, retrieve and intervene early; keep the system of record authoritative. Observability and automated rollback matter more as generative variance enters the path. Shadow traffic, canaries and explicit human-in-the-loop bands for high-impact actions keep the architecture honest. The goal is not to move the bank, the insurer or the book onto a chatbot. It is to make every high-value request slightly smarter, faster and more local than it was last quarter.
The board case: tokens, locality and one control plane
The engineering heat is real: ship Rust to hundreds of cities, bind Grok or a hosted open model in a few lines, keep agents on Durable Objects, store embeddings in Vectorize. The commercial case is sharper. There is no private GPU fleet to size for peak and watch sit idle. You pay for tokens used rather than capacity reserved. One security and logging surface replaces a mesh of model vendors. Time-to-decision falls where latency is margin: trading-adjacent servicing, claims leakage, in-play hold, authorisation fraud windows measured in tens of milliseconds.
Locality is not a slogan. Inference in the city of origin and data that need not leave the region by default strengthen privacy, conduct and resilience narratives that auditors already ask about. Operational resilience improves when the decision path does not depend on a single distant region staying healthy. Talent arguments follow: teams learn to ship governed edge logic rather than integrate yet another remote API. The winners will not be the firms that “add a chatbot”. They will be the ones that put a small, well-governed model next to every high-value request their customers already make.
Immediate next steps for technology leaders
Commission a short inventory of request paths where a distant model hop already burns the latency or locality budget. Pick one bank, insurer or wagering journey with clear commercial stakes and an existing edge front door. Stand up a Worker that keeps deterministic checks in Rust via workers-rs, binds inference through env.AI.run and AI Gateway and measures shadow performance against the central path. Decide explicitly which models stay on Workers AI, which frontier calls such as Grok are justified and how Durable Objects plus Vectorize will be used only where state and retrieval earn their keep.
Give a small cross-functional team authority to ship that path under security and risk oversight, with token budgets and regional data rules written down before traffic rises. Review results in weeks, not quarters: latency, decision quality, regional retention of sensitive context and cost per decision versus idle GPU alternatives. Expand only what clears those gates.
Treating AI as a distant GPU farm you call over a WAN is the low-ambition default. The network in front of the application can already authenticate, route, enforce policy and now infer. Technology leaders who move decision logic into that isolate will define the operating standard for the next product cycle. Those who wait will keep explaining why the score arrived after the window closed.