Edge Intelligence
Inference is quietly migrating off the datacenter and into the device in your hand — and the builders who understand the new latency, privacy and cost math will own the next four years.
The first time I traced a "fast" AI feature end to end, the number that killed me wasn't compute. It was geography. A voice assistant on a phone in Mumbai was sending audio to a GPU cluster in Virginia, getting a transcription, calling a model, and shipping tokens back across an ocean and a continent. The model itself ran in roughly 80 milliseconds. The round trip — TLS handshake, queue wait, two intercontinental hops, the inevitable retry when one packet sulked — ate 600. We had optimized the brain and ignored the nervous system. Everyone does.
Cloud inference has a sticker price and a real price, and they are not the same. The sticker price is per-token billing: clean, predictable, easy to put in a spreadsheet. The real price is the round trip nobody budgets for. Every request leaves the device, climbs the radio stack, crosses your carrier's network, hits a load balancer, waits in a queue behind whoever else landed on that replica, runs, and comes all the way back. On a good day with a warm connection that's 100–150ms of pure overhead before a single token is generated. On a bad day — congested cell, cold autoscale, a region failover mid-request — it's seconds. And users don't experience your median. They experience your worst moments, repeatedly, and they remember those.
Here's the part that separates the demo from the product. Benchmark your cloud feature at batch size one on a clean office Wi-Fi connection and you'll get a number you're proud of. Ship it to a few million phones on real networks and watch the distribution fan out. Your p50 might be a respectable 300ms. Your p99 — the slowest one in every hundred — is 2.8 seconds, because that's the user on a train, behind a TLS renegotiation, hitting a replica that just got three other requests at once. Tail latency isn't an edge case at scale; it's a guarantee. At ten requests per session, a 1-in-100 stall means most sessions hit one. The cloud doesn't make your model slow. It makes your model unpredictable, and unpredictability is the thing UX cannot hide.
Then there's the liability nobody puts in the latency budget. Every one of those round trips is a decision to take a user's words, photos, location, medical question, or half-finished legal document and copy it onto a machine you operate in a jurisdiction they've never heard of. You are now the custodian of that data — for the retention window, for the breach, for the subpoena, for the regulator who wants to know exactly where European health queries are processed. The cheapest data to secure is the data that never left the device. For a growing class of workloads, shipping user input to a GPU farm isn't a performance choice. It's a risk you've quietly underwritten on the user's behalf, and the math on that risk is getting worse, not better.
So here's the thesis this story is built on. For years the device was a thin client and the cloud was the brain — phones were too weak, models too fat, and offloading was simply correct. That assumption is expiring. Phones now ship with neural accelerators measured in tens of TOPS. Quantization has dragged capable models down from "needs a datacenter" to "fits in a few gigabytes of unified memory." And the three forces above — round-trip latency, p99 unpredictability, and privacy liability — all point the same direction: toward the device, not away from it. The cloud isn't disappearing. But for an entire category of inference, it's about to become the exception you reach for, not the default you assume. The builders who internalize that inversion early — who learn the new latency, cost, and privacy math before their competitors — are going to own the next four years. The ones who don't will keep optimizing the brain while the nervous system bleeds them dry.
What does it actually look like when the model moves into your hand? That's where we go next.
Narrow section
Cloud inference has a sticker price and a real price, and they are not the same. The sticker price is per-token billing: clean, predictable, easy to put in a spreadsheet. The real price is the round trip nobody budgets for. Every request leaves the device, climbs the radio stack, crosses your carrier's network, hits a load balancer, waits in a queue behind whoever else landed on that replica, runs, and comes all the way back. On a good day with a warm connection that's 100–150ms of pure overhead before a single token is generated. On a bad day — congested cell, cold autoscale, a region failover mid-request — it's seconds. And users don't experience your median. They experience your worst moments, repeatedly, and they remember those.
Inference is quietly migrating off the datacenter and into the device in your hand — and the builders who understand the new latency, privacy and cost math will own the next four years.
This article explores a hypothetical scenario where OpenAI pivots competitive focus from chatbot quality to automated vulnerability patching. It examines measurable security outcomes, structural risks of automated patching adoption, and the lock-in dynamics that could emerge as enterprises adopt AI-driven security tools.
Inference economics—not model quality—will determine which AI products survive 2026. The fundamental tradeoff between per-token cost and p99 latency is locked in physics: builders can optimize for low cost, low latency, or high throughput, but not all three simultaneously. Most products are priced at the cheap end while their UX demands the expensive end.
Astroscale's ongoing funding needs expose a critical gap in satellite servicing commercialization: the capital required to move from one-time demonstrations to recurring, profitable operations. This isn't a failure of the technology—it's the structural reality of asset-heavy space infrastructure businesses building predictable revenue cadences.
The first time I watched a 7-billion-parameter model generate fluent text on a laptop with the Wi-Fi switched off, I did the thing every engineer does: I assumed I'd misconfigured something and it was secretly hitting an endpoint. It wasn't. The model was running on an M2 MacBook Air — a fanless machine, the kind you buy your parent — pulling tokens out of thin air at a pace fast enough to read along with. The cloud, for that moment, had nothing to do with it. The brain was in my backpack.
Here's the part the spec sheets bury: for single-stream local inference, you are almost never compute-bound. You're memory-bound. Autoregressive generation reads the entire set of model weights from memory for every single token it produces. So the ceiling on your tokens-per-second isn't FLOPS — it's how fast you can stream weights out of memory. Do the napkin math. A 7B model quantized to 4 bits is roughly 3.8GB of weights. An M2's unified memory delivers about 100GB/s of bandwidth. Divide and you get a theoretical ceiling near 26 tokens/second; in practice you'll see high-teens to low-twenties after overhead. An M2 Max, at ~400GB/s, quadruples that ceiling. Same model, same quantization — the only thing that changed is how fast the silicon can move bytes.
This is why Apple Silicon blindsided everyone. Unified memory means the CPU, GPU and Neural Engine all read from the same pool — no copying a 4GB tensor across a PCIe bus to feed a discrete GPU, no agonizing over whether your model fits in 8GB of VRAM while 32GB of system RAM sits idle next to it. Qualcomm's Snapdragon X chips and the latest flagship phone SoCs are chasing the same architecture: a dedicated NPU (neural processing unit — a matrix-multiply accelerator tuned for low precision) sitting next to a wide, shared memory bus. The NPU does the heavy linear algebra at single-digit watts; the unified memory keeps it fed. That combination is what turns "technically possible" into "runs while you're on battery in a coffee shop."
None of this matters if shrinking the model destroys it — and for years, that was the catch. Then llama.cpp and the GGUF format turned quantization from a research curiosity into a Tuesday-afternoon workflow. The trick is that not every weight deserves the same precision. Modern quantization schemes — k-quants, and now the imatrix-calibrated variants — spend their precious bits where the model is sensitive and starve the parts that aren't. A 4-bit quantization (call it ~4.5 bits effective once you account for the more important layers) of a good 7–8B model lands within a hair of the full 16-bit version on most evals, while cutting the memory footprint — and therefore the bandwidth bill on every token — by roughly 4x. On the Apple side, the MLX framework does the same dance natively, with the added luxury of keeping everything in that unified pool.
So what can you actually do with this today, on hardware a normal person owns? A 3B model — Phi-3-mini, a small Llama, a Gemma — fits in well under 3GB quantized and will hold a coherent conversation, summarize a document, draft an email, or power autocomplete on a mid-range phone. An 8B model on a current MacBook handles RAG over your local files, code completion, and classification tasks at a quality that would have been a flagship cloud API eighteen months ago. The honest caveat: these aren't GPT-4-class reasoners, and they'll fall apart on long multi-step logic. But for the high-frequency, latency-sensitive, privacy-sensitive 80% of what people actually ask software to do, the gap has narrowed from "embarrassing" to "you might not notice."
The silicon caught up to the ambition faster than almost anyone forecasted. Which raises the question the next chapter has to answer — because once the brain can live on the device, the economics and the privacy math stop being theoretical, and the people writing the checks start asking why they're still paying per token for things that could run for free in your pocket.
The demo always works. Someone holds up a phone in airplane mode, types a question, and a 3-billion-parameter model answers in under a second. The room claps. What the demo never shows you is the second question, asked ninety seconds later, when the device has warmed to the temperature of a regrettable life decision and the throttle governor has quietly clipped your clock speed by 40%. Local inference is not hard because the first token is hard. It's hard because the thousandth token, on a thermally soaked phone, with a half-full KV cache and a background app fighting you for RAM, is a completely different machine than the one in the keynote.
Start with the most boring number, because it's the one that kills products: model-load latency. A 4-bit quantised 7B model is roughly 4GB on disk. Memory-mapping that into a constrained address space, paging weights in on first access, and warming the compute graph is not free — on a mid-tier device you can spend two to five seconds before the first forward pass even begins. Users will not wait. So you keep the model resident, which means you're now holding multiple gigabytes hostage against an OS that is aggressively looking for memory to reclaim. On iOS, that's the jetsam killer; cross a threshold and your process is simply gone, and your "instant" assistant now has a cold start every time the user switches back from Messages. The engineering work isn't the inference. It's negotiating with the operating system for the right to stay alive.
Here's the part the parameter count hides. Your weights are a fixed cost, but the KV cache — the stored keys and values for every token in context, the thing that lets you avoid recomputing the whole prompt on each step — grows linearly with sequence length. On an 8GB phone where the OS already claimed three-plus and your weights ate four, an 8K-token context can push you off the cliff entirely. The honest answers are unglamorous: shorter context windows, grouped-query attention to shrink the cache, cache quantisation down to int8 or int4, and eviction policies that throw away the oldest tokens and pray the conversation didn't need them. Every one of those is a quality trade. You are choosing, explicitly, to make the model slightly worse so that it runs at all — and that trade is invisible in every benchmark published at batch size one on a tethered, fan-cooled dev board.
Sustained generation is where the laptop demo lies hardest. A phone SoC can hit impressive tokens-per-second for a burst, but it cannot dissipate that heat indefinitely, and there is no fan. Run continuous decode for ninety seconds and the throttle governor steps your performance cores down to protect the silicon — I've watched real throughput fall by a third to a half mid-paragraph, the generation visibly stuttering as the chip cooks. Battery is the same story told in coulombs: heavy local inference is one of the most power-hungry things a phone does, and "I drained 18% writing two emails" is a churn event, not a feature. This is precisely why the interesting metric isn't peak throughput — it's sustained throughput under thermal load, the on-device equivalent of p99 tail latency. Architecture decisions get paid for in the second minute, not the first second.
And you get to fight all of this across a fractured toolchain. Core ML gives you the Apple Neural Engine and genuinely good power efficiency — if you can coax your model through the conversion and accept that you don't fully control what runs where. ONNX Runtime is your cross-platform hedge, broad and pragmatic, weaker on bleeding-edge ops. ExecuTorch is PyTorch's serious bet on edge, promising and still maturing. And llama.cpp remains the workhorse that proved this was possible at all, with the quantisation formats everyone borrows and the portability nobody else matches. There is no single runtime that is best on every device, every model, and every OS version, which means a real local product ships multiple inference backends and a routing layer to pick between them. That is the unglamorous truth: "running local" is a portfolio, not a switch.
So be honest about the boundary. Local owns the work that is small, frequent, latency-sensitive and private: autocomplete, summarisation of text already on the device, intent classification, the structured glue between a user's tap and a server's heavy lift. It breaks the moment you need a frontier-class model, a 100K-token context, or sustained reasoning that would melt the phone. The win isn't choosing a side — it's a router that runs the cheap, instant, private path on-device and escalates to the cloud only when the task actually demands it, ideally without the user ever knowing a network hop happened. Get that routing logic right and you get the best of both: the privacy and instant-feel of local, the raw capability of the datacenter, and a cost curve that doesn't bankrupt you. Get it wrong and you've built the worst of both — a phone that's hot, slow, and still calls home.
That router is the real product. And in the next chapter, we follow the money to understand why the people building it are about to reshape the entire economics of inference.
There's a number every AI company learns to fear, and it lives in a spreadsheet cell labeled "COGS per active user." For a cloud-served LLM feature, that cell is a slow leak in the hull. Every query you serve costs you GPU-seconds you rent from someone else — and unlike storage or bandwidth, that cost doesn't amortize as you scale. It compounds. Ten million daily users sending ten queries each is a hundred million inferences a day, and at even a fraction of a cent each, you're watching your margin get eaten by the very engagement your growth team is paid to manufacture. The dirty secret of the generative-AI gold rush is that success and bankruptcy point in the same direction.
Now move that inference onto the device. The arithmetic inverts. When a 3-billion-parameter model runs on the user's A18 or Snapdragon 8 Elite, the marginal cost of the ten-millionth query is the same as the first: zero to you. The user paid for the silicon when they bought the phone. They pay for the electricity. Your COGS-per-inference line, the one that used to climb with every successful product launch, flattens against the floor. This is not a rounding error — it's a different business. A consumer app that bleeds money at scale in the cloud becomes a fixed-cost software business again, the kind investors understood before "tokens" entered the vocabulary. You ship the model once in an app update; from then on, usage is free.
It's not free everywhere, and pretending otherwise is how you ship a feature that melts batteries and torches your App Store rating. The crossover math has three variables: model size, query volume, and quality tolerance. If you need a frontier 400B model for legal reasoning, the cloud still wins — no phone runs that, full stop. But the band where on-device makes sense is widening fast, because the workloads people actually do all day — summarize this thread, rewrite this email, classify this receipt, answer from these documents — are increasingly served well by 3B-to-8B quantized models. The honest way to find your crossover is to plot annual cloud inference spend per cohort against the one-time engineering cost of an on-device pipeline. For a high-engagement consumer feature, that line crosses inside a quarter. For a feature three users touch a month, never bother. The skill is knowing which one you're building before you've spent the year finding out.
Then there's the cost that never shows up as a cost: data you no longer have to hold. When inference happens on-device, the user's prompt — their medical question, their half-drafted resignation letter, their kid's homework — never crosses your network boundary. There's nothing to log, nothing to leak, nothing a subpoena can compel, nothing a breach can spill. For regulated enterprise, this rewrites the compliance story from "we encrypt and audit and pray" to "the data physically never left the endpoint." A hospital deploying on-device transcription isn't asking for a BAA and a residency guarantee; there's no third party in the loop to govern. GDPR, HIPAA, the EU AI Act's data-handling provisions — a whole category of risk simply evaporates when the brain is local, and the legal team that used to be your slowest gate becomes the team that approves your launch.
And it works on a plane. It works in a tunnel, in a rural clinic with one bar of signal, in a factory basement, during the next regional cloud outage that takes half the internet's AI features down with it. Offline resilience used to be a nice-to-have bullet point; in an era where the cloud is the product's intelligence, it becomes the difference between a feature and a dependency. The builders who internalize all of this — zero marginal cost, vanishing data liability, latency you control, uptime you don't outsource — aren't just shaving a line item. They're discovering an entirely new unit-economics surface, one where the device does the expensive work and the cloud becomes optional infrastructure rather than the beating heart. Which raises the obvious next question: if the economics are this lopsided, why hasn't everyone already moved? The answer lives in the engineering — and it's harder than the spreadsheet makes it look.
Here is the prediction I'll stake my name on: in four years, "calling the cloud" will feel like calling a database — something you do deliberately, for a specific reason, not the default posture of every interaction. The architecture that wins isn't on-device or cloud. It's a router. A small, fast, ruthlessly specialized model sitting on the NPU handles the common case — the autocomplete, the summarization, the "rewrite this more politely," the intent classification that decides what you actually want. It answers in 40 milliseconds because it never left the silicon. And for the hard tail — the genuinely novel reasoning, the 70B-parameter judgment call, the thing your 3B local model knows it can't do — it escalates. The interesting engineering is no longer "which model," it's "which model when," and getting that routing decision wrong by even ten percent is the difference between a magical product and an expensive one.
The part builders consistently underestimate is on-device fine-tuning. Right now personalization means stuffing your history into a context window and paying for those tokens on every single call — a tax that scales linearly with how well the product knows you, which is exactly backwards. The on-device steady state inverts this. A LoRA adapter — a small set of trainable weights bolted onto a frozen base model — gets nudged overnight, on your phone, on your data, while it charges on the nightstand. No round trip. No data egress. The model that wakes up tomorrow is measurably more yours than the one that went to sleep, and it cost the platform nothing because the compute was yours and the electricity was yours. This is the agent that actually lives in your pocket: not a chatbot with a cloud umbilical, but a resident process that knows your calendar, your writing voice, your codebase conventions, and never once shipped them to someone else's GPU. Privacy stops being a compliance checkbox and becomes the architecture — which, conveniently, is also the cheapest place it's ever been.
If you're designing today for the world that exists in 2028, three decisions compound in your favor. First, instrument your escalation path before you need it. Treat the cloud call as the exception and log every time the local model punts — that escalation rate is your single most important cost metric, and you want a year of data on it before it matters. Second, design your data layer for local-first. If personal context lives on the device and syncs selectively, you've pre-built for on-device inference whether or not you've shipped it; if it all lives in your cloud, you've architected a migration you'll dread. Third, quantize and measure now — pull a 3B or 7B model down to 4-bit, run it on an actual phone, and find out what your common case really costs in latency and battery. The numbers will be worse than the demo and better than you fear, and either way you'll stop guessing.
The builders who internalize this aren't waiting for a hardware moment that's already half-arrived. They're shipping the router, logging the tail, and quietly moving the brain closer to the user one workload at a time. The cloud doesn't disappear — it gets promoted, reserved for the genuinely hard problems instead of squandered on autocomplete. And that reallocation, more than any single model release, is what the next four years are actually about.
We've spent this series taking the brain out of the datacenter and putting it in your hand — the silicon that made it possible, the quantization that made it fit, the latency and cost math that made it inevitable. But a model on a device is still just a model. The thing that turns it into a product — that decides, that acts, that earns the word "agent" — is what we build on top. That's where we're going next.
