WHAT IS EDGE AI?

• 3B parameters on-device, plus a 20B sparse model held in flash, in Apple’s third-generation Foundation Models (June 2026).
• 5B and 8B raw parameters squeezed into 2 GB and 3 GB of memory by Google’s Gemma 3n (May 2025).
• 0.24 Wh per median Gemini Apps text prompt, per Google’s own serving measurement (August 2025).
• 415 TWh of data-centre electricity in 2024, projected to reach about 945 TWh by 2030 in the IEA’s Base Case (April 2025).
• About a quarter of on-device attack papers target model theft, while half of defence papers do, per the May 2026 systematic review of on-device inference security.

What Is Edge AI and How Does It Differ from Cloud AI?

Edge AI is machine-learning inference, and sometimes training, that runs on or near the device producing the data instead of inside a central cloud data center. Cloud AI ships every input across a network to a remote cluster and waits for the answer to return. Edge AI keeps the model next to the sensor, the phone or the factory floor, so the data stays put and the round trip disappears.

The idea is older than the chatbot era. A 2019 Proceedings of the IEEE survey called edge intelligence the “last mile” of AI: billions of phones and IoT devices were already generating data at the network’s edge, so the sensible move was to bring the models to that data. The 2024 systematic review of edge AI, which covers work from 2014 onward, puts it this way:

Edge AI is a network of interconnected systems and devices that “receive, cache, process, and analyze data in close communication with the location where the data is captured.”

So where exactly is “the edge”? It depends who you ask. In practice it’s a spectrum with four tiers:

ETSI’s Multi-access Edge Computing terminology standard, dated January 2019, defines that last tier as an IT service environment with cloud-computing capabilities at the edge of the access network, close to the users it serves. That’s a very different thing from a 100 kB model on a thermostat, yet both get called edge AI.

One warning on vocabulary. Plenty of writers use “edge computing” and “edge AI” as if they were the same term. Others treat edge AI as a subset of edge computing, and the 2024 review folds fog and cloud tiers into its taxonomy as well. When a vendor promises “AI at the edge,” ask which tier they mean: the device itself, a server down the hall, or a rack inside a carrier’s network. The engineering trade-offs are different at each one.

How Edge AI Works: Inference on Devices, Gateways, and Local Servers

Edge AI works by training a model where compute is plentiful, shrinking it until it fits the target hardware, and then running inference locally through a runtime that drives the device’s CPU, GPU or neural processing unit. Training almost never happens at the edge. Inference almost always does.

Most production deployments follow the same five-stage pipeline:

1. Train or fine-tune in the cloud. The full-precision model is built on data-center GPUs, often as a smaller sibling of a larger model. If you need the model to behave a specific way on your own data, this is where fine-tuning an LLM happens, well before anything touches a device.
2. Compress the model. Quantization lowers the numeric precision of the weights. Pruning removes connections that contribute little. Apple trains its 2026 on-device models with quantization-aware training, so the model learns to tolerate the lower precision instead of being squeezed afterwards. Google’s Gemma 3n preview from May 2025 uses a trick called Per-Layer Embeddings to fit 5B and 8B raw-parameter models into roughly 2 GB and 3 GB of memory, the footprint of 2B and 4B models.
3. Package it for a runtime. The compressed model is converted into a format an on-device engine can load. Google’s LiteRT (the renamed TensorFlow Lite) and LiteRT-LM fill this role for Gemini Nano and Gemma across phones, laptops, Chrome and Pixel Watch as of September 2025. On Android, AICore manages the model and isolates each request.
4. Execute on local silicon. The runtime schedules work across the CPU, GPU and, where present, the NPU. Which one gets used depends on the operator types in the model and what the device driver supports.
5. Send only what needs to travel. A compact result (“person detected”, a transcript, a classification) goes upstream. In federated learning the device trains locally and sends only model updates to an aggregator, never the raw data.

Stage two is where first deployments stall. Teams expect compression to be a checkbox and discover it’s a measurement loop: quantize, test accuracy on your own data, adjust, repeat. Budget engineering weeks for it.

At the far end of the spectrum sits TinyML. The MLPerf Tiny benchmark, published in 2021 with more than 50 industry and academic partners, defines the regime as models of roughly 100 kB and below running on sensor data, with keyword spotting, visual wake words and tiny image classification as its reference tasks. A 2023 IEEE Circuits and Systems Magazine review from Song Han’s group identifies memory as the main obstacle at this scale: architectures designed for phones or the cloud simply don’t fit on a microcontroller, so the algorithm and the system have to be designed together, the approach behind MCUNet.

Which Real-World Problems Edge AI Is Built to Solve

Edge AI exists to solve four problems that cloud inference handles badly: slow or absent networks, data that can’t leave the device, bandwidth too expensive to fill with raw sensor streams, and per-query bills that pile up on repetitive work. The surveys list all four as motivations. Treat them as design arguments rather than guarantees, because each holds only when the engineering matches the workload.

Latency and offline operation. A vehicle’s camera can’t wait for a data-center round trip, and a wake-word model has to answer in milliseconds while listening all day on a battery. Android’s Gemini Nano documentation, updated September 2026, states that prompts run locally and need no network connection. Keyword spotting and visual wake words, the two headline MLPerf Tiny tasks, are the purest examples: tiny models that must respond instantly and never phone home.

Privacy. The raw data never leaves the device. The same Android documentation says AICore isolates each request and keeps no inputs or outputs after processing. Federated learning stretches this argument to training. A 2025 survey of privacy-preserving federated learning describes clients that train locally and send only model updates to an aggregator, while flagging the open problems: non-identically-distributed data across devices, hardware heterogeneity, communication overhead, and the continuing need for differential privacy and secure aggregation on top.

Bandwidth. A camera running a visual wake word model sends “person present” instead of a video stream. Only compact results travel, which matters most when thousands of sensors share one uplink.

Per-task cost on repetitive work. NVIDIA Research’s June 2025 position paper argues that small language models are 10 to 30 times cheaper and lower-latency than large ones for repetitive agentic tasks, and that they can run on consumer devices. That figure is an argued position, with no measured benchmark behind it, so quote it as a claim rather than a result.

Notice what’s missing from that list: better answers. Nobody moves a model to the edge to get smarter output. They move it to get faster, cheaper or more private output from a model that is good enough for the task, and “good enough” is a measurement you have to make yourself.

What Does Edge AI Actually Cost to Deploy and Run?

Edge AI shifts cost from a recurring per-query bill to an upfront engineering and hardware spend, and whether that trade pays off depends on query volume and how long the device stays in service. There is no single price tag. What you can do is separate the two cost buckets and put numbers on each for your own workload.

Upfront costs land before the first inference runs:

• NPU-capable hardware, or a fleet refresh if the devices you already own can’t run the model.
• Model compression work: quantization, pruning and the accuracy testing loop around them.
• Toolchain friction: every tier has its own runtime, converter and driver quirks, and the 2023 IEEE review of TinyML from Song Han’s group lists thin compiler and inference-engine support on bare-metal devices as a standing problem.

Recurring costs are where the cloud bill lives: per-query inference charges, bandwidth for moving raw inputs upstream, and the energy behind both.

How much does one cloud query cost in energy terms? Google’s August 2025 measurement of its own Gemini serving stack puts the median Gemini Apps text prompt at 0.24 Wh, 0.03 gCO2e and 0.26 mL of water. Tiny per prompt. Multiply by a million repetitive requests a day and it stops being tiny, which is the arithmetic behind every on-device business case.

The system-level backdrop is large and growing. The IEA’s April 2025 Energy and AI analysis estimates data-centre electricity use at about 415 TWh in 2024, roughly 1.5% of global demand, and projects about 945 TWh by 2030 in its Base Case.

Two cautions on using those figures. The IEA gives no edge-versus-cloud comparison, and as of 2026 no rigorous public study does, so “edge is greener” remains an assumption rather than a finding. Google’s 0.24 Wh is also a vendor’s own number for its own stack, so it won’t convert cleanly into a per-device watt budget. Market-size forecasts for on-device AI circulate widely too, but they come from analyst and vendor reports. Read the original before one goes into a board deck.

Edge AI vs Cloud AI vs Hybrid: Choosing the Right Architecture

The right architecture for most production systems is hybrid: a small model on the device handles frequent, latency-sensitive or private tasks, and a large cloud model takes whatever the small one can’t. Pure edge and pure cloud both exist, but each is the right answer for a narrower set of workloads than vendors admit.

Apple’s third-generation Foundation Models, announced June 8, 2026, show the pattern in a shipped product. A 3B-parameter model runs on the device, a 20B-parameter sparse “Core Advanced” model keeps its weights in flash and activates 1 to 4B parameters at a time, and anything heavier goes to Private Cloud Compute. That’s three tiers of capability behind one assistant, with the user never choosing which one answers.

The academic picture matches. A July 2025 survey on collaboration between edge small language models and cloud LLMs, accepted at ACM Computing Surveys for 2026, sorts hybrid designs into five families:

• Task assignment: a router sends each whole request to the edge or the cloud.
• Task division: one request is split, with the device handling part and the cloud the rest.
• Token-level mixture: both models contribute to a single output, token by token.
• Speculative decoding: the small model drafts, the large model verifies.
• Resource-aware offloading: routing shifts with battery, thermal state or network quality.

Which family fits you is an engineering decision with real consequences for cost and privacy, and it’s worth settling before any model gets compressed. Teams that skip the routing design tend to end up with an on-device model that answers everything badly or a cloud model that answers everything expensively. An AI integration audit of the actual request mix (what share is repetitive, what share is private, what share needs frontier reasoning) usually settles the question faster than any benchmark table.

One honest caveat. Frontier-scale models stay in the cloud for the foreseeable future, so “hybrid” in practice means the cloud remains in the architecture. Edge removes the cloud from the hot path, and that’s the whole gain.

Where Edge AI Breaks Down: Latency, Power, Accuracy, and Model Drift

Edge AI breaks down at six points: hardware ceilings, a quality gap against cloud models, accuracy lost in compression, fragmented toolchains, training that can’t easily happen on the device, and a model sitting inside the attacker’s reach. Each one is manageable. None is optional.

Hardware ceilings. Memory, power and thermal limits bound what fits and how long it can run. A phone that throttles after ninety seconds of sustained NPU load gives you a latency figure that looks great in a demo and falls apart in a shift-long deployment. Measure under sustained load, with the battery below half.

The quality gap. On-device models are smaller than frontier models and perform like it. Apple’s June 2026 report gives human-preference results only against its own previous generation (45.6% versus 23.3% on text prompts). No vendor source supports parity with large cloud models, and you should assume there isn’t any.

Compression loss. A rule of thumb circulates that 4-bit quantization saves 60 to 70% of memory for 2 to 5% accuracy loss. No primary paper backs that exact figure. A May 2025 systematic evaluation of on-device LLMs studies the quantization, performance and resource trade-offs directly, and the lesson is that loss varies by model and task. Test on your data.

Fragmented toolchains. Compiler and inference-engine support for bare-metal devices stays thin, per the 2023 TinyML review, and each hardware tier has its own runtime quirks.

Learning on-device. Federated learning carries heterogeneity and communication costs, as the 2025 federated learning survey lays out. A frozen on-device model also drifts quietly: nobody sees the accuracy slide because nobody is logging the inputs. Plan a feedback channel before launch.

The model in the attacker’s hands. A May 2026 systematic review of on-device inference security groups threats into model theft and extraction, adversarial attacks, and data breaches. Roughly a quarter of attack papers concern IP theft, while half of all defence papers target it. Adversarial attacks make up about a third of the attack literature and have comparatively few defences. The reviewed countermeasures are trusted execution environments, homomorphic encryption, obfuscation and differential privacy. Shipping the weights means shipping them to everyone, including whoever wants to copy or poison them.

Which Hardware and Frameworks Power Edge AI Today

Edge AI in 2026 runs on three classes of silicon (microcontrollers, the NPUs inside phones and PCs, and Linux boards or on-premises servers) with a thin layer of runtimes and compilers on top. The hardware is the easy part to buy. The software is where the hours go.

At the microcontroller tier, MLPerf Tiny is the yardstick. It was built with more than 50 industry and academic partners in 2021, and it’s the only widely shared way to compare a kilobyte-scale model across chips. MCUNet, from Song Han’s group, is the reference approach for making a network fit at all: design the architecture and the inference engine together.

The NPU is the chip that made on-device language models practical. A figure of 40 TOPS is often quoted as the NPU floor for Microsoft’s Copilot+ PCs, but that number circulates in secondary coverage, so confirm it on Microsoft’s own pages before you design a fleet around it. In practice the TOPS rating matters less than driver coverage. A model that falls back to the CPU because one operator has no NPU kernel runs far slower than the spec sheet promises, and you only find out by profiling.

Above the phone sits the board tier. LiteRT-LM, as of September 2025, targets Android, Linux, macOS, Windows and Raspberry Pi with CPU, GPU and NPU backends, which makes a Pi-class gateway a cheap place to prototype before committing to custom hardware.

The models themselves are now shipped artifacts rather than research previews. Gemini Nano arrives through AICore on Android. Gemma 3n handles audio, image and text. Microsoft’s Phi-4-mini, announced in February 2025, has 3.8B parameters and is built to run locally. Apple’s 3B on-device model ships inside its 2026 foundation model family.

Who Should Adopt Edge AI, and Who Should Stay in the Cloud

Adopt edge AI when a workload is latency-critical, must run offline, can’t let data leave the device, or repeats millions of times a day; stay in the cloud when you need frontier reasoning, want to swap models every week, or have no embedded engineers. Most enterprises have both kinds of workload. The decision gets made one workload at a time.

The strongest edge candidates look like this:

• A warehouse scanner that has to classify labels inside a building with dead Wi-Fi zones.
• A bedside monitor that must flag an anomaly without waiting on the hospital network.
• A document pipeline that runs the same classification on every incoming file, all day, at a volume where per-query cloud pricing hurts.
• A camera fleet where sending video upstream costs more than the insight is worth.
• Any input a regulator or a customer contract says may never leave the premises.

Cloud stays the right answer when the task needs multi-step reasoning a 3B-parameter model can’t deliver, when the model changes faster than a device fleet can be updated, or when nobody on the team has shipped firmware or profiled an NPU before. That last one is underrated. Compression and runtime work is a skill, and hiring for it takes longer than most roadmaps allow.

One adopter surprises people: the SaaS company with no hardware at all. LiteRT-LM has run generative models inside Chrome since September 2025, so a browser-based product already has an edge target without a single device purchase.

A five-question self-assessment:

• Does the task fail if a response takes longer than a network round trip?
• Does it have to work with no connection?
• Is the input data contractually or legally pinned to the device?
• Does the same request repeat at a volume where cloud cost compounds?
• Can a small, task-specific model hit the accuracy bar on your own test set?

Three or more yeses and edge is worth a pilot. Fewer than two and the cloud is cheaper to build and cheaper to change.

What Changed in Edge AI This Year: Smaller Models, NPUs, and On-Device LLMs

Between early 2025 and October 2026 the edge moved from running classifiers to running language models with billions of parameters, and the central question shifted from “does it fit” to “how good is it, and who can steal it.” Four shifts explain the year.

Multi-billion-parameter models on phones became ordinary. Phi-4-mini brought 3.8B parameters to local devices in February 2025. Gemma 3n followed in May 2025, fitting 5B and 8B raw-parameter models into 2 GB and 3 GB of memory. Then Apple’s third-generation foundation models, announced June 8, 2026, paired a 3B on-device model with a 20B-parameter sparse model that activates 1 to 4B parameters at a time and keeps its weights in flash storage. That flash trick is the quiet headline. It moves the ceiling on model size from DRAM to storage, and storage is the one resource phones have in abundance.

Runtimes reached watches and browsers. LiteRT-LM, released in September 2025, put generative models in Chrome, Chromebook Plus and Pixel Watch. A language model on a wrist would have sounded like a joke in 2023.

TinyML grew into tiny deep learning. A June 2025 survey tracks the microcontroller tier moving from classic sensor classifiers toward deep networks, which is exactly what the 2023 IEEE Circuits and Systems Magazine review from Song Han’s group predicted in its closing line:

“Today’s large model might be tomorrow’s tiny model.”

On-device security became its own field. The May 2026 systematic review of on-device inference security is the first to map attacks and defences across the whole stack, and it exposes an imbalance: defences cluster around model theft while adversarial attacks go comparatively unanswered. Expect that gap to shape procurement questions through 2027.

What didn’t change matters too. Frontier-scale models still live in the cloud, and no vendor has shown an on-device model matching one. The edge got dramatically more capable this year. It did not get smart enough to work alone.

Frequently Asked Questions About Edge AI

These are the questions people type into a search box about edge AI, answered in a few sentences each.

Is edge AI the same as edge computing?

No. Edge computing is the broader practice of placing compute and storage near where data is produced, and edge AI is the specific case of running machine-learning inference or training there. Many writers blur the two, and the 2024 systematic review of edge AI even folds fog and cloud tiers into its taxonomy. ETSI’s Multi-access Edge Computing terminology standard defines the telecom-network flavor without mentioning AI at all.

Is edge AI more secure than cloud AI?

It is more private and less secure at the same time. Raw data never leaves the device, which removes a whole class of transit and server-side exposure. The model itself, though, now sits in the attacker’s hands: the May 2026 systematic review of on-device inference security catalogs model theft, adversarial attacks and data breaches as the three threat groups, with trusted execution environments, homomorphic encryption, obfuscation and differential privacy as the defences.

Is edge AI greener than cloud AI?

Nobody has shown that it is. The IEA’s April 2025 Energy and AI analysis puts data-centre electricity at about 415 TWh in 2024 and projects about 945 TWh by 2030, but it offers no edge-versus-cloud comparison, and as of October 2026 no rigorous public study does. Treat “greener” as a hypothesis to test on your own fleet.

Can edge AI run large language models?

Yes, up to a few billion parameters. Microsoft’s Phi-4-mini (3.8B parameters, February 2025), Google’s Gemma 3n (May 2025) and Apple’s 3B on-device model (June 2026) all run locally on phones and laptops. Frontier-scale models still live in the cloud, and no vendor has shown an on-device model matching one.

What is TinyML?

TinyML is machine learning on microcontrollers and sensors, the smallest tier of edge AI. The MLPerf Tiny benchmark, published in 2021, defines the regime as models of roughly 100 kB and below handling tasks like keyword spotting and visual wake words. Memory is the binding constraint, which is why models for this tier are co-designed with the hardware they run on.

Does edge AI need an internet connection?

Pure on-device inference does not. Android’s Gemini Nano documentation, updated September 2026, states that prompts run locally with no network required. Hybrid systems do need a connection for whatever they route to the cloud, and a good one degrades to local-only mode when the link drops.

What is an NPU?

An NPU, or neural processing unit, is a block of silicon built for the matrix arithmetic that neural networks run on, sitting beside the CPU and GPU in phones and PCs. It’s the chip that made multi-billion-parameter models practical on consumer hardware. Its headline TOPS rating matters less than driver coverage, because any operator without an NPU kernel falls back to the slower CPU.

Where to Start with Your First Edge AI Pilot

Start an edge AI pilot with one workload, one device tier and one accuracy number, and let everything else follow from those three choices. Six steps get you from idea to a go or no-go decision:

1. Pick one workload that is latency-bound, privacy-bound or repeats at high volume. Skip anything that needs frontier reasoning.
2. Benchmark it in the cloud first. That gives you the accuracy bar and the per-query cost the edge version has to beat.
3. Choose a target tier and runtime, from microcontroller to on-premises server, based on where the data is produced.
4. Quantize, then measure accuracy loss on your own test set, under sustained load and a half-drained battery.
5. Define the hybrid fallback: which requests route to the cloud, and what happens when the link drops.
6. Set security and drift requirements before scaling: who can extract the weights, and how you’ll notice when accuracy slides.

Budget weeks for step four. If you’d rather have a team that has already shipped on-device models scope that pilot with you, talk to AlphaCorp AI.