Most weeks the AI news is a bigger model or a cheaper token. This week Google shipped something structurally different: HEIR — an open-source compiler that turns ordinary machine-learning models into versions that run inference on encrypted data the server never decrypts. It is early, it is slow, and it is one of the more important things to happen to AI privacy in years. All three of those are true, and the interesting work is holding them together honestly.
What Fully Homomorphic Encryption actually promises
Fully Homomorphic Encryption (FHE) is the cryptographic dream that refuses to die: perform arbitrary computation directly on ciphertext, such that the result, once decrypted by the data owner, is exactly what you would have gotten by computing on the plaintext — but the party doing the computing never sees the plaintext at all. Applied to AI inference, the flow is: you encrypt your data with your public key, hand the ciphertext to a server, the server runs the model on the encrypted data producing an encrypted result, and only you can decrypt the answer. The server learns nothing. Not your input, not the output.
If that works at scale, it dissolves the central bargain of cloud AI. Today, using a hosted model means trusting the host with your plaintext — your medical record, your legal document, your customer data. FHE replaces "trust the host" with "the host is mathematically incapable of seeing it." That is not a better privacy policy. It is the removal of the need for one.
Why a compiler is the actual news
FHE has been theoretically possible since Gentry’s 2009 breakthrough. It did not go anywhere useful for over a decade for a mundane reason: writing FHE programs by hand is brutal. You are not writing code; you are hand-managing ciphertext noise budgets, bootstrapping, parameter selection, and data layout — a specialist art form that maybe a few hundred people worldwide practice fluently. That is the bottleneck HEIR attacks. It is an MLIR-based toolchain that automates the lowering from a normal model (a TensorFlow Lite network, say) down to optimized FHE, handling the arithmetization and parameter choices that used to require a cryptographer-in-the-loop.
This is the same pattern that made every prior hard technology usable: the breakthrough is not the primitive, it is the tooling that lets non-specialists wield the primitive. GPUs mattered when CUDA arrived; deep learning mattered when autodiff frameworks arrived. HEIR is trying to be the compiler that turns FHE from a cryptography-PhD craft into something a normal ML team can target. Google explicitly wants it to become the industry-standard FHE compiler, and open-sourcing it is how you make a standard.
The honest performance reality
Now the cold water, because this is where hype lives. Google’s own showcase number: a 3-layer neural network, compiled with HEIR, produces a private inference in about 16 seconds. Sit with that. Sixteen seconds, for a network so small it is a toy by 2026 standards. The general FHE overhead is roughly 100x to 1000x slower than plaintext computation, depending on the operation. You are not running an LLM homomorphically in real time. You are not close.
So calibrate. What HEIR makes practical today is small-model inference on high-value, high-sensitivity data where 16 seconds is completely acceptable because the alternative is "we legally cannot send this to a cloud model at all." A fraud-scoring model on an encrypted transaction. A classifier on an encrypted medical signal. Narrow, valuable, latency-tolerant. What it does not make practical today is homomorphic ChatGPT. The trajectory — hardware acceleration is narrowing the gap, and dedicated FHE accelerators are a live research area — points toward that gap closing over years, not months.
Why I think this matters more than a model release
I have spent much of this month writing about the erosion of privacy from every direction: face scans to prove humanity, robotaxis mapping streets, search that reads everything. FHE is the rare vector pointing the other way — a technology that makes privacy a mathematical property of the computation rather than a promise you have to trust. And the domains where it lands first are exactly the ones currently frozen out of the AI boom by regulation and liability: healthcare, finance, law, government, anything under GDPR’s special categories. Those sectors have enormous latent demand for AI and enormous reasons not to hand plaintext to a third party. FHE is the key that could unfreeze them.
There is also a strategic reading. If encrypted inference becomes practical, it reshapes the cloud-AI trust model in a way that favors whoever controls the compiler and the accelerators. Google open-sourcing HEIR is not charity; it is an attempt to own the standard layer of a privacy-preserving-AI stack it expects to matter. Watch who builds the FHE hardware, because the 100–1000x tax is a hardware problem, and hardware is where durable moats form.
What I would take from this
- Add "encrypted inference" to your architecture vocabulary now. Not to deploy this quarter, but because within a couple of years "can we run this model without ever seeing the data" will be a real design option for regulated workloads, and the teams that understood it early will move first.
- Match the tool to the latency budget. FHE is for narrow, sensitive, latency-tolerant inference — not interactive chat. If someone pitches you homomorphic real-time LLM inference in 2026, they are selling the 2030 version.
- Regulated sectors should be paying the closest attention. Health, finance, legal, public sector: this is the technology that could let you use AI on data you currently cannot legally expose. Start scoping which workloads are small, valuable, and blocked purely by data-exposure rules.
- Follow the compiler and the silicon, not the demo. The primitive is old news. The usability (HEIR) and the speed (FHE accelerators) are the two variables that decide whether this is a 2027 reality or a permanent "five years away."
Homomorphic encryption has been "five years away" for fifteen years. What changed this week is not the math — it is that the math got a compiler aimed at ordinary ML engineers. That does not make private AI fast. It makes it buildable, by people who are not cryptographers, for the first time. In a month full of AI eroding privacy, here is AI being taught to compute on what it is not allowed to see. That is worth more attention than another point on a benchmark.