How On-Device AI Autocomplete Works on a Mac

What “autocomplete” means here

The kind of autocomplete this page is about is the sort programmers have had in their editors for years: you type, and a faint grey continuation appears ahead of your cursor showing how the sentence could carry on. Press a key to accept it, or keep typing and it quietly gets out of the way. That grey preview is usually called ghost text, and it never becomes real text in your document until you accept it.

This is different from a chat assistant you paste writing into, and different from a grammar checker that marks up text you’ve already written. It’s a prediction about the words you haven’t typed yet, offered while you’re still typing them.

Why it can run on your Mac now

Three things changed at roughly the same time.

Small models got good. The language models that make headlines have hundreds of billions of parameters and need a rack of servers. But predicting the next few words of an ordinary sentence is a much narrower job than answering arbitrary questions, and models small enough to sit on a laptop are now genuinely competent at it.

Quantization shrank them further. A model’s parameters are stored as numbers, and storing them at lower precision — 6 bits each rather than 16 or 32 — makes the model several times smaller and faster with a modest loss of quality. This is why a capable model can be a download of a few gigabytes rather than tens of them.

Apple Silicon has unified memory. On a traditional PC, the graphics card has its own separate memory, and a model has to be copied into it before the GPU can work on it. Apple Silicon gives the CPU and GPU one shared pool of memory, so a model loaded once is immediately available to the GPU with no copying. For a workload that has to respond while you type, avoiding that copy matters a great deal.

The speed problem

This is the constraint that shapes everything else. A cloud assistant that takes two seconds to answer feels fine, because you asked it a question and you’re waiting. A suggestion that takes two seconds to appear while you’re mid-sentence is useless — you’ve already typed the words yourself.

So an on-device autocomplete is working to a budget measured in a few tens of milliseconds, on a machine that is simultaneously running your email client, your browser and everything else. That budget rules out large models regardless of how much memory you have, and it means the surrounding machinery — reading the text field, deciding whether to suggest at all, filtering out a bad suggestion — has to be at least as fast as the model itself.

It also means a local model has one structural advantage over a cloud one that no amount of server hardware can erase: there is no network round trip. Even a fast connection adds tens of milliseconds before the remote machine has started thinking.

What the model actually sees

Less than people usually assume. To predict your next few words, the model needs the words immediately before your cursor, and it helps to know roughly what kind of writing this is. It does not need your document history, your other applications, or anything you typed yesterday.

TypeSeer reads the text of the field you are currently typing in, through the same macOS Accessibility interface that screen readers use, and uses it to condition the prediction. When the model has produced a continuation, that text is drawn on screen as an overlay — it is not inserted into your document unless you accept it.

TypeSeer deliberately stays quiet in password fields and secure text entry, and in fields whose own completion already uses the Tab key, such as address bars and search boxes. The privacy page sets out exactly what is and isn’t read.

What running locally actually costs you

Honesty is more useful than salesmanship here, so:

There is a real download. The model has to arrive on your Mac once. For TypeSeer’s default model that is roughly 3.5 GB, fetched after you confirm it, and it stays on your disk afterwards.
It uses memory while it runs. A model held ready to answer in milliseconds is a model held in memory. That is the price of not waiting for a server.
There is a warm-up. The first suggestion after launch is slower than the ones that follow, because the model has to be loaded and the initial context processed.
A small local model is not a frontier cloud model. For finishing an ordinary sentence in your own voice it is very good. For reasoning about a complicated document it is not trying to compete, and any tool that claims otherwise about a 3 GB model is overselling.
It needs Apple Silicon. The unified-memory architecture is doing real work here. TypeSeer requires an Apple Silicon Mac running macOS 14 or later.

What you get in exchange

The trade is a good one for anything you’d rather not hand to a server.

✓ Your writing is never transmitted, so it cannot be logged, breached, or used as training data.
✓ It works with no internet connection at all once the model has been downloaded.
✓ There is no per-word cost, so there is no usage cap and no meter running.
✓ Suggestions do not degrade when the network is slow or an API is having a bad day.
✓ Personalization can learn from your actual writing without that writing going anywhere, because the learning happens on the same machine.

That last point is the one people tend to find most interesting. A cloud tool that adapts to your writing style has to keep your writing somewhere to do it. A local one simply doesn’t have that problem — and in TypeSeer, personalization is optional, learns only from sentences you finish, and can be inspected and deleted line by line.

How on-device AI autocomplete works