VLMs on your Mobile Phone

Two streams this past weekend. On saturday’s stream we looked at Vision Language Models (or VLMs, though some call them LVLM and even MMLMs). These are basically still just a vision encoder (usually CLIP, but sometimes DINO) connected via an MLP to an LLM. You train the “connector” first, freezing both the encoder and the llm. Then you can train the whole thing, though mainly just the LLM. There are many variants of this pattern but none seem to be much better than the others. One interesting idea is that using a combination (or ensemble) of different vision encoders works better, if you want to pay the extra inference cost.

On Sunday we examined some tiny models that run on mobile devices. You can run little image generators (diffusion model in latent space) on a mobile device with some optimizations in model arch and things like quantization. We also looked at a couple agent-ish papers where you can get GPT4V to basically click through most phone UIs quite successfully: send emails, comment on social media posts, etc.

If you combine the results from both of these I think we are quite close to a VLM running locally on your mobile phone that operates as an agent using a simple Sense-Plan-Act loop. This would be almost impossible to distinguish from a human and instantly decentralize power to all the current owners of a cell phone, which is basically all humans. At which point AGI spreads like a virus to all the existing devices in the world.

Preprint

Preprint

VLMs on your Mobile Phone