AI language model running on an ESP32-S3 – 28.9 million parameters

slvDev has presented an interesting project on GitHub – a port of a text-generating AI model to the ESP32-S3 microcontroller, all on a single chip with 512KB SRAM, 8MB PSRAM and 16MB flash. The model weighs in at 14.9MB, with 25 million parameters stored in flash. It achieves 9.88 tokens per second. The whole thing is based on the Per-Layer Embeddings technique (from Google’s Gemma 3n model). During generation, only around 450 bytes are fetched on the fly from flash memory for each token.
How was this achieved? The author has cleverly utilised the three memory layers. The fastest SRAM (512KB) handles the dense core of the model (559K parameters). The output head, which is scanned in its entirety for each token, has been offloaded to external PSRAM. The bulk of the data, namely 25 million parameters, is stored in slow flash memory. According to the code analysis, reading these parameters takes just 0.12 ms per token – a mere 0.7% of the total memory load, which means that the flash memory is by no means a bottleneck. The success is based on a large vocabulary (vocab 32,768), which enables the model to accurately interpret rare lines rather than relying on a massive core. The engine is a direct port of llama2.c.

How do you get it running? All you need is a board with an ESP32-S3 chip in the N16R8 version (16MB flash and 8MB PSRAM), which you can buy for around $8. Installation simply involves running two scripts. First,

fetch_model.sh

fetches and verifies the data, and then

deploy.sh

compiles the firmware and flashes everything onto the board. The model is stored on a dedicated flash partition at address 0x110000.

What remains to be done? The author himself admits that the project could be developed in two directions. The first is optimisation – the ESP32-S3 has hardware vector instructions (SIMD) that have not yet been utilised in the code, which is a straightforward way to boost speed. The second is usability. At present, the model only writes simple stories, but the memory trick alone paves the way for more powerful offline tasks. The author is aiming for a local command parser, text prediction or a pocket-sized translator. As he rightly points out, such a powerful model is simply not needed for basic automation.
Can you see any applications for language models running on the ESP?

Sources:
Project repository: https://github.com/slvDev/esp32-ai
Author’s post: https://www.reddit.com/r/arduino/comments/1v4...89m_llm_running_on_a_esp32s3_9_toks_no_cloud/