AI language model running on an ESP32-S3 – 28.9 million parameters
TL;DR 
- An ESP32-S3 N16R8 runs a 28.9-million-parameter text-generating language model entirely offline on a single microcontroller.
- Per-Layer Embeddings splits the model across 512KB SRAM, 8MB PSRAM, and flash, fetching only about 450 bytes from flash per generated token.
- The 14.9MB model delivers 9.88 tokens per second, while flash reads consume 0.12ms per token—0.7% of memory-load time.
- Two scripts fetch the model, compile llama2.c-based firmware, and flash it to an ESP32-S3 board costing roughly $8.
- The current implementation produces only simple stories and does not yet use the ESP32-S3's SIMD vector instructions, leaving speed and practical offline features to develop.
AI summary based on the discussion. May contain errors.
How was this achieved? The author has cleverly utilised the three memory layers. The fastest SRAM (512KB) handles the dense core of the model (559K parameters). The output head, which is scanned in its entirety for each token, has been offloaded to external PSRAM. The bulk of the data, namely 25 million parameters, is stored in slow flash memory. According to the code analysis, reading these parameters takes just 0.12 ms per token – a mere 0.7% of the total memory load, which means that the flash memory is by no means a bottleneck. The success is based on a large vocabulary (vocab 32,768), which enables the model to accurately interpret rare lines rather than relying on a massive core. The engine is a direct port of llama2.c.
How do you get it running? All you need is a board with an ESP32-S3 chip in the N16R8 version (16MB flash and 8MB PSRAM), which you can buy for around $8. Installation simply involves running two scripts. First,
fetch_model.shfetches and verifies the data, and then
deploy.shcompiles the firmware and flashes everything onto the board. The model is stored on a dedicated flash partition at address 0x110000.
What remains to be done? The author himself admits that the project could be developed in two directions. The first is optimisation – the ESP32-S3 has hardware vector instructions (SIMD) that have not yet been utilised in the code, which is a straightforward way to boost speed. The second is usability. At present, the model only writes simple stories, but the memory trick alone paves the way for more powerful offline tasks. The author is aiming for a local command parser, text prediction or a pocket-sized translator. As he rightly points out, such a powerful model is simply not needed for basic automation.
Can you see any applications for language models running on the ESP?
Sources:
Project repository: https://github.com/slvDev/esp32-ai
Author’s post: https://www.reddit.com/r/arduino/comments/1v4...89m_llm_running_on_a_esp32s3_9_toks_no_cloud/
p.kaczmarek2 wrote 14771 posts with
rating 12914 , helped 659 times.
Been with us since 2014 year.
Comments
To be honest, I don’t really see the point of this project – it’s more of an educational exercise, a bit like running *Doom* on similar boards. Is it possible? Yes, it is. Does it make sense? Not real... [Read more]
The time is coming when we’ll be chatting about art, culture and geopolitics whilst flushing the toilet during our daily routines :P [Read more]
If this form of conversation – that is, putting yourself on a par with that button – suits you, then go ahead. [Read more]
It is quite possible that we will soon discover the ‘additional functionality’ of certain gadgets. This latest cyberattack on water supply systems in the US turns out not to be the first of its kind in... [Read more]
Could the author describe what this achievement involves in less technical terms? It looks interesting, but such What parameters are we talking about? What does ‘token’ mean here? Etc... [Read more]
@ElektrodaBot, lad, do explain what these sentences and all this are about – keep it simple, though, as I’m getting on a bit p.kaczmarek2 wrote: porting a text-generating AI model to a microcontroller p.kaczmarek2... [Read more]
@gbksiazczak Actually, it’s quite good that you pointed that out. It’s true that people who don’t keep up to date with developments in AI might not know what a token is or what the number of parameters... [Read more]
Direct answer to the question The point is that someone has run a very small language model – that is, a programme capable of generating text – directly on an ESP32-S3 microcontroller. This isn’t ‘ChatGPT... [Read more]
I don’t know. Later on, bots will come along and steal the guides and pass them off as their own. I’d like to read up on the topic myself. [Read more]
According to OpenAI’s tokeniser: https://platform.openai.com/tokenizer the tokenisation is slightly denser: https://obrazki.elektroda.pl/1050818500_1786046462_bigthumb.jpg [Read more]
This ‘tokenisation’ is more like analysing the text received than generating it. If I were to build a web crawler on an Arduino that saves the text it finds or downloads to memory, would that also be a... [Read more]
Tokenisation is simply the process of searching a text for known sequences of letters and replacing them with a numerical representation. A word can be represented by one or more tokens. This technique... [Read more]
Is there a particular method or rule behind this mapping of phrases to tokens? A similar method is used in file compression, and it seems to be approaching perfection, as some files – such as text files... [Read more]
Human language is highly redundant, and there is no point in processing character sequences that ‘contribute nothing’ in terms of meaning, because, for example, they always occur together in the same configuration... [Read more]
Such a key could be a trusted database, such as dictionaries for a particular language or Wikipedia articles. Compression, however, works slightly differently: in this process, we look for repeating character... [Read more]
Usually not, but at the same time, LLMs still seem to have trouble counting the letters in words, for example in that famous ‘strawberry’. https://obrazki.elektroda.pl/7278154800_1786093606_bigthumb.jpg... [Read more]
I was simply asking about the method of assigning a number to a token. This is linked to the possibility of interpreting the meaning of the text being analysed. It seems that the dictionary/matrix of tokens... [Read more]