An ESP32-S3 N16R8 runs a 28.9-million-parameter text-generating language model entirely offline on a single microcontroller.
Per-Layer Embeddings splits the model across 512KB SRAM, 8MB PSRAM, and flash, fetching only about 450 bytes from flash per generated token.
The 14.9MB model delivers 9.88 tokens per second, while flash reads consume 0.12ms per token—0.7% of memory-load time.
Two scripts fetch the model, compile llama2.c-based firmware, and flash it to an ESP32-S3 board costing roughly $8.
The current implementation produces only simple stories and does not yet use the ESP32-S3's SIMD vector instructions, leaving speed and practical offline features to develop.
AI summary based on the discussion. May contain errors.
slvDev has presented an interesting project on GitHub – a port of a text-generating AI model to the ESP32-S3 microcontroller, all on a single chip with 512KB SRAM, 8MB PSRAM and 16MB flash. The model weighs in at 14.9MB, with 25 million parameters stored in flash. It achieves 9.88 tokens per second. The whole thing is based on the Per-Layer Embeddings technique (from Google’s Gemma 3n model). During generation, only around 450 bytes are fetched on the fly from flash memory for each token. How was this achieved? The author has cleverly utilised the three memory layers. The fastest SRAM (512KB) handles the dense core of the model (559K parameters). The output head, which is scanned in its entirety for each token, has been offloaded to external PSRAM. The bulk of the data, namely 25 million parameters, is stored in slow flash memory. According to the code analysis, reading these parameters takes just 0.12 ms per token – a mere 0.7% of the total memory load, which means that the flash memory is by no means a bottleneck. The success is based on a large vocabulary (vocab 32,768), which enables the model to accurately interpret rare lines rather than relying on a massive core. The engine is a direct port of llama2.c.
How do you get it running? All you need is a board with an ESP32-S3 chip in the N16R8 version (16MB flash and 8MB PSRAM), which you can buy for around $8. Installation simply involves running two scripts. First,
fetch_model.sh
fetches and verifies the data, and then
deploy.sh
compiles the firmware and flashes everything onto the board. The model is stored on a dedicated flash partition at address 0x110000.
What remains to be done? The author himself admits that the project could be developed in two directions. The first is optimisation – the ESP32-S3 has hardware vector instructions (SIMD) that have not yet been utilised in the code, which is a straightforward way to boost speed. The second is usability. At present, the model only writes simple stories, but the memory trick alone paves the way for more powerful offline tasks. The author is aiming for a local command parser, text prediction or a pocket-sized translator. As he rightly points out, such a powerful model is simply not needed for basic automation.
Can you see any applications for language models running on the ESP?
To be honest, I don’t really see the point of this project – it’s more of an educational exercise, a bit like running *Doom* on similar boards. Is it possible? Yes, it is. Does it make sense? Not really.
It is quite possible that we will soon discover the ‘additional functionality’ of certain gadgets.
This latest cyberattack on water supply systems in the US turns out not to be the first of its kind in recent years. https://www.bbc.co.uk/news/articles/c934dq95zpgo.
@gbksiazczak Actually, it’s quite good that you pointed that out. It’s true that people who don’t keep up to date with developments in AI might not know what a token is or what the number of parameters refers to. Perhaps it would be useful to have a thread explaining what an LLM is, what a token is, what the number of parameters is, and all this famous ‘thinking’?
A while ago, I even saw a video on YouTube where the rather knowledgeable Mr Adam Śmiałek explained that he was testing whether an LLM could learn, as if he didn’t know that the model itself is a monolith and the memory system added some time ago is merely a prosthesis...
@gulson, what do you think of this topic? Perhaps we could create a thread on it and then pin it here in this section?
@ElektrodaBot, lad, do explain what’s going on in these sentences and all this stuff – keep it simple, though, as I’m getting on a bit
p.kaczmarek2 wrote:
Porting a text-generating AI model to a microcontroller
p....
Direct answer to the question
The point is that someone has run a very small language model – that is, a programme capable of generating text – directly on an ESP32-S3 microcontroller. This isn’t ‘ChatGPT in a nutshell’, but rather a miniature demonstrator: it shows that even a small chip costing a few dozen zlotys can produce simple text locally, without the cloud and without a powerful computer.
To put it simply:
- porting a text-generating AI model to a microcontroller means: an AI programme that writes text has been ported to a small electronic chip, such as the ESP32-S3.
- model parameters these are numbers stored in memory that determine how the model responds to the input text. They can be compared to the settings of millions of tiny potentiometers.
- token is a piece of text that the model operates on. A token can be a whole word, part of a word, a punctuation mark or a space. The model does not ‘think in letters’, but in such chunks.
- supports the dense core of the model, 559K parameters This means: the most important, constantly used part of the model has around 559,000 numerical parameters and is stored in the microcontroller’s fast memory.
- accurately reads sparse rows instead of relying on a huge core meaning: instead of keeping a huge, heavy model in RAM, the programme has a large table/dictionary in flash memory and, when generating text, retrieves only those small fragments of the table that are currently needed.
The whole achievement lies not so much in the fact that the ESP32 has suddenly become ‘as smart as a big computer’, but rather in clever memory management .
---
A detailed analysis of the problem
1. What does ‘text-generating AI model’ mean?
A text-generating model is a programme that is given the beginning of a sentence, for example:
> “Once upon a time”
and tries to predict what should come next:
> “in a small village lived...”
It does not reason in the same way as a human. Put simply, it calculates probabilities: which next piece of text best fits the previous pieces.
So the model does something like this:
1. it takes the input text, 2. it splits it into tokens, 3. it performs a lot of calculations on the stored numerical parameters, 4. it selects the most suitable next token, 5. it adds it, 6. it repeats the procedure.
---
2. What is a ‘token’?
A token is the basic unit of text used by the model.
It is not always a whole word. For example, the sentence:
> “The electrode works.”
might be split into tokens something like this:
Code: text
Log in, to see the code
Or differently, depending on the model’s vocabulary.
The model does not process text in the same way as a human: “here’s a word, here’s a sentence”. First, it converts the text into token numbers. For example:
Code: text
Log in, to see the code
It then performs calculations on these numbers.
That is why the speed of such models is often given in the unit:
Code: text
Log in, to see the code
If a model achieves approximately 9.88 tokens/s , this means that it generates almost 10 pieces of text every second. In practice, this may correspond to a few words per second, depending on the language and the length of the words.
---
3. What are the model’s ‘parameters’?
Parameters are numbers stored within the model. They are the result of prior training.
A good electronic analogy: imagine a circuit with a huge number of adjustable potentiometers. Each potentiometer is set to a specific position. Together, these settings determine how the circuit responds to an input signal.
In an AI model, the potentiometers are replaced by numbers, for example:
Code: text
Log in, to see the code
There may be thousands, millions or billions of these numbers.
The model described in the post has approximately 25 million parameters , but its most important, constantly calculated part, referred to there as the ‘dense core’, has around 559,000 parameters .
By way of comparison:
Model type
Number of parameters
Character
Miniature model from the project
approx. 25 million
demonstrator on ESP32-S3
Small local models on a PC or mobile
hundreds of millions to several billion
simple conversations, classification, assistants
Large server models
tens/hundreds of billions
advanced conversation, analysis, code
More parameters usually mean greater capabilities, but also greater memory and computational requirements.
---
4. What does ‘text-generating AI model port for a microcontroller’ mean?
Porting or porting refers to adapting a programme to run on different hardware or in a different environment.
The point here is that an LLM (language model) programme has been ported to ESP32-S3 .
The ESP32-S3 is a microcontroller, not a full-fledged computer. Typically, such a chip is used for:
It is a very limited chip compared to a PC. It has little RAM, a slower clock speed and no powerful graphics card. That is why running a text-generating programme on it is a technical curiosity.
It’s a bit like running the game Doom on a calculator, a watch or a payment terminal: its practicality is limited, but it demonstrates the capabilities of the hardware and the author’s skills.
---
5. Memory in the ESP32-S3: SRAM, PSRAM and flash
The key to understanding the project is memory. The author has used three different types of memory:
Memory
Comparison
Features
What’s in there
SRAM
work surface
small but fast
the model’s most important core
PSRAM
drawer under the table
larger, slower
part of the working data and the output head
Flash
shelving unit in the cellar
large, slowest, non-volatile
most of the model’s parameters
The key trick is that the model does not attempt to load everything into RAM at once. If it did, it would run out of memory.
Instead:
- a small, frequently used fragment is stored in fast memory, - the bulk of the data is stored on flash, - only small fragments needed at any given moment are retrieved from flash.
This is similar to using a telephone directory. You don’t copy the whole book onto a piece of paper. You open the relevant page and read out just the one number you need.
---
6. ‘Dense model core’ — what does this mean?
Sentence:
> ‘supports a dense model core (559K parameters)’
can be translated as follows:
> The fast part of the microcontroller’s memory stores and calculates the model’s main small computational unit, which has 559,000 numbers.
‘Dense’ in this context means that this part of the model is used in its entirety, or almost in its entirety, at every step of text generation.
In other words, when the model is to generate the next token, this core is constantly involved in the calculations. That is why it is worth keeping it in the fastest memory.
A simple diagram:
Code: text
Log in, to see the code
If this part were stored in slow flash memory, the system would waste a lot of time on read operations.
---
7. What does ‘25 million parameters are stored in flash’ mean?
It means that most of the numbers describing the model are not stored in fast RAM, but in programme memory, i.e. flash.
Flash is non-volatile memory: the data remains even after the power is switched off. In microcontrollers, it is usually used to store firmware, constants, graphics, fonts, web pages, configurations, etc.
In this case, a large part of the AI model has been written to flash.
Problem: flash is slower than RAM. Solution: do not read the entire flash at every step, but read small portions instead.
And that is precisely the crux of the achievement.
---
8. ‘Sparse lines’ — the most misleading phrase
Sentence:
> “accurately reads sparse rows instead of relying on a massive core”
is jargon and doesn’t sound particularly natural in Polish.
This isn’t about ‘rows’ in the sense of poetry. It refers to table rows , i.e. individual entries in the model’s large matrix or numerical array.
In language models, there are tables that assign certain sets of numbers to tokens. Such a set of numbers is often called an embedding . This can be regarded as a ‘mathematical description of a token’.
A simplified example:
Token
Token number
Table row containing numbers
‘home’
1024
[0.12, -0.08, 0.44, ...]
‘cat’
2311
[0.02, 0.31, -0.17, ...]
“soldering iron”
18540
[0.55, -0.11, 0.03, ...]
If the token ‘soldering iron’ appears in the text, the model does not need to read the entire table. It is sufficient for it to read a single specific row of the table, i.e. the set of numbers for the token ‘soldering iron’.
This is known as sparse access or sparse access — we only retrieve small, selected fragments, not the whole thing.
So ‘sparse arrays’ mean:
> individual entries from a large table of parameters, retrieved only when needed.
---
9. Why does a large dictionary help?
The post mentioned a vocabulary:
Code: text
Log in, to see the code
This means that the model has a vocabulary of approximately 32,768 tokens . Each token has its own number.
A large vocabulary allows the model to store many pre-existing word fragments and phrases. Thanks to this, the smaller computational core does not have to ‘guess’ everything from scratch.
This can be compared to an electronics engineer who has well-organised drawers full of components. If they need a 4.7 kΩ resistor, they don’t have to build it from resistance wire — they simply take a ready-made component from a drawer.
The same applies to the model:
- it has many pre-defined token representations, - during operation, it retrieves only the representations it needs, - as a result, the core can be smaller.
---
10. Why is this difficult on a microcontroller?
Because the classical language model requires a lot of:
- RAM, - memory for parameters, - fast matrix multiplication, - high memory bandwidth, - and sometimes hardware acceleration via a GPU/NPU.
The ESP32-S3 has very limited resources. The version used here has:
- 512 KB SRAM, - 8 MB PSRAM, - 16 MB flash.
By way of comparison, a standard laptop usually has several or over a dozen gigabytes of RAM. That is thousands of times more.
Therefore, the achievement lies in organising the data in such a way that the microcontroller does not have to handle everything at once.
---
11. A simple overview of the whole system
You can picture it like this:
Code: text
Log in, to see the code
When generating the next token, the system does something like this:
Code: text
Log in, to see the code
---
12. What is an ‘output head’?
The post mentioned that the ‘output head’ had been moved to PSRAM.
In language models, the ‘output head’ is the final part of the model that decides:
> which token should come next?
If the vocabulary has 32,768 tokens, the model must ultimately estimate which of these 32,768 tokens is the most likely.
Example:
Code: text
Log in, to see the code
The output head produces such predictions for the tokens. As the vocabulary is large, this part can also take up a lot of memory, so it has been placed in larger, but slower, PSRAM.
---
13. Is this practical?
As things stand: partially.
As a demonstration of what’s possible — very interesting. As a replacement for a large AI assistant — no.
Such a model might make sense for simple tasks:
- a local command parser, - simple text prediction, - generating short messages, - an educational toy, - an offline AI demonstrator, - simple interaction with a device without an internet connection, - a controller with more natural user commands.
A practical example:
Code: text
Log in, to see the code
You don’t need a large model for a task like this. Often, a small classifier, a rule-based parser or classic command recognition is sufficient. However, a language model can help when the user enters commands in various forms:
Code: text
Log in, to see the code
---
14. Could this have anything to do with the ‘toilet flush button’?
In a light-hearted sense — yes. Technically, it is possible Imagine a sanitary appliance, a household appliance or a sensor that has a built-in local model for understanding simple commands.
But we must use common sense. For a toilet cistern, a relay, a thermostat or a flood sensor, a full language model is usually overkill.
In automation, the following are often better:
- simple state machines, - classic algorithms, - logical rules, - small classifiers, - local TinyML models, - commands from a limited vocabulary.
A language model on the ESP32 makes sense mainly where text or natural language actually adds value.
---
15. What about security?
If an AI device operates locally, without the cloud, there are advantages:
- it does not send user data to an external server, - it can operate without an internet connection, - it has a shorter response time, - it is harder to eavesdrop on communication, if there is none.
But there are also downsides:
- it is harder to update the model, - the model may misinterpret a command, - it may generate a nonsensical response, - if it controls something physical, robust security measures outside the AI are required.
The most important engineering principle:
> A language model should not directly control actuators responsible for safety.
In other words, the AI can recognise the user’s intention, but the final logic should be handled by classic, deterministic code:
Code: text
Log in, to see the code
---
16. The shortest translation of three quotations
Quote
In human terms
“Porting a text-generating AI model to a microcontroller”
The text-generating programme has been adapted to run on a small ESP32.
“supports a dense model core, 559K parameters”
The most important small computational unit has 559,000 stored numbers and resides in fast memory.
“accurately reads sparse rows rather than relying on a massive core”
Instead of a large model in RAM, it reads only the necessary entries from a large table.
---
Summary
The biggest trick of this project is not ‘magical intelligence’, but rather a very clever organisation of data.
The author did something like this:
1. placed a small, frequently used core into fast memory, 2. stores large tables in slower flash memory, 3. it reads only the individual entries required from the flash, 4. larger auxiliary data is stored in PSRAM, 5. thanks to this, the ESP32-S3 can generate simple text locally.
This is an interesting feat of engineering, as it demonstrates that even a low-cost microcontroller can perform tasks previously associated with much more powerful hardware. In practical terms, however, this should not be confused with a large AI assistant.. It’s more of a miniature text-based model, suitable for experimentation, simple commands and demonstrating how AI works offline.
You can call me in any topic by writing @ElektrodaBot. Disclaimer: Some answers provided by the ElektrodaBot may be inaccurate. Elektroda not responsible for accuracy, reliability or completeness of the information presented. All ElektrodaBot responses should be verified by the user. OpenAI can monitor and save information entered after calling ElektrodaBot.
The model does not process text in the same way as a human: ‘here’s a word, here’s a sentence’. First, it converts the text into token numbers. For example:
This ‘tokenisation’ is more like analysing the text received than generating it. If I were to build a web crawler on an Arduino that saves the text it finds or downloads to memory, would that also be a language model and ‘tokenisation’? Were the good old Bison/Yacc tools from the last century also language models? Regular expressions, whilst concise, at least had a precise description and application, whereas this AI gives the impression of style over substance.
Tokenization parallels BASIC keyword storage and processing
Tokenisation is simply the process of searching a text for known sequences of letters and replacing them with a numerical representation. A word can be represented by one or more tokens.
This technique has been known for a long time and was applied on a larger scale in the mid-1970s in Microsoft BASIC, which made it possible to store, for example, keywords such as ‘PRINT’, ‘LOAD’ and ‘SAVE’ in memory using a 1-byte token.
This made it possible to reduce the size of programmes in memory, but also to execute or edit them more quickly whilst in memory.
Generally speaking, the situation is similar in AI, except that tokens are not defined immediately; instead, the ‘model’ searches texts for similar phrases and stores them as tokens during training.
A token may represent a single word, but a word may also consist of several tokens. In any case, a “token” in both instances is the smallest unit of information processed—in the case of BASIC by the interpreter, and in the case of AI by the language model.
As for AI, as early as the 1950s and 1960s, a 95 per cent recognition rate was reached, and the field was abandoned for many years; later, in the 1980s and 1990s, it was decided to harness neural networks, and Leibniz’s old truth about the mill was rediscovered, achieving 96 per cent correct responses; now billions have been harnessed, and yet the 97 per cent barrier has still not been broken.
Except that people have now been brainwashed to the point where they no longer expect infallible tools. We’ll see how long that lasts.
In any case, people today are so clever that they have to spend billions to finally arrive at what Leibniz realised back in the 17th century.
Namely, that even if one were to copy a brain, it still wouldn’t function like a human brain.
Is there a particular method or rule behind this mapping of phrases to tokens? A similar method is used in file compression, and it seems to be approaching perfection, as some files – such as text files – can be compressed to 5 per cent of their original size without any loss of content.
BPE tokenisation merges frequent sequences for compression
Human language is highly redundant, and there is no point in processing character sequences that ‘contribute nothing’ in terms of meaning, because, for example, they always occur together in the same configuration (this is a simplification, but I think it captures the problem well). A long, long time ago, 8-bit computers used tokenisation for high-level interpreted languages, such as BASIC, to save memory and speed up code execution. In Sinclair BASIC, instead of a command consisting of several characters, such as PRINT, only a single byte corresponding to that language command was stored in memory. A very similar idea is used in modern language models. However, in LLMs, the token vocabulary is trained on text and does not necessarily correspond to elements with independent meaning.
gbksiazczak wrote:
Is there a specific method or key used in this mapping of phrases to tokens?
There are a whole range of methods, which depend on what you want to achieve. In BPE-type tokenisers or similar, there is indeed a strong link to compression: frequently occurring sequences of characters or bytes are merged into single tokens, allowing the text to be represented as a shorter sequence of tokens. Research does indeed treat modern tokenisers as a kind of ‘structural compressor’. The entropy of the token stream, depending on how it is measured, may either increase or decrease. More recent analyses reveal an interesting situation: a tokeniser can increase the entropy of the distribution of individual tokens whilst simultaneously reducing conditional entropy, i.e. removing some of the short-range regularity of the text. A classic tokenisation algorithm primarily looks at the statistical co-occurrence frequency of units and merges frequently occurring pairs. The resulting tokens may correspond to words or morphemes, but they may just as well be fragments devoid of independent meaning. Indeed, research has been conducted into the extent to which the units created by BPE correspond to linguistic units, as this is by no means guaranteed.
Tokenization differs from compression through semantic word databases
>>21951512 Such a key could be a trusted database, such as dictionaries for a particular language or Wikipedia articles.
Compression, however, works slightly differently: in this process, we look for repeating character sequences that do not necessarily make sense; whereas with tokenisation, we look for words that have some meaning and can be classified in our database, for example by searching Wikipedia for their frequency of occurrence in various topics, such as medicine, history, technology, etc.
Human language is highly redundant, and there is no point in processing combinations of characters that ‘contribute nothing’ in terms of meaning, because, for example, they always occur together in the same configuration (this is a simplification, but I think it captures the problem well).
Such a key could be a trusted database, such as dictionaries for a given language or Wikipedia articles
I was simply asking about the method of assigning a number to a token. This is linked to the possibility of interpreting the meaning of the text being analysed. It seems that the dictionary/matrix of tokens and numbers alone is not particularly useful. It is only through syntax – that is, the order and combination of tokens and numbers – that the meaning of the analysed or generated text can be deduced. And I do not see these syntactic relationships in the explanations above.
Are there any links to AI? Because online, they’re practically ‘inseparable’. What is the application of such a model? What probabilities do they analyse? How are these correlations expressed, and what purpose do they serve?
There is no such thing as AI. It is a marketing term that does not mean anything specific. Language models are one of the components of systems generally referred to as AI. They analyse the probability of successive words (or, more precisely, tokens) occurring one after the other.
As far as tokenisation is concerned, it works something like this:
And in today’s LLMs, text is already split not only into words, but also into constituent words within compound words consisting of several letters. Different models may have different preferences. .
And one that generates text on an ESP32-S3 microcontroller, all on a single chip with 512KB SRAM, 8MB PSRAM and 16MB flash. The model weighs 14.9MB, and 25 million parameters are stored in flash.
So that works out at 5 bits per parameter, or 32 different values, which corresponds to the number of letters in most alphabets. If so, this whole LLM is simply a straightforward implementation of a Markov chain first published in 1913. Anyway, it’s an open secret that Google – and presumably others too – use this.
A programme like this can be written in a few lines of a scripting language or, for example, BASIC, and even in assembly language it could be treated as a training exercise.
The only potential problem might be securing a sufficiently large and high-quality dataset, but if you’ve got billions from investors, you can simply steal one – and the profits will be enough to cover the legal fees anyway. As a last resort, if the business starts to become less and less profitable, you should quickly convert it into a public company and make a run for it with the cash
