logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

More powerful than ESP32 MCU for UDP handling and packet parsing

mateos2 2052 41
Best answers

How can I reliably receive and parse frequent UDP broadcast packets from many Wi‑Fi thermostats without dropping packets on an Arduino-compatible MCU?

There is no magic “faster ESP32” fix here; the main wins are to remove heap-heavy `String` parsing and to increase UDP receive buffering, or move to a platform with a stronger network stack [#21190975][#21191019][#21192167] Parse the packet directly from the received buffer with `strstr`/similar functions or `std::string_view` instead of `malloc` + `String` + `substring()` copies [#21190737][#21190975] If you stay on ESP32, use ESP-IDF rather than the Arduino SDK, because Arduino does not let you freely change lwIP settings such as `CONFIG_LWIP_UDP_RECVMBOX_SIZE`; one reply notes the default UDP mailbox is only 6 and suggests raising it to 64 in ESP-IDF [#21191019][#21191347][#21192167] Another likely bottleneck is Wi‑Fi broadcast itself: broadcast UDP is much slower and more lossy than unicast, so the traffic pattern may be the real problem [#21191464] If you want the least fight for headroom, several replies suggest moving to a Raspberry Pi Zero W or Pi 4 instead of another MCU [#21191347][#21190927]
Generated by the language model.
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
Report a violation of the law
Reply | New topic
  • #1 21189769
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    Welcome

    I have a working ESP S3 module that collects data from 40 Wi-Fi thermostats that broadcast their readings every second with a UDP broadcast. The module receives the packets, parses them and transmits the statistics to the BMS (rs485) and to the web server (http API).

    This works well, but the UDP packet parsing time is large - up to 500us, so adding more thermostats will increase the problem of missed packets.
    (I assume 500us is great and it will be hard to optimise - a faster MCU remains)

    Trying to swap the esp32 for an Arduino opta 485 connected over Ethernet ended in failure - module flooded with UDP broadcasts stops responding.

    Do you perhaps know of another module or version that can handle UDP reception AND parsing faster?
    Parsing is looking for keys and values in a string.
    The platform is of course Arduino.
  • ADVERTISEMENT
  • #2 21189788
    JacekCz
    Level 42  
    Posts: 8670
    Help: 760
    Rate: 1460
    mateos2 wrote:
    which can handle UDP reception and parsing faster
    .

    These are two profoundly different issues, and should not be discussed "in the same breath".
    One has a lot to do with hardware (+operating system in case there is one) and nothing to do with application code, the other 100% user code and no hardware involvement


    mateos2 wrote:
    The platform is, of course, the Arduino.


    A good opportunity to sp....
    If it's a typical Arduino philosophy, with delays, port polling in a loop (rather than interrupts), it kills more activity

    Added after 2 [minutes]:

    mateos2 wrote:
    I assume that 500us is a great result and will be hard to optimize

    mateos2 wrote:
    Parsing is looking in string for keys and values.


    Hmmm, naive Arduino-style code can be a deal-breaker.
    Without seeing the code, the answer is "not necessarily" / "it depends".
  • #3 21189810
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    JacekCz wrote:
    Good opportunity to screw this up....
    If this is the typical Arduino philosophy, with delays, port polling in a loop (rather than interrupts), it kills more activity


    As I wrote - it uses a TCP stack AND simple string parsing. There's no magic to it, simple C portable to multiple platforms.

    Example of parsing code:
    Code: text Expand Select all Copy to clipboard
    
void parseudp(AsyncUDPPacket packet) {
long sn ;
int pos;
int pose;
int rssi;
String valt; 

char* tmpStr = (char*) malloc(packet.length() + 1);
memcpy(tmpStr, packet.data(), packet.length());
tmpStr[packet.length()] = '\0'; 
String data = String(tmpStr);
free(tmpStr);

    pos = data.indexOf("\"sn\"",pos)+6;
    pose = data.indexOf(",",pos)-1;
    valt = data.substring(pos+5,pose);
    sn = valt.toFloat();
    .

    String class used, because the values are then sent to the server as string anyway (to avoid string -> float -> string)
  • #4 21189871
    pikarel
    Level 39  
    Posts: 4959
    Help: 407
    Rate: 1819
    Since it's just a readout, with no need for quick feedback to some heater/motor controller - it can take as long as 20ms to read one sensor (as long as one frame on TV) and you won't even notice it, and the refresh of data from the readout to the display can be every 10-20s, because nothing depends on it.
  • ADVERTISEMENT
  • #5 21189898
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    But the idea is that I don't do any reading just catch the messages of the thermostats.
    I set ttl to 120s and if the thermostat hasn't sent anything for 120 seconds then I remove it from the list.
    Currently some thermostats are caught even after 15 seconds, which means adding more will do more UDP frames, more analysis time and they will start to drop out.

    The controller controls the heat pumps in groups and this can cause them to stop and start for the wrong reasons.

    I have no control over sending frames so the MCU has to catch all packets as much as possible.
    E.g. 40 thermostats will send packets in 100ms and I would like to catch as many as possible
  • #6 21189962
    lopr_pol
    Level 32  
    Posts: 1691
    Help: 161
    Rate: 476
    Do you seriously need a temperature every 1sec? Who made these thermostats that they are haying communication like this? I would ignore readings more frequent than 2-5min and that's it.
  • #7 21190241
    pikarel
    Level 39  
    Posts: 4959
    Help: 407
    Rate: 1819
    Quote:
    data from 40 Wi-Fi thermostats that report their readings
    .
    Why are you doing a reading of these thermostats?
    For fun, i.e. to look at them for yourself?
  • #8 21190270
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    lopr_pol wrote:
    Who made these thermostats that they are haying communication like this?

    Chinese :) sometimes it is the case that you find a solution and have to adjust

    Added after 3 [minutes]:

    pikarel wrote:
    Why are you doing a reading of these thermostats?
    .
    Because the reading from them is used to control the heating, ventilation and cooling of the office building.
    By knowing the thermostats and the volume they serve, you can select the minimum operating point of the systems saving the client massive amounts on bills.
  • #9 21190357
    pikarel
    Level 39  
    Posts: 4959
    Help: 407
    Rate: 1819
    Do you know what the thermostat is for, what its role is?
    Do you know how complex the control algorithms inside their own processor are?
    You by looking at the readout of their operation - because they have a rogue - want to save money?
    Good.
  • #10 21190400
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    pikarel wrote:
    Do you know how complex the control algorithms inside your own processor are?
    .

    But I understand that this is ironic?
    They have an algorithm for 3 lines of code:) bimetal simulator

    pikarel wrote:
    You looking at the readout of their work - because they have a rogue - you want to save money?
    .
    You didn't read my post
  • #11 21190737
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    mateos2 wrote:
    JacekCz wrote:
    Good opportunity to screw this up....
    If it's typical Arduino philosophy, with delays, querying ports in a loop (rather than interrupts), it kills more activity


    As I wrote - it uses a TCP stack AND simple string parsing. There's no magic to it, simple C portable to multiple platforms.

    Example of parsing code:
    Code: text Expand Select all Copy to clipboard
    
void parseudp(AsyncUDPPacket packet) {
...
char* tmpStr = (char*) malloc(packet.length() + 1);
memcpy(tmpStr, packet.data(), packet.length());
tmpStr[packet.length()] = '\0'; 
String data = String(tmpStr);
free(tmpStr);

    pos = data.indexOf("\"sn\"",pos)+6;
    pose = data.indexOf(",",pos)-1;
    valt = data.substring(pos+5,pose);
    sn = valt.toFloat();
    .

    String class used, because the values are then sent to the server as string anyway (to avoid string -> float -> string)

    You've gone a bit overboard. So many memory allocations to chunk one data.
    The packet comes as probably void*, search for items with the strstr function that act like indexof, then there won't be two resource allocations. It's always a bit of microseconds gained.

    As for the original question, you could port the nanrpi zero W code, price very similar, but the ESP32 has a fair bit of processing power too. It is possible that the Arduino framework is killing it. Isn't he based on rtos?

    Added after 7 [minutes]: .

    mateos2 wrote:
    40 thermostats will send packets in 100ms and I would like to catch as many as possible
    .
    The stack should hold them all until read.
    The idea of discarding packets coming from the same sensor is a good one. Know which dp socket the packet is coming from, note the time and discard the rest according to a counter or timer
  • #12 21190771
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    michal.zd wrote:
    .
    You've gone a bit overboard. So many memory allocations to extract one data.


    I extract 6 data from each UDP packet - format is unstable (JSON similar), only keys are stable
    neither memory allocation nor fragmentation hurts me - there is a spare
    working on char type didn't speed up much, so I left String

    I'll have a look how exactly these packets are handled, and maybe I'll try to do some FIFO queuing
    or I'll finally get around to delegating parsing to another core than TCP handling :) but then say goodbye to easy portability
  • #13 21190777
    pikarel
    Level 39  
    Posts: 4959
    Help: 407
    Rate: 1819
    To clarify; a thermostat is a device that autonomously stabilises a set temperature by operating the devices that provide this stabilisation, e.g. motors, heaters. Some thermostats are equipped with communication for remote reading of this temperature.
    You have trouble reading it, besides, your ESP does not feedback control to any of the thermostats.

    What I'm asking is not ironic; I'm asking why you're doing it, because if not for control, then just for the fun of reading it "because you can".

    Systems with remote desktop control have their own operating programs for this, with service authorisation, with no access to parameters without that authorisation.
  • ADVERTISEMENT
  • #14 21190810
    JacekCz
    Level 42  
    Posts: 8670
    Help: 760
    Rate: 1460
    mateos2 wrote:
    Memory allocation or fragmentation doesn't hurt me - there's a spare
    working on char type didn't speed up much so I left String
    .

    The hard points of the issue are not where you see them.
    Allocation is not just "aggregate memory usage", but each "piece" must be handled in the manager, e.g. releasing costs non-trivial time. And it's that time that's most valuable when it's hottest.
    Invoking a new String, each piece, is the same. Every visible String invoked, substr() etc ...

    michal.zd wrote:
    You've gone a bit overboard. So many memory allocations to chunk one data.
    .
    michal.zd wrote:
    Possible that the Arduino framework is killing it.
    .

    Reasonable assumptions

    Added after 8 [minutes]:

    mateos2 wrote:
    Working on the char type didn't speed up much so I left String
    .

    You can optimise such a theme beautifully on char. As much as I am not a fan of C, in such limited applications it has prospects of
    All reception to those few characters of output data without/with one allocation

    You didn't have a significant gain because it's not done well. Rescue, stronger processor needed.
  • #15 21190884
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    JacekCz wrote:
    You have not had a significant gain because it is not well done. Rescue, stronger processor needed.


    of course yes , but .... $ vs time
    spending 100-300zl on each MCU is for me sometimes cheaper than optimising the code :) .
    with 10 pieces of product per year the optimisation is too expensive

    the times when I wanted to push code with TCP stack on Arduino uno are gone - now I respect time more + if someone will come to develop the code, it's easier to find someone who can handle string in Arduino than someone who can handle pure C
  • #16 21190913
    JacekCz
    Level 42  
    Posts: 8670
    Help: 760
    Rate: 1460
    mateos2 wrote:
    with 10 units of product per year, optimisation is too expensive
    .

    The code I have in my imagination is no hackneyed optimization, just normal decent knowledge of C, not more lines of code at all
    I have a not-so-wild rule: I don't practice commercialism on tools I don't know professionally (I'm not talking about perfection, because it doesn't exist).
    In my world, it wouldn't be ethical for me to perform appendectomies, play guitar commercially, or program in languages I don't know very well (PHP, JS, Basics).

    mateos2 wrote:
    if someone comes to develop code, it's easier to find a person who can handle strings in Arduino than one who can handle pure C
    .

    I can't imagine a processional worker programming an arduino commercially, and not knowing C. But then again, I'm probably out of date
    Let's make a bowel movement because someone will continue it (and I do). Typical arguments in arduino climates
  • #17 21190918
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    JacekCz wrote:
    mateos2 wrote:
    with 10 pieces of product per year, optimisation is too expensive
    .
    I have a not-so-wild rule: I don't practice commercialism on tools I don't know professionally (I'm not talking about perfection, because it doesn't exist)


    unfortunately playing around with BMSs requires being an automation specialist, coder, electrician, electronics engineer, plumber, pumping engineer, PV engineer, HVAC engineer, etc. at the same time.
    a bit of everything, so nothing too deep - the 80/20 rule
  • #18 21190927
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    mateos2 wrote:
    than optimisation
    .
    Sorry, but this is not a good approach. From the outset you need to keep in mind the overheads of cppi to properly match the tool to the capabilities.
    It's like no processor without mmu, acct the string class without continuous memory allocation can't do much.
    But in your case, with everything already in memory, there is no point in reassigning it several times, base it on the available c functions, or write a string class that operates on ready data. I did this once, my own implementation of the string class running on an already allocated buffer, added the functions I needed. Not much needed.

    Added after 6 [minutes]:

    mateos2 wrote:
    everyone will come to develop the code
    .
    If you keep the code simple and clear, then you have nothing to worry about. It won't be your fault if a 'developer' rather than a programmer gets into your code. As it stands, the code is not simple and clear, it would not pass review.

    Added after 11 [minutes]:

    Back on topic.
    If you are assuming extending the controller with more sensors, you might actually go straight to another platform. I would choose a raspberry pi even if only a zero W. An additional plus is the greater stability of production, in a few years there will still be some minicomputer, rpi zero probably also, Linux too, so the code will be possible to possibly recompile to another architecture.
    Scalability basically unlimited, extension of functionality also.
    What are these sensors? Don't they have the ability to communicate with an MQTT server? This would be the best option.
  • #19 21190956
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    michal.zd wrote:
    What are these sensors? Don't they have the ability to communicate with an MQTT server? This would be the best option.
    .

    TMS812 sensors from China - the customer already has these mounted and connected throughout the site - ESP8266 inside
    manufacturer support ends on sale :)

    I had the choice to either
    1/ connect to the 230V output of each - i.e. 3 point control
    2/ select 20% of those represented and connect the cables
    3/ rev-enginnering

    50 or even 10 points means hundreds of metres of cable to pull, so I chose option 3
    it worked out that I would either sniff HTTPS, or use UDP broadcasts
    (probably the Chinese were planning an aggregate management system of some kind).
    UDP is bare text so I just went that way

    thanks to this I not only have the information whether to heat or cool, but also the actual temperature and by the serial number for comparison I have the exact information how many m2 or m3 I have to cool or heat

    and this is valuable information
    because if I have to heat 20% of a room by 0.2 degrees, it's a completely different power than if I have to heat 80% of a room by 2 degrees.
  • #20 21190961
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    mateos2 wrote:
    UDP is bare text so I just went that way
    .
    Very good idea. I'm curious if the data resembles the mqtt framework. Is it json from the start, or are there a few odd tags?

    Added after 56 [seconds]:

    mateos2 wrote:
    hysteresis of 0.2 degrees is quite
    unrealistic. I have doubts that half a degree of hysteresis can be maintained on a large volume.
  • #21 21190975
    Anonymous
    Level 1  
  • #22 21190994
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    khoam wrote:
    Std::string_view objects can also be directly created on char*,
    .
    I gave a plus for this. This is a very good idea. I wonder if Arduino will let you use the library std.
  • #23 21190996
    Anonymous
    Level 1  
  • #24 21191007
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    michal.zd wrote:
    Very good idea. I wonder if the data resembles a mqtt frame. Is it json from the start, or are there a few odd tags?
    .

    There are some stamps and from ~16th place the text "key" starts: "value" ...

    I looked at the times and it looks like something is wrong with the UDP frame buffering.
    Parsing now takes 60 to 300us
  • ADVERTISEMENT
  • #25 21191019
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    mateos2 wrote:
    I looked at the times and it looks like something is wrong with the UDP frame buffering.
    Parsing now takes 60 to 300us
    .
    Without going through the functions from the Arduino you won't deduce much.
    A couple of years ago, when I was playing around with embedded on the esp8266, the network handling, ip frames, was done after the loop() function exited. The framework for esp32 was supposed to be based on rtos. What is it really like?
    In any case, the Arduino is suitable for simple programs, send data from attached sensors to a server, handle relays. But for such a server/concentrator type project it is better to use esp idf with rtos if you are going to stay on that hardware.
  • #26 21191021
    Anonymous
    Level 1  
  • #27 21191022
    michal.zd
    Level 31  
    Posts: 1614
    Help: 83
    Rate: 262
    mateos2 wrote:
    There are some stamps a from ~16th place
    .
    That's a bit much for mqtt.
  • #28 21191044
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    khoam wrote:
    what buffering specifically
    .
    Buffering for UDP frames
    I need to review the library that I use: asyncudp
  • #29 21191115
    xury
    Automation specialist  
    Posts: 7071
    Help: 876
    Rate: 1486
    What are these 'thermostats' ? Since they are based on esp8266 it might be worth reflashing them to Tasmot ? I also don't understand what the problem is with collecting UDP frames.
  • #30 21191160
    mateos2
    Level 13  
    Posts: 151
    Help: 1
    Rate: 7
    >>21191115
    ad1/ as I already wrote TMS812 - it controls the chilled water circuit, underfloor heating and ventilation damper
    ad2/ it is possible, but I will not learn what Tasmota is and why it works + I will not take responsibility for the fact that it works or not
    ad3/ with the amount - how to collect 50-100 UDP broadcasts of 150 bytes in 50-100ms
Reply | New topic
Report a violation of the law

Topic summary

✨ The discussion addresses the challenge of handling and parsing frequent UDP broadcast packets from approximately 40 Wi-Fi thermostats using an ESP32-S3 module programmed via the Arduino platform. The main issue is the high UDP packet parsing time (up to 500 microseconds), which risks packet loss as the number of thermostats increases. Attempts to replace the ESP32 with an Arduino Opta 485 over Ethernet failed due to flooding and unresponsiveness. The parsing involves extracting key-value pairs from unstable JSON-like strings, currently implemented using Arduino String objects, which may cause overhead and heap fragmentation. Suggestions include optimizing parsing by using C-style string functions (e.g., strstr), employing std::string and std::string_view for efficient memory usage, and delegating parsing to a separate core on the ESP32-S3. The limitations of UDP broadcast over Wi-Fi are highlighted, noting that broadcast packets are slower and more prone to loss compared to unicast UDP. Increasing LWIP UDP receive mailbox size and using ESP-IDF instead of Arduino SDK for better stack configuration are recommended. Alternatives such as using a Raspberry Pi Zero W or Raspberry Pi 4 for better network stack handling and processing power are proposed. The thermostats (TMS812 with embedded ESP8266) send data every second without control over transmission timing, complicating packet reception. The system runs on a dedicated VLAN to reduce network noise. Tools like Packet Sender and Wireshark are suggested for traffic analysis. Overall, the consensus is that while ESP32-S3 is capable, the Arduino framework and UDP broadcast nature impose constraints, and stronger hardware or optimized software stacks may be necessary for reliable high-frequency UDP packet handling and parsing.
Generated by the language model.
Today's popular

FAQ

TL;DR: Handling 40 thermostats pushes ESP32-S3 UDP parsing to 500 µs per packet [Elektroda, mateos2, post #21189769]; "not necessarily" speed-limited, argues an expert [Elektroda, JacekCz, post #21189788] Switching to larger UDP mailboxes and lean C parsing cuts latency to 60-300 µs [Elektroda, mateos2, post #21191007]

Why it matters: Fewer missed packets means steadier HVAC control and lower energy bills.

Quick Facts

• ESP32-S3 dual-core Xtensa @ 240 MHz, 512 kB SRAM [Espressif Datasheet]. • Arduino-ESP32 default CONFIG_LWIP_UDP_RECVMBOX_SIZE = 6 slots [Elektroda, mateos2, post #21191290] • ESP-IDF allows 64-slot mailbox after recompilation [Elektroda, khoam, post #21192167] • Practical Wi-Fi broadcast cap: ≈7 packets/s vs 800 unicast packets/s [Elektroda, rb401, post #21191464] • Raspberry Pi Zero W: 1 GHz CPU, ~US$15, full Linux TCP/IP stack [Raspberry Pi Ltd. Specs].

Why is my ESP32 dropping UDP broadcast packets from many thermostats?

Broadcast packets share one small LWIP mailbox. Arduino-ESP32 ships with only six queue slots, so bursts of forty 150-byte frames overflow it and older frames are lost [Elektroda, mateos2, post #21191290] Wi-Fi also slows broadcast delivery, grouping them inside DTIM windows, so throughput collapses when all nodes transmit together [Elektroda, rb401, post #21191464]

What faster parsing method can I use in pure C/C++?

Keep the UDP payload in its original buffer, then:
  1. Locate keys with strstr()/memmem().
  2. Create std::string_view slices—zero allocations.
  3. Convert numbers only when needed. This removes temporary copies and cuts heap traffic [Elektroda, michal.zd, post #21190927]

Can I enlarge the UDP mailbox under Arduino?

Not directly. Arduino uses a pre-built ESP-IDF where CONFIG_LWIP_UDP_RECVMBOX_SIZE is fixed. Recompile the framework as an ESP-IDF component, set the parameter to 64, then flash the custom build [Elektroda, khoam, post #21192167]

Will a stronger microcontroller solve the issue?

Sometimes. A Raspberry Pi Zero W can sustain hundreds of UDP packets per second thanks to its full Linux stack and 512 MB RAM [Raspberry Pi Ltd. Specs]. ESP32-S3 offers two cores but still shares the same small mailbox unless reconfigured [Elektroda, khoam, post #21192167]

Do I really need 1-second updates from every thermostat?

Not for room-scale HVAC. Colleagues note that 10–60 second intervals keep comfort while easing traffic, and ignoring extra packets cuts CPU load [Elektroda, pikarel, #21189871; lopr_pol, #21189962].

How can I track ‘alive’ thermostats without parsing every packet?

Use the sender’s IP last octet as an index into a 255-byte array. Increment the slot on each arrival; only parse every Nth packet or when the counter changed. Reset counters with millis() to detect drop-outs [Elektroda, mateos2, post #21191438]

What edge-case failure should I watch for?

Heap fragmentation can leave large free space yet no contiguous block for a new String, triggering std::bad_alloc and reboot loops—a silent failure that mimics packet loss [Elektroda, khoam, post #21190975]

How do I move parsing to the second ESP32 core?

Use FreeRTOS:
  1. Create a task pinned to core 1: xTaskCreatePinnedToCore(parseTask,…,1);
  2. In the UDP callback, push packets into a lock-free queue.
  3. Let parseTask pop and process. This decouples radio handling from parsing and raises resilience [Elektroda, mateos2, post #21190771]

What’s the quickest way to benchmark Wi-Fi packet loss?

Install Packet Sender, fire 100 × 150-byte UDP broadcasts at 10 ms intervals, and watch captured counts in Wireshark on the ESP AP. This reveals mailbox overflows within minutes [Elektroda, khoam, post #21192254]

Could switching to unicast eliminate most losses?

Yes. Tests show ESP can handle ~800 unicast UDP packets per second, versus ≤7 broadcast packets before drops [Elektroda, rb401, post #21191464] Re-flashing thermostats for unicast or MQTT would greatly raise reliability.
Generated by the language model.
ADVERTISEMENT