logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

More powerful than ESP32 MCU for UDP handling and packet parsing

mateos2 864 41
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
Reply | New topic
  • #1 21189769
    mateos2
    Level 12  
    Welcome

    I have a working ESP S3 module that collects data from 40 Wi-Fi thermostats that broadcast their readings every second with a UDP broadcast. The module receives the packets, parses them and transmits the statistics to the BMS (rs485) and to the web server (http API).

    This works well, but the UDP packet parsing time is large - up to 500us, so adding more thermostats will increase the problem of missed packets.
    (I assume 500us is great and it will be hard to optimise - a faster MCU remains)

    Trying to swap the esp32 for an Arduino opta 485 connected over Ethernet ended in failure - module flooded with UDP broadcasts stops responding.

    Do you perhaps know of another module or version that can handle UDP reception AND parsing faster?
    Parsing is looking for keys and values in a string.
    The platform is of course Arduino.
  • ADVERTISEMENT
  • #2 21189788
    JacekCz
    Level 42  
    mateos2 wrote:
    which can handle UDP reception and parsing faster
    .

    These are two profoundly different issues, and should not be discussed "in the same breath".
    One has a lot to do with hardware (+operating system in case there is one) and nothing to do with application code, the other 100% user code and no hardware involvement


    mateos2 wrote:
    The platform is, of course, the Arduino.


    A good opportunity to sp....
    If it's a typical Arduino philosophy, with delays, port polling in a loop (rather than interrupts), it kills more activity

    Added after 2 [minutes]:

    mateos2 wrote:
    I assume that 500us is a great result and will be hard to optimize

    mateos2 wrote:
    Parsing is looking in string for keys and values.


    Hmmm, naive Arduino-style code can be a deal-breaker.
    Without seeing the code, the answer is "not necessarily" / "it depends".
  • #3 21189810
    mateos2
    Level 12  
    JacekCz wrote:
    Good opportunity to screw this up....
    If this is the typical Arduino philosophy, with delays, port polling in a loop (rather than interrupts), it kills more activity


    As I wrote - it uses a TCP stack AND simple string parsing. There's no magic to it, simple C portable to multiple platforms.

    Example of parsing code:
    Code: text Expand Select all Copy to clipboard
    
void parseudp(AsyncUDPPacket packet) {
long sn ;
int pos;
int pose;
int rssi;
String valt; 

char* tmpStr = (char*) malloc(packet.length() + 1);
memcpy(tmpStr, packet.data(), packet.length());
tmpStr[packet.length()] = '\0'; 
String data = String(tmpStr);
free(tmpStr);

    pos = data.indexOf("\"sn\"",pos)+6;
    pose = data.indexOf(",",pos)-1;
    valt = data.substring(pos+5,pose);
    sn = valt.toFloat();
    .

    String class used, because the values are then sent to the server as string anyway (to avoid string -> float -> string)
  • #4 21189871
    pikarel
    Level 38  
    Since it's just a readout, with no need for quick feedback to some heater/motor controller - it can take as long as 20ms to read one sensor (as long as one frame on TV) and you won't even notice it, and the refresh of data from the readout to the display can be every 10-20s, because nothing depends on it.
  • #5 21189898
    mateos2
    Level 12  
    But the idea is that I don't do any reading just catch the messages of the thermostats.
    I set ttl to 120s and if the thermostat hasn't sent anything for 120 seconds then I remove it from the list.
    Currently some thermostats are caught even after 15 seconds, which means adding more will do more UDP frames, more analysis time and they will start to drop out.

    The controller controls the heat pumps in groups and this can cause them to stop and start for the wrong reasons.

    I have no control over sending frames so the MCU has to catch all packets as much as possible.
    E.g. 40 thermostats will send packets in 100ms and I would like to catch as many as possible
  • ADVERTISEMENT
  • #6 21189962
    lopr_pol
    Level 32  
    Do you seriously need a temperature every 1sec? Who made these thermostats that they are haying communication like this? I would ignore readings more frequent than 2-5min and that's it.
  • #7 21190241
    pikarel
    Level 38  
    Quote:
    data from 40 Wi-Fi thermostats that report their readings
    .
    Why are you doing a reading of these thermostats?
    For fun, i.e. to look at them for yourself?
  • #8 21190270
    mateos2
    Level 12  
    lopr_pol wrote:
    Who made these thermostats that they are haying communication like this?

    Chinese :) sometimes it is the case that you find a solution and have to adjust

    Added after 3 [minutes]:

    pikarel wrote:
    Why are you doing a reading of these thermostats?
    .
    Because the reading from them is used to control the heating, ventilation and cooling of the office building.
    By knowing the thermostats and the volume they serve, you can select the minimum operating point of the systems saving the client massive amounts on bills.
  • #9 21190357
    pikarel
    Level 38  
    Do you know what the thermostat is for, what its role is?
    Do you know how complex the control algorithms inside their own processor are?
    You by looking at the readout of their operation - because they have a rogue - want to save money?
    Good.
  • ADVERTISEMENT
  • #10 21190400
    mateos2
    Level 12  
    pikarel wrote:
    Do you know how complex the control algorithms inside your own processor are?
    .

    But I understand that this is ironic?
    They have an algorithm for 3 lines of code:) bimetal simulator

    pikarel wrote:
    You looking at the readout of their work - because they have a rogue - you want to save money?
    .
    You didn't read my post
  • #11 21190737
    michal.zd
    Level 29  
    mateos2 wrote:
    JacekCz wrote:
    Good opportunity to screw this up....
    If it's typical Arduino philosophy, with delays, querying ports in a loop (rather than interrupts), it kills more activity


    As I wrote - it uses a TCP stack AND simple string parsing. There's no magic to it, simple C portable to multiple platforms.

    Example of parsing code:
    Code: text Expand Select all Copy to clipboard
    
void parseudp(AsyncUDPPacket packet) {
...
char* tmpStr = (char*) malloc(packet.length() + 1);
memcpy(tmpStr, packet.data(), packet.length());
tmpStr[packet.length()] = '\0'; 
String data = String(tmpStr);
free(tmpStr);

    pos = data.indexOf("\"sn\"",pos)+6;
    pose = data.indexOf(",",pos)-1;
    valt = data.substring(pos+5,pose);
    sn = valt.toFloat();
    .

    String class used, because the values are then sent to the server as string anyway (to avoid string -> float -> string)

    You've gone a bit overboard. So many memory allocations to chunk one data.
    The packet comes as probably void*, search for items with the strstr function that act like indexof, then there won't be two resource allocations. It's always a bit of microseconds gained.

    As for the original question, you could port the nanrpi zero W code, price very similar, but the ESP32 has a fair bit of processing power too. It is possible that the Arduino framework is killing it. Isn't he based on rtos?

    Added after 7 [minutes]: .

    mateos2 wrote:
    40 thermostats will send packets in 100ms and I would like to catch as many as possible
    .
    The stack should hold them all until read.
    The idea of discarding packets coming from the same sensor is a good one. Know which dp socket the packet is coming from, note the time and discard the rest according to a counter or timer
  • #12 21190771
    mateos2
    Level 12  
    michal.zd wrote:
    .
    You've gone a bit overboard. So many memory allocations to extract one data.


    I extract 6 data from each UDP packet - format is unstable (JSON similar), only keys are stable
    neither memory allocation nor fragmentation hurts me - there is a spare
    working on char type didn't speed up much, so I left String

    I'll have a look how exactly these packets are handled, and maybe I'll try to do some FIFO queuing
    or I'll finally get around to delegating parsing to another core than TCP handling :) but then say goodbye to easy portability
  • #13 21190777
    pikarel
    Level 38  
    To clarify; a thermostat is a device that autonomously stabilises a set temperature by operating the devices that provide this stabilisation, e.g. motors, heaters. Some thermostats are equipped with communication for remote reading of this temperature.
    You have trouble reading it, besides, your ESP does not feedback control to any of the thermostats.

    What I'm asking is not ironic; I'm asking why you're doing it, because if not for control, then just for the fun of reading it "because you can".

    Systems with remote desktop control have their own operating programs for this, with service authorisation, with no access to parameters without that authorisation.
  • #14 21190810
    JacekCz
    Level 42  
    mateos2 wrote:
    Memory allocation or fragmentation doesn't hurt me - there's a spare
    working on char type didn't speed up much so I left String
    .

    The hard points of the issue are not where you see them.
    Allocation is not just "aggregate memory usage", but each "piece" must be handled in the manager, e.g. releasing costs non-trivial time. And it's that time that's most valuable when it's hottest.
    Invoking a new String, each piece, is the same. Every visible String invoked, substr() etc ...

    michal.zd wrote:
    You've gone a bit overboard. So many memory allocations to chunk one data.
    .
    michal.zd wrote:
    Possible that the Arduino framework is killing it.
    .

    Reasonable assumptions

    Added after 8 [minutes]:

    mateos2 wrote:
    Working on the char type didn't speed up much so I left String
    .

    You can optimise such a theme beautifully on char. As much as I am not a fan of C, in such limited applications it has prospects of
    All reception to those few characters of output data without/with one allocation

    You didn't have a significant gain because it's not done well. Rescue, stronger processor needed.
  • #15 21190884
    mateos2
    Level 12  
    JacekCz wrote:
    You have not had a significant gain because it is not well done. Rescue, stronger processor needed.


    of course yes , but .... $ vs time
    spending 100-300zl on each MCU is for me sometimes cheaper than optimising the code :) .
    with 10 pieces of product per year the optimisation is too expensive

    the times when I wanted to push code with TCP stack on Arduino uno are gone - now I respect time more + if someone will come to develop the code, it's easier to find someone who can handle string in Arduino than someone who can handle pure C
  • #16 21190913
    JacekCz
    Level 42  
    mateos2 wrote:
    with 10 units of product per year, optimisation is too expensive
    .

    The code I have in my imagination is no hackneyed optimization, just normal decent knowledge of C, not more lines of code at all
    I have a not-so-wild rule: I don't practice commercialism on tools I don't know professionally (I'm not talking about perfection, because it doesn't exist).
    In my world, it wouldn't be ethical for me to perform appendectomies, play guitar commercially, or program in languages I don't know very well (PHP, JS, Basics).

    mateos2 wrote:
    if someone comes to develop code, it's easier to find a person who can handle strings in Arduino than one who can handle pure C
    .

    I can't imagine a processional worker programming an arduino commercially, and not knowing C. But then again, I'm probably out of date
    Let's make a bowel movement because someone will continue it (and I do). Typical arguments in arduino climates
  • #17 21190918
    mateos2
    Level 12  
    JacekCz wrote:
    mateos2 wrote:
    with 10 pieces of product per year, optimisation is too expensive
    .
    I have a not-so-wild rule: I don't practice commercialism on tools I don't know professionally (I'm not talking about perfection, because it doesn't exist)


    unfortunately playing around with BMSs requires being an automation specialist, coder, electrician, electronics engineer, plumber, pumping engineer, PV engineer, HVAC engineer, etc. at the same time.
    a bit of everything, so nothing too deep - the 80/20 rule
  • #18 21190927
    michal.zd
    Level 29  
    mateos2 wrote:
    than optimisation
    .
    Sorry, but this is not a good approach. From the outset you need to keep in mind the overheads of cppi to properly match the tool to the capabilities.
    It's like no processor without mmu, acct the string class without continuous memory allocation can't do much.
    But in your case, with everything already in memory, there is no point in reassigning it several times, base it on the available c functions, or write a string class that operates on ready data. I did this once, my own implementation of the string class running on an already allocated buffer, added the functions I needed. Not much needed.

    Added after 6 [minutes]:

    mateos2 wrote:
    everyone will come to develop the code
    .
    If you keep the code simple and clear, then you have nothing to worry about. It won't be your fault if a 'developer' rather than a programmer gets into your code. As it stands, the code is not simple and clear, it would not pass review.

    Added after 11 [minutes]:

    Back on topic.
    If you are assuming extending the controller with more sensors, you might actually go straight to another platform. I would choose a raspberry pi even if only a zero W. An additional plus is the greater stability of production, in a few years there will still be some minicomputer, rpi zero probably also, Linux too, so the code will be possible to possibly recompile to another architecture.
    Scalability basically unlimited, extension of functionality also.
    What are these sensors? Don't they have the ability to communicate with an MQTT server? This would be the best option.
  • #19 21190956
    mateos2
    Level 12  
    michal.zd wrote:
    What are these sensors? Don't they have the ability to communicate with an MQTT server? This would be the best option.
    .

    TMS812 sensors from China - the customer already has these mounted and connected throughout the site - ESP8266 inside
    manufacturer support ends on sale :)

    I had the choice to either
    1/ connect to the 230V output of each - i.e. 3 point control
    2/ select 20% of those represented and connect the cables
    3/ rev-enginnering

    50 or even 10 points means hundreds of metres of cable to pull, so I chose option 3
    it worked out that I would either sniff HTTPS, or use UDP broadcasts
    (probably the Chinese were planning an aggregate management system of some kind).
    UDP is bare text so I just went that way

    thanks to this I not only have the information whether to heat or cool, but also the actual temperature and by the serial number for comparison I have the exact information how many m2 or m3 I have to cool or heat

    and this is valuable information
    because if I have to heat 20% of a room by 0.2 degrees, it's a completely different power than if I have to heat 80% of a room by 2 degrees.
  • #20 21190961
    michal.zd
    Level 29  
    mateos2 wrote:
    UDP is bare text so I just went that way
    .
    Very good idea. I'm curious if the data resembles the mqtt framework. Is it json from the start, or are there a few odd tags?

    Added after 56 [seconds]:

    mateos2 wrote:
    hysteresis of 0.2 degrees is quite
    unrealistic. I have doubts that half a degree of hysteresis can be maintained on a large volume.
  • #21 21190975
    khoam
    Level 42  
    mateos2 wrote:
    memory allocation or fragmentation doesn't hurt me
    .
    With heap fragmentation in the ESP32, it's not as loose as you think. There is no defragmentation mechanism, so you might run out of heap space ... even if, in theory, there will be. "Arduino's" String class with is a good way to make that memory on the heap run out if you "gleefully" create lots of temporary copies of strings or parts of strings.

    It is already better to use std::string from C++, and to perform the operations of reading or parsing string fragments using std::string_view. Std::string_view objects can also be directly created on char*, thus accessing a number of methods that make searching easier, for example - they are faster and do not need memory allocation on the heap.
  • #22 21190994
    michal.zd
    Level 29  
    khoam wrote:
    Std::string_view objects can also be directly created on char*,
    .
    I gave a plus for this. This is a very good idea. I wonder if Arduino will let you use the library std.
  • #23 21190996
    khoam
    Level 42  
    michal.zd wrote:
    I wonder if the Arduino will let you use the std library.
    .
    There is a complete std library in the toolchain for ESP32* and it is available in the Arduino environment. You can also (if you wish) perform conversions between String and std::string objects both ways although in my opinion you should primarily use std::string_view objects where possible.
  • ADVERTISEMENT
  • #24 21191007
    mateos2
    Level 12  
    michal.zd wrote:
    Very good idea. I wonder if the data resembles a mqtt frame. Is it json from the start, or are there a few odd tags?
    .

    There are some stamps and from ~16th place the text "key" starts: "value" ...

    I looked at the times and it looks like something is wrong with the UDP frame buffering.
    Parsing now takes 60 to 300us
  • #25 21191019
    michal.zd
    Level 29  
    mateos2 wrote:
    I looked at the times and it looks like something is wrong with the UDP frame buffering.
    Parsing now takes 60 to 300us
    .
    Without going through the functions from the Arduino you won't deduce much.
    A couple of years ago, when I was playing around with embedded on the esp8266, the network handling, ip frames, was done after the loop() function exited. The framework for esp32 was supposed to be based on rtos. What is it really like?
    In any case, the Arduino is suitable for simple programs, send data from attached sensors to a server, handle relays. But for such a server/concentrator type project it is better to use esp idf with rtos if you are going to stay on that hardware.
  • #26 21191021
    khoam
    Level 42  
    mateos2 wrote:
    I looked at the times and it looks like something is wrong with the buffering of the UDP frames.
    .
    Which buffering specifically are you referring to i.e. which buffer?

    michal.zd wrote:
    Framework for esp32 was supposed to be based on rtos. How is it really ?
    .
    It is based on FreeRTOS. The loop() function is one of the threads.
  • #27 21191022
    michal.zd
    Level 29  
    mateos2 wrote:
    There are some stamps a from ~16th place
    .
    That's a bit much for mqtt.
  • #28 21191044
    mateos2
    Level 12  
    khoam wrote:
    what buffering specifically
    .
    Buffering for UDP frames
    I need to review the library that I use: asyncudp
  • #29 21191115
    xury
    Automation specialist  
    What are these 'thermostats' ? Since they are based on esp8266 it might be worth reflashing them to Tasmot ? I also don't understand what the problem is with collecting UDP frames.
  • #30 21191160
    mateos2
    Level 12  
    >>21191115
    ad1/ as I already wrote TMS812 - it controls the chilled water circuit, underfloor heating and ventilation damper
    ad2/ it is possible, but I will not learn what Tasmota is and why it works + I will not take responsibility for the fact that it works or not
    ad3/ with the amount - how to collect 50-100 UDP broadcasts of 150 bytes in 50-100ms
Reply | New topic
Report a violation of the law

Topic summary

The discussion addresses the challenge of handling and parsing frequent UDP broadcast packets from approximately 40 Wi-Fi thermostats using an ESP32-S3 module programmed via the Arduino platform. The main issue is the high UDP packet parsing time (up to 500 microseconds), which risks packet loss as the number of thermostats increases. Attempts to replace the ESP32 with an Arduino Opta 485 over Ethernet failed due to flooding and unresponsiveness. The parsing involves extracting key-value pairs from unstable JSON-like strings, currently implemented using Arduino String objects, which may cause overhead and heap fragmentation. Suggestions include optimizing parsing by using C-style string functions (e.g., strstr), employing std::string and std::string_view for efficient memory usage, and delegating parsing to a separate core on the ESP32-S3. The limitations of UDP broadcast over Wi-Fi are highlighted, noting that broadcast packets are slower and more prone to loss compared to unicast UDP. Increasing LWIP UDP receive mailbox size and using ESP-IDF instead of Arduino SDK for better stack configuration are recommended. Alternatives such as using a Raspberry Pi Zero W or Raspberry Pi 4 for better network stack handling and processing power are proposed. The thermostats (TMS812 with embedded ESP8266) send data every second without control over transmission timing, complicating packet reception. The system runs on a dedicated VLAN to reduce network noise. Tools like Packet Sender and Wireshark are suggested for traffic analysis. Overall, the consensus is that while ESP32-S3 is capable, the Arduino framework and UDP broadcast nature impose constraints, and stronger hardware or optimized software stacks may be necessary for reliable high-frequency UDP packet handling and parsing.
Summary generated by the language model.
Popular Topics
ADVERTISEMENT