logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

More powerful than ESP32 MCU for UDP handling and packet parsing

mateos2 864 41
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
Reply | New topic
  • #31 21191228
    michal.zd
    Level 29  
    mateos2 wrote:
    take 50-100 UDP broadcasts of 150 bytes each in 50-100ms
    .
    But what is the problem. I'm confused.
    We've agreed that these are objects with quite a lot of inertia, and certainly this applies to office spaces, one measurement per minute is perfectly adequate, for other devices let it be a few dozen seconds. That's how much time you have to react.
    You don't mean to say that the stack is losing frames?
    The ip stack should buffer tcp/ip frames until the application layer receives them, adding to the previous comment I don't see where the problem is. You just receive the tcp frames one at a time, discard every one as not carrying a change without parsing, fix the code for this function, you should be ok. It doesn't have to be an immediate response.

    Added after 4 [minutes]: .

    mateos2 wrote:
    something is wrong with UDP frame buffering.
    Parsing now takes 60 to 300us
    .
    300 microseconds x 100 frames is 30 msec. This is pretty good for an Arduino framework.
  • ADVERTISEMENT
  • #32 21191290
    mateos2
    Level 12  
    michal.zd wrote:
    one measurement per minute is quite sufficient


    Ok then, to systematise
    1/ UDP frames are sent by thermostats over which I have no control. They do this every second.
    2/ in the worst case they all send data at the same time
    3/ apart from reading the data, it is important for me to count the number of active thermostats - hence I count the time for a given serial number from the last frame - if there has been no packet for a minute, I consider the thermostat to have dropped out and stop counting it
    The problem I have is that a thermostat drops out not because it is deactivated, but because its frames have been missed
    With every additional thermostat added, the time of missing frames increases - currently it is reaching 15-20 seconds....
    After adding more I either have to lengthen the ttl or catch frames more efficiently.

    As the thermostats have already been purchased, fitted and connected by the customer, I have to manage this somehow. It is unfortunately a problem to do the installation without thinking about automation.

    If I can't figure it out, I'll either have to wire it up, but that's about 1-2km of cable to lay, or I'll estimate based on pressure/flow and temperature sensors.

    Added after 21 [minutes]: .

    I think I found the cause in lwip
    The default mailbox for UDP is 6
    As RAM like ants, I increase it to 64 and it protests
  • #33 21191347
    khoam
    Level 42  
    mateos2 wrote:
    1/ UDP frames are sent by thermostats over which I have no control. They do this every second.
    2/ in the worst case they all send data at the same time
    .
    I don't think you can handle that kind of traffic with a single ESP32 with online parsing, or even ESP32-S3. I think you need to go towards a Raspberry Pi 4 or something similar.

    Added after 12 [minutes]: .

    mateos2 wrote:
    I think I found the cause in lwip
    The default mailbox for UDP is 6.
    As RAM is antsy, increase it to 64 and it protests
    .
    You won't change the LWIP settings if you are using the Arduino SDK - in this case sdkconfig is unconfigurable as the already compiled ESP-IDF is used.
    https://espressif-docs.readthedocs-hosted.com/projects/arduino-esp32/en/latest/faq.html
  • #34 21191373
    michal.zd
    Level 29  
    mateos2 wrote:
    Default mailbox for UDP is 6
    ridiculously low.
    So the opinion on Arduino is confirmed, for simple stuff ok.
    khoam wrote:
    I think you need to go towards Raspberry Pi 4
    .
    That's what I've been suggesting all along, but the rpi zero W will also be suitable. This is more about a decent network stack.

    Added after 5 [minutes]:

    mateos2 wrote:
    count the time for a given serial number
    .
    You don't need to parse the data, each sensor has a unique udp socket ie address:port.
    You can skip the next few packets after reading the udp packet without parsing.
    But by setting up the network stack in this way, it will lead to packet loss itself.
  • ADVERTISEMENT
  • #35 21191438
    mateos2
    Level 12  
    michal.zd wrote:
    You don't have to parse the data
    .
    O And this is a mega interesting option
    Either the last octet of the address or parsing to the serial of the thermostat to control the flag that the thermostat is alive, and every 10 or 20 packets do the parsing
    The octet will be the most efficient, because it will be enough to allocate the byte[255] table and only when the octet exceeds 20 packets will it parse.

    However, if it is the stack that loses it, I don't know if it will help much because the gain may be too small
  • #36 21191464
    rb401
    Level 39  
    mateos2 wrote:
    data from 40 Wi-Fi thermostats that broadcast their readings every second with a UDP broadcast.
    .

    There is a general problem with WiFi technology that just broadcast UDP, runs much slower than UDP to a specific IP. This is due, generally speaking, to the way broadcast UDP is transmitted not immediately but in some packets a few times per second (which can be adjusted somewhat in some network devices but does not solve the problem). I won't say the details for now because I can't remember but I used to do trials, practical comparisons of UDP broadcast transfer and a specific IP address. The difference was colossal. Specifically, with ESP it was possible to send a maximum of 6-7 UDP broadcast packets per second, because if more were sent, they were lost. But specifically addressed UDP packets could be sent 800 per second without any problem.
    Anyway, the topic of poor UDP broadcast bitrates on WiFi is quite often reported on the Internet, e.g:
    https://forum.arduino.cc/t/broadcasting-udp-over-local-ap-poor-performance/580383
    and changing to UDP with a specific address helps definitively.
    I therefore advise against using broadcast UDP for cyclic data. It is better to use UDP broadcast for data on a specific IP of the receiver and use UDP broadcast at the receiver only to broadcast its IP address once every few seconds for the transmitters that have just joined the network.
  • #37 21191556
    michal.zd
    Level 29  
    rb401 wrote:
    it was possible to send
    .
    In this case it is all about receiving, sending from devices, as the author claims, is not affected.

    Added after 15 [minutes]:

    mateos2 wrote:
    Table allocation byte[255]
    .
    Basically if they are all on the same subnet then yes. Check what type of sender address is returned in the LWIP. It is best to operate directly on this type. Note that the ipv4 address is four bytes, which is theoretically a uint32_t type, so it's better to make sure how it is in the Arduino.

    Added after 26 [minutes]: .

    One more note. if this is going to work on a corporate network, there is a lot more udp and tcp rubbish on such a network than your broadcasts. Some portion of it will be directly routed to your device, making things even worse. Ideally, the wifi network for your needs should be separate. However, everything assumes that after these changes the number of lost packets will drop heavily, which may not be possible. Any way you look at it, a queue of six packets is stingy.

    Added after 6 [minutes]: .

    I would, however, opt for rpi and not fight problems that don't exist in normal systems (like qqrna socialism)

    Added after 6 [minutes]:

    mateos2 wrote:
    Do you know perhaps another module or version that can handle UDP receive AND parsing faster?
    The platform is of course the Arduino.
    .
    Unless the last sentence is a design assumption. Well then it will be difficult, even impossible.
    You have already met the limitation.
  • #38 21191624
    mateos2
    Level 12  
    michal.zd wrote:
    Preferably the wifi network for your needs is dedicated.
    .
    The whole IoT is on a dedicated vlan with rubbish isolation, of course the performance of the wi-fi itself will also be affected, but in general no more rubbish over UDP will reach the ESP.

    Added after 1 [minute]:

    rb401 wrote:
    This is generally due to the way the UDP broadcast is transmitted not immediately but in some packets
    .
    Apmdu as far as I remember - UDP frame aggregation
    Generally ESP supports this, but poorly :) .
    The biggest problem will be mailbox from what I'm looking at

    Added after 1 [minute]:

    michal.zd wrote:
    I would choose rpi though
    .
    I'm sticking with Arduino and ESP because of backwards compatibility - a lot of the code and building blocks I have on this is finished
  • ADVERTISEMENT
  • #39 21192167
    khoam
    Level 42  
    mateos2 wrote:
    Generally ESP handles this, but poorly
    .
    Not really. The CONFIG_LWIP_UDP_RECVMBOX_SIZE parameter can be increased to 64. Of course, then the program has to be written directly using ESP-IDF and not the Arduino SDK.

    Added after 2 [minutes]: .

    mateos2 wrote:
    I'm sticking with Arduino and ESP because of backwards compatibility - a lot of the code and bricks I have on this is ready
    .
    Alternatively, you can use the Arduino SDK as a component and then freely set the system parameters for ESP-IDF. However, this requires a recompilation of the entire SDK.
    https://espressif-docs.readthedocs-hosted.com...rduino-esp32/en/latest/esp-idf_component.html
  • #40 21192229
    rb401
    Level 39  
    michal.zd wrote:
    In this case it is about receiving, sending from the devices, as the author claims, is not affected.
    .

    Agreed. But the crux of the problem may be that the mechanism for transmitting UDP packets over WiFi networks is based on a decidedly different method for addressing specific IP and broadcast addressing.
    Although I'm not able to practically research this problem myself, I'm just signalling e as UDP broacast is something you need to be particularly careful about.
    But I am rather more inclined to see the cause of the problem, as e.g. my colleague khoam points out here, in the size of some buffers or the structure of the program, so that some practical tests with simple test applications on a PC transmitting such UDP packet streams and observing the result would be rather useful.
  • ADVERTISEMENT
  • #41 21192254
    khoam
    Level 42  
    rb401 wrote:
    so it would be rather useful to have some practical trials with simple PC test applications transmitting such UDP packet streams and observe the result.
    .
    I completely agree. For such purposes I use the program Packet Sender which I recommend. Link .
  • #42 21192273
    mateos2
    Level 12  
    >>21192254 .
    I'll start by looking at how it arrives at a single host
    Generating packets from a single host certainly looks different to packets collected by 15 or 20 different wifi Aps - aggregation works differently then.
    So that wire shark I will fire up and see what that traffic looks like.

    But I'll have to enlarge the mailbox anyway, especially as the tx and rx TCP buffers are terribly small and communication over VPN is poor in terms of throughput - at 100ms latency 200kbps is the maximum.
Reply | New topic
Report a violation of the law

Topic summary

The discussion addresses the challenge of handling and parsing frequent UDP broadcast packets from approximately 40 Wi-Fi thermostats using an ESP32-S3 module programmed via the Arduino platform. The main issue is the high UDP packet parsing time (up to 500 microseconds), which risks packet loss as the number of thermostats increases. Attempts to replace the ESP32 with an Arduino Opta 485 over Ethernet failed due to flooding and unresponsiveness. The parsing involves extracting key-value pairs from unstable JSON-like strings, currently implemented using Arduino String objects, which may cause overhead and heap fragmentation. Suggestions include optimizing parsing by using C-style string functions (e.g., strstr), employing std::string and std::string_view for efficient memory usage, and delegating parsing to a separate core on the ESP32-S3. The limitations of UDP broadcast over Wi-Fi are highlighted, noting that broadcast packets are slower and more prone to loss compared to unicast UDP. Increasing LWIP UDP receive mailbox size and using ESP-IDF instead of Arduino SDK for better stack configuration are recommended. Alternatives such as using a Raspberry Pi Zero W or Raspberry Pi 4 for better network stack handling and processing power are proposed. The thermostats (TMS812 with embedded ESP8266) send data every second without control over transmission timing, complicating packet reception. The system runs on a dedicated VLAN to reduce network noise. Tools like Packet Sender and Wireshark are suggested for traffic analysis. Overall, the consensus is that while ESP32-S3 is capable, the Arduino framework and UDP broadcast nature impose constraints, and stronger hardware or optimized software stacks may be necessary for reliable high-frequency UDP packet handling and parsing.
Summary generated by the language model.
Popular Topics
ADVERTISEMENT