logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

ESP8266 Wemos D1 - Handling String Overflow and Reboots with Large Files

jaskol 1017 16
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
Reply | New topic
  • #1 19847872
    jaskol
    Level 12  
    Hello everyone.
    I'm struggling myself, but I'm sure someone has had this problem before and will give some suggestions. I'm loading a file, 20kB, into a string (SPIFFS or LittleFS - makes no difference), substituting some strings and saving in a second file. As long as the file was small (I can't remember, maybe 10kB, but I'm guessing), it all worked fine. Now I see that once every few writes the processor reboots. I've turned on debug and I get the message every so often:
    [String] Reallocating large String(191 -> 192 bytes) [.......] Then I guess the stack overflows, in any case a reboot occurs.
    I can split this file into several smaller ones, but it messes up my concepts. I would prefer to know how to handle this. I'm not a programmer, and I'm sure there's someone here who knows right away whether to put this in some kind of array or solve it in yet another way.

    I'm not a programmer, please don't shout ;) .
    It simply loads the contents of the file into a string variable:

    String Config_Webpage = "";

    File file_init = LittleFS.open("/config_init.html", "r");
    file_init.setTimeout(5000); // could be 1000, could be 5000, makes no difference - anticipate the question why so much

    if (file_init) {
    Config_Webpage = file_init.readString();
    }

    I thought the program was crashing when writing to the file, but I did the traps and debug after RS showed that it was already crashing at the file read stage. Does anyone know how much you can "fit" in a String type variable ?

    I'd just like to add that if I declare an array with a fixed size, this won't quite solve the problem either, as this file can have a different size (+/- 1kB). I wouldn't want to permanently allocate an area of memory, because it's known that there's not enough of that all the time. Unless I can somehow neatly allocate and release.
    Please help, because I don't want to compromise myself any further, and something tells me that such problems were solved ages ago....

    Regards,
    Mariusz
  • ADVERTISEMENT
  • #2 19848021
    mpier
    Level 29  
    Hello,
    Now you are just trying to "gracefully allocate and release". Read all the data into a buffer previously reserved, but don't release it after use. Do you even need a whole file in RAM to swap a few bytes?

    Greetings.
  • ADVERTISEMENT
  • #3 19848058
    jaskol
    Level 12  
    mpier wrote:
    Hello,
    Now you are just trying to "gracefully allocate and release". Read all the data into a buffer previously reserved, but don't release it after use. Do you even need a whole file in RAM to swap a few bytes?

    Greetings.
    .
    Hi.
    I felt something like that, but was afraid to speak out loud ;) .
    I've already started doing this buffer because no one was responding, we'll see.
    If I don't slow it down, at least nothing will occupy that area of memory and maybe there'll be less chance of a "dump" in the future.
    I don't know how else to bite it.
    What I've done is that I have an html master file (config_init.html) where I have a web page with the device configuration.
    I have keywords in it, in which I change the values, and I've already imported the new ones into the second file, the one displayed to the user.
    You make changes on the website, a form is sent, ESP ingests it into a String, the "replace" function changes what it needs to in this String (depending on what you changed on the website) and saves it to the config.html file.
    I've been combining to load a few lines at a time, but it's pointless, as it takes a long time and the String can overflow anyway.
    On the web you enter your email for example, so the size of this file could change.
    I declared a String and thought I'd have peace of mind, because the processor would already take care of it, to "stuff" it properly in RAM.
    Well it does stuff it, but up to a certain point ;) .
    The question is whether it's known how many bytes will fit in there, because maybe I could control that and then step in ?
    I am making this buffer but would love to know if there is not a better way to solve my problem, sometimes a person's eyes flap....

    M.
  • #4 19848067
    mpier
    Level 29  
    If it is known in advance that there is not enough memory, it is safe to assume that the programme will not work properly anyway, and if it does work, there is not too little memory.

    You should be able to fit as much as you reserve. See reserve(). Judging by the name it makes sense, you need to check in the sources if it will be ok.

    Can't you save the data separately? You could then populate the data on the fly and leave the HTML template read-only. There must be some ready-made library for this.
  • ADVERTISEMENT
  • #5 19848620
    jaskol
    Level 12  
    mpier wrote:
    If it is known in advance that there is not enough memory, it is safe to assume that the programme will not work properly anyway, and if it does, the memory is not too little.

    You should be able to fit as much as you reserve. See reserve(). Judging by the name it makes sense, you need to check in the sources if it will be ok.

    Can't you save the data separately? You could then fill in the data on the fly and leave the HTML template read-only. There must be some ready-made library for this.
    .

    I do tests and fire up the program with:
    Free heap size: 20992 (bytes)
    From what I've read, that's just my amount of free RAM.
    20kB, that's a bit low. I read that the ESP8266 has 80kB at the start.
    So indeed a miracle is not going to happen. I'll either split the file into smaller parts, or switch to ESP32.
    I don't know how I would populate the data on the fly, I have it stored in an array and this is how I extract it from it,
    I look for a string in the file loaded into String, replace the string with that data and so on until the search is complete.
    Then I save the whole thing to a file.
    I have to load the whole file to search it and replace the data in one go, replace() function.
    I could line by line, but I still have to put everything together later to save to the file - saving after each change rather
    would be time-consuming.
    I could still rigidly make the file and I'd know which line I have which variable to replace, but that would limit me because any expansion of that file would involve re-analysing what's in which line, what's moved, etc.
    The only thing I've come up with now is that I'll make a start and end tag for the "proper" data and instead of loading the file from the beginning, I'll load it, but discard everything that was before the start tag (html headers and such).
    Just how much will I save ? Some small :( .
    I keep testing, but I can see that this is a major limitation.
    You write yourself a program, the memory is running out, although you think the data takes up 66% according to the compiler.
    Later you count and it turns out that there are a few K left and death.
    I have versions with ESP32 and PSRAM, probably time to get interested....

    Mariusz
  • ADVERTISEMENT
  • #6 19849190
    mpier
    Level 29  
    After all, you don't have to load the whole file at once. You don't have space for 20kB, but you can easily fit 2kB. You can send the text from the file with the substituted variables directly to the client or write it to a file.
    You have nothing to analyse after changing the contents of the file, if you write, for example, like this:
    Code: HTML, XML
    Log in, to see the code
    .
  • #7 19849244
    khoam
    Level 42  
    Why are you using file_init.readString()? Doing so can actually overflow the heap when operating on long Strings, and repeated loading of Strings also fragments that heap. Better to use file_init.readBytes() and load the data into a static array (stack allocation).
    Link .
  • #8 19849315
    jaskol
    Level 12  
    mpier wrote:
    After all, you don't need to load the whole file at once. You don't have space for 20kB, but you can easily fit 2kB already. You can send the text from the file with the substituted variables directly to the client, or save it to a file.
    You don't have anything to analyse after changing the contents of the file if you write, for example, like this:
    Code: HTML, XML
    Log in, to see the code
    .
    The problem is that this file has quite a lot of text in HTML+javascript to make it look like something.
    That's where the 20kB comes from.
    It doesn't want to upload directly, because it's such a MENU and tabbed file - different options divided into sections, tabs switch, but everything happens in one file. There are, for example, 5 or 6 independent forms. Generally, it's rare to change anything in the configuration of the device and you can afford to then wait, say, 3 seconds for a full zap and information on the OLED that "saved".
    If I were to split this into chunks of 2kB each, not quite enough time, but still another code to handle.
    What if in that 2kB I hit half of the name of some variable of mine and read %NAZ, and in the other 2kB is the rest, i.e. WISKO% ?
    It's possible to read in a few lines at a time, waiting for "\n", but I can already see that in a while I'll run out of memory for other things.
    I've started a program that shows me the occupancy with each line loaded from the file, and I can see how it gets off - e.g. when the declared array String buffer[500] overflows. It will load >500 and it falls down. If I give 600, it's still not enough and it crashes, but later.
    It's even the case that if I have the entire target batch and check it anyway, it already does a reboot when RAM is at 400B, and if I throw everything away and leave just the routine to test, I can get down to 4B of memory and it still works.
    I'm already switching a bit to the ESP32, but it's killing me again, because although the RAM occupancy is 14%, but the program memory is 72% ! And it's the same program. And as far as I remember, in ESP8266 I had a Flash of 4MB, while in ESP32 I have LOLIN D32 PRO and 16MB Flash ! Partition scheme I change, but somehow I still feel that some misery remains. 16MB and so busy ?

    Chip is ESP32-D0WD-V3 (revision 3)
    Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
    Crystal is 40MHz
    Auto-detected Flash size: 16MB

    Someone comfort me that I'm doing something wrong or reading something wrong, or will it turn out that I gain RAM and run out of FLASH in a while ?

    Thanks,
    Mariusz

    Added after 1 [minute]: .

    khoam wrote:
    Why are you using file_init.readString()? Doing so can actually overflow the heap when operating on long Strings, and there is also fragmentation of that heap when loading strings repeatedly. Better to use file_init.readBytes() and load the data into a static array (stack allocation).
    Link [/quote

    Thanks, I'll check how it behaves, but anyway, that space for that 20kB has to be found, sooner or later.
    Is it possible to somehow release the occupied space in the array when I no longer need it ?
    I've done this quickly, but I have the impression that if I fill up such an array String buffer[600] with the contents of the file,
    then the subsequent String buffer[600]="" didn't do anything.
    Is there an instruction to "clear" such an array entirely and free up memory?

    Mariusz
    .
  • #9 19849584
    khoam
    Level 42  
    jaskol wrote:
    khoam wrote:
    Is it possible to somehow free up the occupied space in the array when I no longer need it ?
    .
    How often will you use this memory area in the program? Just once? You can allocate and release the desired memory area dynamically (using new/delete), but in the case of 20kB I "thinly" see this on the ESP8266 (the program itself also uses the heap). It is certainly better to use a char[] array instead of String class objects.

    jaskol wrote:
    String buffer[500].
    .
    This way it's bound to be problems. Each String can be of different length , there will be 500 areas of different size allocated on the heap and it will run out of space.
  • Helpful post
    #10 19849593
    janek_wro
    Level 29  
    jaskol wrote:
    The problem is that this file has quite a lot of text in HTML+javascript to make it look like something.
    That's where the 20kB comes from.
    It doesn't want to upload directly, because it's such a MENU and tabbed file - different options broken down into sections, tabs switch, but it all happens in one file.
    .
    Maybe you are approaching this from the wrong side. Why manipulate a sizable file on this ESP?
    Make one file (hmtl+js), fully static content. Put the variables in a separate js file. Smaller.
  • #11 19850386
    mpier
    Level 29  
    jaskol wrote:
    And how in these 2kB will I hit half of the name of some variable of mine and read %NAZ, and in the other 2kB will be the rest, that is WISKO% ?
    .
    You won't hit the half because the %NAZ will always be at the beginning of the buffer, and the %CHANGENAME% will be much shorter than the length of that buffer.

    If you are writing in JS, you can substitute variables in the browser too.

    You must have some other big bug in your program that such problems are coming out.
  • #12 19850480
    janek_wro
    Level 29  
    mpier wrote:
    You won't hit half because the %NAZ will always be at the beginning of the buffer
    Further substantiate this claim. Because somehow I don't feel what you mean. After all, such a placeholder can occur anywhere in the template file, including where the later "break" will fall after reading a fixed number of bytes into the buffer.
    I guess that the reading into the buffer should only occur up to the occurrence of % (or whatever character(s) the template uses). And the rest in the next round.
    Or read 1 line at a time, i.e. up to CRLF. As long as the template has a CRLF, because it does not have to.

    mpier wrote:
    If you write in JS, you can substitute variables in the browser too.
    If the intended use assumes that the client will be online, in the sense of accessing the world, then you can even take the jquery/ui link from the CDN, and use its benefits. Because it's probably not enough space to build in a device?
  • #13 19851191
    jaskol
    Level 12  
    janek_wro wrote:
    jaskol wrote:
    The problem is that this file has quite a lot of text in HTML+javascript to make it look like something.
    That's where the 20kB comes from.
    It doesn't want to upload directly, because it's such a MENU and tabbed file - different options broken down into sections, tabs switch, but it all happens in one file.

    Maybe you're approaching this from the wrong side. Why manipulate a sizable file on this ESP?
    Make one file (hmtl+js), fully static content. Put the variables in a separate js file. Smaller.

    YES, this could be the solution to my problem !
    Well, and it will be fast.
    Thanks, I'll check it out, although I've already switched to ESP32 and as soon as I get rid of some urge in that webserver to serve everything in "gz" and in filesystem to add some prefixes (I've set what I need to "1" and it still adds "/littlefs", then maybe I'll stay with ESP32.
    However, if it turns out that with the ESP8266 and 4MB Flash I was at 60% occupancy, and with the ESP32 and 16MB I'm at 70%, I'll get really mad. I'm about to start implementing your idea, on this ESP32, here too it's better to have less to write and faster running code.

    Thanks,
    Mariusz

    Added after 4 [minutes]: .

    khoam wrote:
    jaskol wrote:
    khoam wrote:
    Is it possible to somehow free up the occupied space in the array when I no longer need it ?
    .
    How often will you use this memory area in the program? Just once? You can allocate and release the desired memory area dynamically (using new/delete), but in the case of 20kB I "thinly" see this on the ESP8266 (the program itself also uses the heap). It is certainly better to use a char[] array instead of String class objects.

    jaskol wrote:
    String buffer[500].
    .
    This way it's bound to be problems. Each String can be of different length , there will be allocated 500 areas of different size on the heap and it will run out of space.

    I understand, it's just that char was "not working out" for me something in practice.
    Even once it's ok, I'm worried that in a while I'll have 10kB of free memory left and it'll be over.
    If I free up memory and then use it up with something else and there's not enough to load the file, it'll all blow up too.
    As if they couldn't give a bit more of that RAM, after all, I would pay extra ;) .

    Mariusz

    Added after 8 [minutes]:

    janek_wro wrote:
    mpier wrote:
    You won't hit half because %NAZ will always be at the beginning of the buffer
    Further substantiate this claim. Because somehow I don't feel what you mean. After all, such a placeholder can occur anywhere in the template file, including where the later "break" will fall after reading a fixed number of bytes into the buffer.
    I guess that the reading into the buffer should only occur up to the occurrence of % (or whatever character(s) the template uses). And the rest in the next round.
    Or read 1 line at a time, i.e. up to CRLF. As long as the template has a CRLF, because it does not have to.

    I use string.replace() and it seems to work by looking for occurrences of the given string in the string.
    This is how I replace variables with values. That is, it has to find the whole string %NAME% to replace that with KOWALSKI.
    Yes, I've written about being able to load line by line, but if I go through each line each time and load the next line, the whole thing gets rather nightmarishly long.

    mpier wrote:
    If you're writing in JS, then you can substitute variables in the browser as well.
    If the intended use is that the client will be online, in the sense of accessing the world, then you can even take the jquery/ui link from the CDN, and use its benefits. Because in a device is probably not enough space to build in?
    .
    Yes, I've got jquery in the header because I've found that it works faster when taken from the net than when read from the filesystem.
    I wanted to be independent, I had everything locally and it was definitely running slower than when it was "from the net".
    It's also about page caching, browser caching and such, so it turns out that it's more efficient to load from the web.
    If the network is down, then I'll still be testing, because everyone assumes that the network always works....

    Mariusz
  • #14 19868860
    kaczakat
    Level 34  
    I have the web page saved in the INO file as follows:
    Code: text Expand Select all Copy to clipboard
    WebSocketsServer webSocket = WebSocketsServer(81);

static const char PROGMEM INDEX_HTML[] = R"rawliteral(
<!DOCTYPE html>
<html>
<head>
......
</html>
)rawliteral";
    .
    My first working example is THERE .

    It passes it to the client in its entirety, and then it is the client who substitutes the content for itself via Websockets. You can also move it to a separate project file and conveniently swap it with another one. In ESP, you can also simply use an html file uploaded to SPIFFS, in which case the page can be split into separate files. Attached is a game made for the World Wide Web according to the Youtube channel's HTML course "The Passion of Computer Science", html, css, js, graphics, core ESP 2.5.0. with SPIFFS you upload the ESP code separately and the contents of the flash drive separately.
    Attachments:
    Helpful post? Buy me a coffee.
  • #15 19868908
    jaskol
    Level 12  
    kaczakat wrote:
    I have the web page saved in an INO file like this:
    Code: text Expand Select all Copy to clipboard
    WebSocketsServer webSocket = WebSocketsServer(81);

static const char PROGMEM INDEX_HTML[] = R"rawliteral(
<!DOCTYPE html>
<html>
<head>
......
</html>
)rawliteral";
    .
    My first working example is THERE

    It passes it to the client in its entirety, and then it is the client who substitutes the content for itself via Websockets. You can also move it to a separate project file and conveniently swap it with another one. In ESP, you can also simply use an html file uploaded to SPIFFS, in which case the page can be split into separate files. Attached is a game made for the WWW according to the Youtube channel's HTML course "Passion for Computer Science", html, css, js, graphics, core ESP 2.5.0. with SPIFFS you upload the ESP code separately and the contents of the flash drive separately.
    .


    Thanks, great, I'll have a look, but from the method using PROGMEM, or keeping it in the MCU itself at all (to put it in a nutshell, as the file system is also in the same flash, after all), it doesn't quite work for me, as I modify the web page instantly, check how it looks and only when everything is ok do I upload it to SPIFFS. On top of that, in javascript it generates, for example, tables with 20 rows in a loop, while in fact there are only a few lines and a "for" loop in the file. Sure, you could also do it in MCU, but testing would be tedious, making changes too.
    Through PROGMEM I do, for example, replies to the client that something was successful or not. Simple and repeatable things.
    I implemented the idea with variables in javascript and it works great. The whole page can be huge and look nice (using Bootstrap, for example) and when the HTML is loaded, the javascript kicks in and substitutes variables, on the fly, for values. Clear, fast, simple. I'll stick with that, but thanks for the feedback, I'll of course look at your files out of curiosity and try to fish something out for myself ;) .
    Now I'm struggling with writing structures to EEPROM, because while put and get worked for me so far on single variables, when I write whole structures, after reading I don't have what I've written...but I'm still "researching" it, because there might be a problem again with overlapping addresses and such.
    And anyway, once something works, I start to find pleasure in optimising the code, and suddenly it turns out that with all these changes, memory is not so scarce after all ;) .

    Regards,
    Mariusz
  • #16 19868988
    kaczakat
    Level 34  
    For testing, the web page can also be on a PC in an HTML file, just change the line in it to "websock = new WebSocket('ws://192.168.1.30:81/');" and it displays the data talking to this ESP with ip 30.
    Helpful post? Buy me a coffee.
  • #17 19869001
    jaskol
    Level 12  
    kaczakat wrote:
    So for testing this web page can be and on a PC in an HTML file, just change the line in it to "websock = new WebSocket('ws://192.168.1.30:81/');" and it displays the data talking to this ESP with an ip of 30.
    .

    I'll be testing, as it's certainly ideal for sending real-time sensor data.
    As for the rest, I'll test that too, because so far, in this version of mine, everything works, but I make no secret of the fact that it's slow.
    Maybe with websockets it would be faster, and that could already be tempting....
    I'm reading, watching, checking. I will definitely use something from this ;) .

    Thanks to,
    Mariusz
    • Reply | New topic
    Report a violation of the law

    Topic summary

    The discussion revolves around handling string overflow and reboots on the ESP8266 Wemos D1 when processing large files (20kB) using SPIFFS or LittleFS. The user experiences reboots due to memory issues when manipulating large strings. Suggestions include using a buffer for data storage instead of loading the entire file into memory, utilizing static arrays, and dynamically allocating memory. Alternatives such as splitting the file into smaller parts or using JavaScript for variable substitution on the client side are also proposed. The user considers switching to the ESP32 for better memory management and performance. The conversation highlights the importance of efficient memory usage and the potential of WebSockets for real-time data handling.
    Summary generated by the language model.
    Popular Topics
    ADVERTISEMENT