logo elektroda
logo elektroda
X
logo elektroda

Efficient hosting of a simple HTML page on a microcontroller - GZip compression in HTTP

p.kaczmarek2 459 4
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
📢 Listen (AI):
  • Screenshot of a web application window for uploading files, showing a file list and a Gzip compression option. .
    OpenBeken has long had integration of the LittleFS file system allowing files to be stored and managed directly in the microcontroller's Flash memory. This allows you to conveniently create device-specific configuration panels, such a panel itself is implemented in HTML and its communication with the firmware is done through a REST interface, which I have already presented in the topic OpenBeken as a mini HTTP hosting - writing pages in Javascript, Tasmota's REST API etc . Here, in turn, I would like to show some very simple but efficient optimisation that I have recently implemented for this system.

    One of the goals of the exercise is to efficiently host a simple Web application written in VUE - all directly on the device. Let's see what we are dealing with - 11 files, sizes up to 38kB.
    Web application window with a compression button and a list of files ready for upload to LittleFS. .
    That's 167kB in total, a lot for a flash memory file system. On the BK7231 we already have trouble with this, as it interferes with the section from the OTA and for larger builds the update removes the LFS. However, there is a solution...

    HTTP has a built-in and generally supported mechanism to reduce the size of transmitted data based on Gzip compression. It requires packing the transferred bytes with this compression and adding the appropriate parameter to the packet header:
    
    	poststr(request, "Content-Encoding: gzip");
    
    .
    Worse, however, is the fact that you now have to compress the sent data on the server side, doesn't it?
    This is where the second optimisation comes in. Our microcontroller is only supposed to host the page, it doesn't need to edit it, so we can compress the files before putting them in LittleFS .
    I've created a helpful button in the web app for this, meaning the compression is done in the browser rather than on the MCU:
    Screenshot of a table summarizing web app files: 11 files, largest 38 kB, total size 167 kB, file names and sizes shown on the right. .
    Let's see the implementation of the compression:
    Code: Javascript
    Log in, to see the code
    .
    There really is only enough - Javascript does the rest. The MCU itself doesn't even know that a gz file is being uploaded. Zero changes to the firmware.

    However, you still need to change the sending of files from the MCU to the browser. Here you just need to detect the gz extension and add the information to the HTTP header:
    Code: C / C++
    Log in, to see the code
    .
    It's time to see how much memory this will save us:
    Screenshot showing a web interface for uploading and compressing files on the OpenBeken system using LittleFS. .
    167kB versus 40kB is a very big difference - that's more than four times less! In this way, it is not only possible to save space in the flash memory, but also to speed up the transfer, because less data is sent faster. All this at practically zero cost.

    You could end here, but I will take this opportunity to present some other side ideas and improvements I have concerning the LFS.
    1. LFS profiler .
    This simple program checks the timing of sending and receiving files from the LFS via HTTP GET and POST requests. The times and results of the operations are shown in the table. Below are the results for LFS on an external flash drive connected by software.
    Screenshot of a web tool for file compression and uploading to LittleFS, showing a file list and upload buttons with a “Zip before sending” option. .
    2. Supporting an external Flash bone .
    At the moment I have realised this based on software SPI, as the hardware SPI is not derived on the WB3S/CB3S, only on the CBU...
    Additional Flash memory for measurements for free? Communication protocol, write, read, erase .
    3. LFS optimisation .
    One of the simpler LFS optimisations is to increase the size of the working buffer. This allows faster read/write operations (more bytes are processed at once) at the cost of slightly higher RAM usage:
    Screenshot showing OpenBeken web panel for uploading files to LittleFS with Gzip compression support, featuring Select files and Upload compressed (GZ) buttons, and a list of uploaded files with sizes. .
    4. Execution of Berry scripts on the "server side" .
    Another interesting possibility implemented in the firmware is to use Berry as a PHP equivalent. My system allows its scripts to be inserted into HTML code. The scripts are then executed at the GET/POST request handling stage, all on the source MCU. Below is an excerpt from the associated test:
    Code: C / C++
    Log in, to see the code
    .
    Berry in the PHP role will be discussed in a separate topic.

    In summary , that famous one simple trick for hosting larger pages and scripts on the MCU is to store an already compressed version of the file and serve it with the appropriate header. The MCU essentially doesn't have to do anything here, well, except for that one HTTP header field what it says about compression.
    In further topics I will show what I have used such a system for and what I have managed to host.

    Cool? Ranking DIY
    Helpful post? Buy me a coffee.
    About Author
    p.kaczmarek2
    Moderator Smart Home
    Offline 
    p.kaczmarek2 wrote 12080 posts with rating 10096, helped 579 times. Been with us since 2014 year.
  • ADVERTISEMENT
  • #2 21602704
    sq3evp
    Level 37  
    What is the size of the compressed page?
    How much does the binary take to handle?
    Interesting theme and all in all will probably work.
    Simple pages don't take up much in memory.
  • ADVERTISEMENT
  • #3 21602879
    p.kaczmarek2
    Moderator Smart Home
    sq3evp wrote:
    What is the size of the compressed page?
    .
    The 'Web App' page used for the experiment is an overlay on OpenBeken and communicates with the firmware via a tiny sized REST interface. Normally it is downloaded from Github with GH Pages, but it is so small that it can potentially be hosted on the device.
    Well, and answering - without compression 167kB approximately, after gz compression 40kB.


    sq3evp wrote:
    .
    How much does the operating binary take up?
    .
    Adding GZ file serving to an existing firmware binaries is just a dozen lines of code, just detecting the gz extension and giving the appropriate entry in the HTTP header. Probably just 100 bytes or so. Remember that the compression itself is done before uploading to the device, that's the whole trick of the theme.


    sq3evp wrote:
    .
    The theme interesting and all in all will probably work.
    Simple pages don't take up much in memory.

    CSS style can be quite heavy, and now a small chart library would probably fit on the device too. The Web App in question is also quite rich and has a lot of features, for example a fully done in JS GPIO Doctor for testing the IO of the WiFi module:
    Efficient hosting of a simple HTML page on a microcontroller - GZip compression in HTTP .
    Or toolsy.
    Efficient hosting of a simple HTML page on a microcontroller - GZip compression in HTTP .
    Helpful post? Buy me a coffee.
  • ADVERTISEMENT
  • #4 21602930
    sq3evp
    Level 37  
    A simple page with a simple button will already be easier and lighter.
    Some text and bardz simple pages can probably be realised?
  • #5 21602941
    p.kaczmarek2
    Moderator Smart Home
    Simple examples of a button page on my firmware were discussed here:
    OpenBeken as a mini HTTP host - writing pages in Javascript, Tasmota's REST API etc .
    It shows how such a page on the MCU communicates with the backend on the device via a REST interface.

    And here is a practical example of use (see what kind of page the user made):
    Alternative HTML control page for TOMPD-63-WIFI .
    Helpful post? Buy me a coffee.
📢 Listen (AI):
ADVERTISEMENT