logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)

jkwim 3579 56
ADVERTISEMENT
Reply | New topic
  • #31 20947236
    jkwim
    Level 12  
    I tried to build a small shell script to do the minification using my laptop by accessing the API's given for the tools.
    It works except I am having an issue with the large chunk of javascript found in filesystem.vue.
    Anyway that is trivial as I don't mind having to do that manually.

    With your help I now have a means of running OpenBK devices in a closed WiFi network and have access to Logs & Filesystem interfaces for on and off maintenance.

    I even tested configuring the other devices also to use same FS. So these files need to be stored in one device only.

    I also have Tasmota device with a RTC and it is running a NTP Server.

    So for my specific closed network I can now freely add more OpenBK devices.
  • ADVERTISEMENT
  • #32 20955109
    Angel0fDeath
    Level 13  

    @jkwim I'm very glad everything is okay with you.
    Finally had some time to test it - I confirm it works like a charm... although I'm not going to use it.
    Please consider writing a tutorial and posting it.

    EDIT: To be clear - I used your tar file for the test
  • ADVERTISEMENT
  • #33 20955148
    p.kaczmarek2
    Moderator Smart Home  
    That's a very good news, it indeed deserves a tutorial! I would be able to link it from a Github to your work @jkwim then.
    Helpful post? Buy me a coffee.
  • #34 20955352
    Angel0fDeath
    Level 13  

    @p.kaczmarek2 I'm pretty sure without your help (RTOS delay) this couldn't happen, so thank you for the work you are doing and keep going. Personally, I will help with whatever I can for further development. Thank you.

    EDIT: And I already have a list for you to implement, but you probably already know my list from my posts
  • #35 20955878
    p.kaczmarek2
    Moderator Smart Home  
    Angel0fDeath wrote:

    @p.kaczmarek2 I'm pretty sure without your help (RTOS delay) this couldn't happen, so thank you for the work you are doing and keep going. Personally, I will help with whatever I can for further development. Thank you.

    Do you know how to do online builds? It would be great if you had time to try playing around this RTOS delay, maybe it's possible to get better settings/better stability somehow, but the delay may need to be adjusted. Maybe delay must be more frequent or longer? You can do it fully online, without having any toolchain on your PC.
    See:
    OpenBeken online building system - compiling firmware for all platforms (BK7231, BL602, W800, etc)


    Angel0fDeath wrote:

    EDIT: And I already have a list for you to implement, but you probably already know my list from my posts

    I like your suggestions, but I am just a single person and it's sometimes hard for me to track things. So, please, feel free to repost your requests or open a separate topic about them. Don't worry, it will be considered helpful and not rude, altough the final decision of implementation depends on many factors, for example, on useability versus implementation difficulty ratio.

    In general, to anyone reading - any suggestions and help is welcome, feel free to post questions and requests on our forum. We're here to help. It's just as I said, I'm just a one person, it's hard to reply to every request quickly. You can support me via: https://www.paypal.com/paypalme/openshwprojects
    Helpful post? Buy me a coffee.
  • #36 20955948
    Angel0fDeath
    Level 13  
    Will check it :)
    Concerning the list - it is not so long... currently:
    1. Smaller block size for Little FS if possible. 4K block size is too much for small partitions
    2. Flags 0 doesn't clear all flags. I also don't know why this command behaves differently on T and N devices setting different flags on...
    3. Flags page in device - the number in the beginning is not giving any useful information, unless you are very good in converting the number from dec to bin and see which bits are on - without calculator of course. More informative will be just to list the numbers of active flags, if someone can remember the meaning of all of the flags.... In general not needed...
  • ADVERTISEMENT
  • #37 20956315
    jkwim
    Level 12  
    p.kaczmarek2 wrote:
    That's a very good news, it indeed deserves a tutorial! I would be able to link it from a Github to your work @jkwim then.


    Give me a couple of days. I will try to put things together for a tutorial. For me it works to my liking now.
  • #38 21587296
    p.kaczmarek2
    Moderator Smart Home  
    I am working on a feature that may be related to this topic.

    I am going to try moving LFS to external flash chip, currently interfaced by software SPI. I am not sure about the perfomance yet, but I'll just check this once it works.

    This would mean that we could get like even 4MB or 8MB of LFS.
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)
    Anyone interested in testing that? @divadiow do you have any flash chips? You know, like from old laptop, etc?
    Helpful post? Buy me a coffee.
  • #39 21587304
    insmod
    Level 25  
    Why flash chips and not SD? SD can be used over SPI.
    With them we can get gigabytes of space, and adapters are usually very cheap.
  • #40 21587324
    p.kaczmarek2
    Moderator Smart Home  
    SD card is also good idea, but I chose SPI Flash chips for a start because I just have lots of them from various scrap devices and they are also easier for install to me - no need to buy adapter, just a piece of proto board is enough (sample: https://www.elektroda.pl/rtvforum/topic3993012.html )

    My personal end use for such large LFS would be something like a measurements history, so I don't really need tens of megabytes, 4 or 8MB will be enough.

    Still, everything depends on speed and stuff, I will probably try to rewrite read/write routines to be faster and non-portable. As far as I remember, when testing bit bang WS2812, I managed to narrow down BK7231 port operations down to registers, and I guess they will be much faster than using SPI_Write HAL with platform HAL and Beken HAL.
    Short sample of what I attempted in the past:
    Code: C / C++
    Log in, to see the code
    Helpful post? Buy me a coffee.
  • #41 21587342
    divadiow
    Level 35  
    p.kaczmarek2 wrote:
    @divadiow do you have any flash chips?

    not as many as I would like. 512k, 2mb, 1mb, 8mb maybe

    I do have BK7252N/U, TXW817 (😁) and of course XF16 cams with micro-SD in place
  • #42 21587410
    p.kaczmarek2
    Moderator Smart Home  
    It seems it's almost working. Currently I decided to hardcode LFS switch to SPI chip, but I am not sure.... but still, who else would use it? It's in obk_config.h
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)
    I am doing final fixes, then I will check if it's even useable in current state.
    Helpful post? Buy me a coffee.
  • #43 21587412
    Angel0fDeath
    Level 13  
    Hm. 4 - 8 MB of data is not so much. You can store your history on home automation server. Bigger LFS will be great if SD card used.. In this case FS will not be deleted (hopefuly) during OTA update. But in general idea is good to move FS outside of the flash.
  • #44 21587418
    p.kaczmarek2
    Moderator Smart Home  
    I can think about SD card solution for you later, as @insmod also requested it, but for my target purpose (measurements storage for history) flash chips are perfect.

    However, as far as I remember, I will probably need to pick some existing FAT32 library for SD card. Still, it was some time since I did SD card support on PIC32 or whatever it was.

    Added after 1 [minutes]:

    PS: I love this tool:
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)
    With this (and OBK simulator) I can developer for OBK even while extremaly low on free time.

    Added after 4 [minutes]:

    Interesting, it writes in 16 byte chunks internally.
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)

    Added after 13 [minutes]:

    http_rest_post_lfs_file has a delay... inside,
    Helpful post? Buy me a coffee.
  • ADVERTISEMENT
  • #45 21587442
    Angel0fDeath
    Level 13  
    >>21587418 16 bytes chunks I think is ok for this cpu. Maybe more cash is needed for more. But in general flash chips dont give big performance... Btw, Im not going to use obk devices as hard drives 😀. Just remembered an year ago a guy wanted to host web app on one of his devices and we both helped him. Thats the reason Im saying it will be better with SD card. And make this feature optional or via flag, i.e. not by default

    Added after 5 [minutes]:

    And why you use http to write over spi, no spi fs library?
  • #46 21587455
    p.kaczmarek2
    Moderator Smart Home  
    600 kB file uploaded... but it took way too long.
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)


    I am using my own SPI write/read bitbang code (not yet optimized, I am going to optimize it A LOT) and I write to it through LittleFS
    Helpful post? Buy me a coffee.
  • #47 21587458
    Angel0fDeath
    Level 13  
    >>21587455 I think the problem is with the size of your internal variables. An year ago I told you you dont host the flags correctly, because variables are max 32 bits. So for larger values you should make 2 (or 3) variables and to calculate them correctly. Negative values in your post mean variable is overflowed and not clear how much times.
    Each overflow of the variable leads to negative values, if not declared as byte or word, but even if declared on some cpus after overflow it is interpreted as negative value
  • #48 21587487
    p.kaczmarek2
    Moderator Smart Home  
    What? This has nothing to do with flags and it's not negative. It's by @btsimonh design the size is shown after a dash. Sample:
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)

    I was just showing the size of uploaded file. 600kB.
    Helpful post? Buy me a coffee.
  • #49 21587490
    Angel0fDeath
    Level 13  
    >>21587487 ok, then sorry. My mistake 😀
    Didnt been here more than an year - personal problems... Sorry
  • #50 21587493
    p.kaczmarek2
    Moderator Smart Home  
    Don't worry, it's nice to see you here again.

    There was (or maybe still is?) a flags issue, but as far as I know it's related to printing 64-bit variable to string. Not sure about it.

    Ok, I'm trying to make bit bang SPI faster now.
    Helpful post? Buy me a coffee.
  • #51 21587517
    Angel0fDeath
    Level 13  
    >>21587493 with the flags - not only printing, clearing also (didnt tested). ClearFlags command didnt clear all. Maybe already fixed...
    Nevertheless, Im back, and thank you for all your work.
  • #52 21587542
    p.kaczmarek2
    Moderator Smart Home  
    Fast bitbang SPI path for Beken almost ready:
    Code: C / C++
    Log in, to see the code

    Maybe I will also unwind and inline read/write loops
    Helpful post? Buy me a coffee.
  • #53 21587591
    Angel0fDeath
    Level 13  
    >>21587542 the code looks good,and probably will do the trick. Unfortunately cannaot help you (testing) - no flash chips 😞
  • #54 21587596
    p.kaczmarek2
    Moderator Smart Home  
    Well, are you able to get access to some scrap TV, DVD player, laptop, PC, anything? That's where I get most of my flash chips.
    Helpful post? Buy me a coffee.
  • #55 21589234
    p.kaczmarek2
    Moderator Smart Home  
    I think the 16 byte writes are caused by LFS cache limited to 16 bytes. I will try increasing it. Maybe we should also increase it in the main tree?
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)

    Added after 30 [minutes]:

    Comparison, first with 16 cache, second 128. LFS on Software SPI Flash.
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)

    Added after 4 [minutes]:

    For comparison - stock OBK. LFS on internal flash with cache 16. Crashed on the last entry.
    Hosting OpenBekenIOT Web App's 'Logs' function in the LittleFS of the Device (BK7231T)
    Helpful post? Buy me a coffee.
  • #56 21589333
    DeDaMrAz
    Level 20  
    Well this also seems rather promising in relation to the topic of this thread, hosting webapp on a device.

    I have some chips as well 😁 so adding some of them for testing will probably be soon. Can be great for the purpose of logging and storing thing on the module.

    But overall I'll second the idea of an SD card instead of the chips just because of space and possibly speed. With that we can host the webapp and have "unlimited" log space which is always nice.

    @Angel0fDeath

    Glad to see you back on the project!
  • #57 21589422
    p.kaczmarek2
    Moderator Smart Home  
    Large files are still slow, I need to check how would it behave with many smaller files.

    I want to use it to store measurements history on device.
    Helpful post? Buy me a coffee.
Reply | New topic
Report a violation of the law

Topic summary

The discussion focuses on hosting the OpenBekenIOT Web App's 'Logs' function within the device's LittleFS filesystem on BK7231T and BK7231N devices. The default LittleFS size is 32kB, which is insufficient for hosting multiple web app files such as startup.js, httpVueLoader.js, vue.min.js, and logs.vue. Users experimented with increasing LittleFS size up to 256kB and 200kB (0x32000), with mixed success due to device reboots and low heap warnings during large file uploads. A key finding was that adding RTOS delays during file upload improved stability, as implemented in a test firmware build. Firmware upgrades to versions around 1.17.404 and above were recommended for better performance. Minification of JavaScript and Vue files significantly reduced storage requirements, enabling hosting within the constrained LittleFS space. The memory layout of BK7231T and BK7231N devices was reviewed to understand partitioning and avoid overwriting OTA or application partitions. Alternative hosting solutions suggested include running the web app on a NAS or router with OpenWrt. Further development includes moving LittleFS to external SPI flash chips (4-8MB) or SD cards to expand storage capacity for logs and web app files. Fast bitbang SPI implementations and increased LittleFS cache sizes were tested to improve performance. The community encourages creating tutorials for these solutions and ongoing firmware improvements to enhance stability and usability of hosting web apps directly on BK7231 devices.
Summary generated by the language model.
Popular Topics
ADVERTISEMENT