logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

OBK Docker Local Build: Reducing SDK Copy Time and Improving Build Speed

DeDaMrAz 234 14
ADVERTISEMENT
Report a violation of the law
Reply | New topic
  • #1 21830409
    DeDaMrAz
    Level 21  
    OBK local build via Docker.

    I was in need to build N platform builds locally for testing things and gave Docker build a try. What I found out is that SDK files will be copied every time into docker and then build will be serialized and that all takes approximately 20 minutes because for some reason all the SDK's are being copied every time (not cached).

    I gave my Container 16 cores to play with and I am in a process of creating a local build script that may reduce that time to ~105 seconds on clean build (first copy of selected SDK - time depends on SDK size) but subsequent builds are just 15 seconds!?!

    Has anybody been playing with this ever??

    I am in testing phase right now but damn this time reduction seems unreal 😁

    Script is not yet ready but I'll share once finalized, just curious to see if somebody played around with this at all.
    From Display to Complete System: How the Solution Team Creates Value at Unisystem [ad]
  • ADVERTISEMENT
  • #2 21830930
    DeDaMrAz
    Level 21  




    Is anybody else willing to test this out?

    Added after 4 [hours] 6 [minutes]:

    Script will now honor all available platforms and respected drivers but I need somebody else to test it on their machines.

    Requirements are:

    - Docker desktop installed and running
    - Python 3.xx installed and added to PATH
    - Git installed and added to PATH
    - about 25Gb of free space (all cached files volumes + build container without ESP-IDF!!!)
    - about 30-45 minutes of time for initial volume and container builds

    I think this is a significant build time update but again more testing is needed not just by me. For example, clean build is less than 30 seconds, driver change updates and subsequent builds are below 15 second (PC spec dependent!!)

    Thanks.

    Added after 6 [hours] 30 [minutes]:

    ESP32C6 test build time...
    Code: text Expand Select all Copy to clipboard
    Wrote 0x8e340 bytes to file output/3DRIVERS/OpenESP32C6_3DRIVERS.factory.bin, ready to flash to offset 0x0
cp ./platforms/ESP-IDF/build-c6/OpenBeken.bin output/3DRIVERS/OpenESP32C6_3DRIVERS.img
Build artifacts copied to output folder.
Build task finished in 23.20 seconds.

PS C:\Users\DeDaMrAz\Desktop\Docker tests>


    BK7231N build time test....

    Code: text Expand Select all Copy to clipboard
    rm sdk/OpenBK7231N/platforms/bk7231n/toolchain/../bk7231n_os/tools/generate/OpenBK7231N_*.rbl || /bin/true
rm sdk/OpenBK7231N/platforms/bk7231n/toolchain/../bk7231n_os/tools/generate/OpenBK7231N_*.bin || /bin/true
make[1]: Leaving directory '/app/build'
Build artifacts copied to output folder.
Build task finished in 14.55 seconds.

PS C:\Users\DeDaMrAz\Desktop\Docker tests>


    The only downside of this speed up is a large disk space required (more than 32Gb) to cache all the files on first build but after that build in 23 seconds or 14.5.... no contest.

    It can probably be optimized more but I am very satisfied with the result. Will add a PR for this in the coming days once I clean up all the mess I made because I didn't know about berry prebuild.... thanks @p.kaczmarek2 for the tip to fix that 😉
  • ADVERTISEMENT
  • Helpful post
    #3 21831694
    divadiow
    Level 38  
    Windows File Explorer list showing mostly OpenBK7231N .bin files plus .asm, .axf, .map, and .rbl

    Windows cmd/PowerShell window showing OpenBK7231N firmware build log with “COMPILE SUCCESS”.

    I wasn't timing the initial build but subsequent ECR6600 with 1 random driver took less than a minute. Timer result is 0s though

    Windows console screenshot showing OpenBeken firmware build logs and generated binary files
  • ADVERTISEMENT
  • Helpful post
    #5 21834395
    p.kaczmarek2
    Moderator Smart Home  
    I think you should integrate PROTA mechanism directly - rewrite it in Python:
    https://github.com/openshwprojects/OBKotaTool
    This way your tool can auto update target devices.
    Helpful post? Buy me a coffee.
  • #6 21834411
    DeDaMrAz
    Level 21  
    Console screenshot with test summary: 33 total, 17 success, 16 failed, plus lists of successful and failed platforms.


    First run of the test all builds, not bad I would say, most of the failed ones are my mistake some of which I fixed already like RTL8720 and 8721, adding IDF v5.5.2 to the cached SDK's o fix ESP32C5 and ESP32C61 now and will tackle BK7231U, 38, 52 next and rerun all the tests again.

    Once I get all the platforms to build 100% with all available drivers I'll post a PR update and consider it ready to merge ( I would hope so )

    Then I will think about creating local PROTA tool, that is a good idea.

    Status updates will go here, getting closer one platform at a time ;)


    Terminal screenshot with test summary: 33 total, 21 successes, 12 failures, plus lists of successful and failed platforms.

    Enough for tonight... success rate climbed to 84.8% + I'll fix the BL602 before I call it a night and continue tomorrow


    Terminal screenshot showing test summary: 33 total, 28 success (84.8%), 5 failed, 17.5 minutes

    No it is not enough, my brain would not let me sleep so I've wrestled with problematic builds and my own stupidity... came out clean on the other side (points for everyone who gets the reference)

    Full test --clean build with all drivers available on all 33 platforms is about 18 minutes (I can already see how I can speed it up even more - by not downloading certain thing for example but have them ready) each platform is building now with 2Mb default flash size ( yes I know and will do a full-nuclear test with everything form the start and flash sizes corrected at some point but I needed this win NOW!!!) After all the tests are done and completed I'll post a finished commit in the PR and I would call it done. When I find time I'll improve on the build speeds on certain platforms and maybe even create a ready made docker container from docker hub but it can wait... for now and our OBK community this is rather significant improvement (in my eyes at least), you can use python script as is or if you prefer GUI, that is available as well.

    MAJOR WARNING!!! I have tested builds on certain platforms only - not all of them!!! I tried to replicate the CI build as much as possible but be warned again - prepare for bricked modules!!!


    Console screenshot showing test summary: 33/33 success, platform list, and 18.4 minutes runtime

    TXW build is now "fixed" and it is fast!!!


    Terminal screenshot showing build logs, copying .bin and .img files, and a build finished message.

    Next big thing is correct caching of all the big tar files that influence ESP and RTL builds the most so I can shave more time for those build as well... right now RTL8721DA takes about 50 seconds, I am going to test and see if my idea is correct....

    My idea is correct and RTL build went down from 54 to under 19 seconds, continuing.... as long as there is coffee...
  • #7 21834566
    p.kaczmarek2
    Moderator Smart Home  
    @DeDaMrAz maybe do Visual Code extension next?
    Helpful post? Buy me a coffee.
  • #9 21834717
    p.kaczmarek2
    Moderator Smart Home  
    Maybe restrict flash sizes to only ones that really exist? Per platform.
    Helpful post? Buy me a coffee.
  • #10 21834723
    DeDaMrAz
    Level 21  
    Well that may be a problem as you and I had a conversation some time ago to add SPI to BK platform and use it for testing so 2Mb will not apply there :)

    Also I don't actually know all the specifications for each platform, for example I know BL has 1 and 2Mb versions, BK is 2Mb default but not all (??), but the rest is somewhat hard to find exactly so I'd rather leave the option to choose the flash size than to go into that rabbit hole now :) I'll test again the BMP280 driver now to see what the problem was on some of my previous runs.
  • ADVERTISEMENT
  • #11 21835246
    p.kaczmarek2
    Moderator Smart Home  
    SPI? You mean in Easy Flasher? That's a separate thing.
    Helpful post? Buy me a coffee.
  • Helpful post
    #12 21838691
    max4elektroda
    Level 23  
    I'f I didn't do something wrong there's some todo for all platforms with "variants" @DeDaMrAz.

    Simple example: I build a BK7238 firmware with drivers
    DS1820_FULL
    I2C
    SHT3x

    Trying to start SHT3x results in:
    Code: text Expand Select all Copy to clipboard
    
Info:MAIN:Driver sht3x is not known in this build.
Info:MAIN:Available drivers: 
Info:MAIN:I2C
Info:MAIN:, PWMToggler
Info:MAIN:, DoorSensor
Info:CMD:[WebApp Cmd 'startdriver sht3x' Result] OK


    The driver code seems not present, only the "IOroles":


    Code: text Expand Select all Copy to clipboard
    strings test_OUT/OpenBK7238_QIO_test.bin  | grep SHT
SHT3X
"SHT3X":
SHT3X_SDA
SHT3X_S



    but it seems reasonable, looking into the temporary obk_config.h:



    Code: text Expand Select all Copy to clipboard
    #elif PLATFORM_BEKEN

//#define ENABLE_LFS_SPI                  1
// #define ENABLE_DRIVER_TESTSPIFLASH   0 // Disabled by build_tool

[...snipp...]

// #define ENABLE_DRIVER_HT16K33   0 // Disabled by build_tool
// #define ENABLE_DRIVER_MAX72XX   0 // Disabled by build_tool
// #define ENABLE_DRIVER_ADCBUTTON   0 // Disabled by build_tool
#define ENABLE_I2C   1
// #define ENABLE_TEST_COMMANDS               1
#define ENABLE_CALENDAR_EVENTS               1
// #define ENABLE_DRIVER_BRIDGE   0 // Disabled by build_tool

[...snipp...]

#if (OBK_VARIANT == OBK_VARIANT_SENSORS || OBK_VARIANT == OBK_VARIANT_BATTERY)
// #define ENABLE_DRIVER_BMP280   0 // Disabled by build_tool
// #define ENABLE_DRIVER_BMPI2C   0 // Disabled by build_tool
#define ENABLE_DRIVER_SHT3X   1
#define ENABLE_DRIVER_DS1820_FULL   1
#endif

#if (OBK_VARIANT == OBK_VARIANT_HLW8112)

[...snipp...]

#endif

#elif PLATFORM_LN882H


    Since we define our own config, the variants don't make sense in my eyes.

    Here's my idea to solve this:

    Code: text Expand Select all Copy to clipboard
    diff --git a/docker/GUI_build_tool.py b/docker/GUI_build_tool.py
index d6af62b2..95ea79eb 100644
--- a/docker/GUI_build_tool.py
+++ b/docker/GUI_build_tool.py
@@ -1115,7 +1115,8 @@ class OBKBuildToolGUI(tk.Tk):
             # Cleanup temp config
             try:
                 if os.path.exists(temp_config_path):
-                    os.remove(temp_config_path)
+                    pass
+                    #os.remove(temp_config_path)
             except Exception:
                 pass
 
diff --git a/docker/build_tool.py b/docker/build_tool.py
index 52700b71..efff66df 100644
--- a/docker/build_tool.py
+++ b/docker/build_tool.py
@@ -457,7 +457,7 @@ def get_available_drivers(platform_name):
     driver_regex = re.compile(r'^\s*#\s*define\s+(ENABLE_\w+)\s+1')
     
     # Special non-driver names we want to capture
-    special_names = ["ENABLE_NTP", "ENABLE_I2C", "ENABLE_TASMOTADEVICEGROUPS"]
+    special_names = ["ENABLE_NTP", "ENABLE_I2C", "ENABLE_TASMOTADEVICEGROUPS", "ENABLE_OBK_BERRY"]
 
     for line in lines:
         line = line.strip()
@@ -567,8 +567,21 @@ def create_custom_config(selected_drivers):
     new_lines = []
     # Match any #define ENABLE_... or #define ENABLE_DRIVER_... followed by a number
     driver_regex = re.compile(r'#\s*define\s+(ENABLE_\w+)\s+(\d+)')
+    VARIANT_regex = re.compile(r'^#if \(OBK_VARIANT ==')
+    undef_regex = re.compile(r'^#undef ')
+    endif_regex = re.compile(r'^#endif')
+    VAR_found = 0
 
     for line in lines:
+        if VARIANT_regex.search(line):
+            VAR_found = 1
+            continue
+        if VAR_found == 1 and undef_regex.search(line):
+            continue
+        if VAR_found == 1 and endif_regex.search(line):
+            VAR_found = 0
+            continue
+
         match = driver_regex.search(line)
         if match:
             driver_name = match.group(1)
@@ -576,7 +589,7 @@ def create_custom_config(selected_drivers):
             # IS this a driver we are managing?
             # We manage anything starting with ENABLE_DRIVER_ or our special list
             is_managed = driver_name.startswith("ENABLE_DRIVER_") or \
-                         driver_name in ["ENABLE_NTP", "ENABLE_I2C", "ENABLE_TASMOTADEVICEGROUPS"]
+                         driver_name in ["ENABLE_NTP", "ENABLE_I2C", "ENABLE_TASMOTADEVICEGROUPS", "ENABLE_OBK_BERRY"]
             
             if is_managed:
                 if driver_name in all_drivers:
@@ -915,7 +928,8 @@ def main():
     finally:
         # Cleanup
         if os.path.exists(temp_config_path):
-            os.remove(temp_config_path)
+            pass
+            #os.remove(temp_config_path)
             
         elapsed = time.time() - start_time
         print(f"\nScript execution time: {elapsed:.2f} seconds")


    Remove all "OBK_VARIANT" lines and the matching "#endif" lines plus all "#undef" inside the block.

    Also added "berry" as an additional choice.

    Screenshot of firmware build tool: BK7238 platform, driver checklist, and log showing “Build completed”

    At least for BK7238 it works now ...

    Code: text Expand Select all Copy to clipboard
    
Info:SENSOR:SHT : Status : FF FF
Info:MAIN:Started sht3x.
Info:CMD:[WebApp Cmd 'startdriver sht3x' Result] OK
Info:MAIN:Time 77, idle 372242/s, free 108032, MQTT 0(5), bWifi 1, secondsWithNoPing 11, socks 2/24 
Info:MAIN:Time 78, idle 738719/s, free 116336, MQTT 0(5), bWifi 1, secondsWithNoPing 12, socks 2/24 
Info:MAIN:Time 79, idle 361863/s, free 108032, MQTT 0(5), bWifi 1, secondsWithNoPing 13, socks 2/24 
Info:MAIN:Driver UNKNOWN is not known in this build.
Info:MAIN:Available drivers: 
Info:MAIN:I2C
Info:MAIN:, PWMToggler
Info:MAIN:, DoorSensor
Info:MAIN:, CHT83XX
Info:MAIN:, SHT3X
Info:MAIN:, DS1820_FULL
Info:CMD:[WebApp Cmd 'startdriver UNKNOWN' Result] OK


    Added after 1 [minutes]:

    Keeping temporary config is meant just for testing
  • #13 21840000
    DeDaMrAz
    Level 21  
    @max4elektroda

    Thanks for the test and report and I 100% agree on the suggestion. I was focused on getting the build local out not so much on obk_config.h, I'll look into that next and add drivers that were also disabled by default and add them to selection for testing.
  • #14 21840005
    max4elektroda
    Level 23  
    That's nothing, I just did one step ahead for I needed this actually. I really like the new possibility to get a fully customized image without editing obk_config.h.
    And it just came in time for me. So, thanks again, this new possibility is really a big help.
  • #15 21840108
    DeDaMrAz
    Level 21  
    PR updated and core drivers revised (MQTT, HA and scripting for example) - incorporate @max4elektroda suggestions, no need to rebuild anything in docker just run and test the script.
Reply | New topic
Report a violation of the law

Topic summary

The discussion focuses on optimizing OBK platform builds using Docker to reduce SDK copy time and overall build duration. The initial issue identified was that SDK files were copied into the Docker container on every build without caching, causing build times of approximately 20 minutes. A new local build script was developed to address this, achieving a clean build time of around 105 seconds for the first SDK copy, with subsequent builds completing in about 15 seconds by leveraging caching and parallel processing with up to 16 CPU cores. The script supports multiple platforms and drivers, requires Docker Desktop, Python 3.x, Git, and sufficient disk space (~25GB) for cached volumes and containers. Initial volume and container setup takes 30-45 minutes, but subsequent builds are significantly faster, with clean builds under 30 seconds and driver updates below 15 seconds, depending on PC specifications. Further testing by other users is requested to validate these improvements.
Summary generated by the language model.
Today's popular
ADVERTISEMENT