Beken BK7231/BK7252 SPI flashing and recovery - new flasher tool and protocol specs

thanks all




Added after 13 [minutes]:

but now I want to revise the links and wording on the Docs tab more 😈

Can we merge what we have first?
This empty place looked indeed better with more info, haha:

<sigh> OK. It is back

RTL8710B custom stub (with r/w compression, efuse and rom read).
https://github.com/openshwprojects/BK7231GUIFlashTool/pull/158
Plus made some xmodem adjustments specifically for it. No longer halts for several seconds before stub upload.
Included BL602/616 MAC read.

https://github.com/openshwprojects/BK7231GUIFlashTool/pull/157 fixes cases where Tuya config extraction could include stale KV records as if they were live data.

Added after 10 [hours] 22 [minutes]:

@p.kaczmarek2 please consider merging some/all things so further development with rom/efuse read is clearer

Ok, however one PR requires conflicts resolve, also, what are stale KVs?

p.kaczmarek2 wrote:

Ok, however one PR requires conflicts resolve

fixed

Added after 3 [minutes]:

p.kaczmarek2 wrote:

what are stale KVs?

old Tuya key/value records that are still present in flash and still decrypt/CRC-check correctly, but are outside the active KVStorage window for the current config. They appear to be leftover records from previous storage pages/config writes, so the extractor code would merge them into the displayed JSON as if they were current live values.

I checked 508 Tuya-named dumps. Of those, 491 produced valid Tuya config extractions, and 178 of those extracted JSON outputs were affected by this stale-KV surfacing.

removed some bloat. thanks @insmod
https://github.com/divadiow/BK7231GUIFlashToo...mmit/35a05d1f8f1c1dc1ae785e71a37904c7b2a6ef8c

https://github.com/openshwprojects/BK7231GUIFlashTool/pull/159
conflicts resolved

Great, merged. It seems we will need a separate topic guide/progress update for OPL soon.

bad test harness used when bulk testing PR 157. some non-Beken KV extractions are getting truncated extractions, so: https://github.com/openshwprojects/BK7231GUIFlashTool/pull/160

divadiow wrote:

there are 3 files in FlashDumps that contain config outside of the KV

one of these is actually beken-crc but plain text.

there are also 21 dumps where the pin mapping config is in the encrypted app. eg Tuya_Windows_Door_Vibration_(schemaID-e1kc5844)_ljwxe1cyevzjt8m2_CBU_1.0.3.bin




toying with added block to json output for these

Maybe more automatic tests are needed? Something that could run with each build?

Good idea!

Maybe after W800 custom stub. Just watchng Codex write, build and flash to a connected W800 at its own pace. Crazy

Is new stub faster than the current one?

not sure yet, but it's more than that - efuse/rom read ability. compression. still in testing. not that there's anything particularly wrong with the current one.

Added after 1 [minutes]:

well, W806 read doesn't work with it, but I don't know if that's a stub problem or EF.

Why not port ECR stub?

hmm dunno. it's not directly from ECR6600 stub code but it's been developed from the shared protocol conventions in all the other custom stubs, so effectively a port?

getting there I think. miniz too

compressed 2mb W800 and 1mb W806 read


accel crc too:


ROM read. W806 and W800 have identical ROMs

That's a great result, what's currently the bottleneck? UART speed or CRC or compression/decompression? Can you show the current source?

hmm. dunno tbh. the 2mbps CH340N on my dev boards? I should do proper Upgrade Tools stub vs custom stub vs compressed tests but I'm away for the next couple of days.

https://github.com/divadiow/w800_stub
https://github.com/divadiow/BK7231GUIFlashTool/tree/w800stub

Added after 8 [hours] 2 [minutes]:

https://github.com/openshwprojects/BK7231GUIFlashTool/pull/161

BK7258 efuse/rom/otp read

Thank you, merged.

divadiow wrote:

Upgrade Tools stub vs custom stub vs compressed tests

w800 currently (921600):

Operation	Stub/mode	Time	Difference vs original
Read 2 MiB	Original Upgrade Tools	34.679 s / 34,679 ms	Baseline
Read 2 MiB	Custom, uncompressed	33.992 s / 33,992 ms	2.0% faster
Read 2 MiB	Custom, compressed	16.011 s / 16,011 ms	53.8% faster
Write 2,088,960 bytes	Original Upgrade Tools	51.965 s / 51,965 ms	Baseline
Write 2,088,960 bytes	Custom, uncompressed	58.081 s / 58,081 ms	11.8% slower
Write 2,088,960 bytes	Custom, compressed	25.764 s / 25,764 ms	50.4% faster
Erase writable flash	Original Upgrade Tools	4.184 s / 4,184 ms	Baseline
Erase writable flash	Custom	12.996 s / 12,996 ms	210.6% slower

so compression is the obvious winner. I'll tweak stub more to at least match original stub for non-compressed operations.

Added after 4 [hours] 2 [minutes]:

OK. improved.

Operation	Stub/mode	Updated time	Difference vs original
Read 2 MiB	Original Upgrade Tools	35.263 s / 35,263 ms	Baseline
Read 2 MiB	Custom, uncompressed	33.183 s / 33,183 ms	5.9% faster
Read 2 MiB	Custom, compressed	16.072 s / 16,072 ms	54.4% faster
Write 2,088,960 bytes	Original Upgrade Tools	50.669 s / 50,669 ms	Baseline
Write 2,088,960 bytes	Custom, uncompressed	42.447 s / 42,447 ms	16.2% faster
Write 2,088,960 bytes	Custom, compressed	25.646 s / 25,646 ms	49.4% faster
Erase writable flash	Original Upgrade Tools	4.208 s / 4,208 ms	Baseline
Erase writable flash	Custom	4.931 s / 4,931 ms	17.2% slower

A trace showed that corrupt XMODEM packets were detected and retried as intended. However, if the one-byte ACK or NAK response was lost, the stub waited approximately 10.02 seconds before retransmitting. This delay occurred in three of six successful baseline Easy Flasher reads; two other baseline attempts failed.
The custom stub’s actual erase command takes approximately 1.35 seconds. The table measures the complete operation, including setup and uploading the 13.3 KiB custom stub, leaving it 0.723 seconds slower overall than the factory-loader operation. When the custom stub is already resident, the erase operation is substantially faster.

Changes made:
Mixed 4 KiB sector and 64 KiB block erasing.
A full W800 writable-area erase now uses 14 sectors and 31 blocks.
Raw writes erase the destination efficiently before XMODEM reception.
Completely blank 0xFF pages are not unnecessarily programmed.
Nonblank pages still receive immediate readback comparison.
Easy Flasher still performs its final hardware CRC32 verification.
XMODEM packet size and ACK/NAK protocol remain unchanged.
The stub’s ACK-response timeout was reduced from approximately 10 seconds to approximately 2 seconds.
Easy Flasher now requests retransmission after one second on W800 reads.
The stub’s packet retry allowance was increased from 10 to 30.

In final testing, 15/15 uncompressed and 10/10 compressed cold-upload reads completed with matching CRC32 values, without the 10-second delay or truncated transfers.

p.kaczmarek2 wrote:

Maybe more automatic tests are needed? Something that could run with each build?

something like this? https://github.com/openshwprojects/BK7231GUIF...vadiow:BK7231GUIFlashTool:tuyaconfig-selftest

it tests quite a lot, including matching human-readable output. tests would need updating if tweaks were made that affected anything. could be annoying, but it is more complete?

Added after 2 [minutes]:

test binaries are all real dumps but are all FF apart from KV/PSM areas. same offset as original. gzipped

build.bat


Added after 5 [minutes]:

>>21941637

automated W800 tests with current code custom stub



lol. just realised the w800 was blank

Update GitHub Actions for Node.js 24 https://github.com/openshwprojects/BK7231GUIFlashTool/pull/162

Added after 9 [hours] 8 [minutes]:

switch to Semantic Release so Easy Flasher release number is sequential rather than based on workflow number? So no more release number jumps like for eg v440 straight to v444.

The repository’s old highest SemVer tag is v1.1.7 so start from v1.1.447.

The current release being "BK7231GUIFlashTool build 446".



https://github.com/openshwprojects/BK7231GUIFlashTool/pull/164

divadiow wrote:

something like this? https://github.com/openshwprojects/BK7231GUIF...vadiow:BK7231GUIFlashTool:tuyaconfig-selftest

https://github.com/openshwprojects/BK7231GUIFlashTool/pull/165

Added after 8 [hours] 49 [minutes]:

couple of more tweaks. ive put lan scanner pr into draft for some more work to be done on it finished

https://github.com/openshwprojects/BK7231GUIFlashTool/pull/166

Nice, but two suggestions/questions.
1. Do we really need to check 1:1 human readable output? Maybe we do, but if someone formats it more, it will fail self test:

Still, maybe it makes sense
2. Do validation of json output checks existence of object fields, or does it blindly compare strings?

p.kaczmarek2 wrote:

1. Do we really need to check 1:1 human readable output? Maybe we do, but if someone formats it more, it will fail self test:

well, not sure. it does add complexity, but then it may also catch things. depends I guess how annoying it becomes to update when a change it made that requires self-test updates. Maybe it won't be frequent enough to be an issue...

p.kaczmarek2 wrote:

Do validation of json output checks existence of object fields, or does it blindly compare strings?

It performs an exact golden-output comparison, ignoring only line ending differences. It doesn’t inspect fields individually, but missing, added, reordered, or changed fields will all fail because the complete output must match the known-good baseline.

divadiow wrote:

something like https://github.com/openshwprojects/OpenBK7231...ain...divadiow:OpenBK7231T_App:enableddrivers

https://github.com/divadiow/OpenBK7231T_App/b...29e03e01f00fa683ea4b5/docs/enabledFeatures.md Added after 1 [hours] 19 [minutes]:better https://github.com/divadiow/OpenBK7231T_App/blob/enableddrivers/docs/enabledFeatures.md

https://github.com/openshwprojects/OpenBK7231T_App/pull/2147

reworked this.

more accurate, simpler output file.

enabledFeatures.md

is now generated from the compiler state of every real platform and variant build. Previously, js attempted to reproduce the build rules and infer which features should be enabled. The new method records which macros are actually active during compilation, so platform-, SDK- and variant-specific settings are included automatically.

It does create temporary manifest artefacts before the

enabledFeatures.md

is generated, but these are deleted before completion.



rehearsal run https://github.com/divadiow/OpenBK7231T_App/actions/runs/30686747902
rehearsal enabledFeatures https://github.com/divadiow/OpenBK7231T_App/blob/main/docs/enabledFeatures.md

Added after 27 [minutes]:

a little snag. cleanup can't be performed when pr builds are built before merge in upstream eg: https://github.com/openshwprojects/OpenBK7231T_App/actions/runs/30687368843

so skip cleanup if no write access, set retention on manifest files to 1 day. https://github.com/openshwprojects/OpenBK7231T_App/actions/runs/30687969003