Developing a Charts Driver for OpenBeken to Display Measurement History

Question

It's a great time to get ESP32 support, I've started working recently on a charts driver for OBK so we will be able to display some measurements history on the OBK device itself.

max4elektroda · Answer

I've seen that approach. It's a coincidence I had the same idea but started with the other side ;-)

I think one point is how to store a history of values and how many memory can be used. I would propose a storage of temperatures in 8 bits, it should be sufficient to store rounded temperatures in this.

Personally I think it might be an alternative not to use an external library but make a simple own implementation (I would think of converting the data to an svg image).
But maybe we should separate this topic...

p.kaczmarek2 · Answer

The data can be stored in much more efficient manner but in case of my "charts" driver it was not the design requirement. I've been thinking rather about a flexible method to generate any charts, without worrying that much about space.

If we were to design a super efficient charts driver, we could have done much more than just using 8 bit type for temperature. For example, if we, let's say, know that we want to store temperature+humidity, we could have used 7 bits for humidity and 9 bits for temperature and use a short type (16 bits), etc, but that would make things too unflexible.

I've been thinking about some kind of flexible charts API, idk, like a command to setup N series (like: "setup chart with humidity and temperature") and a command to add samples (from script). The charts driver wouldn't have anything hardcoded, it would just display the data from the ring buffer. Optionally, ring buffer could be saved to LittleFS or something.

For now, I've chosen a float data type (32 bits per sample) because I can't tell in advance what will be stored there, and storing voltage may be different than storing, I don't know, power consumption.

In theory, we could have allow user to set a type for chart, like "setup a chart with series of bytes and series of floats" and handle it in code (type enumeration?), but that complicates the driver code

I am creating multiplatform open source firmware (Tasmota replacement), right now supporting BK7231T, BK7231N, XR809, BL602, W800, W600, LN882H and soon supporting RTL and W701:
https://github.com/openshwprojects/OpenBK7231T
If you like my work, support me at: https://paypal.me/openshwprojects

Helpful post? Buy me a coffee.

max4elektroda · Answer

I'll try to share my first thoughts and ideas tomorrow.

Regarding the charts: usually the device showing the OBK gui will have internet access, then it's no problem to download a library for charts.
On the other Hand, I would usually prefer a gui that is "self containing" and doesn't need to download content from remote.

p.kaczmarek2 · Answer

The best thing about the drivers and online build system is that we can actually make multiple drivers with separate goals in mind. An offline lightweight driver and online flexible charts driver.

Alternatively, we could just try to support fetching this charts lib from LittleFS....

p.kaczmarek2 · Answer

@divadiow are you able to get any OBK platform device with DHT11 or anything working for some tests?

https://github.com/openshwprojects/OpenBK7231T_App/blob/main/src/driver/drv_charts.c

Added after 3 [minutes]:

This:



startDriver charts
// chart with max 16 samples, 3 variables and two separate vertical axes
chart_create 16 3 2
// set variables along with their axes
chart_setVar 0 "Room T" "axtemp"
chart_setVar 1 "Outside T" "axtemp"
chart_setVar 2 "Humidity" "axhum"
// setup axes
// axis_index, name, flags, label
chart_setAxis 0 "axtemp" 0 "Temperature (C)"
// flags 1 means this axis is on the right
chart_setAxis 1 "axhum" 1 "Humidity (%)"
// for demonstration purposes, add some data at fixed times
// First argument is NTP time value
chart_add 1725606094 20 15 89
chart_add 1725616094 22 16 88
chart_add 1725626094 26 17 91
chart_add 1725636094 30 14 92
chart_add 1725646094 28 13 92
chart_add 1725656094 27 15 91

gives:

This:


startDriver charts
// chart with max 16 samples, 3 variables and 3 separate vertical axes
chart_create 16 3 3
// set variables along with their axes
chart_setVar 0 "Voltage" "axvolt"
chart_setVar 1 "Current" "axcurr"
chart_setVar 2 "Power" "axpower"
// setup axes
// axis_index, name, flags, label
chart_setAxis 0 "axvolt" 0 "Voltage (V)"
chart_setAxis 1 "axcurr" 1 "Current (A)"
chart_setAxis 2 "axpower" 2 "Power (W)"
chart_add 1725606094 12 0.5 6
chart_add 1725616094 11.8 0.52 6.14
chart_add 1725626094 11.5 0.54 6.21
chart_add 1725636094 11.3 0.55 6.22
chart_add 1725646094 11.1 0.56 6.22
chart_add 1725656094 10.9 0.58 6.32

gives:

I am creating multiplatform open source firmware (Tasmota replacement), right now supporting BK7231T, BK7231N, XR809, BL602, W800, W600, LN882H and soon supporting RTL and W701:
https://github.com/openshwprojects/OpenBK7231T
If you like my work, support me at: https://paypal.me/openshwprojects

Helpful post? Buy me a coffee.

divadiow · Answer

yes. I see your PRs and the releases but startdriver charts doesn't seem to start Code: TextLog in, to see the code

p.kaczmarek2 · Answer

You need to enable it for your platform in obk_config.h

divadiow · Answer

my bad. assumed it was ready to rock. neat.

p.kaczmarek2 · Answer

Try this sample, but you need to manually refresh page for it to refresh measurements, not sure why:


startDriver charts
startDriver NTP
//waitFor NTPState 1
chart_create 16 1 1
chart_setVar 0 "Number" "ax"
chart_setAxis 0 "ax" 0 "Number"


again:
// EDIT: fixed
setChannel 5 $rand*0.001
chart_addNow $CH5
delay_s 1
goto again

or we can try to use a real DHT and capture measurements over a day, idk, one per 15 minutes?

divadiow · Answer

this in autoexec verbatim gives a single line that does refresh itself but never changes I can set something up and leave it running yes. the second autoexec in post above gives:

p.kaczmarek2 · Answer

Auto refresh is annoying, I've added IndexRefreshInterval command.

For now, for DHT11 setup, I'd suggest:



// Sample 8
// DHT11 setup
IndexRefreshInterval 100000
startDriver charts
startDriver NTP
waitFor NTPState 1
chart_create 48 2 2
// set variables along with their axes
chart_setVar 0 "Temperature" "axtemp"
chart_setVar 1 "Humidity" "axhum"
// setup axes
// axis_index, name, flags, label
chart_setAxis 0 "axtemp" 0 "Temperature (C)"
// flags 1 means this axis is on the right
chart_setAxis 1 "axhum" 1 "Humidity (%)"

// every 60 seconds, -1 means infinite repeats
// assumes that $CH1 is temperature div10 and $CH2 is humidity
addRepeatingEvent 60 -1 chart_addNow $CH1*0.1 $CH2

For a start, try with repeating event every 10 seconds or so.

Take care with number of samples, 48 samples with two variables (and 8 bytes time) it's 48*(4+4+8) = 768bytes

I am creating multiplatform open source firmware (Tasmota replacement), right now supporting BK7231T, BK7231N, XR809, BL602, W800, W600, LN882H and soon supporting RTL and W701:
https://github.com/openshwprojects/OpenBK7231T
If you like my work, support me at: https://paypal.me/openshwprojects

Helpful post? Buy me a coffee.

divadiow · Answer

>>21222830 1345_merge_b0d7de0b4570 Added after 3 [minutes]:

p.kaczmarek2 · Answer

Hmm at first it may look confusing, but after second glance I guess it will look better with some more measurements data?

divadiow · Answer

>>21222948 was going to setup a second with longer intervals. leave it going also

p.kaczmarek2 · Answer

Great! By the way, we can also check if my ring buffer implementation works, but I think it should be okay.

The next question might be how to handle the saving of the data. Whether to use LittleFS or something else. We don't want to waste flash cycles. Maybe we could dedicate few flash sectors and use some kind of logic that allows appending new measurement without an erase cycle, so we could get, I don't know, one erase per 1024 writes (sector is 4096 as far as I remember, and I am assuming we're writing a 4 bytes integer).

p.kaczmarek2 · Answer

Are you saying that something has broken, or do you have a perfectly stable temperature in your room? We can try larger samples size.

divadiow · Answer

well. not entirely sure tbh. I don't believe the temperature hasn't changed at all. I'll warm it up a bit

Added after 3 [minutes]:

oh no, it's OK. maybe it has stayed constant. held it in my hands for a while and it climbed to 27

Added after 3 [minutes]:

im about to harvest another CB3S to sit alongside with the DHT22. want any specific setup for that apart from maybe event every 5 or 10 mins?

also, if it's every 60s, how come the time reading is every ~2mins?

p.kaczmarek2 · Answer

Well, with two variables and time, we get 4+4+8 , which is 16 bytes per sample. My sample BK7231 device has currently.... let me check... free 83024 bytes of RAM. Let's say we want a one day of measurements, and measurements every 15 minutes, so it's 24*4 = 96 samples, somewhat 1.5kB memory. Still not much.
So, set 96 samples in chart_init and change event time to 15 minutes, which is, 15*60=900 seconds.
Maybe we can already set it for two days, then it will be 192 samples.

Regarding the timing issue - I have no idea. It seems we may need to check this. But... no, actually it's not every 2 minutes! Take a closer look:

It's just that every second label is skipped. Probably chart determined that it has not enough space for all of them.

I am creating multiplatform open source firmware (Tasmota replacement), right now supporting BK7231T, BK7231N, XR809, BL602, W800, W600, LN882H and soon supporting RTL and W701:
https://github.com/openshwprojects/OpenBK7231T
If you like my work, support me at: https://paypal.me/openshwprojects

Helpful post? Buy me a coffee.

divadiow · Answer

ah yes

this is running with DHT11 now

Log in, to see the code

DHT22 doesn't seem to work on the pins I've tried on CB3S or WB3S. I get one reading rarely, then nothing. oh well.

OK in Tasmota

Added after 8 [hours] 25 [minutes]:

and for temp/hum comparison purposes the DHT22 reports similar right now

Log in, to see the code

max4elektroda · Answer

Sorry, I was busier than expected, first day back from holiday is hard I like the actual possibility,especially because it's open to all sorts of date. But still I hope it's ok if I share my idea, not as far as your working code: The actual idea is to keep data to store information as low as possible, that needs to make some hard requirements, which I think are acceptable in many cases. - all data is stored periodically with a fixed rate, e.g. every 5 seconds, every minute ... - the data is actual, so we don't need to store timestamps (easily adopted to a slightly lower constraint, that it's complete (no gaps) and we store the timestamp of the latest data) This way we can build our graph with only data and the knowledge of the timestamp of the latest data point. Up to now I'm struggling with a first implementation in OpenBeken, but I can give an idea of how it looks and that it's not so demanding related to memory footprint... I attached a simple HTML page for that, so you can try it out yourself - feedback highly appreciated (it's without a separate .js file to have only one file - just remove .txt extension, couldn't upload HTML file)

max4elektroda · Answer

Finally some result:
LN882H is really good in powerSaving, on BK-N I see no difference in temperature (but it's much lower anyway) and no gain of information on W800 ;-)

Added after 5 [hours]:

p.kaczmarek2 wrote:
The next question might be how to handle the saving of the data.

Maybe I can bring up my approach again?
As soon as you can make sure that samples are taken in fixed rate, you can avoid saving the timestamps. As long as you save the timestamp of the latest data set, you can calculate all other ones from that base.
This way you can save 4 bytes per sample!

To minimize memory footprint I also would suggest to decrease the accuracy or the recorded data: Who needs to know that temperature was 21.18° and not 21.25° or even 21° in case of 0.5° steps?
And can the can you distinguish between this values in the graph, not to bring up the question, if the sensor was correct in measuring this vale?

So I would suggest to go for more data with less granularity and try to avoid storing information that might be calculated.

In case you are not using a ring buffer, for environmental data like temperatures it might be even more efficient to store a temperature and the number of equal readings for (at least if you have a limited accuracy) all temperature changes will usually be very slow (as long as you don't use external heating, warm the sensor in your hands or the sun will suddenly reach it). But in general, you will often have similar (equal) readings for longer periods, making it more efficient to store it this way.
In case you want a ring buffer, this is more challenging, you will need e.g. two bytes per value and "counter", so deleting the oldest entry will have a different impact every time.
In case of volatile data (as e.g. reading CPU temperature), this won't help ,,

So with some deep thinking, it should be able to store quite some data in RAM, if we can live with the fact that ist gone in case of a reboot.

If writing it to flash, only in "large" time ranges to protect the flash memory.

p.kaczmarek2 · Answer

Very nice graphs, are they configurable? We can have two charts drivers, so if you just want, we can integrate yours driver as well.

The size of time_t may be 8 bytes on some platforms, so we may save even more by skipping the time.

Saving the temperature as 2 bytes instead of 4 bytes may also work, but my driver is designed to be flexible and it would be somewhat a hassle to support multiple data types.
Still, maybe we could give it a try.
Introduce a variable types: 8 bit, 16 bit, 32 bit?

We could also use delta encoding. It may be a bit more tricky with a ring buffer, but it still may be possible. We would need to store one bit (somehow) per variable telling us whether given value is skipped (same as before) or changed. We could also store 2 bits per variable, then 00 means "the same", 01 means "small change" (encoded as one byte), and 10 could mean "full new value".

I am creating multiplatform open source firmware (Tasmota replacement), right now supporting BK7231T, BK7231N, XR809, BL602, W800, W600, LN882H and soon supporting RTL and W701:
https://github.com/openshwprojects/OpenBK7231T
If you like my work, support me at: https://paypal.me/openshwprojects

Helpful post? Buy me a coffee.

max4elektroda · Answer

Kind of, since it's my own implementation, at least extending it for two graphs shouldn't be to much hassle. Sorry that I'm to lazy to look myself : Would it be a possibility/idea to use a union to store data? So we can e.g. use the same type of 32bit to store 1, 2 or 4 values in this entry? Yes, that's what I thought, my first idea was (always thinking in 8 bit values ) to us a 16 bit value of two times 8 bit: value first, then up to 255 repetitions of this value. As long as we usually have at least two consecutive equal readings, we wouldn't have overhead.

max4elektroda · Answer

An updated version with some kind of a library. The HTML-File shows, how the library can be used: Place a on the HTML page where the graph shall be call createsvg to initialize a graph (arguments are the div-element, size of the image and an id for the image) Then call draw to actually visualize the data (the argument is an option element, containing the id of the image created in the step before plus information about the data tor draw... Attached is again an HTML file, including a minimized JS version and the full JS (still with some testing and commented out lines. (again: remove .txt to use , it's just to make the attachments possible) But with quite some text to explain the calls ...

Developing a Charts Driver for OpenBeken to Display Measurement History

Post #1

Post #2

Post #3

Post #4

Post #5

Post #6

Post #7

Post #8

Post #9

Post #10

Post #11

Post #12

Post #13

Post #14

Post #15

Post #16

Post #17

Post #18

Post #19

Post #20

Post #21

Post #22

Post #23

Post #24

Post #25

Post #26

Topic summary