Bi-Directional meter for Solar with total Consumption and Export Energy counters (BL0942 based)

Question

Just updating this to reflect the work done on this project. It's now possible to measure reverse and forward energy consumption on devices based on the BL0942. I have also implemented Netmetering, so that we know how much is self consumption and how much is actually exported to the grid....

cdtdsilva · Accepted Answer

Hi p.kaczmarek2! The piece of code I used to save the generation variable is for development purposes. I will use your code, once you're finished. For now i just made sure the correct memory positions were used, so it's backwards compatible with the final code. I may need some help to sync your repo to mine - last time it just over-wrote all my changes (good thing I had a backup). This time, too many files were changed to do it by hand. I'm not a daily github user, so not sure on all the perks there... I noticed power data is saved in 10Wh intervals. That's probably usefull to document. 10Wh increments is probably fine for a smart plug, for whole house power consumption, it will likelly write the flash 1000+ times a day. I sugest users change the default value to 100W and have added a small indicator showing the set value. The variable that writes the flash is the one below, inside src/driver/drv_bl_shared.c. I have re-written it to save whenever either of the values goes above the threshold set by 'changeSavedThresholdEnergy'. Code: text Expand Select all Copy to clipboardif (((sensors[OBK_CONSUMPTION_TOTAL].lastReading - lastSavedEnergyCounterValue) >= changeSavedThresholdEnergy) ||         ((xTaskGetTickCount() - lastConsumptionSaveStamp) >= (6 * 3600 * 1000 / portTICK_PERIOD_MS))  I modified the Periodic statistics feature to fit the needs of Netmetering (the balance between consumption and generation), as currently implemented by power companies. Because both the period and time zone will vary across the globe, the user will need to enable NTP and statistics manually. My start command looks like this, for reference: Code: text Expand Select all Copy to clipboardbacklog Powersave 1; startDriver BL0942; startDriver NTP; ntp_timeZoneOfs 1; SetupEnergyStats 1 60 15 Energy stats sets the period offered by the utility company. 60 seconds and x minutes. The rest should be self-explanatory. The statistics themselves, show the results for consumption, just as before. Netmetering balance is shown in a separate entry. A positive value indicates excess production and negative indicates consumption from the grid. Where there is excess production, I made provision to turn on a remote smart plug. This can be used to turn on an immersion heater or command a heat pump. In Poland, the present regulations offer 60min netmetering period, which is very generous. The relay goes off if either the solar array is not producing above a minimun value or we drop below a certain ammount of excess production, otherwise it remains on. The turn on and off thresholds are different, to accomodate some hysteresis, plus the relay only changes states every 2 minutes. Right now the default values are: > 50W (instant) and >10Wh excess production to turn on the relay,

p.kaczmarek2 · Answer

This is a very good idea. I think you need to edit this file: Namely this array: Code: C / C++Log in, to see the code You also need to update the corresponding enumeration: Code: C / C++Log in, to see the code In order for extra counters like energy today, etc, you need to enable NTP and setup energy counters. See our documentation for more details: Related command is SetupEnergyStats. You can check other threads for more information: Once you do the modification, I can merge in your changes. Let me know if you have any futher questions.

cdtdsilva · Answer

I was hoping you could assist me in enabling the additional statistics, as shown in my previous image (for now, let's disregard time statistics and solely focus on the totals). These statistics should be assigned to a variable, and that's where I'm encountering difficulties in comprehending the code. Based on my investigation, it seems that the correct function is Code: text Expand Select all Copy to clipboardvoid BL_ProcessUpdate(float voltage, float current, float power,                       float frequency, float energyWh) {...} However, this function needs to be modified in order to display a few more elements on the webpage (such as total production, for instance) and somehow store it, similar to the consumption data. Storing wise, this function seems to be responsible for it, Code: text Expand Select all Copy to clipboardHAL_FlashVars_SaveTotalConsumption(sensors[OBK_CONSUMPTION_TOTAL].lastReading); but I'm not sure in which circunstances it saves, How do I ensure I don't write the memory too many times? It appears that this task may exceed my current abilities, despite my sincere efforts to assist.

p.kaczmarek2 · Answer

If we assume that energyWh is negative when we have energy gain, then you don't need to add a variable there. You need to add a conditional statement later down the line to separate the taken from given values. I think you should be able to extend the structures there: Update will break stats for all of the users but maybe we can live with that... Or maybe utilize the reserved 3-byte field? Or maybe store it in the last 4 bytes of: Code: text Expand Select all Copy to clipboard short savedValues[MAX_RETAIN_CHANNELS]; No one ever retains more than 4 channels and MAX_RETAIN_CHANNELS is 12 so you could use that... if you need 4 bytes. The flashvars code works by writing 64 byte blocks many times to a single page and then erasing it only when the page is full. So, assuming page size 4096, then we get 4096/64 = 64 writes per erase cycle. Futhermore I think more pages are utilized. So it's many times more efficient and less prone to flash wear than a simple blind erase+write on each change. I am very happy with your activity so far. I can help more if it's required. Please tell me which new field of which type and size do you need and I can add them for you. I will most likely just use: Code: text Expand Select all Copy to clipboard short savedValues[MAX_RETAIN_CHANNELS]; because, as I said earlier, I can't imagine a scenario where someones does energy metering and has 10 or so relay channels..

cdtdsilva · Answer

Yes, doing that now. I've added it as such, Do you see any potential problems given the larger size? Will anything get overwritten if we use both? Code: text Expand Select all Copy to clipboard/* Fixed size 32 bytes */ typedef struct ENERGY_METERING_DATA { float TotalGeneration; float TotalConsumption; float TodayConsumpion; //float YesterdayConsumption; long save_counter; float ConsumptionHistory[2]; time_t ConsumptionResetTime; unsigned char reseved[3]; char actual_mday; } ENERGY_METERING_DATA; That's great, nothing will change then. Well, we are moving Is there a system timer I can use without the need for NTP? I want to reset the absolute power variable every 15min / 60min. Where is the code for the webpage located? I want to add an extra button to reset the generation statistics, next to the one for the totals

Honsu · Answer

I have autoexec.bat entered in the file startDriver NTP ntp_timeZoneOfs 1 SetupEnergyStats 1 60 60

p.kaczmarek2 · Answer

I am not sure if I missed it, but can you @cdtdsilva specify how much bytes do you need? Adding new fields to this structure will cause all users to lose energy stats on reboot and cause faster flash wear. My suggestion currently is: Please use those fields instead of adding new ones. Just get a pointer and cast it to integer or what kind of type do you want? Do you want to use it for testing purposes only or for production? There is addRepeatingEvent command that can be used for development testing. Please do not add any special buttons to native HTTP page. We need to preserve flash memory. Some devices have only 1 MB. You only need to add console command like here: Then you can edit our Javascript web app code here: in order to add a button to call this command, like I did here:

cdtdsilva · Answer

I need 8 bytes. The total counters are saved as a float - using a short, I'm guessing we only have 4 bits available, which would cause them to overflow somehow quicker. I was thinking in using the 'Yesterday Consumption' field or commenting it out, such as on the example, to give space to a variable of the same size. Technically then, that's the only field that would be broken, the other memory positions remain unchanged. That only saves last 24h consumption, so it would affect users minimally. See above. I want to use it for production, without NTP. Is there a system tick counter used for non blocking delays?

p.kaczmarek2 · Answer

I can try to add two floating point fields for you in the place of last 4 retained channels, which no one is using. How should I name them? Two 32-bit floats, so 2*4 = 8 bytes. How precise timing do you need?

cdtdsilva · Answer

Thanks. I only need one variable called TotalGeneration. It should be a 64bit / 8 byte float. The timing doesn't need to be very precise, just roughly 15min or 60min options, as those are the standard net metering times offered by utilities. I said I didn't want to use NTP... For clarification - I don't mind using it - But I want to be able to use it without the internet. So would the driver still run, with the time not synced (ie starting to count from 1900)? I will be doing is getting the number of seconds and schedulling the next clear of the absolute energy counter to 15 or 60 min ahead in time. This would ideally be synced to the start of the hour, if the device was connected to the internet, but correct timezone, day, week, etc is not required. Energy metering is now working, just neet to add the net metering feature and change the 'TotalGeneration' to the new variable you will be creating plus some tidy up of the code

p.kaczmarek2 · Answer

All our math is done on 32-bit floats and 32-bit float is used to store power consumption, why do you suddenly want to use 64-bit double for power generation? How much power do you expect to generate that you think it will overflow the 32-bit type? If you are worried about floating point precision, you could also settle on 32 bit unsigned integer and assume that it's a value in Wh or something, depending on how much precision you need. Regarding timing - maybe just do: Code: C / C++Log in, to see the code in the driver code? In the on every second tick update... Added after 1 [minutes]:Also - can you respect Flag 25 in your changes? To be more precise - only run your changes when this flag is set? I am asking becaus we had reports saying that some power metering chips are reporting negative values randomly...

cdtdsilva · Answer

I was making a number of assumptions, but I see your point now. 4 bytes is OK. Would it be enough to initialize it as Code: text Expand Select all Copy to clipboard#define MAX_RETAIN_CHANNELS 10 and then add our new variable on the top of Code: text Expand Select all Copy to clipboardtypedef struct ENERGY_METERING_DATA {float TotalGeneration; ...} so it ocupies those first 4 bytes? Ill try that for a first go. Yes, that shouldn't be a problem.

p.kaczmarek2 · Answer

Ok ok, but it's 4 byte, not bit. It's just two letters but it matters. Your solution may be more elegant but I'd still opt for a bit more hacky approach that keeps MAX_RETAIN_CHANNELS at 12 but still allows you to overwrite last two values (2 * 2 bytes = 4 bytes) with your value. This is because I think there is almost no chance of conflict between those two usages so we can keep both those features, just I will document that they occupy the same space. Ok, so, now, can I add that function to store a 4-byte float for you? I will add it to the main tree already, but it won't be called anywhere, I will leave it up to you.

cdtdsilva · Answer

Ups... Corrected! Yes, please go ahead. I was thinking to write the bytes in a loop to the memory locations, but then I was worried that a power loss might increase the risk of data corruption. I'm sure you'll have a better idea Added after 10 [hours] 18 [minutes]: Ok - Changes are complete. I did the changes I proposed earlier, and then we can use this new function, once it's complete. src/hal/hal_flashVars.h Code: text Expand Select all Copy to clipboard#define MAX_RETAIN_CHANNELS 10 ... typedef struct ENERGY_METERING_DATA {float TotalGeneration; ...} NTP works fine witout internet, so I used the minute variable, after enabling it. This does not break compatibility with existing stats (Example PowerSave 1; startDriver NTP; SetupEnergyStats 1 60 60) - These work as they did. They measure consumption and remain 0'd with generation. Flag 25 disables the netmetering statistics and generation counter. Active energy is 0 and the consumption counter is not incremented. Current, Voltage and var are still displayed, but nothing to do with my code changes - They probably always were, as if you attach a capacitor to the load (lots of current/var, nil consumption). It seems like the total generation is not saved very often. A restart does save it - but pulling the plug out doesn't. Consumption typically saves it in 0.01 or 0.02KW intervals. For the net metering functionality I would like to be able to set a variable to define the actual time (15min, 60min, etc.) - Can I do this using the console? Or can it be added on the webpage? The variable is called Code: text Expand Select all Copy to clipboardnet_metring_interval Is it possible to run a command based on the value of a given variable, for example, close the relay, when generation is above a certain threshold? Any examples of this, I can refer to?

p.kaczmarek2 · Answer

As I said, I'd prefer to keep MAX_RETAIN_CHANNELS at 12 and just use last two retain channels as a float with a cast, but I will add a sample of this for you later. I am not currently sure, I need to check the bl_shared.c code. This must be done via console command and later you can use our Vue/JS page to create a button for that. I think you can use addChangeHandler for that: Added after 5 [hours] 36 [minutes]:@cdtdsilva I have a little update. I have checked the code and I now know where the data is saved. The following feature was added by our contributor so I couldn't just figure it out from the memory. You need to search for BL09XX_SaveEmeteringStatistics See: There is a conditional statement in the bl_shared.c which determines the frequency of power metering data saving.

p.kaczmarek2 · Answer

Here is my proposal of save function: You can sync changes from mains to get my changes on your fork. Merge conflicts can be indeed hard to resolve. Did you use Github gui? Maybe try using a dedicated diff/merge tool? I think there may be some online tutorials for conflicts resolve: Alternatively you can just rewrite/copy paste your changes to the main tree.... It would be a bit difficult for me to do that with your fork, as I don't have a good understanding of your code and I am also not the main contributor of power metering features, as far as I know they were introduces by @valeklubomir and later some other user improved them. This is very good if it's all enabled by flag 25. I am not sure about that diversion relay part. I will need to check this once it's in PR. My main question is - how much flash space it takes and why not just use script with addChangeHandler? But again, maybe if it's not taking much space, then it's a good idea... or we can just #ifdef it and allow users to enable it in obk_config.h so we can save flash space when it's not needed... I want to keep all your features, just as I said, I have only few requirements: - the energy export must be enabled by flag 25 or similiar (and this is already implemented, right?) - if you add some features that requires a lot of flash memory/space, consider adding option to disable them to obk_config.h - you can't break too much compatbility with older versions - and as you said, we need some basic docs I will do a final code review once you manage to get the conflicts resolved. I can also do some basic testing with @DeDaMrAz and maybe @divadiow before merging. Added after 1 [minutes]:EDIT: The energy step for save could be made configurable via console command

cdtdsilva · Answer

Hi p.kaczmarek2 I don't have any conflicts, just last time I tried to merge, it somehow overwritten my changes. I don't use these tools on an everyday basis, so I find it extremely time consuming to figure it all out. Now that the easter holidays are over I'm not so easy on time, so I will have to review at a later time. The 'diversion relay' part of the code is an if loop with conditional statements, which calls out the Send Get command to send an HTTP request to turn on or off a remote plug. All the code is contained inside That's makes use of the already included code inside selftest_http_client.c, example: Code: text Expand Select all Copy to clipboard///CMD_ExecuteCommand(SendGet  You might have a good idea of the memory use of that feature. The rest, I think it's negligible. A few more variables and loops. 90% of the aditions are inside src/driver/drv_bl_shared.c. I did some previous changes on how metering data is obtained from the BL0942 inside /src/driver/drv_bl0942.c The rest is adition of variables here and there. I'm pretty confident no backwards compatibility was lost, but due to the memory structure, the generation data is probably sent via MQTT regardless of flag 25 (As a 0 value). That needs to be tested, to ensure nothing goes out of place on HA, etc. I know all of the variables can be changed via the console. Are you kindly telling me there is no space to make the changes? I would find it more user friendly to have them on the webinterface, but understand if space is an issue. We could also make a separate version, a bit like tasmota (with an #ifdef) specific for users that require reverse metering. It's quite a niche application, after all. Either way I will prepare some details covering the various variables of interest. I will probably forget them in a few weeks, so better have them documented Meanwhile, these work fine via MQTT on home assistant.

p.kaczmarek2 · Answer

@cdtdsilva are you using Github GUI? If there are no conflicts, I could just merge your changes on my site, but then you won't be present in Github history as a contributor, and I think your presence there is deserved as you did good work. It's up to you. If you think you can reasonably test this things on your side then I can open PR for you once you send me your code... I think it should be scripted, we can't have that in code, but you can leave it commented out in the code as example. Solution is very simple: 1. make a console command 2. add interface to our web app: for example like: BL602 is very close to OTA size limit. Futhermore, BK7231s have issue because OTA file is growing and it's sometimes reaching LFS size and overwrites LFS on OTA If possible and easy, yes. That looks very good and promising!

cdtdsilva · Answer

@p.kaczmarek2 I'm compiling using the github web page. I have absolutelly no problem it not having my name there as a contributor. I'm happy enough that others make use of this code. More and more, it seems one needs to have a doctorate degree to understand the power bill, let alone all the solar stuff and it's rules! That actually reminds me that the home assistant implementation of 'net energy' is probably useless for most people, because under current schemes, there is a grace period one can consume the produced energy, after which is sold or given away to the grid. And so, when I look at the figures, at the end of the day, I can see what was exported, but I don't know how much of it was during the netmetering period or outside. I may add another counter to diferentiate between energy consumed within the netmetering period and energy outside, which was either exported or sold to the grid. I think there is a serious gap on the market to offer something that allows people to understand exactly how they are being billed and how to better make use of their own generated energy... I also realized, that I need a way to override the netmering for my diversion load. For example, if it is a water cylinder or a heat pump, I may want it to run on a timer, if there is no production at the end of the day. So this needs to override the energy based rules. With that, I was trying to find out how to read the button presses. Would you be able to tell me where is this function? And also, how to toggle the status LED? I have played arround with setChannel x 'state' but I can't get consistent results with that. So any guidance to the relevant documentation would be great. I will ditch the 2 minute histerysis to turn the diversion load on and off, and instead rely on the net accumulated energy... Using a 1000W load, as an example, translates to roughly 17W per minute Setting a 40W 'reserve' before the relay turns on provides satisfactory results, without unnecessary delays. I'm presently working with 15min net times, which are the most challenging, but those in poland can choose higher values as they have a whole hour to use any leftovers! Now Back to the code... Can you tell me how this scripting works? Is it the autoexec.bat? I'm thinking that even if flag 25 is left out, the code will still be in the memory. The only true way to save space is to comment the lines at compilation time? But we would need an easy way to generate the binaries. Maybe a separate compilation with some rules changed? That's interesting. So is this webapp pointing to an outside page, instead of being stored on the flash? That would explain why I can't acess it when connected to the Becken hotspot I'm going to do some more fine tuning to the code over the next few days, or whenever we have some sun. It would be great if you could clarify how to read the button presses and toggle the status LED.

cdtdsilva · Answer

@p.kaczmarek2 With the aim of making the code as compatible as possible to different devices, can you please add a new flag to invert direction of AC power? That's because some devices measure consumption as positive or negative. I have have repurposed flag 40 on my tests and that works well, but need one to add to the final code. Something like: 'Invert direction of AC power Flow' Thanks

p.kaczmarek2 · Answer

Sure, here is the flag you requested: Please merge your changes and it's ready to use. I am very happy with the recent progress, when do you expect to be ready for the final PR merge?

cdtdsilva · Answer

Thanks for your help. Possibly this weekend. I'm logging data on two units and need to compare it with supplier data to check accuracy. Everything seems to be working OK. Out of curiosity - Looking at the flag code, with the current flash alocation - How many more flags can we add?

p.kaczmarek2 · Answer

Flags are currently stored as two 32 bit values, so there is a maximum of 64 flags I guess.

cdtdsilva · Answer

Ok. Would it be acceptable for me to use a few more on some future release? I'm thinking as such: Enable cooling Fan (When power generation greater than setpoint) Remote Relay (The flag would enable or disable a storage heater or heat pump based on excess generation) Storage system: When there is no consumption, at the end of the netmetering period, it enables a battery storage, to ofset consumption. Disable net meteing: Shows Import and Export exactly as metered. If not, I will probably run a separate variant with them. On another point, I've been having issues compiling to the BL602. the error messages are rather confusing do you have an idea what the problem might be? Link

miegapele · Answer

That strange error might be due to conflicting bool definition. Try replacing bool definition with stdbool as I've done here

cdtdsilva · Answer

That didn’t work.

miegapele · Answer

new_cfg.h needs to be included as it is in many other drivers. bl602 build seems strict on implicit function declarations

cdtdsilva · Answer

Thanks, that did the trick!

p.kaczmarek2 · Answer

This is a lot of flags. Maybe we should consider just giving you an extra byte for power flags in config struct? That would give you 8 flags. But I am not sure, I would need to think about it. Futhermore... are those strictly related to power metering? SHouldn't be the enable cooling fan be in external driver? The same goes for remote relay. Hmm

Bi-Directional meter for Solar with total Consumption and Export Energy counters (BL0942 based)

Post #1

Post #2

Post #3

Post #4

Post #5

Post #6

Post #7

Post #8

Post #9

Post #10

Post #11

Post #12

Post #13

Post #14

Post #15

Post #16

Post #17

Post #18

Post #19

Post #20

Post #21

Post #22

Post #23

Post #24

Post #25

Post #26

Post #27

Post #28

Post #29

Post #30

Topic summary