ESP12 recorder with html interface and external flash - concept

Hello
The project I'm currently sitting on is storing frequent measurements from SPI sensors in an external W25Q128FV flash (because the internal one is too small).
I came up with the idea to put the HTML and the configuration file in the internal flash with SPIFFS. To the external one will go the data.
I plan to operate this from the phone via an interface written in HTML. The interface is supposed to be able to set configuration variables, control the esp code, display "live" some sensor data, start writing data to the flash, download previously saved data (csv or txt file), delete previously downloaded measurements (I plan to store max 2 measurement sessions - files).

I have the sensors already figured out (I have made myself my mini libraries from the catalogue notes with only the functions I will use).
I have html written. I also have the general execution algorithm of the code devised, but to the point....

I would need advice/guidance in terms of the optimal concept of how to tie this together best.
1. organisation of the external flash - the fastest write would probably be to split it into 2 parts and feed it with raw bytes. But as I will then want to pull this onto the phone to a csv or txt file from the browser, it is possible that it would be better to have a file system (FS or a second SPIFFS if possible) and write to files. Such a data set (file) may be 2-3 MB in size so I also need to take this into account to make room for RAM when downloading to the phone. I will not write this library myself. I will have to use some library (any light/easy ones you recommend)?
2. WiFi - I understand that the appropriate ESP mode is AP?
3. HTML live - in order not to refresh the page manually or at some interval I would probably have to harness websocket for this?
4. the HTML is about 25kB. Should I load it into a variable (I'm not sure yet whether I can fit it in) or is it possible to pull it live from SPIFFS? Since the html will also contain changing data, buttons etc., I'm probably just going to have to load it into a variable anyway, right?

Thank you in advance for any suggestions, and I will be happy to clarify the information if I am unclear.
Yours sincerely
Przemo

prm_ex wrote:

But as I will then want to pull this onto the phone to a csv or txt file from the browser, it might be better to have a file system (FS or a second SPIFFS if possible) and save to files. Such a data set (file) may be 2-3 MB in size so I also need to take this into account to make room for RAM when downloading to the phone. I will not write this library myself. I will have to use a library (any light/easy ones you recommend)?

.
https://github.com/esp8266/Arduino/tree/master/cores/esp8266/spiffs
https://github.com/pellepl/spiffs/wiki

So I understand not messing around with raw bytes to flash just setting up a file system.
Hopefully it will be fast enough that I can manage to write 55 character lines say 100-150/s.
I need to read up on the linked documentation because at first glance it's a mare.
So far I can't see how I'd set up the CS and which SPI I want to let it go to (I'll have a second SPI besides the one for the internal flash) but I'll probably find one.

A file with constants and variables + html of 25-30kB isn't too much so I probably won't use both flashes just batch, including data, better to do on the external one, eh?

I haven't used it yet but I've been trying on the SerialFlash library.
Seems so user friendly for a new user. Do you know/use it? Would it give any advice?
Or is the one given in the post above the most efficient?

prm_ex wrote:

I haven't used it yet, but I was trying on the SerialFlash library

.
But it is more for handling external flash memory.

Well, that is what I am writing about. Maybe I didn't make it clear enough.
4MB internal is not enough. Even if I gave 3 on SPIFFS.
I have a W25Q128FV on the second SPI and two sessions of data writes (100-150 lines of characters, 150Hz) which can give files of about 2-3MB each + html 25kB + configuration file (e.g. ajax) with constants and variables will then fit me and I will leave the whole 4 internal for code (or 3/1 because I don't see the 4M no spiffs option).

Then maybe consider using ESP32 version ESP32-WROVER instead of ESP-12. You will have 4MB SPI flash plus 8MB PSRAM.

With a file system it might not be enough anyway. Besides, ESP's12 and flash's I already have.
Well ok, it stood to reason that, as intended in the first post, it would be best to put a file system on the flash.
And what about the rest?
Mode AP + websocketserver or is there another solution?

prm_ex wrote:

Outside of that, ESP's12 and flash's I already have.

.
Ok, but remember that the maximum (addressable) amount of memory for the ESP8286 is 16MB, or 128Mb. In the case of the ESP32, this is 4x16MB of flash memory.

I don't think we understand each other very well. .
I don't want to unsolder and replace the "internal" flash.
The ones I have (W25Q128FV) are supposed to be an additional external flash (1 for one ESP), operated over SPI (12,13,14) using, as we have already established, a serialflash.
Greetings

prm_ex wrote:

I don't think we understand each other very well. I don't want to unsolder and replace the "internal" flash.

.
Why? I assume you want to connect the external flash memory over SPI to the ESP8266, so what I wrote in post #8 is valid. I suggest you take a look at the documentation for the ESP8266.

Source - page 7.

Quote:

ESP8266EX uses external SPI flash to store user programs, and supports up to 16 MB
memory capacity theoretically.

.
hmmm, and by any chance, doesn't the referenced documentation talk about the external (for the microcontroller), in our case "internal" flash, where we put the code and where the bootloader is located? Isn't it the one that has this limitation, because such support is contained in that bootloader?
In the case of a completely external flash, on a separate SPI, is it not the library used in the code that will limit the size supported?
That's how I would understand it, but I'm not completely insistent. I'm still a thin bole in these blocks.
Anyway, if this applies to any flash, by all means supported, then statically my W25Q128FV (128Mb=16MB) still catches on.

prm_ex wrote:

In the case of a completely external flash, on a separate SPI, will the library not used in the code limit the size supported?

.
The size of the external flash is limited by the ESP8266's ability to address its memory cells.

I think the topic of flash can be considered solved and using it (16MB max) with the file system will be optimal for me, in this case, I understand.
For the interface on the phone without having to navigate to subpages, would the mentioned wbsocket be the only and optimal one or any other alternative I should consider?

prm_ex wrote:

Interface should be able to set configuration variables, control esp code, display "live" some sensor data, start writing data to flash, download previously saved data (csv or txt file), delete previously downloaded measurements (I plan to store max 2 measurement sessions - files).

.

prm_ex wrote:

For an interface on the phone without having to navigate to sub-pages, would the only and optimal one be the mentioned wbsocket or maybe some other alternative I should consider?

.
Wouldn't it be easier for you to use off-the-shelf solutions like blynk, thingspeak, virtuino. They have relevant libraries for the ESP8266. I personally prefer virtuino because it does not require a cloud server.

The device is to be mobile without internet access.
Just ESP and phone. On its own. I have the html skeleton already written (no scripts from buttons or variables on the page yet) and no separate apps to operate on the phone I envisage. I walk up to the device with my smartphone, know the IP and password, connect to the wifi, display the html in the browser and with this I configure the device, run some procedures and walk away. (The wifi can be put to sleep). The ESP runs itself and saves to flash some time. Sometime I come back and load myself a csv then delete it.

As I mentioned for the sensors I already have my minimal implementations written, for flash I will use some lightweight ready-made one (e.g. serialflash).
I have the impression that there is "not much" left so I would prefer to avoid completely changing the approach.
(And these ready-made solutions I will certainly look into at some point).

Hello again, the topic has been lying dormant for a while out of necessity but I have taken it up again.
I already have the circuit on a board from China instead of a contactor, for the flash however I finally used my implementation from the SPImemory library and will be storing the data in bytes.
Well here is a concept I would like to consult.
For example, one circuit gives a string like this (if displayed in the terminal): 123.876;12372;21.3;-0.23;1.23;-2,72;11.7;6;3;EOL

Since I'd like to write bytes in flash, I'm looking for the optimal way to do this.
Admittedly I'll need variables in float, int andt form for calculations, comparisons, filtering etc but I'm wondering if it might be optimal, given the max sampling, to store raw bytes received from sensors.

For example, I get the value -0.23 from the above data string from the accelerometer, from the function:
.

So I would have bytes c1 and c2 ready to write to flash. Yes this would require decoding when downloading to a phone or PC.

Well, the alternative would be what?
1. from a data string (string), after creating it from variables, you could change the ->ASCII->byte character by character, which is a bit pointless and a waste of time. Admittedly, it would be quicker and easier later on with the download, but the priority is writing and its speed.
2. variables I could change to bytes (e.g. float to 4 bytes etc) which is also a bit tedious and also requires decoding on download.

Are there any other sensible alternatives for such a case?
Is there anything else to consider for the concept?

prm_ex wrote:

for the flash, however, I eventually used my implementation from the SPImemory library and will store the data in bytes.

.
I understand that you somehow wanted to simplify the issue of addressing the stored data in the external SPI memory. The SPImemory library itself also directly supports writing/reading float, string, various int types and byte arrays .

prm_ex wrote:

I could change variables to bytes (e.g. float to 4 bytes etc) which is also a bit tedious

.
This is actually quite simple, e.g. for float using union:

khoam wrote:

I understand that you somehow wanted to simplify the issue of addressing stored data in external SPI memory. The SPImemory library itself also directly supports writing/reading float, string, various int types and byte arrays.

By writing "my implementation" I meant using the library as a resource to learn how to communicate, peek at the functions and write my own. That is, trimming the library to the minimum used by me. I did the same with the LIS3DH accelerometer function shown. Based on the found, working library, I wrote the kind of function I presented two posts above.
Yes, I have seen different forms of writing and reading in SPImemory. One of them I will choose and write my function/e/. But I am considering just what way to load the readings into flash memory. I just wanted to consult the various concepts with you here.
So, it's not so much a question of simplifying addressing as of finding such addressing which would be fastest and at the same time useful later.
To reiterate, the priority is speed (sampling, calculations, writing) to take place at the highest frequency. If the optimal concept is chosen, I can handle writing the function (if only using the union you propose - thank you).

I will also mention that ultimately, the target use of the stored data is to have a csv file on a smartphone. So I am also considering writing strings with consecutive lines of reads (about 50 characters);
If the loss on the speed of the code (writing consecutive lines of data) would be small then I guess the strings would be easier to read and transfer to the smartphone, if only via SPIFFS with FAT, from where the download to the phone could be realised straight to the csv file without separate software on the phone side (controlled from the html page - link).
kind regards

prm_ex wrote:

I would reiterate that the priority is speed (sampling, calculations, writing) are to take place at the highest frequency.

.
I understand Could the writing to the external SPI memory be block-based after each sampling and calculation operation i.e. one sample - one write of all necessary values of different types?

prm_ex wrote:

So I am also considering writing strings with consecutive lines of reads (about 50 characters);

.
But then you would have to format these strings properly before writing, which will affect the write speed to the SPI flash.

khoam wrote:

I understand Could the writing to the external SPI memory be block-based after each sampling and calculation operation, i.e. one sample - one write of all necessary values of different types?

.
Maybe a bit of specifics as I am describing so laconically that I may be muddling through.
I collect data from LIS3DH sensors (3 x acceleration + 3xADC) , MS5607 (temp + altitude), ESP's ADC (battery voltage).
On top of that I get the current sampling time of the series (millis from a certain point), the status of the program.
So in effect I get such a sample series without units of course: 999,999(s);24,5(*C);12423(m);3,12(g);0,5(g);0,4(g);7,6(V);4(status);5(uint8_t);EOL
So if you're talking about the block as a whole such series then of course that could be the case, and indeed it's advisable not to connect to the flash repeatedly and jump around the SPI bus unnecessarily often.
That's what I'm thinking to read the whole series of data into variables, which in between will be used for calculations and various decisions of the circuit, and a record is needed to read the data after the whole algorithm has executed and be able to analyse at the desk in excel.

My proposal is as follows.
Define a Params structure that contains fields assigned to individual sensor values: Define a union: Access to individual parameters would be as follows (example): Writing/reading to an external SPI flash could be implemented in this way (using the SPIMemory library): You could, of course, even more simply, without using the unionParams .
.
You could construct a text string based on a function defined in a Params structure - in C++ struct and class are almost the same thing.

Thank you very much for your suggestions. (Unfortunately, I can no longer contribute points ) in this topic.
So summing up the variables though, rather than the raw bytes from the sensors -> well, sure, that was a sick idea .
I still need to think about whether by writing according to the above examples as writeByteArray or writeAnything I will be able to control the address pointer.
To write and especially to read the memory addresses where the next data series starts.

I have a question by the way (probably trivial). Is there any point in writing your selective functions based on bilibraries?
I.e. if a library has 15 functions and a pile of structures and variables, and I only use e.g. 2 simple functions and only some of the variables in my code, will the hex be smaller after compilation anyway and the program use less memory than if I use all the functions and variables of the library?
Because maybe there is no point in this "pruning".

prm_ex wrote:

I still need to think if by writing according to the above examples as writeByteArray or writeAnything I will be able to control the address pointer.

.
This should not be too difficult. Subsequent address values will be incremented by sizeof(Params) and this is a fixed value for the specified parameter set.

prm_ex wrote:

Is there any point in writing your selective functions based on bilabels?

.
Why not, but I would suggest that you first use the SPIMemory library and see how it works. Later you can optimise the program, by replacing the obtained functionality with your own functions. There is some memory in the ESP8266 for such experiments .

prm_ex wrote:

If a library has 15 functions and a pile of structures and variables, and I use in my code only e.g. 2 simple functions and only a part of variables then after compilation the hex will be smaller anyway and the program will use less memory than if I use all functions and variables of the library?

.
Unless we are dealing with class or function templates, the entire library will be included in the program code, regardless of how much it is "used", except for those functions whose entire definition is in the header file - the unused ones will not occupy memory.

Hello after a "short" break .
Me again with a request for a hint. I am already on the final straight with the project but one hurdle still to jump.

Finally, with reference to the above posts, I have the matter solved in such a way that I have the html and config (json) on SPIFFs on the "internal" flash, and on the external flash I save without a file system the raw data collected from the sensors. I have dropped the need to save several readout sessions and the flash only contains data from one acquisition session. This removes the need to control addressing, ranges, etc. The write is done (after clearing) from the beginning, adding more bytes from the string. (However, this is the way I did it - it's manageable for me and has enough samples to write).
The reading is supposed to be from the beginning of the flash until 255 is encountered.
And this is where my hurdle in the final straight comes in.
I can read the data from the flash and dump it to the terminal.

.

Only that this is good as debugging or testing while having the terminal hooked up to the uart. As a reminder, ultimately the uart is not connected to any terminal.
All the fun is between the ESP WiFi_AP and the browser on the smartphone.
The communication running is on websocket. Dynamic display of some live variables, button handling, settings with config entries to SPIFFS (collaboration with script in html) - works fine.
I am missing one thing: I have a button on the html page with which I would like to download data saved in flash to a csv file e.g. in a folder downloaded on a smartphone.
If I had it somewhere in some path in the form of a csv file then <a href...> would suffice, but I don't have such an option. Creating one on the ESP is out of the question due to the size of SPIFFS or memory.
How can this be implemented most simply?
For example, in response to some onmessage from html, could the ESP send a header as if it were starting to send a csv, then in portions send a string of characters and finally send some footer that it's over and the browser would save it as a csv?
Asking for suggestions.
regards