[Solved] ESP8266 (Arduino): Optimisation of server polling frequency in lighting control

Let me start by describing the problem. I have a lot of ESP at home. The whole control model is based on connectivity to a server, specifically PHP scripts. The WEB application connects to the server and sends or receives information (e.g. lights a lamp, reads the temperature, etc.). On the other hand, I have ESPs that respond accordingly. If they just send data (e.g. temperature) then there is no problem – they do it at a certain interval. Worse when they are actors – then they ask the server if there is something to do e.g. whether to switch on a relay. In the case of lighting control, however, I don't want to wait long for a response, hence the polling needs to be frequent e.g. every 0.5s. This is not a problem for the server, but nevertheless generates unnecessary traffic and even so these 0.5s are noticeable. Of course, the solution would be to set the ESP in the position of the server, but here other problems arise, e.g. time control – for example, if someone turns on an outdoor light and forgets to turn it off, then with regular polling, there is no problem for the PHP server to conclude that the sun has risen and there is no need to shine.
So I came up with a solution that combines the advantages of both solutions (client and server) for actors. It involves cyclically querying at a longer interval (e.g. 20s), while at the same time using the web server function in ESP exclusively to signal – „stop waiting and query, because I have something for you”. In theory, this approach doesn't change the philosophy and doesn't require a lot of software redesign, yet it should be fast and effective.
My idea is very simple (below is the skeleton of the idea):
.
And now the questions – is such an immediate server shutdown (without downloading) correct? Will everything allow itself and will there be no memory leak? Can I use the „client” variable immediately for uploading?

Creating and deleting an object client class WiFiClient in a loop() for each loop pass is not very practical and may involve increased heap fragmentation. .

Added after 8 [minutes]:

krzbor wrote:

And now questions – is such an immediate server shutdown (without downloading) correct?

.
What kind of "shutdown" of the server are you referring to? Where is it in the code?

I mean:
.
That is, I check if something is there and don't even read it, but immediately client.stop()

As for the location of the WiFiClient (inside the loop) you are absolutely right. I think I used a not very good example once.

Hello,
in the past if I remember correctly, http and www were used to serve pages to the browser. If you chose php because you have a good understanding of the issues, then you could use websocket for two-way communication. If not necessarily, then the easiest thing would be to write a separate server to communicate with clients (I mean service, because whether the "server" is a server or a client is irrelevant), here it's all the same whether in php, c or other. You solve both problems in one go.

krzbor wrote:

I mean I check if something is there and I don't even read it, but immediately client.stop()

.
You are not connecting to the server (no client.connect()), so no extra buffer is created in memory (ClientContext and for TCP). All you should do is check what value client.stop() returns - if false, then the client has not been properly closed. This is unlikely in this particular case From the code above, it appears that you can call the client.stop() command without checking the client - the stop() function already does that.

EDIT: The connection to the server may have been created in a previous iteration of loop(), so checking what value client.close() returns is needed. It would also be good to give the client some time to close the connection - parameter maxWaitMs .

I am not explicitly allocating anything. The point is that when server.available() is called, some memory will already be allocated because some data has already been received (unless I'm thinking wrong). I don't read that data, I just call stop(). Second point - from your description I conclude that it's better to use two objects of the WiFiClient class - one for receiving and one for sending then I won't have to worry about the time for closing.

krzbor wrote:

I am not allocating anything.

.
This is done by client.connect() in some iteration.

krzbor wrote:

that it's better to use two objects of the WiFiClient class - one for receiving and one for sending then I won't have to worry about time to close.

If both clients are "chatting" to exactly the same server, it won't change much. Instead, it will increase RAM usage - two client contexts instead of one.

Maybe I have described the idea wrong. In a loop which is called quite often (10ms) at first I check if there was a call to the ESP server (server.available(); ) and then instead of reading from it and being interested I want to immediately ignore everything that came in and close the call. I don't want to read anything from it. It's just supposed to be this kind of 'trigger'. Then there is my actual connection (which I have been using for years) - this time ESP is the client, not the server. I started this thread to consider this problem - if the ESP server "reports" server.available(), and I brute-force it immediately with client.stop(); would this be correct.

krzbor wrote:

if the ESP server "reports" server.available() and I brute-force it immediately with client.stop(); will this be correct.

.
Nothing bad will happen - server.available() will either return a reference to a client that already has a connection to it, or create a new object WiFiClient if there was no such connection or it was previously broken. In both cases you can call client.stop() , but since you don't know which of these clients was returned you should check what value it returns client.stop() . If you want to know whether the server already has a connection to the client, before or after calling server.available() , you can check this with the command server.hasClient() .

OK, but it can't return the wrong client to me because one has any port from Apache and port 80 on ESP server, and the other has any port on ESP client and 80 port on Apache.

I didn't write anything about a "bad" client, but about a reference to a client connected to a server. If there is more than one, subsequent calls to server.available() will return further references to those clients or an "empty" client. Whether a client is connected to the server can be checked using client.connected() , client.status() , client.remoteIP() or simply the bool() operator.

Added after 1 [hour] 17 [minutes]: .

krzbor wrote:

OK, but it can't return the wrong client to me because one has any port from Apache and port 80 on the ESP server, and the other has any port on the ESP client and 80 port on Apache.

.
So there is only one external client connecting to the local server on the ESP? I don't quite understand this statement. How about a simple diagram?

khoam wrote:

So only one external client connects to the local server in the ESP? I don't quite understand this statement. Maybe some simple diagram?

.
Exactly as you wrote.
Currently it is as follows:
1. I turn on the light in the browser (app).
2. this information flies to the main server (to PHP).
3. PHP writes the information "you need to turn on the light for 7 minutes" into a file.
4. completely asynchronously ESP connects to the same server and asks "do you need to turn on the light?".
5. the server replies "yes light on" or "no light on" (depending on the time)
6. the ESP performs the specified action.
In order for the whole thing to be responsive, the ESP needs to ask itself frequently, e.g. every 0.5s, and this is what I wanted to change.

After the change it should be like this:
1. in the browser (app) I turn on the light.
2. this information goes to the main server (to PHP).
3. PHP writes the information "you need to turn on the light for 7 minutes" into a file and additionally sends a signal to the actual ESP "stay awake and get on with it"
4. completely asynchronously the ESP connects to the same server and asks "do you need to turn on the light?".
5. the server replies "yes light up" or "no light up" (depending on the time)
6. the ESP performs the specified action.
Now the ESP can connect every 20s (e.g. to find out that it's already daytime and it's pointless to light up), because an additional signal (trigger) has forced the immediate triggering of point 4. As you can see, this additional trigger can, in my conception, only be fired from the main server. It also does not carry any data, it is simply to force ESP to immediately execute 4.

I know the whole concept can be remodelled, but it works for me. I just want to optimise it. I am currently adding an OTA to my solutions. When ESP starts, there is a version check and a possible firmware swap (the version is also checked by PHP). On this occasion, I will be collecting the IP addresses of the modules so that PHP at point 3 knows which ESP to urge.

Since only one client is supposed to connect to the server on the ESP, you already have the answer in post #9. The only thing I would gently change is: That is, give a short timeout to break the connection and clear the buffers.

Added after 40 [minutes]: .

When you call the command client.connect () this first implicitly calls the function client.stop () and buffers are released if this client has already been connected to any server . Operator= for WiFiClient works analogously: closes the previous network connection.

You are operating on a single object of class WiFiClient , hence my comment .

@khoam, thank you for your guidance. In the end, I decided on 2 WiFiClient objects (declared outside the loop of course) - one for the server and one for the client, and left the client.stop without parameters. The server client will have a lot of time because when there was a request to the ESP server, there is an ESP client action (and that action is running on another object), which is after all ongoing. Then the start of the loop has a 10ms delay. After the initial perplexities of not working (I just forgot about server.begin(); ) the whole thing works wonderfully - I built a test circuit with an LED - virtually instant responsiveness.

It was time to move the test solution to the existing solution. Overall it works well - instant responsiveness as it was during testing. Unfortunately there is a noticeable delay every so often (once every few dozen actions). I estimate it to be about 1s. I have analysed the PHP code and it is not the cause. The ESP remains. It is definitely not related to my "trigger" getting lost, because the delays would be random and much longer (up to 20s). The only place is:
.
There is a 1 second delay here. I changed the program and instead of delay(1000) I gave delay(100). Since then I have not seen a delay once. Of course, the problem itself probably still occurs, but the 100ms is not noticeable, and the next call already happens. I'm writing about this because I was treating the line "if (!client.connect(host, httpPort))" as a really fallback solution. And here it appears that a few percent of calls may be affected. Is this normal?

krzbor wrote:

I'm writing about this because I was treating the line "if (!client.connect(host, httpPort))" as a really fallback. And here it appears that a few percent of calls may be affected. Is this normal?

.
Why not? Without seeing the whole program, I can only guess that in certain situations there is not enough memory on the heap for the client context or because of a timeout.
Also, client.connect () defaults to setNoDelay (defaultNoDelay) and this can be changed.

I think I have found the reason. Well, when the WiFiClient is inside the function, as I read, it is not necessary to call client.stop() (it is called when releasing the object). When the WiFiClient object is brought outside the loop function client.stop() is mandatory, and I didn't have it.
For now, I went back to delay(1000) to see if the delay would show up again. However, it works flawlessly.

krzbor wrote:

When you elevate the WiFiClient object outside the loop function client.stop() is mandatory, and I didn't have it.

.

khoam wrote:

When you call the client.connect() command, the client.stop() function is implicitly called first and buffers are released if that client was already connected to any server. The operator= for WiFiClient works analogously: it closes the previous network connection.

.
Of course, as you don't want to wait for the next call to client.connect() to invalidate the previous client context, you can do it earlier.

I was based on this thread: Link Several people commented that without the "stop" something didn't work for them. I can only confirm that "stop" has also improved my situation.

krzbor wrote:

Based on this thread: Link Several people commented that without "stop" something didn't work for them. I can only confirm that "stop" has also improved my situation.

.
These comments were for any of the 2.4.0-rc versions of Arduino Core for ESP8266, from early 2017. In that version, actually client.connect() was not releasing resources for the previous client context - this was clearly a bug.

The communication problem has finally been solved. After moving WiFiClient outside of the function (global object) the setting client.setTimeout(10); applies not only to subsequent calls (client.readString() but also to other operations, in particular client.connect(host, 80); Here these 10ms are often not enough and therefore did not always work. Now before client.connect(host, 80) I added client.setTimeout(300); and it is OK. In general, I have to say that the concept of periodically checking the state every 20-30 seconds with an additional wakeup by calling the server on the ESP works well - instant responsiveness, and if anything there is a status query every 20 seconds anyway.
Thank you @khoam Topic closed