How to delay writing to web storage data page

I have code in a function (see below) that accepts a 'status' text message from my main code and writes it to a web server based log file ( by Posting to an online php script ).

The code is working well, but I am finding that while the function is posting the text to the web server, it delays execution of the rest of my sketch. Any delays on the internet is slowing the connect and post operations, making the problem worse.

As the execution of the sketch is, in this case, more important than the logging / posting of the status, I was thinking to accumulate the status text's, and post them to the function only when there has been no other activity (input triggers) in the sketch for, say, 15 seconds.

Question : what is the best way to store these status texts prior to sending to the function ?

LogItWebC("Activating system 1");


void LogItWebC(char* LogTxt){

	char XXbuffer[80]; /* local buffer into which strings are encoded and from which they're output'd */

	if (client.connect(myserver, 80)) {
		client.print("GET http://www.xxx.com/serveit.php?data=");
		client.print(urlencode(XXbuffer,LogTxt));
		client.println(" HTTP/1.1");
		client.println("Host: www.xxx.com");
		client.println();
		delay(250);
		client.stop();
		Serial.println(LogTxt);
	}
}

Question : what is the best way to store these status texts prior to sending to the function ?

You have given no indication about how many status texts you want to store, or how big each one is. Therefore, there is no best way to store random amounts of data.

it might require sending over upd instead of tcp... but then you might miss samples too
and it requires deeper level network programming then simply http....

PaulS:
You have given no indication about how many status texts you want to store, or how big each one is. Therefore, there is no best way to store random amounts of data.

The texts are all 1 liners, max 150 chrs, and there would be no more than 30 texts.

The texts are all 1 liners, max 150 chrs, and there would be no more than 30 texts.

So, never more than 4500 characters, or bytes. You have no where near enough room in memory for them, nor do you have enough room in EEPROM for that much data. That leaves an SD card. Storing on the SD card takes time, as does reading the data from the SD card. So, using an SD card is not going to speed up sending data. Doing so is not even moving in the right direction.

PaulS:
So, never more than 4500 characters, or bytes. You have no where near enough room in memory for them, nor do you have enough room in EEPROM for that much data. That leaves an SD card. Storing on the SD card takes time, as does reading the data from the SD card. So, using an SD card is not going to speed up sending data. Doing so is not even moving in the right direction.

OK. How about I change the logic - instead of POSTing to the PHP a long text, like "Activating Main Control System 1", I change to a "code number" like 'c001' . Then I code the receiving php script to convert the received code into the required text, and update the online log file. This way, the max length of the code reduces from 150 characters to only 4.

So assuming that I would now have enough memory ( ps.. using a Mega 2560 ), how do I store these ready to be POSTed after xx seconds of inactivity.

I know how to measure the inactivity time since the last event - I can simply add a LastActivityMillis variable and set LastActivityMillis = millis(); as each input trigger is received, and then check if millis() - LastActivityMillis > xx seconds. The question is how to store the (max) 30 codes - what type of array (?) should I use.

So assuming that I would now have enough memory ( ps.. using a Mega 2560 ), how do I store these ready to be POSTed after xx seconds of inactivity.

In an array. Two arrays, actually - one a character array that contains the message type (c) and another int array that contains the value (001 also known as 1).

The question is how to store the (max) 30 codes - what type of array (?) should I use.

See above.

The code is working well, but I am finding that while the function is posting the text to the web server, it delays execution of the rest of my sketch. Any delays on the internet is slowing the connect and post operations, making the problem worse.

What actual time frame do you consider "slow"? What is the length of a delay that is unacceptable?

zoomkat:
What actual time frame do you consider "slow"? What is the length of a delay that is unacceptable?

Hi Zoomkat :slight_smile:

Here's an example.

I have motion sensors around the house that log each time they are activated.

I want to go out, exit the house, lock the door, and enable the alarm system with a remote control.

Walk to the gate.

Can't open the gate because the motion sensor detected my movement, and is posting the data to the online log file.
I have to stand still for a few seconds before I can open the electric gate ( gate remote connected to Arduino, and Arduino controls a relay to trigger the gate release ).

I have checked the alarm system code and no delays there ( and nothing in the code that would / could delay the code loop ), so it has to be the uploading of the data to the log file ( ethernet card ) where the delay is happening.

Second Example :

At night I arm the alarm system.
We have PIR sensors in the living area of the house, but because our young kids get up before us, I have a set of infra red beams in the hallway.
If they walk from the bedrooms to the lounge, they break the beam, and that disables the PIR in the lounge. However, every once in a while, when they break the beams, the write of the data to the online log file takes a fraction too long, and the kids get to the lounge and the PIR activates, alarm goes off, wife not husband friendly when woken up at 5:30am by the alarm, etc, etc

Your problem is probably with uncontrolled sensor overload and churn in the code you don't show.

zoomkat:
uncontrolled sensor overload

?? are you talking about, for example, when a PIR sensor is triggered, and it closes a circuit for around 1 second, during which time the Arduino sketch loops enough times for it to register a long string of signals on the input pin ?

Not the case here, as I use variables to remember the last state of all inputs, and only act on the input once it changes, and thereafter the last state is changed, so the code for an activated input / PIR is only run once when activated.

DaveO:
Can't open the gate because the motion sensor detected my movement, and is posting the data to the online log file.
I have to stand still for a few seconds before I can open the electric gate ( gate remote connected to Arduino, and Arduino controls a relay to trigger the gate release ).

I'm only guessing because I don't understand the details of your system, but are you trying to log the sensor event before you respond to it? Maybe it would be more responsive to do it the other way round.

Do you have any rate limiting to slug the event rate if one of your sensors starts flapping?

PeterH:
I'm only guessing because I don't understand the details of your system, but are you trying to log the sensor event before you respond to it? Maybe it would be more responsive to do it the other way round.

Hi PeterH

No, the online log entry only happens after the sensor trigger and code in response to the trigger. What I believe is happen, however, is that the time taken to log the trigger of a PIR sensor, is delaying the loop ( = a pause in the loop code ) , which is preventing the code from reacting to an input on a different pin.

PeterH:
Do you have any rate limiting to slug the event rate if one of your sensors starts flapping?

Not yet, but that's a good idea. All I need is a bunch of ants or other insects setting up camp in/on the sensor and my code is going to think we're being invaded.

Do you know how often these events are being logged and how long they could realistically take to be logged?

If they (can) take a significant time then the ideal solution would be to make either the sensor detection or the network I/O asynchronous, so that you can do both jobs in parallel. Since I'm not familiar with the code for either, I don't know how easy it would be to do that.