Managing multiple HTTP calls

I looked at the repeating client example, and it' done me pretty well. My issue is that the httprequest function calls client.stop() when it runs. I understand that's needed and if I take that out, I get a no sockets available error.

When I leave it in however, I run the change of having some calls not go through if they are done in parallel based on the timer variables I have. I resorted to just messing with the timers themselves, and not having nice whole numbers. That seems to work for me, but there has to be a more elegant way of doing this.

Is there a way to queue these things? Maybe I can build an array that stores the outgoing data, then loop through it to make the calls? I'm not really sure here. I'd be curious how you guys would handle it. Code below.

#include <NewPing.h>
#include "DHT.h"
#include <SPI.h>
#include <WiFi.h>

unsigned long currentMillis = 0;

//WIFI VARIABLES
char ssid[] = "xxxxx";
char pass[] = "xxxxxx";
int keyIndex = 0;
int status = WL_IDLE_STATUS;
WiFiClient client;
IPAddress server(xxxxxxx);


//DO NOT USE PINS 4, 7 or 10 (AGAIN)

// DHT22 VARIABLES
#define DHTPIN A0 //green
#define DHTTYPE DHT22
DHT dht(DHTPIN, DHTTYPE);
const long tempInterval = 8937;
unsigned long previousTempMillis = 0;
int tempLoops = 0;
String lastTempString = "";
String tempString = "default";
String tempPath = "recordData/temp/";

//SR04 VARIABLES
const long distInterval = 2000;
unsigned long previousDistMillis = 0;
#define TRIGGER_PIN 3
#define ECHO_PIN 2
#define MAX_DISTANCE 100
NewPing sonar(TRIGGER_PIN, ECHO_PIN, MAX_DISTANCE);
String lastDistString = "";
String distString = "default";
String distPath = "recordData/dist/";

//HCSR501 VARIABLES
int motionPin = 8;
const long motionInterval = 50;
const long motionReportingInterval = 10000;
unsigned long previousReportingMotionMillis = 0; 
unsigned long previousMotionMillis = 0;
int motionDetected = 0; 
int motionPoints = 0; 
String motionPath = "recordData/motion/";

//PHOTORESISTOR VARIABLES
const long lightInterval = 523;
unsigned long previousLightMillis = 0;
int lightPin = 5; 
int lastLightVal = 1023; 
int lightVal = 0; 
int lightSensitivity = 100;
String lightPath = "recordData/light/"; 



void setup() {
  Serial.begin(9600);

  //WIFI SETUP
  while (!Serial) {
    ;
  }
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println(F("WiFi shield not present"));
    while (true);
  }
  while (status != WL_CONNECTED) {
    Serial.print(F("Connecting to SSID: "));
    Serial.println(ssid);
    status = WiFi.begin(ssid, pass);
    delay(10000);
  }

  //DHT22 SETUP
  dht.begin();

  //HCSR501 VARIABLES
  pinMode(motionPin, INPUT);

  //CHANGE IF I GET AN SD CARD
  //USED TO DESELECT THE CARD 
  pinMode(4, OUTPUT);
  digitalWrite(4, HIGH);


}

void loop() {


  currentMillis = millis();

  if (currentMillis - previousTempMillis >= tempInterval) {
    previousTempMillis = currentMillis;
    buildTempString();
    if (lastTempString != tempString) {
      lastTempString = tempString;
      httpRequest(tempPath,tempString);
    }
  }

  if (currentMillis - previousDistMillis >= distInterval) {
    previousDistMillis = currentMillis;
    distString = String(sonar.ping_in());
    if (lastDistString != distString) {
      lastDistString = distString;
      httpRequest(distPath,distString);
    }
  }

  if (currentMillis - previousMotionMillis >= motionInterval) {
    previousMotionMillis = currentMillis;
    if(digitalRead(motionPin) == HIGH){
      motionDetected = 1; 
      motionPoints ++;
      //Serial.println("Motion Deteted, waiting to send");
    }
  }

  if(currentMillis - previousReportingMotionMillis >= motionReportingInterval){
    previousReportingMotionMillis = currentMillis; 
    httpRequest(motionPath, String(motionInterval*motionPoints));
    motionDetected = 0; 
    motionPoints = 0; 
  }

  if (currentMillis - previousLightMillis >= lightInterval) {
    previousLightMillis = currentMillis;
    lightVal = analogRead(lightPin);
    delay(1);
    if(lightVal >= lastLightVal + lightSensitivity || lightVal <= lastLightVal - lightSensitivity){
      lastLightVal = lightVal;
      httpRequest(lightPath, String(lightVal));

    }
  }

}

void buildTempString() {
  float h = dht.readHumidity();
  float f = dht.readTemperature(true);
  tempString = "{\"t\":"+String(f) + ",\"h\":" + String(h)+"}";
}

String httpRequest(String sensor_path, String sensor_data) {

  client.stop();
  
  if (client.connect(server, 80)) {

    client.print("GET /myhouse/"+sensor_path+sensor_data);
    client.println(" HTTP/1.1");
    client.println("Host: xxx,xxx,xxx,xx");
    client.println("User-Agent: ArduinoWiFi/1.1");
    client.println("Connection: close");
    client.println();
    Serial.println("GET /myhouse/"+sensor_path+sensor_data);
    } else {
      Serial.println("FAIL - " + sensor_path);
    }
    return "done";
  }

It makes no sense to have httpRequest() return anything when it ALWAYS returns the same useless thing.

When I leave it in however, I run the change of having some calls not go through if they are done in parallel based on the timer variables

I can't imagine how you came to that conclusion, or why you need client.stop() at all.

When you make a request, AND read all the data returned by the server, the connection will be properly closed, and the socket made available again. When you make a request, and don't read all the data, THEN you need to use,client.stop() to make the socket available again, by dumping the server response without reading it.

Dumping random amounts of unread data is rarely a good idea.

PaulS:
It makes no sense to have httpRequest() return anything when it ALWAYS returns the same useless thing.
I can't imagine how you came to that conclusion, or why you need client.stop() at all.

When you make a request, AND read all the data returned by the server, the connection will be properly closed, and the socket made available again. When you make a request, and don't read all the data, THEN you need to use,client.stop() to make the socket available again, by dumping the server response without reading it.

Dumping random amounts of unread data is rarely a good idea.

Yeah the return was there because I'm a C++ newbie. I took that out and declared that function as void. Like I said, client.stop() was in the examples that came with the library. It's there to make sure that the request you are making goes out even if there is a connection already.

Also, I'm not really getting any response from the server. I'm using those endpoints to pass variables through the URL to my node + mongo server that saves the data and that's hooked up with socket.io to a front end.

I did find a quick and dirty solution to my problem though. If client is connected, then wait a random amount of time between 0-1000ms. Not the most elegant, but it solved all my problems.

Sorry you misunderstood my original question :slight_smile:

Also, I'm not really getting any response from the server.

You ARE getting a response. You may not care what the response is, but the server doesn't know that.

Sorry you misunderstood my original question

I didn't.

PaulS:
You ARE getting a response. You may not care what the response is, but the server doesn't know that.
I didn't.

PaulS:
You ARE getting a response. You may not care what the response is, but the server doesn't know that.
I didn't.

So do you have any tips on how I can queue multiple calls?

So do you have any tips on how I can queue multiple calls?

No, because I don't see what the problem is. The way your code is structured now, there will only ever be zero or one http requests in the queue. When there is none, there isn't a problem. When there is one, the httpRequest() function blocks until it is complete, so no more requests can be queued.

Perhaps that is the real problem?

PaulS:
No, because I don't see what the problem is. The way your code is structured now, there will only ever be zero or one http requests in the queue. When there is none, there isn't a problem. When there is one, the httpRequest() function blocks until it is complete, so no more requests can be queued.

Perhaps that is the real problem?

Yup, that's what I meant by "I run the chang

e of having some calls not go through if they are done in parallel based on the timer variables I have." Looking back, it wasn't super clear. 

So let's say that I have my temp recording function making this call every 10 seconds. I have another function trying to make this call every 60 seconds. Once a minute, one of these calls will get stopped because they'll both be trying to make it at the same time. 

What I mean by queued is that there's logic in there that when two places call that function at the same time, it says, OK, let me finish with this temp dude first, and then I'll do you. The delay works for now, because every time this happens, it pushes one call down the road, and if it's still busy, it'll do that again. What I'm trying to figure out is if there's a way that I can queue these to execute one after another, in a more logical way than just adding a delay.

Once a minute, one of these calls will get stopped because they'll both be trying to make it at the same time.

No. If loop() determines that the 10 second cycle is at an end, and a request needs to be made, it will be. loop() won't see, while the httpRequest() is being made, that the 60 second cycle is also at the end. Once the httpRequest() is complete, then loop() will see that the 60 second cycle is past the end, so it will make another httpRequest() (a bit late).

If it matters that a request is a bit late, you need to make one request that handles all the data, with some flags indicating whether to store, or not, the data passed in the request.

PaulS:
No. If loop() determines that the 10 second cycle is at an end, and a request needs to be made, it will be. loop() won't see, while the httpRequest() is being made, that the 60 second cycle is also at the end. Once the httpRequest() is complete, then loop() will see that the 60 second cycle is past the end, so it will make another httpRequest() (a bit late).

If it matters that a request is a bit late, you need to make one request that handles all the data, with some flags indicating whether to store, or not, the data passed in the request.

I have outside variables that are deciding wether or not to make that http request. In the same loop iteration, it's possible for two requests to that same function to be made. I also have some event based requests that have conflicted with the timed ones.

It doesn't matter if a request is late, but there were requests not making it through because they were being made while another one was happening, and them stopped by that line of code. Delaying if there's an active request seems to have solved my problems for now.

Does the arduino http library support multiple TCP stacks?

The pattern would be:

void loop() {
  while there's a new incoming connection
    make a new client object and stuff it into my array of live connections
  
  loop through all live connections {
    deal with client[i] by calling the server function
    if the client has closed, clean up the array of live connections
  }
  
}

I make multiple http requests all the time with no problems with nothing special. Check your code, that's the problem.

mistergreen:
I make multiple http requests all the time with no problems with nothing special. Check your code, that's the problem.

PaulMurrayCbr:
Does the arduino http library support multiple TCP stacks?

The pattern would be:

void loop() {

while there's a new incoming connection
    make a new client object and stuff it into my array of live connections
 
  loop through all live connections {
    deal with client[i] by calling the server function
    if the client has closed, clean up the array of live connections
  }
 
}

This looks like a good way to do it. I'll try to implement it. Thanks!

mistergreen:
I make multiple http requests all the time with no problems with nothing special. Check your code, that's the problem.

It's about making parallel requests and what happens when two are fighting to go out at the same time. Do you have any code that I can look at to see how you handle it?

Why would you want to make parallel requests?
One at a time is fine for most applications.
Maybe you should code it smarter where you gather all sensors inside one loop with one request.
That's how I do it.