Buffering data and sending it to the server

Projects Programming
1

Hello there,
I have a SparkFun Simultaneous RFID Reader - M6E Nano reader connected to a Redboard UNO, these two read RFID tags and send the readings to an ESP8266, and then the ESP sends these data to a server to be shown on a website.
the problem is that the ESP8266 sends the data every 800ms, and the RFID reader reads a new tag every 500ms. so while the ESP sends a packet of data to the server, there might be new readings on the RFID reader, so we lose that data.
My idea to fix this was to make the code in such a way that the Redboard saves the RFID readings somewhere and sends them to the ESP for example every 5 seconds. this way I wouldn't lose any data and ESP would have more time to send the data to the server, and also I could reduce the RFID reading time even more to have more readings. but this is a bit out of my knowledge and I could use some help. I'll share the Redboard and ESP code below.

Redboard Qwiic code:

#include <SoftwareSerial.h>  //Used for transmitting to the device

SoftwareSerial softSerial(2, 3);  //RX, TX
SoftwareSerial mySerial(7, 8);    //RX, TX

#include "SparkFun_UHF_RFID_Reader.h"  //Library for controlling the M6E Nano module
RFID nano;                             //Create instance

///#define BUZZER1 9
///#define BUZZER2 10

char c;
#define EPC_COUNT 50
#define EPC_ENTRY 12
uint8_t EPC_recv[EPC_COUNT][EPC_ENTRY];
bool enable_clr = false;
byte type_s;
byte addr[8];
int val1;

unsigned long previousMillis = 0;  // last time update
long interval = 2000;              // interval at which to do something (milliseconds)
#define PRMDEBUG 0


void setup(void) {
  Serial.begin(115200);
  mySerial.begin(115200);

  initialize_RFID();

  init_array();
}

void loop(void) {

  Check_EPC();
}

void initialize_RFID() {
retry:
  Serial.println(F("try to connect to RFID reader"));

  //Configure nano to run at certain speed
  int RunStatus = setupNano(38400);

  if (RunStatus == 0) {
    serialTrigger(F("Module failed to respond. Please check wiring."));
    yield();
    goto retry;
  }

  if (RunStatus == 1) {  // was not yet in continues mode
    nano.setRegion(REGION_EUROPE);
    nano.setTagProtocol();  //Set protocol to GEN2
    nano.setAntennaPort();  //Set TX/RX antenna ports to 1
    nano.setReadPower(2000);  //10.00 dBm. Higher values may cause USB port to brown out
    //Max Read TX Power is 27.00 dBm and may cause temperature-limit throttling
  }

  Serial.println("RFID shield initialized");
}

/* initialize/empty the array to capture detected EPC */
void init_array() {
  for (val1 = 0; val1 < EPC_COUNT; val1++) EPC_recv[val1][0] = 0;
  Serial.println("done");
}

void Check_EPC() {

  uint8_t myEPC[EPC_ENTRY];  // Most EPCs are 12 bytes
  //if (nano.check() == true) { // Check to see if any new data has come in from module

  byte responseType = nano.parseResponse();  //Break response into tag ID, RSSI, frequency, and timestamp

  if (responseType == RESPONSE_IS_KEEPALIVE) {
    if (PRMDEBUG > 1) Serial.println(F("scanning"));
    Serial.println("hi");
  }

  else if (responseType == RESPONSE_IS_TAGFOUND) {

    for (byte x = 0; x < EPC_ENTRY; x++) {
      myEPC[x] = nano.msg[31 + x];
    }
    add_ECP_entry(myEPC, EPC_ENTRY);

    if (PRMDEBUG > 0) {
      //If we have a full record we can pull out the fun bits
      int rssi = nano.getTagRSSI();  //Get the RSSI for this tag read

      long freq = nano.getTagFreq();  //Get the frequency this tag was detected at

      long timeStamp = nano.getTagTimestamp();  //Get the time this was read, (ms) since last keep-alive message

      byte tagEPCBytes = nano.getTagEPCBytes();  //Get the number of bytes of EPC from response

      Serial.print(F(" rssi["));
      Serial.print(rssi);
      Serial.print(F("]"));

      Serial.print(F(" freq["));
      Serial.print(freq);
      Serial.print(F("]"));

      Serial.print(F(" time["));
      Serial.print(timeStamp);
      Serial.print(F("]"));

      //Print EPC bytes,
      Serial.print(F(" epc["));
      for (byte x = 0; x < EPC_ENTRY; x++) {
        if (myEPC[x] < 0x10) Serial.print(F("0"));  //Pretty print adding zero
        Serial.print(myEPC[x], HEX);
        Serial.print(F(" "));
      }
      Serial.println("]");
    }
  } else {
    //Unknown response

    if (PRMDEBUG > 1) nano.printMessageArray();  //Print the response message. Look up errors in tmr__status_8h.html
  }
}



//Gracefully handles a reader that is already configured and already reading continuously
int setupNano(long baudRate) {

  if (PRMDEBUG == 3) nano.enableDebugging(softSerial);

  nano.begin(softSerial);  //Tell the library to communicate over software serial port

  //Test to see if we are already connected to a module
  //This would be the case if the Arduino has been reprogrammed and the module has stayed powered
  softSerial.begin(baudRate);  //For this test, assume module is already at our desired baud rate

  while (!softSerial) yield();  //Wait for port to open

  //About 200ms from power on the module will send its firmware version at 115200. We need to ignore this.
  while (softSerial.available()) softSerial.read();

  val1 = 0;

  while (val1 < 2) {
    nano.getVersion();

    if (nano.msg[0] == ERROR_WRONG_OPCODE_RESPONSE) {
      if (PRMDEBUG > 1) Serial.println(F("Module continuously reading. Not Asking it to stop..."));
      return (2);
    }

    else if (nano.msg[0] != ALL_GOOD) {
      if (PRMDEBUG > 1) Serial.println(F("Try reset RFID speed"));

      //The module did not respond so assume it's just been powered on and communicating at 115200bps
      softSerial.begin(115200);  //Start software serial at 115200

      nano.setBaud(baudRate);  //Tell the module to go to the chosen baud rate. Ignore the response msg

      softSerial.begin(baudRate);  //Start the software serial port, this time at user's chosen baud rate
    }

    else {
      return (1);  // Responded, but have to be set in the right mode
    }

    val1++;
  }

  return (0);
}

void add_ECP_entry(uint8_t *msg, byte mlength) {
  byte j;
  val1 = 0;
  bool found;

  // as long as not end of list or end of array
  while (val1 < EPC_COUNT && EPC_recv[val1][0] != 0) {
    found = true;

    // check each entry for a match
    for (j = 0; j < mlength, j < EPC_ENTRY; j++) {
      if (EPC_recv[val1][j] != msg[j]) {
        found = false;  // indicate not fuond
        j = EPC_ENTRY;
      }
    }

    // if found
    if (found == true) {
      if (PRMDEBUG > 1) {
        Serial.print(F("FOUND EPC in array entry "));
        Serial.println(val1);
      }
      return;
    }

    val1++;  // next entry
  }

  // Not found, can we still add?
  if (val1 == EPC_COUNT) {
    if (PRMDEBUG > 0) Serial.println(F("Can not add more to array"));
    return;
  }

  // debug info
  if (PRMDEBUG > 1) {
    Serial.print(F("New entry number is "));
    Serial.println(val1);
  }

  // add entry to the array
  for (j = 0; j < mlength, j < EPC_ENTRY; j++) EPC_recv[val1][j] = msg[j];
}

void serialTrigger(String message) {
  Serial.println();
  Serial.println(message);
  Serial.println();
  while (!Serial.available())
    ;
  while (Serial.available()) Serial.read();
}

ESP8266 Code:

#include <ArduinoJson.h>
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <SoftwareSerial.h>

const char *ssid = "...";
const char *password = "...";

SoftwareSerial mySerial(D1, D2); // RX, TX

String httpGET(String path, String header, String val1, String val2);

void ESPSendData(int tempe, int humid);

int connectionCounter = 0;
int rebootCount       = 0;

void setup() {
  mySerial.begin(115200);
  Serial.begin(115200);
  Serial.println();

  for (uint8_t t = 4; t > 0; t--)
  {
    Serial.printf("[SETUP] BOOT WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);


  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  WiFi.begin(ssid, password);
  delay(300);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
    connectionCounter++;
    if (connectionCounter == 10) {
      WiFi.begin(ssid, password);
      connectionCounter = 0;
      Serial.println();
      Serial.print("Connecting");
    }
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

}

void loop() {
  Serial.println("listen");
  String payload = "";
  bool readingFlag = false;
  char s = 0;
  while (1) {
    if (mySerial.available()) {
      s = mySerial.read();

      if (s == '$') {
        break;
      }
      if (readingFlag) {
        payload += s;
        mySerial.println(s);
        Serial.println(s);
      }
      if (s == '@') {
        readingFlag = true;
      }
    }
    delay(2);
  }

  if (payload.length() != 0) {

    //Serial.println(payload.length());
    String val1 = payload.substring(0, payload.indexOf('#'));
    String val2 = payload.substring(payload.indexOf('#') + 1, payload.length());
    //Serial.println(val1);
    Serial.print('\n');
    Serial.println(val1 + "," + val2);
    httpGET("", "", val1, val2);
    delay(800);
  }


}

String httpGET(String path, String header, String val1, String val2) {
  String response = "";
  if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
    WiFiClient client;
    HTTPClient http;
    String payload = "http://api';
    //payload += "api_key";
    payload += "zone_number=1";
    payload += "&tag_id=";
    payload += val1;
    payload += "&rssi=";
    payload += val2;
    http.begin(client, payload); //Specify request destination
    http.addHeader("Content-Type", "application/json"); //Specify content-type header

    int httpCode = http.GET(); //Send the request
    response = http.getString(); //Get the response payload

    Serial.println(httpCode); //Print HTTP return code
    Serial.println(response);  //Print request response payload

    http.end(); //Close connection
    
  }
  else {
    Serial.println("Error in WiFi connection");
    WiFi.disconnect();
    WiFi.begin(ssid, password);
    delay(300);
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
      connectionCounter++;
      if (connectionCounter == 10) {
        WiFi.begin(ssid, password);
        connectionCounter = 0;
        Serial.println();
        Serial.print("Connecting");
      }
    }
    Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
  }
  return response;
}

The Redboard and the ESP connected together in this way:

  • Pin 7 of Redboard <-> Pin D2 of ESP8266
  • Pin 8 of RedBoard <-> Pin D1 of ESP8266
  • GND of RedBoard <-> G of ESP8266 (GROUNDS)
2

Sorry, it isn't clear enough to me.

First, AFAIK SoftwareSerial is unreliable at speeds above 38400 (at least on UNO) and you here have two of them! I don't suggest to set it up at 115.200 baud, and IMHO this could be the main cause of some data loss.

Second, if the incoming data has a 500 ms pace (two per second), I don't know if they come continuously or randomly with 500ms as minimum time between reading.

Third, I wonder why (if) you timingly send the data from ESP8266 and don't work on data events (e.g. I send a packet only when I receive it).

Fourth, is there any particular reason why you need to use a UNO, send data to an ESP and then send it to the server, instead of doing everything with just an ESP (and avoiding any serial issues)?

If I have got it right, I think you just moved the problem forward in time. If you store 5 seconds of data you'll need more time to send it to the ESP, and are you sure you won't receive any other data in the meantime (send every 5 seconds means 10 packets to be sent, thus the communication time will be 10 times than before)?

3

Thanks, I'll try to fix that.

I didn't quit understand what you meant, but: this RFID reader is capable of reading 150 tags per second, but that way it'd heat up and throttle. so there's a 500ms delay to prevent that.

I think it's on events, I actually didn't write the code (nor design the hardware) so I don't really understand it well that's why I'm asking here, but in the serial, the ESP says "listening..." after connecting to WIFI and sends a packet every time the Redboard receives a message.

Honestly IDK, I think the reason is that the guy who made it bought everything as a pack and decided to do it that way, but also the RFID reader board itself is a UNO shield so it might be that as well.

That's the idea, to save the readings while the ESP is sending them away, or while the Redboardis sending the to the ESP. something like this:
read1->stored
read2-> stored
read3->stored
five seconds reached->send to ESP - meanwhile: read4->stored
still sendng to ESP - meanwhile: read5->stored
and so on...

4

I think we need to know the final target environment and see some code (even if an example) to give you better and useful answers...

Good, but I still can't get why you decided for such timings and delays... Read an RFID, immediately send the packet to ESP via serial (the time required depends on the packet size and -obviously- the serial speed), then the ESP will send it to the server via TCP or whatever. Remember serial interfaces have a buffer, so even if timings are critical, it might not be an issue if the data to be sent via serial is a few bytes long (e.g. the RFID ID?). And in this case the serial connection should be cleared quickly enough, with or without implementing some kind of FIFO buffer (if you know what it means you could find some code samples on how you can implement it, or there are even some libraries for that).

So, I think you start making some performance tests with a revised version of your code as a POC (trying to get rid of any delay() calls), then we'll get into it and see where bottlenecks could be (if any).

5

How do you imagine that working? If you're trying to buffer data that is coming in once every 500ms and only sending it out once every 800ms, don't you think the buffer will eventually fill up? Buffers like this are only use full for smoothing out time-variant rates of data arrival and/or transmission. However, on average, the rate of data entering the buffer must be exactly equal to the rate of data exiting the buffer. Otherwise, you will get buffer underflow / overflow.

6

I was thinking of saving the readings in the buffer and then sending all of them at once to the server

7

Code of what? if you mean the codes that are on my boards right now, I posted both of them in the main topic.

8

Code of some kind of test to reproduce the issue on a smaller code we could try and see what happens in detail, without any other element/device to be connected. It's often directly derived from the main code by removing sections not necessary. I also said:

So, I think you start making some performance tests with a revised version of your code as a POC (trying to get rid of any delay() calls), then we'll get into it and see where bottlenecks could be (if any).

By the way, is it the only thing you read of my latest post?