enc28j60 - How to send data to remote server faster/non blocking

Hi,
I am building new set of wireless temperature monitors with rfm69hw wireless modules. Wireless communication is great - great range, basically no lost packets. But I have problem on the gatweay which receives the data and then sends it to remote server for storing into database.
Gateweay setup is:

1 --> I2C --> 2 --> ethernet --> php script

So when 1 receives payload from the wireless node, it sends it via I2C using wire library to 2.
[2] is using Wire.onReceive(handler) and in the handler function I read the data and store it in a struct, afterwards I am constructing URL from the struct's data and calling a function which initiates ethernet client and calls php script using the constructed URL which stores the data in to remote database.

It works with 1 remote node and send interval of 2 seconds (this is for testing purposes, in reality I will use longer intervals) and I have basically no dropped values. The problem starts with 2 remote nodes. [1] receives all data from both nodes, but [2] is not able to store them all via ethernet.

Do anybody have an idea how to design this setup to be more reliable? I think the ethernet sending part is quite slow and causes drops of some received payloads. Or maybe some of the sendings over ethernet are interrupted by further Wire.onReceive events?

Thanks for any ideas, suggestions.

Post your code (for the moment that of the gateway should be enough)!

Here is code for 1:

#include <RFM69.h>
#include <Wire.h>
#include <RFM69_ATC.h>

#define NODEID      999
#define NETWORKID   100
#define FREQUENCY   RF69_433MHZ //Match this with the version of your Moteino! (others: RF69_433MHZ, RF69_868MHZ)
#define KEY         "iBooriqueinaenae" //has to be same 16 characters/bytes on all nodes, not more not less!
#define SERIAL_BAUD 115200
#define IS_RFM69HW_HCW  //uncomment only for RFM69HW/HCW! Leave out if you have RFM69W/CW!

#define ENABLE_ATC    //comment out this line to disable AUTO TRANSMISSION CONTROL

#ifdef ENABLE_ATC
RFM69_ATC radio;
#else
RFM69 radio;
#endif

typedef struct {
        float temp;
        float humi;
        float volts;
        int nodeId;
        int counter;
        int16_t rssi;
} Payload;
Payload theData;

void setup() {
        Serial.begin(SERIAL_BAUD);
        delay(10);
        radio.initialize(FREQUENCY,NODEID,NETWORKID);
#ifdef IS_RFM69HW_HCW
        radio.setHighPower(); //must include this only for RFM69HW/HCW!
#endif
        radio.encrypt(KEY);
        Wire.begin(); // join i2c bus (address optional for master)
}


void loop() {
        if (radio.receiveDone())
        {
                if (radio.DATALEN != sizeof(Payload))
                        Serial.print("Invalid payload received, not matching Payload struct!");
                else
                {
                        theData = *(Payload*)radio.DATA; //assume radio.DATA actually contains our struct and not something else
                        theData.rssi=radio.RSSI;
                }

                if (radio.ACKRequested())
                {
                        radio.sendACK();
                }

                Wire.beginTransmission(8); // transmit to device #8
                Wire.write((byte *)&theData, sizeof(theData));
                Wire.endTransmission(); // stop transmitting
        }
}

And here is the code for 2:

#include <UIPEthernet.h>
#include <Wire.h>
#include <avr/wdt.h>

typedef struct {
        float temp;
        float humi;
        float volts;
        int nodeId;
        int counter;
        int16_t rssi;
} Payload;
Payload payload;

char universal_buff[8];
bool havedata = false;
bool sent = 0;
unsigned int retries = 0; // variable for controlling retrying of sending via ethernet
#define MAX_RETRIES 3 //max amount of attempts for ethernet sending

//
// Ethernet config
//
byte mac[] = {  0x6a, 0xce, 0x22, 0x09, 0xb8, 0x25 };
IPAddress ip(192,168,2,2);
IPAddress gateway(192, 168, 2, 106);
IPAddress subnet(255, 255, 255, 0);
EthernetClient client;
char pageAdd[94];
//
// End - Ethernet config
//

void setup() {
        // Start ethernet
        Ethernet.begin(mac, ip, gateway, subnet);
        wdt_enable(WDTO_8S); //watchdog enable, 8 seconds
        Wire.begin(8);        // join i2c bus with address #8
        Wire.onReceive(receiveEvent); // register event
        //Serial.begin(115200);   // start serial for output
        wdt_reset();
}

void loop() {
        wdt_reset();
        if (havedata == true) {
                //prepare and send the data via eth
                dtostrf(payload.temp,1,2,universal_buff);
                sprintf(pageAdd, "/add.php?temp=%s&", universal_buff);
                dtostrf(payload.humi,1,2,universal_buff);
                sprintf(pageAdd + strlen(pageAdd), "humid=%s&", universal_buff);
                dtostrf(payload.volts,1,4,universal_buff);
                sprintf(pageAdd + strlen (pageAdd), "voltage=%s&counter=%d&auth=12345678&s=%d&rssi=%d", universal_buff, payload.counter,payload.nodeId,payload.rssi);

                //
                // Ethernet send part
                //
                sent = false;
                retries = 0;
                do
                {
                        sent = sendData(pageAdd);
                        if ( retries++ > MAX_RETRIES )
                                sent = true;
                }
                while ( sent == 0 );
                //
                // END - thernet send part
                //
                havedata = false;
        }
}


void receiveEvent(int howMany) {
        // read into structure
        byte * p = (byte *) &payload;
        for (byte i = 0; i < sizeof(payload); i++)
                *p++ = Wire.read ();
        havedata = true;
}

bool sendData(char *page)
{
        char outBuf[107];
        bool cconn=false;
        unsigned int connectLoop=0;
        do //try connecting until connected or timeout
        {
                connectLoop++;
                cconn = client.connect(IPAddress(192,168,2,106),80);
                if ( connectLoop > 10000 ) return 0; //timeout if too many not succsssfull attempts, return 0
        }
        while ( cconn == false );

        if ( cconn == 1 )
        {
                sprintf(outBuf,"GET %s HTTP/1.1",page);
                client.println(outBuf);
                client.println(F("Host: 192.168.2.106"));
                client.println(F("Connection: close\r\n"));
                connectLoop = 0;
                while (!client.available())
                {
                        connectLoop++;
                        if ( connectLoop > 10000 ) return 0; //timeout if client was not available for long time, return 0
                }
                connectLoop = 0;
                while (client.read() > 0)
                {
                        connectLoop++;
                        if ( connectLoop > 10000 ) return 0; //timeout if reading data took too long, return 0
                }
                client.stop();
                return 1;
        }
}

payload must be declared volatile because it's changed in interrupt context. Same for havedata.

You don't check for pageAdd to overflow.

Do anybody have an idea how to design this setup to be more reliable? I think the ethernet sending part is quite slow and causes drops of some received payloads. Or maybe some of the sendings over ethernet are interrupted by further Wire.onReceive events?

Use just one Arduino for both tasks. That way you eliminate the I2C communication which interrupts your Ethernet chip quite often. And it might be useful to use a WizNet5100 based Ethernet connection because that chip does much more in hardware than the ENC28j60.

Hi,

thanks for the hint with the volatile qualifier.

Regarding using only 1 arduino for both tasks on gateway - I am not sure if it is possible to use rfm69 and anc28j60 on 1 arduino (I remember I wanted this setup but then I decided to use 2 arduinos, but I am not sure now what was the reason). And second thing - I already have PCB with all the components (I used 2 arduino pro minis, enc28j60, rfm69, etc.. on the same pcb board). So theoretically I could create another PCB and use Wiznet5100, but I do not want to spend more time and money now. I will simply do the best optimizations to the current setup and will live with its drawbacks.

Today I played a bit with the code and I modified it to use payload as a fixed size array - so all new received data via I2C are stored into a "free index" of the array and in the main loop the array is looped and in case there are new data ready there, it is being send via ethernet. My synthetic tests shown success. It looks it could be much more reliable now and the loss of data will be lower. Let's see in the reality - when I will deploy couple of the remote nodes.