Most efficient way to transmit data via ethernet?

system · October 20, 2014, 4:06pm

This project is my first experience with Arduino. My project is to create a Motorcycle computer that handles a bunch of different data and logs it. I have done a ton of research and have been through countless topics where other people have implemented something similar, but I still have a few questions about my implementation.

The setup is an Arduino Uno with ethernet shield connected to a small WiFi router located under my motorcycle seat. Then an Android phone app that I created using App Inventor sits on my dash displaying all the information that I want.

At this point I have it working pretty well and I am at the stage where I want to refine some things, so that leads me to my questions.

Right now I am communicating between the Android App and Arduino using /GET requests and responses occurring on timed intervals. I have different addresses called by the App which returns different data, but this means sometimes multiple requests go out at the same time and things start to overlap and it gets messy, especially at the high refresh rate that I want (ideally 250-500ms for speed). I want the speed data to refresh as fast as I can get, but the 0-60 data only needs to refresh every 5ish seconds or less. One thing I could do is put all the information into the same GET response, but then I have to send all the data at a high rate even though I don't need all of it that quickly. This would prevent any possibility for conflicts though.
Another feature which I am starting to work on now is utilizing the Position sensors in the Android to get lean angle information. I'm really not sure the best way to get that information from the Android app back into the arduino so it can be logged to the SD card. I would want this to be at a fairly high rate also, similar to the speed data.

As you can see there is starting to be a lot of data that I want to pass back and forth between the Android and Arduino at a fairly quick rate, and I suspect that I am probably not handling it efficiently right now. Any advice you can give me on making sure my data is transferred as quickly as possible without conflicts would be very helpful. Also I apologize for the long post, I wanted to give as much info up front as possible.

Here is a short demo video of it working as is Arduino Digital Speedometer on ZRX1200R - YouTube

robtillaart · October 20, 2014, 4:56pm

Use compression
send data in binary format if possible instead of text

system · October 20, 2014, 5:03pm

Is that possible since i'm communicating over http? Could you be a bit more specific?

robtillaart · October 20, 2014, 5:11pm

Example of compression:
instead of "analogRead 3 value 368" you could send "A3 368" using acronyms for often occurring words and skipping words that do not add value.

Using binary data means that a value like 368 is send as a 16 bit integer which only takes 2 bytes.

The Android receiving APP can expand this abbreviated info to a long text for the user.

I do not know how your protocol looks like in detail but I have seen many from which 50-75% of the data can be squeezed out by using above trick.

Can you post your Arduino code please?

system · October 20, 2014, 5:32pm

Hopefully I posted this properly

/* 
* Motorcycle Speedometer Code V2
*/

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

/* Define constants */
const boolean TRUE = 1;
const boolean FALSE = 0;
const int chipSelect = 4;
const int interuptPin = 0;
const int hallsensor = 2;
boolean enableLog = TRUE;
const int pulsesPerRev = 1;

/* RPM to MPH Calculations
 * 5280ft per mile, 60min/hr.  
 * if C = circumference of tire then RPM/MPH = 5280/(60*C) */
const float rpmPerMphFactor = 14.7; //13.45; /* With an estimated tire C of 6.54495. Actual factor is 13.44548 */ /* 71.83 inch C = 5.9858 */
const int timeoutRefreshRate = 1000;
const int dispRefreshRate = 2000;
const int filterRefreshRate = 5; 
const int diffErrorLevelRPM = 25; /* Val of 100 = 330ms 0-60 time. Val of 50 = 700ms 0-60 time. Val of 25 = 1325ms 0-60 time.  */

/* MAC and IP Address */
byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(192,168,0,2);
String readString;
EthernetServer server(80);

/* Global variables */
volatile long rpm_raw = 0;
volatile unsigned int rpmcount = 0;
int filter_Factor = 7;
boolean sdActive = TRUE;
float mph_filtered = 0.0;
int totalDragTime = 0;

int debug = 0;

/* Setup function run once at startup */
void setup()
{
  Serial.begin(9600); 
  
  Serial.println("Initializing SD card...");
  //pinMode(10, OUTPUT); //If only using SD card
  //digitalWrite(10, HIGH); //If only using SD card
  //pinMode(4, OUTPUT); //If only using ethernet
  //digitalWrite(4, HIGH); //If only using ethernet
  
  if (!SD.begin(chipSelect)) 
  {
    Serial.println("Card failed, or not present");
    sdActive = FALSE;
  }
  else
  {
    Serial.println("card initialized.");
  }
  
  /* start the Ethernet connection and the server: */
  Ethernet.begin(mac, ip);
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
  
  pinMode(hallsensor, INPUT); 
  attachInterrupt(interuptPin, interrupt_func, RISING);//interrupt zero (0) is on pin two(2).
  
  /*Setting limits for filter factor to prevent overflow */
  if(filter_Factor > 7)
  {
    filter_Factor = 7;
  }
  if(filter_Factor < 0)
  {
    filter_Factor = 0;
  }
}

/* Main loop */
void loop()
{
  /* Local defines */
  static int rpm_filtered = 0;
  static float old_mph_filtered = 0.0;
  static unsigned int oldRpmCount = 0;
  static long new_rpm_raw = 0;
  static boolean firstTime = TRUE;
  static long dragTimerStart = 0;
  static long dragTimerEnd = 0;
  
  /* Timers */
  static unsigned long lastmillis = 0;
  static unsigned long lastmillis2 = 0;
  static unsigned long lastmillis3 = 0;
  static unsigned long now_millis = 0;
  
  now_millis = millis();
  
  ethernetDispUpdate(); /* Call ethernet function to serve data to the Android App */
    
  /* Set minimum rpm amount and also prevent value sticking after we stop spinning */
  if (now_millis - lastmillis3 >= timeoutRefreshRate) 
  {
    if(rpmcount == oldRpmCount)
    {
      rpm_raw = 0;
      rpm_filtered = 0;
    }
    oldRpmCount = rpmcount; //update oldRpmCount to be used above so we don't display an old value after we stop moving
    lastmillis3 = now_millis;
  }
  
  /* Filter the rpm value, plausability check, mph calculations */
  if (now_millis - lastmillis2 >= filterRefreshRate)
  { 
    /* Plausability check on rpm delta */
    if(rpm_raw < 0)
    {
      new_rpm_raw = 0;
    }
    else if ((rpm_raw - rpm_filtered) > diffErrorLevelRPM)
    {
      new_rpm_raw = rpm_filtered + diffErrorLevelRPM; /* If new value is significantly higher than filtered, only incrememnt by highest plausible amount */
    }
    else if((rpm_filtered - rpm_raw) > diffErrorLevelRPM)
    {
      new_rpm_raw = rpm_filtered - diffErrorLevelRPM; /* If new value is significantly lower than filtered, only decrement by highest plausible amount */
    }
    else
    {
      new_rpm_raw = rpm_raw; /* Normal condition with no plausability issues */
    }
    
    /* Filter the RPM value */
    rpm_filtered = ((filter_Factor) * rpm_filtered + (8 - filter_Factor) * new_rpm_raw) >> 3; // Maximum rpm with this filter resolution of 8 is ~8191r.   
    
    /* Calculate MPH from RPM */
    mph_filtered = rpm_filtered / rpmPerMphFactor;
    lastmillis2 = now_millis; 
  }
  
  /* Calculate 0-60 times */
  if((mph_filtered > 1) && (old_mph_filtered < 1))
  {
    dragTimerStart = now_millis;
    firstTime = TRUE;
  }
  else if((mph_filtered > 60) && (firstTime == TRUE))
  {
    dragTimerEnd = now_millis;
    totalDragTime = dragTimerEnd - dragTimerStart;
    firstTime = FALSE;
    
    storeToSD(totalDragTime);
  }
  
  /* Output data */
  //if (now_millis - lastmillis >= dispRefreshRate)
  //{ 
  //  Serial.print("RPM = ");
  //  Serial.print(rpm_filtered);
  //  Serial.print("   MPH = ");
  //  Serial.println(mph_filtered);
  //  Serial.print("   0-60 = ");
  //  Serial.print(totalDragTime);
  //  Serial.print("ms     ");
  //  Serial.print(debug);
  //  lastmillis = now_millis;
  //}
  
  /* Save old MPH value for next iteration */
  old_mph_filtered = mph_filtered;
}

/* Handle all the ethernet display stuff here */
void ethernetDispUpdate(void)
{
  // Create a client connection
  EthernetClient client = server.available();
  if (client) 
  {
    while (client.connected()) 
    {
      if (client.available()) 
      {
        char c = client.read();
        /* read char by char HTTP request */
        if (readString.length() < 100) 
        {
          readString += c; //store characters to string
          //Serial.print(c);
        }
 
        /* if HTTP request has ended */
        if (c == '\n') 
        { 
          Serial.println(readString); //print to serial monitor for debuging
 
          if(readString.indexOf("?speed") > 0)
          {
            client.println("HTTP/1.1 200 OK");
            client.println("Content-Type: text/html");
            client.println();    
            client.println(mph_filtered, 1);
            Serial.println(mph_filtered, 1);
            delay(1);
            client.stop();
          }
          if(readString.indexOf("?drag") > 0)
          {
            client.println("HTTP/1.1 200 OK");
            client.println("Content-Type: text/html");
            client.println();    
            client.println(totalDragTime);
            Serial.println(totalDragTime);
            delay(1);
            client.stop();
          }
          else if(readString.indexOf("?logging") > 0)
          {
            if(enableLog == TRUE)
            {
              enableLog = FALSE;    // disable logging
              Serial.println("Logging Off");
              client.println("HTTP/1.1 200 OK");
              client.println("Content-Type: text/html");
              client.println();    
              client.println("FALSE");
              Serial.println("FALSE");
              delay(1);
            }
            else if(enableLog == FALSE)
            {
              enableLog = TRUE;    // disable logging
              Serial.println("Logging ON");
              client.println("HTTP/1.1 200 OK"); //send new page
              client.println("Content-Type: text/html");
              client.println();    
              client.println("TRUE");
              Serial.println("TRUE");
              delay(1);
            }
            //stopping client
            client.stop();
          }
          readString=""; //clearing string for next read
        }
      }
    }
  }
}

/* Store Data to SD Card when called as long as logging is enabled and the SD card is present */
void storeToSD(int logData)
{
  if((enableLog == TRUE) && (sdActive == TRUE))
  {
    File dataFile = SD.open("datalog.txt", FILE_WRITE);
    
    if (dataFile) 
    { 
      dataFile.println(logData);
      dataFile.close();
    }  
    else 
    {
      Serial.println("error opening datalog.txt");
    } 
  }
}

/* Interrupt routine that calculates raw rpm value based on time between interupts */
void interrupt_func(void)
{ 
  static unsigned long now = 0;
  static unsigned long interval = 0;
  static unsigned long lastPulseTime = 0;
  
  now = micros();
  interval = now - lastPulseTime;
  if (interval > 2000)
  {
     rpm_raw = 60000000UL/(interval * pulsesPerRev);
     lastPulseTime = now;
     rpmcount++;
  }  
}

robtillaart · October 20, 2014, 5:48pm

you could set Serial.begin(115200) to have faster serial IO gaining some time to do other things.

the data sent can only be marginally be compressed

              client.println("HTTP/1.1 200 OK"); //send new page
              client.println("Content-Type: text/html");
              client.println();    
              client.println("TRUE");

could become

              client.println("HTTP/1.1 200 OK"); //send new page
              client.println("Content-Type: text/html");
              client.println();    
              client.println("T");

You should consider using raw sockets instead of HTTP packets. Than you can get rid of this part

              client.println("HTTP/1.1 200 OK");
              client.println("Content-Type: text/html");
              client.println();

system · October 20, 2014, 6:02pm

Ok thanks i'll try that stuff out.

Do you have any suggestions on how I should handle the data measured on the Android phone that I want to send down to the Arduino? Should I try to have the Android app do a PUT command to the arduino? I'm also not sure how I should handle these different simultanious threads of send/receive data. It's almost like I need some kind of multiplexor setup.

robtillaart · October 20, 2014, 7:04pm

Sockets are bidirectional so it should work in the same way…

system · October 20, 2014, 7:47pm

After reading up on raw sockets a bit I think I am starting to understand how to implement them on the Arduino, but I am not sure how they would work interfacing with the Android app. I realize that is outside the scope of this forum but do you if that is something I can even do on Android without diving too deep into the unix below?

SurferTim · October 20, 2014, 8:56pm

Can you use UDP on the Android app? UDP can be much faster if used correctly, and it is much friendlier to raw (byte and integer) data types.

Just my opinion...

system · October 20, 2014, 10:27pm

SurferTim:
Can you use UDP on the Android app? UDP can be much faster if used correctly, and it is much friendlier to raw (byte and integer) data types.

Just my opinion...

I might be able to, i've seen a few things that indicate I might be able to do that (specifically this http://ai.kittywolf.net/index.php/TinywebIP). This networking stuff is all new to me. The rest of the Arduino and embedded programming is what I do every day, but networking is something I've never touched.

SurferTim · October 21, 2014, 1:11am

If you want to see how it works, try the UdpNtpClient and UDPSendReceiveString examples included with the ethernet library. I use the same theory with my UDP stuff. The "client" device sends a packet to the "server" device, and the "server" sends a response packet back to the "client". I put client and server in quotes because technically there is no client and server in UDP, but to the NTP code, there is a NTP server. ??

I get about 10 packets (48 bytes/packet) per second, but I'm sure I could go faster and bigger. That is all I need right now.

system · October 21, 2014, 2:23pm

Darn, looks like raw sockets and UDP are not possible with App Inventor at this time. I guess I could start digging into lower level android development.... But that is not exactly where I wanted this project to go.

system · May 5, 2021, 5:17pm