High frequency data logging over serial

Hi Arduino Community!

I'm working with Arduino for a few weeks now and i do have some coding knowledge. However, as it seems i cannot figure out the problem i'm facing. Maybe you can help me!

My goal is to use an Arduino UNO with an 200g-accelerometer to sample at a >500Hz rate (for a few minutes) and log the data over a serial connection. After some literature research and following trials im sticking with this:

My Arduino Code:

//some declarations...

[...]

void setup() {
	//Initialize Program on LCD
	LiquidCrystal lcd(0);
	lcd.begin(20,4);
  	lcd.setBacklight(HIGH);
  	lcd.print("  TEST ENVIRONMENT  ");
 	lcd.setCursor(0, 2);
 	lcd.print(" Preparing...       ");
 	lcd.setCursor(0, 3);
 	lcd.print("PRESS ENTER TO START");

	// Serial connection
	Serial.begin(115200);
	delay(2000);
	Serial.println("Starting Test Program");

	// pins
	pinMode(ledPin,OUTPUT);
	pinMode(sensorPin, INPUT);
	
	while(1)	{
		//Serial.println(digitalRead(button));
		if(digitalRead(button)==1) {
		break;
		}
	}

	lcd.setCursor(0, 2);
 	lcd.print("TRANSMISSION ONGOING");
 	lcd.setCursor(0, 3);
 	lcd.print("--------------------");

} // end SETUP


void loop() {

//Sending Sensor data over serial
 	
	//sensValue=analogRead(sensorPin);
	sensValue+=1;
	time_us = micros();

	Serial.print('H'); 		// sends 1 byte
	sendBinary(time_us); 		// sends 4 byte
	sendBinary(sensValue); 	// sends 2 byte
	delay(10);


} // end LOOP





// ==========================================================
// Functions

void sendBinary(int value)
{
	// Send 2 bytes
	Serial.write(lowByte(value));
	Serial.write(highByte(value));

}

void sendBinary(unsigned long value)
{
	// Split 32bit long into lower 16bit integer value...
	int temp = value && 0xFFFF;
	sendBinary(temp);
	// ...and higher 16bit integer
	temp = value >> 16;
	sendBinary(temp);

}

For first debugging purposes i used a counter instead of the sensor data.

Here's the other end of the serial connection using a simple Processing Script:

//Some declarations:
[...]

void setup () { 

  // List all the available serial ports
  println(Serial.list());
  // Check the listed serial ports in your machine
  // and use the correct index number in Serial.list()[]. 
  myPort = new Serial(this, portName, 115200);  //

File f = new File("data.txt");
  if (f.exists()) {
    f.delete();
  }

fileOut = createWriter("data.txt"); 
}


void draw () {

 if ( myPort.available() >= 7) // If at least 7 bytes are available, {
    {
      if (myPort.read() == HEADER)
      {
      // Read in Long (2x2xByte)
      int_longA = myPort.read();
      int_longA = myPort.read() * 256 + int_longA;
      
      int_longB = myPort.read();
      int_longB = myPort.read() * 256 + int_longB;

      longValue = 256*256*int_longB + int_longA;

      // Read in Integer (2xByte)
      intValue = myPort.read();
      intValue = myPort.read()*256 + intValue;

      println( longValue + " " + intValue);             //Console Print Out
      fileOut.println(longValue + ", " + intValue);  //File Print Out
      
      }
    }


  
}

void keyPressed() {
  fileOut.flush();  // Writes the remaining data to the file
  fileOut.close();  // Finishes the file
  exit();  // Stops the program
}

Works fine for delays of 100ms on the Arduino side.
For a delay of 10ms (so i can get a maximum sampling rate of <100Hz) the data shows duplicate values for the time (in microseconds) which is weird.

data.txt:

4915201, 1
4980737, 2
4980737, 3
4980737, 4
4980737, 5
4980737, 6
4980737, 7
4980737, 8
5046273, 9
5046273, 10
5046273, 11
5046273, 12
5046273, 13
5046273, 14
5111809, 15
5111809, 16
5111809, 17
5111809, 18
5111809, 19
5111809, 20
5111809, 21
5177345, 22
5177345, 23
5177345, 24
5177345, 25
5177345, 26
5177345, 27
5242881, 28
5242881, 29
5242881, 30

So i think Arduino does not update the time_us variable, but why not?

  1. Any inputs on how to solve/improve this?
  2. Is the binary approach i'm using here the most efficient for this application?
  3. Where is the bottleneck for such a high frequency data logging system? Arduino, Serial connection? or the receiving end?
  4. I also tried to plot the data in real time (in Matlab) for high frequencies, which didnt work. It froze my peripherals, probably due to a buffer overflow since my Mac couldnt keep up with the arduino (?). Is it even possible to plot this data in real-time? Any thoughts?

Thank you in advance!

If myPort.read() returns and 8-bit value then maybe this is a problem? I'm not that good with Java (the language of Processing).

myPort.read() * 256

It also looks like the byte order is getting reversed as well as some bytes getting zeroed:

Decimal Hexadecimal
5046273 = 00 4D 00 01
5111809 = 00 4E 00 01

5177345 = 00 4F 00 01

Yes, indeed there was something wrong with the way i handled the 32bit-long conversion. I changed that following on the Arduino...

        Serial.write((int)((value & 0xFF000000) >>24 ));
	Serial.write((int)((value & 0x00FF0000) >>16 ));
	Serial.write((int)((value & 0x0000FF00) >>8 ));
	Serial.write((int)((value & 0x000000FF) ));

and following on the receiving side:

      longValue = myPort.read() << 24;
      longValue |= (myPort.read() <<16);
      longValue |= (myPort.read() << 8);
      longValue |= (myPort.read());

Works like a charm! Thank you for the input!

To follow up to my question:

What is the fastest way to read out bytes from the serial and store them in a txt-file? (Some code samples in C, Java or anything else could help)
I tried reading out bytes 1-by-1, but my code can't keep up.

Would this be faster?: Read out serial byte and save it to a variable array. After Serial transmission is completed i save the array to a file or use it further , e.g for visualization.

Any book recommendations on serial communication and its optimization? I'm Using Processing (Java) right now, do you recommend any other language to speed up execution (C++,...)?