need help speeding up serial communication or a different solution

Basically, writing code to sample 4 analog channels at at least 10kHz and for~5 seconds and then write it out to a file using processing. The only way to achieve 10kHz analog sampling was store the values in a buffer and print out in between sampling but with limited memory and 4 int arrays for the 4 analog channels, it can do only 0.6 seconds at a time.

this is okay if the printing sequences were only about 1ms long but the fastest i can print out the int arrays (~(6000 * 4) ints) is about 0.7 seconds. I've tried everything, using serial.write and then bitshifting the two bytes back into an int on the processing side, using the native port SerialUSB.print/SerialUSB.write , .. etc

here is my arduino code. Any creative solutions to get this working? thank you very much!

unsigned long timer;
unsigned long timerb = 0;
//but if not you may have to input each byte individually

const int latchPin = 12;

const int analogPin1 = A0;
const int analogPin2 = A1;
const int analogPin3 = A2;
const int analogPin4 = A3;


const int sampsize = 6000;   //6000;
int myVal1[sampsize];
int myVal2[sampsize];
int myVal3[sampsize];
int myVal4[sampsize];



void setup() {
  // put your setup code here, to run once:


  //delay(2000);//let processing get ready
  REG_ADC_MR = (REG_ADC_MR & 0xFFF0FFFF) | 0x00020000; // speeds up analog read from   http://www.djerickson.com/arduino/
  //now 4 analog reads down to about 20us
  
  Serial.begin(614400);

  pinMode(latchPin, OUTPUT);

  analogReadResolution(12);
  delayMicroseconds(100);//delay 100us before measuring analog
}


void loop() {
  // put your main code here, to run repeatedly:

    
    //timerb = micros();
    for(int i = 0; i<sampsize; i++){


      //timer = micros();    
      myVal1[i] = analogRead(analogPin1); //4 analog reads take about 8-10us
      myVal2[i] = analogRead(analogPin2);
      myVal3[i] = analogRead(analogPin3);
      myVal4[i] = analogRead(analogPin4);
      delayMicroseconds(89);

    }


    timer = micros();

for(int j = 0; j < sampsize; j++){

Serial.write((myVal1[j] >>8) & 0xFF);
Serial.write(myVal1[j] & 0xFF);
//Serial.write(35);

Serial.write((myVal2[j] >>8) & 0xFF);
Serial.write(myVal2[j] & 0xFF);
//Serial.write(35);

Serial.write((myVal3[j] >>8) & 0xFF);
Serial.write(myVal3[j] & 0xFF);
//Serial.write(35);

Serial.write((myVal4[j] >>8) & 0xFF);
Serial.write(myVal4[j] & 0xFF);
//Serial.write(35);
     
    }

  
    timerb = micros();



  while(1);
}

There is a version of Serial.write() that will send a complete array in one go. That is probably faster.

You may be able to operate an Uno or Mega at 1000000 baud.

If you are using a Leonardo or Micro (with an Atmega 32U4) the baud rate is ignored and the communication is very much faster.

...R

Oh sorry should've specified that Im using the arduino Due

I am not familiar with the Due, but my suggestions may still be relevant.

...R

I think the Serial.write() thing you are talking about, it has to be an array of bytes but I have to send an array of ints.

Ive tried every baud rate I could find, and every multiple, 614400 is the only one that works, in fact in most due posts ive seen, its not even supposed to work (according to some people) but it does

maybe this is similar to using the native port on the due where the baud rate is ignored and communication is supposed to be faster.. but its not for my case..

thank you anyways!

Does anybody else have any ideas..

uint32_t buf[1024];
SerialUSB.write((char*)buf,4096);

?

I can achieve 1MB/s+ (yes BYTE not bit) between Due and processing using Native port.

Regards,

Graham

Edit try something like this? But I would recommend the native port.... and also you know that 6000*89 microseconds is over half a second delay?

unsigned long timer;
unsigned long timerb = 0;
//but if not you may have to input each byte individually

const int latchPin = 12;
const int analogPin1 = A0;
const int analogPin2 = A1;
const int analogPin3 = A2;
const int analogPin4 = A3;


const int sampsize = 6000;   //6000;
uint16_t myVal1[sampsize];
uint16_t myVal2[sampsize];
uint16_t myVal3[sampsize];
uint16_t myVal4[sampsize];

void setup() {
  // put your setup code here, to run once:


  //delay(2000);//let processing get ready
  REG_ADC_MR = (REG_ADC_MR & 0xFFF0FFFF) | 0x00020000; // speeds up analog read from   http://www.djerickson.com/arduino/
  //now 4 analog reads down to about 20us

  Serial.begin(614400);

  pinMode(latchPin, OUTPUT);

  analogReadResolution(12);
  delayMicroseconds(100);//delay 100us before measuring analog
}

void loop() {
  //timerb = micros();
  for (int i = 0; i < sampsize; i++) {
    myVal1[i] = analogRead(analogPin1); //4 analog reads take about 8-10us
    myVal2[i] = analogRead(analogPin2);
    myVal3[i] = analogRead(analogPin3);
    myVal4[i] = analogRead(analogPin4);
    delayMicroseconds(89);
  }
  timer = micros();
  Serial.write((char*)myVal1, 12000);
  Serial.write((char*)myVal2, 12000);
  Serial.write((char*)myVal3, 12000);
  Serial.write((char*)myVal4, 12000);
  timerb = micros();
  while (1);
}

Processing code used:-

void setup() {
  String s;
  size(200, 200); // Dummy window for Serial
  println("Scanning serial ports...");
  for (String portname : Serial.list ()) {
    try {
      port = new Serial(this, portname, Serial_SPEED);
    } 
    catch (Exception e) { // Port in use, etc.
      continue;
    }
    print("Trying port " + portname + "...");
    delay(1500);
    if (((s = readLine()) != null) && s.contains("HELLO")) {
      //println("OK");
      break;
    } else {
      println();
      port.stop();
      port = null;
    }
  }
  int mill=millis();
  byte[] inBuffer = new byte[10];
  int z=0;
  while (z<47999) {
    if (port.available() > 0) {
      char l= port.readChar();
      z+=1;
    }
  }
  int end=millis();

  println("Done z="+z+" time="+(end-mill)+"ms");

  while (((s        = readLine())                != null)) {
    println("Time taken (uS)"+s);
  }
}

Just as a sync. You are right about 0.7seconds over serial at 614400, but Kudos for finding that weird and wonderful speed which works.......... measuring micros on the Arduino side puts the transfer at 729ms however............... Using native port for the same test takes

Scanning serial ports...
Trying port COM7...Done z=47999 time=5ms
Time taken (uS)6757

little over 6.5ms....... is that fast enough for you? 8) :wink: :stuck_out_tongue:

Graham

Very much appreciated ghlawrence!! will be trying this just as soon as I can (dont have access to arduino right now) And will let you know

Thank you!

philipjfry:
I think the Serial.write() thing you are talking about, it has to be an array of bytes but I have to send an array of ints.

You can use a UNION to have two different representations of the same data. It can be a very useful tool.

...R

@philipjfry,

Any success?

Regards,

Graham

Hi Graham,

Finally got chance to work on it, looking over your posts again and have some questions

uint32_t buf[1024];
SerialUSB.write((char*)buf,4096);

I get the errors
invalid conversion from 'char*' to 'const uint8_t*'
initializing argument 1 of 'virtual size_t Serial_::write(const uint8_t*, size_t)'

And I get the same errors when I try the example arduino code.. any ideas?

Also, when going from programming to native port, is there anything I need to change aside from moving cable to native port, changing the programming method in the IDE, changing every Serial.write to SerialUSB.write, and having this bit of code
SerialUSB.begin(9600);
while(!SerialUSB); ?

the baud rate in processing doesnt matter correct? you use Serial_SPEED but its not defined in the code. Because processing is not printing anything out when i switch over to the native port perhaps I am missing something

thank you!

oh woops haha just changed it to byte and it seems to be working

uint32_t buf[1024];
SerialUSB.write((byte*)buf,4096);

This will print the data one array at a time which is not what I wanted but is not a big deal IF i can see native port printing this much faster! still trying to get native port working..

The processing code was incomplete, it was just to show you the functional part, Serial_SPEED is defined as a global before setup.....

Try replacing ((char*)buf.... with ((uint8_t*)buf....

It is probably a IDE version difference, but the concept is correct.

At the end of Setup() on the Arduino, you need a SerialUSB.println("HELLO"); It is what processing is looking for before it starts the transfer....

Good luck!

Regards,

Graham

philipjfry:
This will print the data one array at a time which is not what I wanted but is not a big deal IF i can see native port printing this much faster! still trying to get native port working..

To gain the maximum benefit using the native port, you do need to send a block of data.............. the only way around it if you REALLY wish to interleave the 4 arrays........... is say build a dummy array, like

uint16_t tempbuf[256] = array1[0], array2[0], array3[0], array4[0], array1[1], array2[1], array3[1],array4[1] etc....
SerialUSB.write((byte*)tempbuf,512);

Regards,

Graham

PS, since your A/D resolution is only 12 bit, use uint16_t arrays.....

Hi,

tried sending 4 arrays separately

SerialUSB.write((byte*)myVal1, 12000);
  SerialUSB.write((byte*)myVal2, 12000);
  SerialUSB.write((byte*)myVal3, 12000);
  SerialUSB.write((byte*)myVal4, 12000);

and tried interweaving and sending one big array

int index = 0;
    for(int p = 0; p<sampsize*4-1; p = p+4){
    myVals[p] = myVal1[index];
    myVals[p+1] = myVal2[index];
    myVals[p+2] = myVal2[index];
    myVals[p+3] = myVal2[index];
    index++;
    }
    SerialUSB.write((byte*)myVals,48000);

turns out interweaving is a little bit faster even including the for loop but its still at 0.3 seconds.

I do do the processing side a little different but that shouldnt affect arduino side speed correct?

still wondering how you got around 6 ms..

Thank you

Ok, all I can do is let you try the code I used.

Processing:-

int     Serial_SPEED=614400;

/******************************************************************************************************************
 Nothing to see here!!!!!!!
 *****************************************************************************************************************/

import processing.serial.*;
Serial  port = null;
int     capacity;
int     mysize;
int     mytime, mytimeE;
int     tt, tt1, tt2;
int     min;
int     sec;
boolean done = false;
// Wait for line from serial port, with timeout
String readLine() {
  String s;
  int    start = millis();
  do {
    s = port.readStringUntil('\n');
  } while ( (s == null) && ((millis() - start) < 3000));
  return s;
}

void setup() {
  String s;
  size(200, 200); // Dummy window for Serial
  println("Scanning serial ports...");
  for (String portname : Serial.list ()) {
    try {
      port = new Serial(this, portname, Serial_SPEED);
    } 
    catch (Exception e) { // Port in use, etc.
      continue;
    }
    print("Trying port " + portname + "...");
    delay(1500);
    if (((s = readLine()) != null) && s.contains("HELLO")) {
      println("OK");
      break;
    } else {
      println();
      port.stop();
      port = null;
    }
  }
  for (int y=0; y<10; y++) {
    while (((s = readLine()) == null) || !s.contains("HELLO")) ;
    int mill=millis();
    int z=0;
    while (z<47999) {
      if (port.available() > 0) {
        char l= port.readChar();
        z+=1;
      }
    }
    int end=millis();
    println("Done z="+z+" time="+(end-mill)+"ms");
    s        = readLine();
    println("Time taken (uS)"+s);
  }
  exit();
}

Arduino:-

#define serialport SerialUSB
#define serialspeed 614400

unsigned long timer;
unsigned long timerb = 0;
const int latchPin = 12;
const int analogPin1 = A0;
const int analogPin2 = A1;
const int analogPin3 = A2;
const int analogPin4 = A3;
const int sampsize = 6000;   //6000;

uint16_t myVal[sampsize * 4];

void setup() {
  //delay(2000);//let processing get ready
  REG_ADC_MR = (REG_ADC_MR & 0xFFF0FFFF) | 0x00020000; // speeds up analog read from   http://www.djerickson.com/arduino/
  //now 4 analog reads down to about 20us

  serialport.begin(serialspeed);
  while (!serialport); // <- wait for port to open
  pinMode(latchPin, OUTPUT);
  analogReadResolution(12);
  delayMicroseconds(100);//delay 100us before measuring analog
  serialport.println("HELLO");
}

void loop() {
  for (int i = 0; i < sampsize * 4; i += 4) {
    myVal[i] = analogRead(analogPin1); //4 analog reads take about 8-10us
    myVal[i + 1] = analogRead(analogPin2);
    myVal[i + 2] = analogRead(analogPin3);
    myVal[i + 3] = analogRead(analogPin4);
    delayMicroseconds(89);
  }
  serialport.println("HELLO");
  timer = micros();
  serialport.write((char*)myVal, sampsize * 2 * 4);
  timerb = micros();
  serialport.println(timerb - timer);
  //while (1);
}

Results:-

Done z=47999 time=5ms
Time taken (uS)6650

Done z=47999 time=4ms
Time taken (uS)6118

Done z=47999 time=6ms
Time taken (uS)6110

Done z=47999 time=6ms
Time taken (uS)5964

Done z=47999 time=6ms
Time taken (uS)5964

Done z=47999 time=6ms
Time taken (uS)6010

Done z=47999 time=6ms
Time taken (uS)6158

Done z=47999 time=6ms
Time taken (uS)6019

Done z=47999 time=6ms
Time taken (uS)6170

Done z=47999 time=6ms
Time taken (uS)5958

Regards,

Graham

Hi Graham, That is your arduino code when using the programming port! Do you have your code for when you used the Native port?

I.E. when you used SerialUSB.prints instead of Serial.prints

thank you

#define serialport SerialUSB
  serialport.println("HELLO");

Last time I checked, SerialUSB is the native port!!

#define serialport Serial
  serialport.println("HELLO");

Would be the programming port no?

Regards,

Graham

Oh, I have been playing, you can change a variable to decide how many chunks of 48000 bytes to send now, and get statistics at the end of the run, here is a summary of 5 chunks of 48000 bytes :wink: :-

Total Time taken to transfer 240000 bytes =30590(us)
Bytes per second =7845701.5
MB per second =7.482244

And 50 chunks of 48000 bytes :-

Total Time taken to transfer 2400000 bytes =302282(us)
Bytes per second =7939606.0
MB per second =7.5717983

No doubt this will upset you even more, this is using my 'secret' technique which everyone tells me 'makes no difference'! Judge for yourself... :wink: :smiley: :stuck_out_tongue:
5 chunks :-

Total Time taken to transfer 240000 bytes =26394(us)
Bytes per second =9092976.0
MB per second =8.671738

50 chunks :-

Total Time taken to transfer 2400000 bytes =264328(us)
Bytes per second =9079628.0
MB per second =8.659008

Regards,

Graham

I am running out of ideas now and still nowhere near 6 ms.. What is your secret technique!!