Dealing with a lot of data in Arduino...?

Hello all,
Like many that post here, I am new to the Arduino platform and programming micro-controllers in general (I’ve really only done numerical analysis in MATLAB). I’m working on a project that requires me to collect a lot of data, and then subsequently perform analysis in MATLAB. I have a simple (?) but pretty bad problem with my program though: I’m unable to actually collect all the data I need at my desired sampling rate.

In the following code, I’m controlling the ADC (I’m not using the analogRead() function because I found that it was too slow) to make measurements from 3 different sensors. However, in my final program it might end up being more than 3. I only made this code for this post, so that it is easy for everyone to absorb the problem. I am making a measurement every 5 milliseconds (200 Hz sampling rate). What I want to do here is somehow the keep all the data until 10 seconds pass, and then print it to the serial board. It has to print to the serial board because that’s how I’m transferring everything into MATLAB. I asked some friends of mine and a few said that this should be possible, and suggested I post on the Arduino forum for help (I’ve tried to do research on this topic, but surprisingly, haven’t found a lot of people that have encountered a similar issue). Note that I have found that reading and printing as the program goes slows it down for my purposes, which is why I must only save or somehow keep the data, and then print everything to the serial board at the end (after 10 seconds).

Here is the code:

volatile int aval = 0;      // analog value
unsigned long timerInterval= 5; //basically the sampling rate, i.e., make a measurement every 5 milliseconds (sampling rate 200 Hz)
unsigned long previousTimerMillis = 0;
unsigned long currentMillis = 0; 

void setup() { 
  DIDR0 = 0x3F;            // digital inputs disabled
  ADMUX = 0x40;            // measuring on ADC, use the internal 1.1 reference
  ADCSRA = 0xAC;           // AD-converter on, interrupt enabled, prescaler = 16
  ADCSRB = 0x40;           // AD channels MUX on, free running mode
  bitWrite(ADCSRA, 6, 1);  // Start the conversion by setting bit 6 (=ADSC) in ADCSRA
  sei();                   // set interrupt flag
  
  Serial.begin(9600);
}

  void loop() {
  currentMillis = millis();
  data();
}

void data(){
  if (currentMillis - previousTimerMillis >= timerInterval ){
    ADC_select_pin(0);
    ADC_select_pin(1);
    ADC_select_pin(2);
    previousTimerMillis += timerInterval;
 
  }
}
  

void ADC_select_pin(int pin){
 if (pin == 0){
   ADMUX= 0x40; //ADC1, pin 0
   
   Serial.println(aval);
 } 
 if (pin== 1){
   ADMUX= 0x41;// ADC2, pin1
   
   Serial.println(aval);
 }
 if (pin== 2){
   ADMUX= 0x42; //ADC 3, pin 2
   
   Serial.println(aval); 
 }
}

/*** Interrupt routine ADC ready ***/
ISR(ADC_vect) {

  aval = ADCL;        // store lower byte ADC
  aval += ADCH << 8;  // store higher bytes ADC

}

What I originally had done was save each sample into an array (so, 3 arrays for 3 different things I was measuring), and then after 10 seconds had past, I wanted to print the contents of each array to the serial board. While the logic seemed right, I think I had run out of memory (SRAM?). Is there another way to do this? My friend told me to try and keep everything into the buffer, and then print at the end. I’m not sure what this means or how to do this, however. Is my problem solvable, or am I looking at this the wrong way? I would really appreciate any help I can get.

Edit: Perhaps I should have further noted that while this code is with using an Arduino UNO, I also have a MEGA in my possession if that helps.

jalperto34: I am making a measurement every 5 milliseconds (200 Hz sampling rate). What I want to do here is somehow the keep all the data until 10 seconds pass, and then print it to the serial board.

Bad idea.

Why not send each sample after you have obtained the value?

Either sending as a single sample. Or sending as a "data record" of three or more values from different ADCs.

Why not? That would be cheap easy.

There is no way you should be running out of memory, or processor bandwidth, on such a tiny, simple program. You should be able to handle more channels and FAR more data with ease. More likely, perhaps you're not handling the interrupt correctly, and spending all your time in the interrupt handler.

I've never looked at the A/D hardware in any detail, but don't you need to do something in the ISR to clear the A/D interrupt, and start a new conversion? Or does simply reading the data registers do that? Is the interrupt configured for rising edge operation, rather than high level?

Also I see nowhere that you actually USE the data read by the ISR, nor do you have any synchronization between the interrupt handler writing new samples, and loop() reading those samples, so you have no way of knowing when a new sample is ready. You could be missing some samples, and/or reading the same samples multiple times, and you have no way of knowing. You should at least set a boolean flag in the ISR to indicate a new sample is available, then clear it when loop() picks up the new value. If you reach the ISR when the flag is set, simple discard the new sample, and start a new conversion. Alternatively, surround the read of aval in loop() with cli()/sti, so the read can't be interrupted. Otherwise, loop() may sometimes read the low byte of one sample, and the high byte of the previous sample, which will give you garbage data.

Regards, Ray L.

10 seconds, 200 x 3 samples per second, that's 6000 readings.

You need more storage than the Uno has (2k) - you could add a serial SRAM to the system for storage, upgrade to the Mega (8k - might be a squeeze), Due (lots more RAM, but 3V3 though), or perhaps write data to an SDcard. Writing to serial at a decent baudrate will do it too.

MarkT: 10 seconds, 200 x 3 samples per second, that's 6000 readings.

You need more storage than the Uno has (2k) - you could add a serial SRAM to the system for storage, upgrade to the Mega (8k - might be a squeeze), Due (lots more RAM, but 3V3 though), or perhaps write data to an SDcard. Writing to serial at a decent baudrate will do it too.

Could you clarify what you mean by the bolded? Also, I don't think its a good idea for me to invest time in getting it to work with the MEGA, because as you said it may be a squeeze. Furthermore, I will probably be collecting more data in the future, so I'd like to solve it more elegantly.

RayLivingston: There is no way you should be running out of memory, or processor bandwidth, on such a tiny, simple program. You should be able to handle more channels and FAR more data with ease. More likely, perhaps you're not handling the interrupt correctly, and spending all your time in the interrupt handler.

I've never looked at the A/D hardware in any detail, but don't you need to do something in the ISR to clear the A/D interrupt, and start a new conversion? Or does simply reading the data registers do that? Is the interrupt configured for rising edge operation, rather than high level?

Also I see nowhere that you actually USE the data read by the ISR, nor do you have any synchronization between the interrupt handler writing new samples, and loop() reading those samples, so you have no way of knowing when a new sample is ready. You could be missing some samples, and/or reading the same samples multiple times, and you have no way of knowing. You should at least set a boolean flag in the ISR to indicate a new sample is available, then clear it when loop() picks up the new value. If you reach the ISR when the flag is set, simple discard the new sample, and start a new conversion. Alternatively, surround the read of aval in loop() with cli()/sti, so the read can't be interrupted. Otherwise, loop() may sometimes read the low byte of one sample, and the high byte of the previous sample, which will give you garbage data.

Regards, Ray L.

Hi Ray, Hmmm, does the bolded conflict with what was said by the other two posters I've quoted? Also, I'm having trouble understanding your last paragraph. How can it be possible for me to read the same measurements if I wait 5 ms before reading the next? Or is ADC really that slow? Also, with the data I've already been collecting, I looked at it in MATLAB and it seemed correct. However, I would like to implement the safety measures you're talking about to see if they do in fact have any impact on the data I'm reading (although I'm not sure if it solves my problem at hand). Do you have a resource or example you could show me?

jurs: Bad idea.

Why not send each sample after you have obtained the value?

Either sending as a single sample. Or sending as a "data record" of three or more values from different ADCs.

Why not? That would be cheap easy.

I'm not sure what you mean by the bolded? How is that possible? The reason I can't print data as I go is because in my final program I'm simultaneously moving a servo. I found that if I print the data while the servo is moving, it slows everything down (eg, the servo speed itself was very slow). But if I didnt print while the servo moved, it was fine. If you like, I can post the code of my final program, which all it does is it moves a servo back forth while collecting data through the ADC, and then when the servo changes direction, I print all the data to the serial board. However like I said it doesnt work due to the lack of memory.

What I did in another projects is use 2 buffers for one input. One buffer is being filled while the other is dumping on an SD card and vice versa. Inputs came from an interrupt input (light pulse). This was used in a benificial program for a soap factory in Malawi. Why a buffer? Single writes take too long. Dumping say 500 bytes at a time is a lot more efficient than one 32bit number. I can post the code here if anyone is interested, but there is a lot of other stuff in the code as well.

jalperto34: I'm not sure what you mean by the bolded? How is that possible?

This prints one data record ("one line" of data) while it reads the data:

  Serial.print(analogRead(A0));
  Serial.print(',');
  Serial.print(analogRead(A1));
  Serial.print(',');
  Serial.print(analogRead(A2));
  Serial.println();

As each value can be up to 4 digits and you are sending two commas and CR/LF at the end of line, you need to send 3*4+2+2=16 chars in 5 millisonds = 200*16 = 3200 chars per second (max.). This means that the minimum required baud rate would be 32000, so you could send with 57600 or 115200 baud (or more).

jalperto34: The reason I can't print data as I go is because in my final program I'm simultaneously moving a servo. I found that if I print the data while the servo is moving, it slows everything down (eg, the servo speed itself was very slow).

Most likely: Bad programming logic, possibly using "delay()" function in your code.

jurs: This prints one data record ("one line" of data) while it reads the data:

  Serial.print(analogRead(A0));
  Serial.print(',');
  Serial.print(analogRead(A1));
  Serial.print(',');
  Serial.print(analogRead(A2));
  Serial.println();

As each value can be up to 4 digits and you are sending two commas and CR/LF at the end of line, you need to send 3*4+2+2=16 chars in 5 millisonds = 200*16 = 3200 chars per second (max.). This means that the minimum required baud rate would be 32000, so you could send with 57600 or 115200 baud (or more).

Most likely: Bad programming logic, possibly using "delay()" function in your code.

Oh that's good to know! So you're saying, theoretically, that printing data to the serial board while simultaneously moving my servo shouldn't affect the speed of the servo? If so then I must indeed be making an error (your calculation is useful). However I can tell you for sure that I'm not using the delay function in a context that would cause this. Let me try to recreate that problem, and I'll get back to you sometime tomorrow (it'll take some time, I also don't have the servo with me). I'll then post my code for your great analysis and critique :-)

But seriously. Thanks for your help so far. Hopefully I can solve this.

nicoverduin: What I did in another projects is use 2 buffers for one input. One buffer is being filled while the other is dumping on an SD card and vice versa. Inputs came from an interrupt input (light pulse). This was used in a benificial program for a soap factory in Malawi. Why a buffer? Single writes take too long. Dumping say 500 bytes at a time is a lot more efficient than one 32bit number. I can post the code here if anyone is interested, but there is a lot of other stuff in the code as well.

First of all, the bolded is freaking awesome. Now, the problem is that I can't even picture what you're saying. I have no idea how to interact with "buffers" the way you're describing them in Arduino (or at all, haha). So while your description makes it sound like it would work for my application, I can't be 100% sure because I'm not sure how these things are usually implemented. For example, does that technique only work if you're reading in data from just 1 sensor? Basically, can you have multiple buffers like that so you can separate data from different sources? I'm not sure how these generally work. Do you think you can post the code, and then maybe I can see which part does the data acquisition/processing you're describing? I've never written or read data from an SD card either, is that difficult for a beginner like me? I appreciate your unique, alternative idea though. Really goes to show that when it comes to programming, there's probably always more than one solution.

You can do asynchronous analog reads.

http://www.gammon.com.au/forum/?id=11488&reply=5#reply5

An analog read (ADC conversion) normally takes 104 µS, and normally blocks. However by doing it asynchronously you can be sending the previous result while you get the next one. Also you can reduce the resolution of the ADC conversion which speeds it up.

  Serial.begin(9600);

If you want speed, this definitely won't help. Try 115200 baud rate.

I have attached the code (added som DEBUG) stuff. I also created a website (doxygen) here:Malawi project

Malawi_001.ino (12.3 KB)