Go Down

Topic: New fast data logging sketches (Read 12837 times) previous topic - next topic

fat16lib

SPI on avr processors is a bit confusing.  This Atmel application note might help.

http://atmel.com/dyn/resources/prod_documents/doc2585.pdf

keija

#16
Jun 28, 2011, 01:57 pm Last Edit: Jun 28, 2011, 02:13 pm by keija Reason: 1
Hi fat16lib,

your new code is great, especially the truncate function is awesome. Using multiple buffers is an improvement as well (although i don't see the point in using 12 - if your logger doesn't manage to write fast enough when using 3 buffers, it's propably too slow and won't catch up after going through 12 ... or does the writing latency vary in such an extreme way?)

nonetheless, I encounter the same problem that I did with my previous code, because the CAN-Controller and the SD-Card controller share the mosi and miso line.
Im using this shield
http://www.skpang.co.uk/catalog/arduino-canbus-shield-with-usd-card-holder-p-706.html
http://www.sparkfun.com/datasheets/DevTools/Arduino/canbus_shield-v12.pdf

And i have to set ChipSelectLow(SS_PIN) when I want to write to the sd and put it back to high (not selected) afterwards, because otherwise receiving the CAN-Messages also writes nonsense to the SD card

...
If i do not allow interrupts while writing, everything works fine, but i "lose" one or 2 messages during the write process, because the CAN controller can only buffer 2 messages.

Thus, calling cli() right before writing to the sd card works fine
Code: [Select]
     
cli();   //clear interrupts
digitalWrite(SS_PIN, LOW);  // SD on[/i]
// write block to SD
if (!sd.card()->writeData(block)) {
        error("writeData");
}     
digitalWrite(SS_PIN, HIGH);  // SD off
sei(); // enable interrupts[/i]



...

I then tried to enable interrupts at various positions in the writeData() function, and after encountering many write errors I found a position that produced no errors and a "valid" log file (formatwise):

Code: [Select]

/** SPI send block - only one call so force inline */
static inline __attribute__((always_inline))
  void spiSendBlock(uint8_t token, const uint8_t* buf) {
  SPDR = token;
  for (uint16_t i = 0; i < 512; i += 2) {

  // turn writing off and on again to allow interrupts
  chipSelectHigh();
  sei();
  cli();
  chipSelectLow();
  // custom code end


    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i + 1];
  }
  while (!(SPSR & (1 << SPIF)));
}



...

But if I do so, I get an incomplete Log file! somehow a lot of data is lost and I have milliseconds of no data. Maybe because of a buffer not actually being written?! ... i don't receive any errors and the program moves on without a hint..

I'm giving up hope, after testing it with your impressive logger.   Do you have any hint, as to what I could try?


I'm sorry if this seems like I'm asking you to debug my code ... but I've already invested 15+ hours into solving this interference problem and I'm considering giving up my goal of achieving a Real-Time Capable CAN logger...

edit:
do you know, whether calling    chipSelectLow/High during the write process does stuff other than just turning the connection on/off ? does it flush a buffer or something like that? .... is this something to consider debugging?

fat16lib

keija,

First write latency for SD cards varies widely.  Occasionally an SD card has many milliseconds of extra latency so extra buffers help.

Many SD cards require chip select to be held low for the entire 512 byte block write.  You can't suddenly switch use of the hardware SPI bus during a block write.  SD cards only allow CS for go high at certain times.

This is why I use software SPI for the ADC in the interrupt code.

You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.

keija


Many SD cards require chip select to be held low for the entire 512 byte block write.  You can't suddenly switch use of the hardware SPI bus during a block write.  SD cards only allow CS for go high at certain times.
This is why I use software SPI for the ADC in the interrupt code.
You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.


So my guesswork went into the right direction ... sadly ...
Thanks a million for your quick answer!

I'll reconsider my options and use your logging code for now (it's already extremely performant, even when disabling interrupts while writing!).

Keep up the great work fat16, your SD library already works tremendously well! Thank you!

ebw44

Hello,

    according to what I understand of this post it is not possible to have hardware SPI on sdcard and ADC. I've got a project working with arduinoMega analog inputs (multiplexer with datalogging at 10Hz). I added MCP3208 to the project, which is working nicely using hardware spi and spi.h. But if I start using the sdcard I've got wrong readings on the MCP3208, also sdfat is not working nicely as I don't get all files I should on the sdcard. Analog reading is done in a time interrupt to have proper frequency.
    Do I need to unsolder the SPI pin of the MCP3208 to use soft spi or is there another option ?
    Over question is why sdfat doesn't use spi.h , is it for performances ?

Thanks for any info.

robtillaart

Every SPI device has its own select pin and these should differ! You can only have one device selected simultaneously otherwise there will be interference. The most robust way is to wrap every SPI action with a device select/deselect.

some pseudocode

Code: [Select]

digitalWrite(ADC_SS, HIGH);
ADC.start();
digitalWrite(ADC_SS, LOW);

....

digitalWrite(ADC_SS, HIGH);
x = ADC.read();
digitalWrite(ADC_SS, LOW);

...

digitalWrite(SD_SS, HIGH);
SD.write (x)
digitalWrite(SD_SS, LOW);
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

ebw44

But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x). And if you use sei() cli() around sd.write(x) you can drop samples in your adc. That was my assumption for my wrong readings. And I wanted to be sure there was no over option than using soft spi and bit banging for adc.
I switched to soft spi for adc and the sketch is working nicely. But I'm still interested to know if there is not another way to do it, perhaps by forcing SS pin of other spi device when getting into the adc interrupt.


robtillaart

Quote
But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x)

Then sd.write() should block IRQ's ?

Quote
... and the sketch is working nicely ...

Can you post your code ...
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

ebw44

setup is calling this function to setup SPI for MCP3208:
Code: [Select]

void setupExternalADC(){
pinMode(SELPIN, OUTPUT);
pinMode(DATAOUT, OUTPUT);
pinMode(DATAIN, INPUT);
pinMode(SPICLOCK, OUTPUT);
//disable device to start with
digitalWrite(SELPIN,HIGH);
digitalWrite(DATAOUT,LOW);
digitalWrite(SPICLOCK,LOW);
}

there is an header file for pin configuration
Code: [Select]

#define SELPIN 25 //Selection Pin
#define DATAOUT 24//MOSI
#define DATAIN  26//MISO
#define SPICLOCK  22//Clock


interrupt call this function (the MCP3208 from playground):
Code: [Select]

int read_adc(int channel){
  int adcvalue = 0;
  byte commandbits = B11000000; //command bits - start, mode, chn (3), dont care (3)

  //allow channel selection
  commandbits|=((channel-1)<<3);

  digitalWrite(SELPIN,LOW); //Select adc
  // setup bits to be written
  for (int i=7; i>=3; i--){
    digitalWrite(DATAOUT,commandbits&1<<i);
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);   
  }

  digitalWrite(SPICLOCK,HIGH);    //ignores 2 null bits
  digitalWrite(SPICLOCK,LOW);
  digitalWrite(SPICLOCK,HIGH); 
  digitalWrite(SPICLOCK,LOW);

  //read bits from adc
  for (int i=11; i>=0; i--){
    adcvalue+=digitalRead(DATAIN)<<i;
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);
  }
  digitalWrite(SELPIN, HIGH); //turn off device
  return adcvalue;
}


datalogging on the sd card is not done with interrupt but just link in this tutorial the http://www.arduino.cc/en/Tutorial/BlinkWithoutDelay

maslo

Hello,

thank you fat16lib for your time which you are putting to open-source code and helping others! I am a beginner with Arduino and I appreciate it a lot...

I am using your mcpLogger.pde with ADC MCP3201, but for some reason it save only first two lines to the file. First ,,Log Interval usec: 250" and second ,,4095" or ,,0". I was trying to put different voltage at the input or put it to the ground, but it lives by itself live and always save one of those values.
On the serial communication it writes only this:

Quote
Type any character to start.
FreeRam: 308
Log Interval: 250 usec
Logging to: FAST


and then it get stacked. Sometime it ends with FAST08. and sometime just with F. Here comes my question. Do you know, please, where can be the problem? I have Arduino Ethernet and saving it to uSD card. Firstly I thought, that it is because small ram, but this doesnt seem to be the problem. Every time I reset the Arduino board it writes one letter less.

I change the code this way:

Code: [Select]

// SD chip select pin
const uint8_t chipSelect = 4;

#else  // MEGA_TEST
const int8_t MCP_SAR_CLK_PIN   = 0;  // analog pin 0
const int8_t MCP_SAR_DOUT_PIN  = 1;  // analog pin 1
const int8_t MCP_SAR_DIN_PIN   = -1;
const int8_t MCP3201_CS_PIN    = 2;  // analog pin 2
#endif  // MEGA_TEST


I will appreciate any idea where I can look. I spend more than week trying to find out the problem.

Thank you in advance,
Maslo

fat16lib

masloo,

Which pins is the ADC connected to?

maslo

I set again the inputs like that:

Code: [Select]

const int8_t MCP_SAR_CLK_PIN   = A0;  // analog pin 0
const int8_t MCP_SAR_DOUT_PIN  = A1;  // analog pin 1
const int8_t MCP_SAR_DIN_PIN   = -1;
const int8_t MCP3201_CS_PIN    = A2;  // analog pin 2


and it acquire 2863 on the SD card, that is quite accurate to 3,3 volts which I am sending to the ADC.

In the log file is only:

Quote

Log Interval usec: 250
2863


very rare happened that it save 2 values. Anyways it sends over serial communication just this:

Quote

Type any character to start.
FreeRam: 308
Log Interval: 250 usec
Logging to: FAS


and then I have to reset the Arduino to make new measurement. It doesnt look like the problem is with PINs. For some reason the program get stacked after saving first value.

Here is the picture of my boards:


http://imageshack.us/photo/my-images/845/img20120416112356.jpg/

Can you please give me an advice how to get rid of this problem? Thank you in advance.

Yours sincerely,
Maslo

maslo

I changed the code for ADC MCP3204 one channel. The serial communication works, but it saves one value to the file and without waiting for user input it automatically stops.

Quote

Type any character to start.
FreeRam: 294
Log Interval: 250 usec
Logging to: FAST03.CSV
Type any character to stop.
Stopped!


Please, any help or advice is more than welcome.
Maslo

fat16lib

Arduino 1.0 changed the functionality of Serial.flush().  Serial.flush() no longer flushes input.  In 1.0 Serial.flush() waits for output to finish.

Change this:
Code: [Select]

PgmPrintln("Type any character to start.");
  while (!Serial.available());
  Serial.flush();

to this:
Code: [Select]

PgmPrintln("Type any character to start.");
  while (!Serial.available());
  delay(10);
  while (Serial.read() >= 0);

If your chip is a MCP3201 you should use the code for that chip.

maslo

Oh my... Now it works great! Thank you very, very much! No more sleepless nights XD At least for this problem...

Hopefully soon I will be able to help you too.

Take care fat16lib,
Maslo


Go Up