New fast data logging sketches

@fat16lib

You are right! I didn't realize it would still be using the SdFat library I already had installed, and wasn't all contained within your recent code. But makes sense.

I've got a Patriot 2GB card, so I'll figure out it's limits on the Mega & let you know.

So far, so good.

I've got it working with Hard SPI and no overruns at 40kHz (25usec being the default).

I still don't understand properly the CS thing, so I'm going to have to read that again and figure out what I'm doing. I think I'm confusing SS and CS, and am not to sure which is which.

I'm using this 2GB SD card (PEF2G133SD):
http://www.patriotmemory.com/products/detailp.jsp?prodline=4&catid=3&prodgroupid=16&id=311&type=6

SPI on avr processors is a bit confusing. This Atmel application note might help.

http://atmel.com/dyn/resources/prod_documents/doc2585.pdf

Hi fat16lib,

your new code is great, especially the truncate function is awesome. Using multiple buffers is an improvement as well (although i don't see the point in using 12 - if your logger doesn't manage to write fast enough when using 3 buffers, it's propably too slow and won't catch up after going through 12 ... or does the writing latency vary in such an extreme way?)

nonetheless, I encounter the same problem that I did with my previous code, because the CAN-Controller and the SD-Card controller share the mosi and miso line.
Im using this shield
http://www.skpang.co.uk/catalog/arduino-canbus-shield-with-usd-card-holder-p-706.html

And i have to set ChipSelectLow(SS_PIN) when I want to write to the sd and put it back to high (not selected) afterwards, because otherwise receiving the CAN-Messages also writes nonsense to the SD card

...
If i do not allow interrupts while writing, everything works fine, but i "lose" one or 2 messages during the write process, because the CAN controller can only buffer 2 messages.

Thus, calling cli() right before writing to the sd card works fine

cli();   //clear interrupts
digitalWrite(SS_PIN, LOW);  // SD on[/i]
// write block to SD
if (!sd.card()->writeData(block)) {
        error("writeData");
}     
digitalWrite(SS_PIN, HIGH);  // SD off
sei(); // enable interrupts[/i]

...

I then tried to enable interrupts at various positions in the writeData() function, and after encountering many write errors I found a position that produced no errors and a "valid" log file (formatwise):

/** SPI send block - only one call so force inline */
static inline __attribute__((always_inline))
  void spiSendBlock(uint8_t token, const uint8_t* buf) {
  SPDR = token;
  for (uint16_t i = 0; i < 512; i += 2) {

  // turn writing off and on again to allow interrupts	
  chipSelectHigh();
  sei();
  cli();
  chipSelectLow();
  // custom code end


    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i + 1];
  }
  while (!(SPSR & (1 << SPIF)));
}

...

But if I do so, I get an incomplete Log file! somehow a lot of data is lost and I have milliseconds of no data. Maybe because of a buffer not actually being written?! ... i don't receive any errors and the program moves on without a hint..

I'm giving up hope, after testing it with your impressive logger. Do you have any hint, as to what I could try?

I'm sorry if this seems like I'm asking you to debug my code ... but I've already invested 15+ hours into solving this interference problem and I'm considering giving up my goal of achieving a Real-Time Capable CAN logger...

edit:
do you know, whether calling chipSelectLow/High during the write process does stuff other than just turning the connection on/off ? does it flush a buffer or something like that? .... is this something to consider debugging?

keija,

First write latency for SD cards varies widely. Occasionally an SD card has many milliseconds of extra latency so extra buffers help.

Many SD cards require chip select to be held low for the entire 512 byte block write. You can't suddenly switch use of the hardware SPI bus during a block write. SD cards only allow CS for go high at certain times.

This is why I use software SPI for the ADC in the interrupt code.

You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.

fat16lib:
Many SD cards require chip select to be held low for the entire 512 byte block write. You can't suddenly switch use of the hardware SPI bus during a block write. SD cards only allow CS for go high at certain times.
This is why I use software SPI for the ADC in the interrupt code.
You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.

So my guesswork went into the right direction ... sadly ...
Thanks a million for your quick answer!

I'll reconsider my options and use your logging code for now (it's already extremely performant, even when disabling interrupts while writing!).

Keep up the great work fat16, your SD library already works tremendously well! Thank you!

Hello,

according to what I understand of this post it is not possible to have hardware SPI on sdcard and ADC. I've got a project working with arduinoMega analog inputs (multiplexer with datalogging at 10Hz). I added MCP3208 to the project, which is working nicely using hardware spi and spi.h. But if I start using the sdcard I've got wrong readings on the MCP3208, also sdfat is not working nicely as I don't get all files I should on the sdcard. Analog reading is done in a time interrupt to have proper frequency.
Do I need to unsolder the SPI pin of the MCP3208 to use soft spi or is there another option ?
Over question is why sdfat doesn't use spi.h , is it for performances ?

Thanks for any info.

Every SPI device has its own select pin and these should differ! You can only have one device selected simultaneously otherwise there will be interference. The most robust way is to wrap every SPI action with a device select/deselect.

some pseudocode

digitalWrite(ADC_SS, HIGH);
ADC.start();
digitalWrite(ADC_SS, LOW);

....

digitalWrite(ADC_SS, HIGH);
x = ADC.read();
digitalWrite(ADC_SS, LOW);

...

digitalWrite(SD_SS, HIGH);
SD.write (x)
digitalWrite(SD_SS, LOW);

But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x). And if you use sei() cli() around sd.write(x) you can drop samples in your adc. That was my assumption for my wrong readings. And I wanted to be sure there was no over option than using soft spi and bit banging for adc.
I switched to soft spi for adc and the sketch is working nicely. But I'm still interested to know if there is not another way to do it, perhaps by forcing SS pin of other spi device when getting into the adc interrupt.

But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x)

Then sd.write() should block IRQ's ?

... and the sketch is working nicely ...

Can you post your code ...

setup is calling this function to setup SPI for MCP3208:

void setupExternalADC(){ 
 pinMode(SELPIN, OUTPUT); 
 pinMode(DATAOUT, OUTPUT); 
 pinMode(DATAIN, INPUT); 
 pinMode(SPICLOCK, OUTPUT); 
 //disable device to start with 
 digitalWrite(SELPIN,HIGH); 
 digitalWrite(DATAOUT,LOW); 
 digitalWrite(SPICLOCK,LOW); 
}

there is an header file for pin configuration

#define SELPIN 25 //Selection Pin 
#define DATAOUT 24//MOSI 
#define DATAIN  26//MISO 
#define SPICLOCK  22//Clock

interrupt call this function (the MCP3208 from playground):

int read_adc(int channel){
  int adcvalue = 0;
  byte commandbits = B11000000; //command bits - start, mode, chn (3), dont care (3)

  //allow channel selection
  commandbits|=((channel-1)<<3);

  digitalWrite(SELPIN,LOW); //Select adc
  // setup bits to be written
  for (int i=7; i>=3; i--){
    digitalWrite(DATAOUT,commandbits&1<<i);
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);    
  }

  digitalWrite(SPICLOCK,HIGH);    //ignores 2 null bits
  digitalWrite(SPICLOCK,LOW);
  digitalWrite(SPICLOCK,HIGH);  
  digitalWrite(SPICLOCK,LOW);

  //read bits from adc
  for (int i=11; i>=0; i--){
    adcvalue+=digitalRead(DATAIN)<<i;
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);
  }
  digitalWrite(SELPIN, HIGH); //turn off device
  return adcvalue;
}

datalogging on the sd card is not done with interrupt but just link in this tutorial the http://www.arduino.cc/en/Tutorial/BlinkWithoutDelay

Hello,

thank you fat16lib for your time which you are putting to open-source code and helping others! I am a beginner with Arduino and I appreciate it a lot...

I am using your mcpLogger.pde with ADC MCP3201, but for some reason it save only first two lines to the file. First ,,Log Interval usec: 250" and second ,,4095" or ,,0". I was trying to put different voltage at the input or put it to the ground, but it lives by itself live and always save one of those values.
On the serial communication it writes only this:

Type any character to start.
FreeRam: 308
Log Interval: 250 usec
Logging to: FAST

and then it get stacked. Sometime it ends with FAST08. and sometime just with F. Here comes my question. Do you know, please, where can be the problem? I have Arduino Ethernet and saving it to uSD card. Firstly I thought, that it is because small ram, but this doesnt seem to be the problem. Every time I reset the Arduino board it writes one letter less.

I change the code this way:

// SD chip select pin
const uint8_t chipSelect = 4;

#else  // MEGA_TEST
const int8_t MCP_SAR_CLK_PIN   = 0;  // analog pin 0
const int8_t MCP_SAR_DOUT_PIN  = 1;  // analog pin 1
const int8_t MCP_SAR_DIN_PIN   = -1;
const int8_t MCP3201_CS_PIN    = 2;  // analog pin 2
#endif  // MEGA_TEST

I will appreciate any idea where I can look. I spend more than week trying to find out the problem.

Thank you in advance,
Maslo

masloo,

Which pins is the ADC connected to?

I set again the inputs like that:

const int8_t MCP_SAR_CLK_PIN   = A0;  // analog pin 0
const int8_t MCP_SAR_DOUT_PIN  = A1;  // analog pin 1
const int8_t MCP_SAR_DIN_PIN   = -1;
const int8_t MCP3201_CS_PIN    = A2;  // analog pin 2

and it acquire 2863 on the SD card, that is quite accurate to 3,3 volts which I am sending to the ADC.

In the log file is only:

Log Interval usec: 250
2863

very rare happened that it save 2 values. Anyways it sends over serial communication just this:

Type any character to start.
FreeRam: 308
Log Interval: 250 usec
Logging to: FAS

and then I have to reset the Arduino to make new measurement. It doesnt look like the problem is with PINs. For some reason the program get stacked after saving first value.

Here is the picture of my boards:

Can you please give me an advice how to get rid of this problem? Thank you in advance.

Yours sincerely,
Maslo

I changed the code for ADC MCP3204 one channel. The serial communication works, but it saves one value to the file and without waiting for user input it automatically stops.

Type any character to start.
FreeRam: 294
Log Interval: 250 usec
Logging to: FAST03.CSV
Type any character to stop.
Stopped!

Please, any help or advice is more than welcome.
Maslo

Arduino 1.0 changed the functionality of Serial.flush(). Serial.flush() no longer flushes input. In 1.0 Serial.flush() waits for output to finish.

Change this:

 PgmPrintln("Type any character to start.");
  while (!Serial.available());
  Serial.flush();

to this:

PgmPrintln("Type any character to start.");
  while (!Serial.available());
  delay(10);
  while (Serial.read() >= 0);

If your chip is a MCP3201 you should use the code for that chip.

Oh my... Now it works great! Thank you very, very much! No more sleepless nights XD At least for this problem...

Hopefully soon I will be able to help you too.

Take care fat16lib,
Maslo

Fat16Lib,

How are you?

I am a beginner at Arduino programming, I have mostly been using the Lab View Interface to run simple projects. Now I am trying to use a MCP3201 to send info to the Arduino using the SPI.

Can you help me out with a simple code for reading the SPI from the MCP3201?

I am using a Duemilanove 328p connecting SPI using 10 to 13 pins.

I really don't need any fancy speed or logging. I would like to debug a board that we have built for a PT100, but I haven't been able to connect it through SPI.

I am reading your code for the datalogger, but it gets me more confused.

If you can I will appreciate.
Thanks

Fefas

If you just want simple hardware SPI access, read section 6.1 of the datasheet http://ww1.microchip.com/downloads/en/DeviceDoc/21290e.pdf.

Hey fat16Lib,

Thanks, I read that. I am trying to write the program, but I don't have a lot of experience on programming, just started.
From what I understood I got to receive 2 bytes, shift the 2nd byte 1 bit to the right and shift the 1st byte 7 bits to the left. Is that correct?
Can you help with the code for that?

Thanks