New fast data logging sketches

Finally getting some time to work on this.

I'm debating on whether to use your MCP3208 code or use what I was working with.

I'm posting the relevant code to what I was using below. It is simpler for me to understand, but maybe you also made some considerable speed improvements in your code.

#include <SPI.h>

  void setup() {
    pinMode(53,OUTPUT);
    digitalWrite(53,HIGH);
    SPI.begin();
    SPI.setClockDivider(SPI_CLOCK_DIV8);
  }

  void loop() {
    readADC(0);
  }

uint16_t readADC(int channel) {
  uint16_t output;
  //Channel must be from 0 to 7
  //Shift bits to match datasheet for MCP3208
  byte commandMSB = B00000110;
  uint16_t commandBytes = (uint16_t) (commandMSB<<8|channel<<6);
  
  //Select ADC
  digitalWrite(53, LOW);
  //send start bit and bit to specify single or differential mode (single mode chosen here)
  SPI.transfer((commandBytes>>8) & 0xff);

  byte msb = SPI.transfer((byte)commandBytes & 0xff) & B00001111;
  byte lsb = SPI.transfer(0x00);
  
  digitalWrite(53,HIGH);

  // cast before shiting the byte
  output = ((uint16_t) msb) <<8 | lsb;
  return output;
}

Ok, I'm starting to understand your code a bit better.

You are using SOFT_SPI for both the SDfat & the MCP320x.

I'm surprised you can do both, so the hardware SPI & the hardware serial port could still be used for something else, which is quite a feat.

I'll go ahead and try to get it tested with your design first, but I'd like to know the best place to define the SPI pins for hardware SPI to be used with the ADC.

I also realize that if I'd use my prior method of communicating with the ADC, I would have to include SPI.h which would create a bigger footprint probably.

oscarcar,

If your code works you should use it.

I used an oscilloscope to develop the MCM320X code so modifying/debugging it is easier with a scope.

I don't want to share the hardware SPI bus between SdFat and interrupt code that reads the ADC. I can just read the ADC in the interrupt service routine without thinking about SdFat and the hardware SPI bus.

My code is faster but that doesn't matter if your code is fast enough.

I designed the MCP320X code to achieve maximum speed for logging values read in an ISR and written in loop(). I could never log 40,000 data points per second from a MCP3201 in binaryLogger.pde without that kind of optimization.

I use hardware SPI for SdFat and soft SPI for the ADC. This is the fastest since SdFat does more I/O and I use 8 MHz for the SD.

fat16lib:
I use hardware SPI for SdFat and soft SPI for the ADC. This is the fastest since SdFat does more I/O and I use 8 MHz for the SD.

But with the Mega, I thought you did software SPI for that, so that the shields could be used without altering.

Will I need to re-route the pins to the Mega SPI pins? I tried doing that with your last codebase, and I never got it working.

Yes I use hardware SPI on the Mega.

SdFat has supported hardware SPI on the Mega since it was released. I added the option for software SPI in December 2009.

Do not edit SdFat files. Just run wires from the SD shield pins to the Mega hardware SPI pins.

To use a shield for a 328 Arduino on the Mega run short wires like this:

Mega Pin <--> Shield Pin
50 <--> 12 MISO
51 <--> 11 MOSI
52 <--> 13 SCK

If you use the default SS pin for SD chip select, connect these pins.
53 <--> 10 SS

Don't run a wire for chip select if you specify the chip select pin number in the init() call like this.

sd.init(SPI_FULL_SPEED, 8);

The init() call will override the default and ,in this case, pin 8 will be use for SD chip select.

When you choose software SPI pins for the ADC be sure to look at MegaPins.txt.

Hi fat16,

I'm loving your progress and effort! I can't get my interrupt driven dataWriter to work and will try out your solution soon!
Is a mega necessary to reach those write speeds? or does a uno suffice?

keep up the good work!
Greetings, Keija

@fat16lib

Thanks for the tips, but I'm still a bit confused.

Last night I tried it with the shield on the Mega, and it seemed to run fine (except for some dropouts, and the fact that I wasn't using the MCP3208 calls yet).

Since it was wired to 10-13 on the Mega (and working), doesn't that mean it had to be using SOFT_SPI?

And if I routed it to the 51-53 pins, wouldn't it still be trying to use SOFT_SPI, since I wouldn't be changing anything in the code?

I was under the assumption that you had some #ifdefine segments that if it's a Mega, you use SOFT_SPI. I thought that's what I was having a hard time working around before.

Thanks for the help. I'm almost there.

To use software SPI you must have edited SdFatConfig.h and changed MEGA_SOFT_SPI.

Go to about line 85 and change it back to the default:

#define MEGA_SOFT_SPI 0

The SPI will be much faster.

Unless you have a very high quality SD, you will have dropped data. Most SD cards have an occasional write latency of over 100 milliseconds.

Only the best card have sustained low latency and only when used with the multiple block writes as used in binaryLogger.pde.

@fat16lib

You are right! I didn't realize it would still be using the SdFat library I already had installed, and wasn't all contained within your recent code. But makes sense.

I've got a Patriot 2GB card, so I'll figure out it's limits on the Mega & let you know.

So far, so good.

I've got it working with Hard SPI and no overruns at 40kHz (25usec being the default).

I still don't understand properly the CS thing, so I'm going to have to read that again and figure out what I'm doing. I think I'm confusing SS and CS, and am not to sure which is which.

I'm using this 2GB SD card (PEF2G133SD):
http://www.patriotmemory.com/products/detailp.jsp?prodline=4&catid=3&prodgroupid=16&id=311&type=6

SPI on avr processors is a bit confusing. This Atmel application note might help.

http://atmel.com/dyn/resources/prod_documents/doc2585.pdf

Hi fat16lib,

your new code is great, especially the truncate function is awesome. Using multiple buffers is an improvement as well (although i don't see the point in using 12 - if your logger doesn't manage to write fast enough when using 3 buffers, it's propably too slow and won't catch up after going through 12 ... or does the writing latency vary in such an extreme way?)

nonetheless, I encounter the same problem that I did with my previous code, because the CAN-Controller and the SD-Card controller share the mosi and miso line.
Im using this shield
http://www.skpang.co.uk/catalog/arduino-canbus-shield-with-usd-card-holder-p-706.html

And i have to set ChipSelectLow(SS_PIN) when I want to write to the sd and put it back to high (not selected) afterwards, because otherwise receiving the CAN-Messages also writes nonsense to the SD card

...
If i do not allow interrupts while writing, everything works fine, but i "lose" one or 2 messages during the write process, because the CAN controller can only buffer 2 messages.

Thus, calling cli() right before writing to the sd card works fine

cli();   //clear interrupts
digitalWrite(SS_PIN, LOW);  // SD on[/i]
// write block to SD
if (!sd.card()->writeData(block)) {
        error("writeData");
}     
digitalWrite(SS_PIN, HIGH);  // SD off
sei(); // enable interrupts[/i]

...

I then tried to enable interrupts at various positions in the writeData() function, and after encountering many write errors I found a position that produced no errors and a "valid" log file (formatwise):

/** SPI send block - only one call so force inline */
static inline __attribute__((always_inline))
  void spiSendBlock(uint8_t token, const uint8_t* buf) {
  SPDR = token;
  for (uint16_t i = 0; i < 512; i += 2) {

  // turn writing off and on again to allow interrupts	
  chipSelectHigh();
  sei();
  cli();
  chipSelectLow();
  // custom code end


    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = buf[i + 1];
  }
  while (!(SPSR & (1 << SPIF)));
}

...

But if I do so, I get an incomplete Log file! somehow a lot of data is lost and I have milliseconds of no data. Maybe because of a buffer not actually being written?! ... i don't receive any errors and the program moves on without a hint..

I'm giving up hope, after testing it with your impressive logger. Do you have any hint, as to what I could try?

I'm sorry if this seems like I'm asking you to debug my code ... but I've already invested 15+ hours into solving this interference problem and I'm considering giving up my goal of achieving a Real-Time Capable CAN logger...

edit:
do you know, whether calling chipSelectLow/High during the write process does stuff other than just turning the connection on/off ? does it flush a buffer or something like that? .... is this something to consider debugging?

keija,

First write latency for SD cards varies widely. Occasionally an SD card has many milliseconds of extra latency so extra buffers help.

Many SD cards require chip select to be held low for the entire 512 byte block write. You can't suddenly switch use of the hardware SPI bus during a block write. SD cards only allow CS for go high at certain times.

This is why I use software SPI for the ADC in the interrupt code.

You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.

fat16lib:
Many SD cards require chip select to be held low for the entire 512 byte block write. You can't suddenly switch use of the hardware SPI bus during a block write. SD cards only allow CS for go high at certain times.
This is why I use software SPI for the ADC in the interrupt code.
You will need to use software SPI for the SD, which may be too slow or use software SPI for the CAN bus.

So my guesswork went into the right direction ... sadly ...
Thanks a million for your quick answer!

I'll reconsider my options and use your logging code for now (it's already extremely performant, even when disabling interrupts while writing!).

Keep up the great work fat16, your SD library already works tremendously well! Thank you!

Hello,

according to what I understand of this post it is not possible to have hardware SPI on sdcard and ADC. I've got a project working with arduinoMega analog inputs (multiplexer with datalogging at 10Hz). I added MCP3208 to the project, which is working nicely using hardware spi and spi.h. But if I start using the sdcard I've got wrong readings on the MCP3208, also sdfat is not working nicely as I don't get all files I should on the sdcard. Analog reading is done in a time interrupt to have proper frequency.
Do I need to unsolder the SPI pin of the MCP3208 to use soft spi or is there another option ?
Over question is why sdfat doesn't use spi.h , is it for performances ?

Thanks for any info.

Every SPI device has its own select pin and these should differ! You can only have one device selected simultaneously otherwise there will be interference. The most robust way is to wrap every SPI action with a device select/deselect.

some pseudocode

digitalWrite(ADC_SS, HIGH);
ADC.start();
digitalWrite(ADC_SS, LOW);

....

digitalWrite(ADC_SS, HIGH);
x = ADC.read();
digitalWrite(ADC_SS, LOW);

...

digitalWrite(SD_SS, HIGH);
SD.write (x)
digitalWrite(SD_SS, LOW);

But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x). And if you use sei() cli() around sd.write(x) you can drop samples in your adc. That was my assumption for my wrong readings. And I wanted to be sure there was no over option than using soft spi and bit banging for adc.
I switched to soft spi for adc and the sketch is working nicely. But I'm still interested to know if there is not another way to do it, perhaps by forcing SS pin of other spi device when getting into the adc interrupt.

But when you have interrupts using adc, you don't know if you are no gone get interrupt call during sd.write(x)

Then sd.write() should block IRQ's ?

... and the sketch is working nicely ...

Can you post your code ...

setup is calling this function to setup SPI for MCP3208:

void setupExternalADC(){ 
 pinMode(SELPIN, OUTPUT); 
 pinMode(DATAOUT, OUTPUT); 
 pinMode(DATAIN, INPUT); 
 pinMode(SPICLOCK, OUTPUT); 
 //disable device to start with 
 digitalWrite(SELPIN,HIGH); 
 digitalWrite(DATAOUT,LOW); 
 digitalWrite(SPICLOCK,LOW); 
}

there is an header file for pin configuration

#define SELPIN 25 //Selection Pin 
#define DATAOUT 24//MOSI 
#define DATAIN  26//MISO 
#define SPICLOCK  22//Clock

interrupt call this function (the MCP3208 from playground):

int read_adc(int channel){
  int adcvalue = 0;
  byte commandbits = B11000000; //command bits - start, mode, chn (3), dont care (3)

  //allow channel selection
  commandbits|=((channel-1)<<3);

  digitalWrite(SELPIN,LOW); //Select adc
  // setup bits to be written
  for (int i=7; i>=3; i--){
    digitalWrite(DATAOUT,commandbits&1<<i);
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);    
  }

  digitalWrite(SPICLOCK,HIGH);    //ignores 2 null bits
  digitalWrite(SPICLOCK,LOW);
  digitalWrite(SPICLOCK,HIGH);  
  digitalWrite(SPICLOCK,LOW);

  //read bits from adc
  for (int i=11; i>=0; i--){
    adcvalue+=digitalRead(DATAIN)<<i;
    //cycle clock
    digitalWrite(SPICLOCK,HIGH);
    digitalWrite(SPICLOCK,LOW);
  }
  digitalWrite(SELPIN, HIGH); //turn off device
  return adcvalue;
}

datalogging on the sd card is not done with interrupt but just link in this tutorial the http://www.arduino.cc/en/Tutorial/BlinkWithoutDelay