Pages: [1] 2 3   Go Down
Author Topic: New fast data logging sketches  (Read 11179 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I just posted three new fast data logging programs here http://code.google.com/p/beta-lib/downloads/list.

The file is fastLoggerBeta20110623.zip.

The fastest program, binaryLogger.pde, can log 40,000 samples per second when used with the highest quality SD cards.  I have been using 4 GB SanDisk Extreme 30 MB/sec cards.

It's not the speed that's important, it's the very low write latency of these cards.

binaryLogger.pde - Read from a MCP3201 ADC and log the data to a binary file.

fastLogger.pde - Read from analog pin zero and log the data to a text file.

mcpLogger.pde - Read from a MCP3201 ADC and log the data to a text file.

These are demo sketches and will need modification for any real application.

I have included a library for Microchip SAR ADCs that supports MCP3201, MCP3202, MCP3204, and MCP3208 chips.  I will be adding MCP3001, MCP3002, and MCP3301 in the future.

Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Very cool.

I will try and test it out tonight and this weekend.

I've started a sketch for a binary storage format, that I can add-on to this.
Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You mean these for SD cards?
http://www.newegg.com/Product/Product.aspx?Item=N82E16820171376&nm_mc=OTC-Froogle&cm_mmc=OTC-Froogle-_-Memory+%28Flash+Memory%29-_-SanDisk-_-20171376

I'm pretty sure that's it, but I thought I'd post it for reference for others.

I see from Sandisk site, that these exact same model used to be 20MB/sec.
http://kb.sandisk.com/app/answers/detail/a_id/1111/~/sandisk-memory-cards-read%2Fwrite-speed-change
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yes, I use the newegg card. 

Also I use the card with "HD Video" on it. The model is SDSDX3-004-P31.

Both work well.
Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Finally getting some time to work on this.

I'm debating on whether to use your MCP3208 code or use what I was working with.

I'm posting the relevant code to what I was using below. It is simpler for me to understand, but maybe you also made some considerable speed improvements in your code.

Code:
#include <SPI.h>

  void setup() {
    pinMode(53,OUTPUT);
    digitalWrite(53,HIGH);
    SPI.begin();
    SPI.setClockDivider(SPI_CLOCK_DIV8);
  }

  void loop() {
    readADC(0);
  }

uint16_t readADC(int channel) {
  uint16_t output;
  //Channel must be from 0 to 7
  //Shift bits to match datasheet for MCP3208
  byte commandMSB = B00000110;
  uint16_t commandBytes = (uint16_t) (commandMSB<<8|channel<<6);
 
  //Select ADC
  digitalWrite(53, LOW);
  //send start bit and bit to specify single or differential mode (single mode chosen here)
  SPI.transfer((commandBytes>>8) & 0xff);

  byte msb = SPI.transfer((byte)commandBytes & 0xff) & B00001111;
  byte lsb = SPI.transfer(0x00);
 
  digitalWrite(53,HIGH);

  // cast before shiting the byte
  output = ((uint16_t) msb) <<8 | lsb;
  return output;
}

Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Ok, I'm starting to understand your code a bit better.

You are using SOFT_SPI for both the SDfat & the MCP320x.

I'm surprised you can do both, so the hardware SPI & the hardware serial port could still be used for something else, which is quite a feat.

I'll go ahead and try to get it tested with your design first, but I'd like to know the best place to define the SPI pins for hardware SPI to be used with the ADC.

I also realize that if I'd use my prior method of communicating with the ADC, I would have to include SPI.h which would create a bigger footprint probably.
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

oscarcar,

If your code works you should use it.

I used an oscilloscope to develop the MCM320X code so modifying/debugging it is easier with a scope.

I don't want to share the hardware SPI bus between SdFat and interrupt code that reads the ADC.  I can just read the ADC in the interrupt service routine without thinking about SdFat and the hardware SPI bus.

My code is faster but that doesn't matter if your code is fast enough. 

I designed the MCP320X code to achieve maximum speed for logging values read in an ISR and written in loop().  I could never log 40,000 data points per second from a MCP3201 in binaryLogger.pde without that kind of optimization.
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I use hardware SPI for SdFat and soft SPI for the ADC.  This is the fastest since SdFat does more I/O and I use 8 MHz for the SD.
Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I use hardware SPI for SdFat and soft SPI for the ADC.  This is the fastest since SdFat does more I/O and I use 8 MHz for the SD.

But with the Mega, I thought you did software SPI for that, so that the shields could be used without altering.

Will I need to re-route the pins to the Mega SPI pins? I tried doing that with your last codebase, and I never got it working.
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yes I use hardware SPI on the Mega.

SdFat has supported hardware SPI on the Mega since it was released.  I added the option for software SPI in December 2009.

Do not edit SdFat files.  Just run wires from the SD shield pins to the Mega hardware SPI pins.

To use a shield for a 328 Arduino on the Mega run short wires like this:

Mega Pin <--> Shield Pin
50 <--> 12  MISO
51 <--> 11  MOSI
52 <--> 13  SCK

If you use the default SS pin for SD chip select, connect these pins.
53 <--> 10  SS

Don't run a wire for chip select if you specify the chip select pin number in the init() call like this.

Code:
sd.init(SPI_FULL_SPEED, 8);

The init() call will override the default and ,in this case, pin 8 will be use for SD chip select.

When you choose software SPI pins for the ADC be sure to look at MegaPins.txt.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 27
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi fat16,

I'm loving your progress and effort! I can't get my interrupt driven dataWriter to work and will try out your solution soon!
Is a mega necessary to reach those write speeds? or does a uno suffice?

keep up the good work!
Greetings, Keija
Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

@fat16lib

Thanks for the tips, but I'm still a bit confused.

Last night I tried it with the shield on the Mega, and it seemed to run fine (except for some dropouts, and the fact that I wasn't using the MCP3208 calls yet).

Since it was wired to 10-13 on the Mega (and working), doesn't that mean it had to be using SOFT_SPI?

And if I routed it to the 51-53 pins, wouldn't it still be trying to use SOFT_SPI, since I wouldn't be changing anything in the code?

I was under the assumption that you had some #ifdefine segments that if it's a Mega, you use SOFT_SPI. I thought that's what I was having a hard time working around before.

Thanks for the help. I'm almost there.
Logged

0
Offline Offline
Edison Member
*
Karma: 63
Posts: 1604
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

To use software SPI you must have edited SdFatConfig.h and changed MEGA_SOFT_SPI.

Go to about line 85 and change it back to the default:
Code:
#define MEGA_SOFT_SPI 0

The SPI will be much faster.

Unless you have a very high quality SD, you will have dropped data.  Most SD cards have an occasional write latency of over 100 milliseconds.

Only the best card have sustained low latency and only when used with the multiple block writes as used in binaryLogger.pde.

Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

@fat16lib

You are right! I didn't realize it would still be using the SdFat library I already had installed, and wasn't all contained within your recent code. But makes sense.

I've got a Patriot 2GB card, so I'll figure out it's limits on the Mega & let you know.
Logged

SF Bay Area
Offline Offline
Full Member
***
Karma: 1
Posts: 182
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

So far, so good.

I've got it working with Hard SPI and no overruns at 40kHz (25usec being the default).

I still don't understand properly the CS thing, so I'm going to have to read that again and figure out what I'm doing. I think I'm confusing SS and CS, and am not to sure which is which.

I'm using this 2GB SD card (PEF2G133SD):
http://www.patriotmemory.com/products/detailp.jsp?prodline=4&catid=3&prodgroupid=16&id=311&type=6
Logged

Pages: [1] 2 3   Go Up
Jump to: