Show Posts
Pages: [1] 2 3 ... 99
1  Using Arduino / Storage / Re: Iterating through a Directory and storing the maximium file number as an integer on: April 16, 2014, 11:48:17 am
Here is a very fast function to find the "maximium file number as an integer"  It was developed for a user with a huge number of files so one pass through the directory was required.
#include <SD.h>
const uint8_t chipSelect = 10;
// Return -2 if error, -1 not found, else max file number.
long maxFileNumber(const char* dirName, const char* baseName, const char* fileExt) {
  File dir;
  long maxNum = -1; 
  uint8_t len = strlen(baseName);

  dir =, FILE_READ);
  if (!dir || !dir.isDirectory()) return -2;
  while (true) {
    char *ext, *name, *ptr;
    dir_t d;
    // Return result if no more entries.
    if (, 32) != 32 ||[0] == 0) return maxNum;
    // Only check existing regular files.
    if ([0] == DIR_NAME_DELETED || !DIR_IS_FILE(&d)) continue;
    // Cast file extension to char* for ease of use
    ext = (char*) + 8;
    ext[3] = 0;
    // Remove any blank fill.
    if ((ptr = strchr(ext, ' '))) *ptr = 0;
    // Skip if wrong file extension. Use case independent compare.
    if (strcasecmp(ext, fileExt)) continue;
    // Cast to char* for ease of use
    name = (char*); 
    // Don't look at file extension
    name[8] = 0;     
    // Look for baseName with case independent compare.
    if (0 == strncasecmp(name, baseName, len)) {
      // Place for start of number.
      char* bgn = name + len;
      // Iterator
      char* ptr = bgn;
      long n = 0;
      // Get number.
      while (isdigit(*ptr)) n = 10*n + *ptr++ - '0';
      // make sure the end of the file name is a valid number.
      if (ptr != bgn && (*ptr == 0 || *ptr == ' ') && n > maxNum) maxNum = n;
void setup() {
  if (!SD.begin(chipSelect)) {
    Serial.println(F("SD.begin failed"));
  Serial.print(F("ANALOG,BIN: "));
  Serial.println(maxFileNumber("/", "ANALOG", "BIN"));
  Serial.print(F("DATA,TXT: "));
  Serial.println(maxFileNumber("/", "DATA", "TXT"));
  Serial.print(F("SHORT,TX: "));
  Serial.println(maxFileNumber("/", "SHORT", "TX"));
  Serial.print(F("NOEXT,<null>: "));
  Serial.println(maxFileNumber("/", "NOEXT", ""));
  Serial.print(F("<nobase>,TXT: "));
  Serial.println(maxFileNumber("/", "", "TXT"));
  Serial.print(F("MISSING,TXT: "));
  Serial.println(maxFileNumber("/", "MISSING", "TXT"));
   Serial.print(F("<bad dir>: "));
  Serial.println(maxFileNumber("/BADDIR", "DATA", "TXT"));
void loop() {}

I ran the program on an SD that had these files root directory:

Here is the test output:
NOEXT,<null>: 9
<nobase>,TXT: 12345678
<bad dir>: -2

Note: Finding "12345678.TXT" by opening every possible file in the series would take close to forever.
2  Using Arduino / Storage / Re: Read CSV or TXT from SD card into string or array on: April 15, 2014, 06:21:04 am
Here is a simple sketch that reads a CSV file with two numbers on each line.  It doesn't use the dangerous String class.
#include <SD.h>
File file;

bool readLine(File &f, char* line, size_t maxLen) {
  for (size_t n = 0; n < maxLen; n++) {
    int c =;
    if ( c < 0 && n == 0) return false;  // EOF
    if (c < 0 || c == '\n') {
      line[n] = 0;
      return true;
    line[n] = c;
  return false; // line too long

bool readVals(long* v1, long* v2) {
  char line[40], *ptr, *str;
  if (!readLine(file, line, sizeof(line))) {
    return false;  // EOF or too long
  *v1 = strtol(line, &ptr, 10);
  if (ptr == line) return false;  // bad number if equal
  while (*ptr) {
    if (*ptr++ == ',') break;
  *v2 = strtol(ptr, &str, 10);
  return str != ptr;  // true if number found

void setup(){
  long x, y;
  if (!SD.begin(SS)) {
    Serial.println("begin error");
  file ="TEST.CSV", FILE_READ);
  if (!file) {
    Serial.println("open error");
  while (readVals(&x, &y)) {
    Serial.print("x: ");
    Serial.print("y: ");
void loop() {}

Reading this "TEST.CSV" file.

Results in this output
x: 1
y: 2

x: 34
y: 56

x: 987
y: -765

x: 1234
y: -5678

3  Using Arduino / Storage / Re: RamDisk - a file system library for RAM devices like 23LCV1024 on: April 13, 2014, 07:00:04 am

Thanks for the info on your 8 MB RAM. 

It is very helpful to see results for a second type of RAM.
4  Using Arduino / Storage / Re: Try this super fast analog pin logger on: April 13, 2014, 06:55:43 am
Can you help me configure it to sample A0 at 40,000 samples per second ?

First you need an SD card that has very low write latency.

I found this card works very well

Change only these values.
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0};
// Sample rate in samples per second.
const float SAMPLE_RATE = 40000;  // Must be 0.25 or greater.
Here is serial output for recording about five minutes of data.  The proper SDHC card is extremely important.
Sample pins: 0
ADC bits: 10
ADC clock kHz: 1000
Sample Rate: 40000.00
Sample interval usec: 25.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG01.BIN
Max block write usec: 1112
Record time sec: 304.766
Sample count: 12190600
Samples/sec: 39999.87
Overruns: 0

What does 8-bit mode mean for the 2000 hZ clock rate ?
8-bit mode only records the high 8-bits of each ADC  reading .  At 2000 kHz ADC clock rate the ADC only provides about 7.5 significant bits. see the ADC_ENOB.PNG file.  This reduces the amount of date written so it is easier for SD cards to handle the data rate.

I didn't include 8-bit mode in the max data rate table.  Here is a version with 8-bit mode.


ADC clock kHz 8-bit only
125 250 500 1000 2000
1 7692 14286 25000 40000 57143
2 3810 6667 11111 16667 22222
3 2572 4790 8421 13559 19512
4 1942 3636 6452 10526 15385
5 1559 2930 5229 8602 12698
6 1303 2454 4396 7273 10811
7 1119 2111 3791 6299 9412
8 980 1852 3333 5556 8333
9 872 1649 2974 4969 7477
10 786 1487 2685 4494 6780
11 715 1354 2446 4103 6202
12 656 1242 2247 3774 5714
13 606 1148 2078 3493 5298
14 563 1067 1932 3252 4938
15 525 996 1806 3042 4624
16 493 935 1695 2857 4348
Here is serial monitor output for a short recording with an 18 microsecond sample interval.
Sample pins: 0
ADC bits: 8
ADC clock kHz: 2000
Sample Rate: 55555.55
Sample interval usec: 18.0000
Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: ANALOG02.BIN
Max block write usec: 1304
Record time sec: 28.103
Sample count: 1561189
Samples/sec: 55552.39
Overruns: 0

5  Using Arduino / Storage / Re: RamDisk - a file system library for RAM devices like 23LCV1024 on: April 12, 2014, 04:21:26 pm
Regarding latency, what speed is the SPI clock running at?  Can it be increased?

The SPI speed is 8 MHz so it can't be increased.  I am working on reducing latency.  I didn't post the new code yet.

I am rewriting the 23LCV1024 library as a C++ template.  I found that for one or two byte transfers, half of the time is consumed by digitalWrite toggling the chip select pin.  Here is a test sketch that write two byte and then reads them back.
#include <M23LCV1024.h>
const uint8_t RAM_CS_PIN = 9;

M23LCV1024 ram;
uint16_t wr = 1234;
uint16_t rd = 0;

void setup() {
  uint32_t m = micros();
  ram.write(444, &wr, 2);, &rd, 2);
  m = micros() - m;
  Serial.print("micros: ");
  Serial.print("data: ");
void loop() {}

The current version takes 52 microseconds to transfer 4 bytes.  That's just 77 KB/sec.
micros: 52
data: 1234

My first cut rewrite runs almost twice as fast.
micros: 28
data: 1234

I probably won't do much better.  A six bytes transfer on the SPI bus is required to read or write two bytes.  A command byte, three address bytes, and the two data bytes.  So there is actually 12 bytes transferred.

This new driver does not change the basic results for the RamStream class.  Here are the results of the single byte read/write test with the new driver.  I removed code that verified correct read data values for this timing.  This is now a major effect and not part of the read time.
File read/write stats
fileSize: 51000
write millis: 1871
read millis: 1610

Stream getc/putc stats
streamSize: 51000
putc millis: 187
getc millis: 177

So RamStream is still about ten times faster.

I am now taking a break from design and implementation of features and trying to improve performance.
6  Using Arduino / Storage / Re: Try this super fast analog pin logger on: April 12, 2014, 03:54:55 pm
Place the AnalogIsrLogger folder in your sketchbook folder. 

I found this instruction in the readme file but didn't see any folder with that spelling . Is this important ?

I should have just said sketchbook I guess.  I didn't say the Arduino folder since you can select the folder to use.
The Arduino environment uses the concept of a sketchbook: a standard place to store your programs (or sketches). The sketches in your sketchbook can be opened from the File > Sketchbook menu or from the Open button on the toolbar. The first time you run the Arduino software, it will automatically create a directory for your sketchbook. You can view or change the location of the sketchbook location from with the Preferences dialog. 

Which file do I edit with my C++ editor ?
You edit AnalogBinLogger.ino.  for example to select the set of analog pins edit this area:
// Analog pin number list for a sample.  Pins may be in any order and pin
// numbers may be repeated.
const uint8_t PIN_LIST[] = {0, 1, 2, 3, 4};

Does the above mean I can perform any of those four commands after I start running the program ?

Yes just type the letter corresponding to the command you want run.

I thought the bin to csv had to be performed using the command line instructions in the readme file .
It takes a long time to convert large binary files on the Arduino so the PC/Mac command line program is often required.
7  Using Arduino / Storage / Re: RamDisk - a file system library for RAM devices like 23LCV1024 on: April 12, 2014, 08:59:36 am
I have attached a new demo release of RamDisk to this reply.  This is a major change since the last version and has a huge amount of new code so I call it a demo, not a beta.

The previous release was very fast for large binary reads/writes but had little gain over an SD for small reads and writes.  Small writes are often required for printing and small reads are required for parsing files.  Also it is nice to have Unix/Linux style character push-back when parsing files.

RamDisk has been re-factored into several classes.

The key classes are RamVolume, RamBaseFile, RamFile, and RamStream.

The main new features are in the RamStream class which is derived from the RamFile class. The API for this class is based on Unix/Linux stdio streams.

The RamStream class provides an additional level of cache and is based on the Unix/Linux stdio design. The RamStream class is very fast for single byte I/O, formated output, and parsing text input. See the examples for performance tests.

I have also provided the Unix/Linux style character push-back, ungetc, with an definable amount of buffering.

The RamBaseFile or RamFile classes are still fastest for large binary transfers. These classes access the RAM with no intermediate buffering.

Here is the result for single byte IO:
This example demonstrates the factor of ten speedup provided by RamStream for single byte transfers.

File read/write stats
fileSize: 51000
write millis: 2429
read millis: 2222

Stream getc/putc stats
putc millis: 200
getc millis: 225
streamSize: 51000

Here are the the speedup ratios for formatted output comparing the previous RamFile with the new RamStream class.  The speed-up of 5.6 to 9.5 is due to the new stdio style buffering and use of robtillaart's post based upon the divu10() code from the book Hackers Delight1. (I still need an acknowledgement for  robtillaart in the code.)
uint8_t 0 to 255, 100 times
fileSize: 116500
print millis: 6776
stdio millis: 711
ratio: 9.53

uint16_t 0 to 18000
fileSize: 114890
print millis: 6226
stdio millis: 749
ratio: 8.31

uint32_t 0 to 18000
fileSize: 114890
print millis: 6221
stdio millis: 852
ratio: 7.30

uint32_t 1000000000 to 1000010000
fileSize: 120000
print millis: 5975
stdio millis: 1061
ratio: 5.63

float nnn.ffff, 10000 times
fileSize: 100000
print millis: 10049
stdio millis: 1723
ratio: 5.83

I plan to make the stdio code generic and port it back into my SD libraries.  It should give a similar speed up for printing and reading files byte-at-a-time.
8  Using Arduino / Storage / Re: Can't open file on an SD card with .open() from SDFat. on: April 04, 2014, 01:46:13 pm
It looks like you can return from loop without closing logfile.

if (logfile.isOpen()) sd.errorHalt("file is open error");
just before the open that fails to check for this problem.

A far bigger problem is that you cannot get any performance if you open and close files in loop.  You need a rxFile and a txFile that are opened in setup().
9  Using Arduino / Storage / Re: Can't open file on an SD card with .open() from SDFat. on: April 03, 2014, 07:46:59 pm
It may be due to the dimension of
char TxFileName[11];

This is really only room for ten characters since there must be a zero byte to terminate a string.

This string requires 12 bytes when you include the zero byte.
char TxFile[] = "GR_TX00.csv";

This will overwrite an extra byte somewhere.

The same is true for RX files.  This may not be the open problem but it will bite at some point.
10  Using Arduino / Storage / Re: SDFat: Writing and reading binary data file using streams on: April 03, 2014, 02:21:38 pm
I didn't implement read() and write() in file streams.  The default for streams is text mode which inserts a CR before LF for write() and removes the CR before a LF on read().  This would be really slow and not very useful on Arduino.

In a full implementation of streams you can open a file in binary mode and use read() and write().

The SdFile and SdBaseFile classes do binary reads() and writes().  Read the CSV file with a ifstream and write the binary file with a SdBaseFile using write();

Read the binary file with SdBaseFile using read().
11  Using Arduino / Storage / Re: SdFat library with I2C and SoftwareSerial on: April 03, 2014, 07:01:48 am
I substituted  your file name for "test.txt" in the SdFatReadWrite example and it runs so there must be a more subtle problem.  Maybe someone will see something if you post all of the code.

Here is the program:
// Ported to SdFat from the native Arduino SD library example by Bill Greiman
// On the Ethernet Shield, CS is pin 4. SdFat handles setting SS
const int chipSelect = 10;
 SD card read/write
 This example shows how to read and write data to and from an SD card file
 The circuit:
 * SD card attached to SPI bus as follows:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 4
 created   Nov 2010
 by David A. Mellis
 updated 2 Dec 2010
 by Tom Igoe
 modified by Bill Greiman 11 Apr 2011
 This example code is in the public domain.
#include <SdFat.h>
SdFat sd;
SdFile myFile;

void setup() {
  while (!Serial) {}  // wait for Leonardo
  Serial.println("Type any character to start");
  while ( <= 0) {}
  delay(400);  // catch Due reset problem
  // Initialize SdFat or print a detailed error message and halt
  // Use half speed like the native library.
  // change to SPI_FULL_SPEED for more performance.
  if (!sd.begin(chipSelect, SPI_HALF_SPEED)) sd.initErrorHalt();

  // open the file for write at end like the Native SD library
  if (!"GPS01.txt", O_RDWR | O_CREAT | O_AT_END)) {
    sd.errorHalt("opening test.txt for write failed");
  // if the file opened okay, write to it:
  Serial.print("Writing to GPS01.txt...");
  myFile.println("testing 1, 2, 3.");

  // close the file:

  // re-open the file for reading:
  if (!"GPS01.txt", O_READ)) {
    sd.errorHalt("opening test.txt for read failed");

  // read from the file until there's nothing else in it:
  int data;
  while ((data = >= 0) Serial.write(data);
  // close the file:

void loop() {
  // nothing happens after setup

Here is the output.
Type any character to start
Writing to GPS01.txt...done.
testing 1, 2, 3.
12  Using Arduino / Storage / Re: SdFat library with I2C and SoftwareSerial on: April 02, 2014, 12:48:39 pm
SdFat only supports 8.3 file names.  This is not a valid 8.3 name.
char fileName[] = "GPS_Teat_01.txt"

You get zero error code from card since this is not a hardware SD card error.  sd.card() is the raw block device and doesn't know about files.
   sprintf(msg,"Opening file for write failed. Error = %d. Data = %d",sd.card()->errorCode(),
13  Using Arduino / Project Guidance / Re: Why inaccurate tempo? on: March 31, 2014, 10:00:05 pm
Here's the problem.
The UNO use a ceramic resonator rather then a quartz crystal.

Is this true?

Edit: Appears the crystal is for the USB, Atmega16U2.  An Uno will never keep good time.
14  Using Arduino / Project Guidance / Re: Why inaccurate tempo? on: March 31, 2014, 09:46:24 pm
Edit: see the next post.  The Uno does not have a crystal for thr 328.

That's poorer accuracy for the clock than I would have expected from other benchmark results I've seen reported here,
All the Uno boards I measured above are poor time keepers.

The Duemilanove I Measured was good to 35 ppm.

you can compensate by adjusting the interval value in your sketch up or down a little as required to get accurate timing.
It is easy to calibrate the board as I indicated above.   The SQW pulse from a DS1307 and attach Interrupt can be used.  Look at forum topics.

Trial and error adjusting is a pain.
15  Using Arduino / Project Guidance / Re: Why inaccurate tempo? on: March 31, 2014, 08:33:53 pm
You could try to find the correct value for interval to correct for the Uno crystal.

I have used the pulse per second from a gps.  it is accurate to about  1 usec.  I connected it to pin 2 or pin 3 and use attachInterrupt. you can then see how many micros between interrupts.  This allows you to calibrate the Arduino crystal.

You can also use the SQW pulse from a RTC like DS1307.

There is always trial and error.
Pages: [1] 2 3 ... 99