Show Posts
Pages: 1 ... 58 59 [60] 61 62 ... 112
886  Using Arduino / Storage / Re: New SdFat optimized for Mega and Teensy 3.0 on: October 22, 2012, 09:56:07 am
pito,

The SPI clock for SD.h on the Mega was 4 MHz.  The standard Arduino SD library has no option for SPI clock speed.

I did a test at 8 MHz with SD.h by editing the source for begin().  Doubling the SPI clock speed increased the write speed from 265.95 KB/sec to 376.29 KB/sec and the read speed from 314.07 KB/sec to 463.86 KB/sec.
Quote
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 376.29 KB/sec
Maximum latency: 78644 usec, Minimum Latency: 9808 usec, Avg Latency: 10873 usec

Starting read test.  Please wait up to a minute
Read 463.86 KB/sec
Maximum latency: 9940 usec, Minimum Latency: 8784 usec, Avg Latency: 8823 usec

The SPI clock speed for Mega was 8 MHz with the new version of SdFat.

The SPI clock speed for Teensy 3.0 was 24 MHz.

The key to high speed writes is to use a record size that is a multiple of 512 bytes.  This insures that very little data needs to be copied to the cache.  In these tests with the new SdFat, only directory entries and the FAT table need to be cached.

Using a record size that is a power of two increases performance slightly.  This insures that writes are aligned with file clusters.

It is very important to use a freshly formatted SD card so that a contiguous file is created.  SD cards perform very poorly for random writes since the internal flash for most cards have a very large page size.  The entire page must be rewritten to newly erased flash for a random write.

The new SdFat still has only one block of cache so overhead is dramatically increased if write data must be cached.  This means that the cache block must be used for both write data and the FAT table.

I am considering options to use more cache but this adds complexity and there is still the overhead of copying data to the cache.  Adding cache would not improve the above results but would help when record size is not a multiple of 512.
887  Using Arduino / Storage / New SdFat optimized for Mega and Teensy 3.0 on: October 21, 2012, 09:33:59 am
I have posted a new SdFat beta - SdFatBeta20121020.zip http://code.google.com/p/beta-lib/downloads/list.

This beta supports AVR boards and the new ARM Teensy 3.0 board http://www.adafruit.com/products/1044.

Performance is greatly improved for large reads and writes. 

The following benchmarks were done using this ATP 1GB Industrial Grade SD card http://www.newegg.com/Product/Product.aspx?Item=9SIA12K0CT6829.

The results are in increasing order of performance.

Mega 2560 using the standard Arduino SD.h library with 4096 byte reads and writes.
Quote
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 265.95 KB/sec
Maximum latency: 84184 usec, Minimum Latency: 14144 usec, Avg Latency: 15388 usec

Starting read test.  Please wait up to a minute
Read 314.07 KB/sec
Maximum latency: 14752 usec, Minimum Latency: 13020 usec, Avg Latency: 13035 usec

Mega 2560 using the new SdFat library with 4096 byte reads and writes.
Quote
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 658.38 KB/sec
Maximum latency: 65816 usec, Minimum Latency: 6036 usec, Avg Latency: 6210 usec

Starting read test.  Please wait up to a minute
Read 616.40 KB/sec
Maximum latency: 7624 usec, Minimum Latency: 6628 usec, Avg Latency: 6638 usec

Teensy 3.0 at 96 MHz using the new SdFat library with 4096 byte reads and writes.
Quote
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 1776.44 KB/sec
Maximum latency: 65790 usec, Minimum Latency: 2146 usec, Avg Latency: 2300 usec

Starting read test.  Please wait up to a minute
Read 2037.15 KB/sec
Maximum latency: 2356 usec, Minimum Latency: 1999 usec, Avg Latency: 2008 usec

Here is my best Teensy 3.0 result using an old Corsair 1GB SD (no longer available) with 8192 byte reads and writes.
Quote
File size 10MB
Buffer size 8192 bytes
Starting write test.  Please wait up to a minute
Write 2002.05 KB/sec
Maximum latency: 6777 usec, Minimum Latency: 4007 usec, Avg Latency: 4089 usec

Starting read test.  Please wait up to a minute
Read 2121.47 KB/sec
Maximum latency: 4231 usec, Minimum Latency: 3853 usec, Avg Latency: 3860 usec
888  Using Arduino / Storage / Re: Memory leak in openNextFile() on: October 20, 2012, 05:45:44 pm
This is a well known problem with the listFiles example and has been around for more than a year.

In general memory is lost unless a File object is closed before doing another open.

This is due mostly to poor design of the SD.h wrapper for SdFat. 

Maybe at some point the Arduino company will fix it.

You can find many posts about this problem in the forum.
889  Development / Other Hardware Development / Re: Teensy 3.0 on: October 20, 2012, 02:31:43 pm
I have posted a version of SdFat that supports Teensy 3.0 as SdFatBeta20121020.zip http://code.google.com/p/beta-lib/downloads/list

I recommend using an industrial SD like this one http://www.newegg.com/Product/Product.aspx?Item=9SIA12K0CT6829

I ran the SdFat bench.ino example with this card at 96 MHz.  I used the following defines for the test:

Code:
#define FILE_SIZE_MB 10
#define FILE_SIZE (1000000UL*FILE_SIZE_MB)
#define BUF_SIZE 8192

Here is the result of the test with this low cost 1GB ATP card:

Quote
Free RAM: 4207
Type is FAT16
File size 10MB
Buffer size 8192 bytes
Starting write test.  Please wait up to a minute
Write 1934.99 KB/sec
Maximum latency: 11333 usec, Minimum Latency: 4144 usec, Avg Latency: 4228 usec

Starting read test.  Please wait up to a minute
Read 2111.61 KB/sec
Maximum latency: 4220 usec, Minimum Latency: 3870 usec, Avg Latency: 3877 usec
890  Development / Other Hardware Development / Re: Teensy 3.0 on: October 16, 2012, 10:18:54 pm
Teensy 3.0 is great.  The hardware is a little gem and the way Paul did the software makes development really easy.

I now have optimized native SPI running in programmed I/O mode and performance is really good.  Here are my latest results for SdFat.

Quote
Free RAM: 4219
Type is FAT16
File size 10MB
Buffer size 8192 bytes
Starting write test.  Please wait up to a minute
Write 1801.74 KB/sec
Maximum latency: 80495 usec, Minimum Latency: 4398 usec, Avg Latency: 4543 usec

Starting read test.  Please wait up to a minute
Read 2017.82 KB/sec
Maximum latency: 4473 usec, Minimum Latency: 4048 usec, Avg Latency: 4057 usec

This is with the CPU overclocked at 96 MHz.  I used an industrial SD designed for embedded use.
891  Development / Other Hardware Development / Re: Teensy 3.0 on: October 15, 2012, 05:05:20 pm
Thanks to Paul's help, I have first results for SdFat with large reads and writes.

Writes are slow since I am not using the full TX fifo yet.

Here are results for 4096 byte writes and reads using the SdFat bench.ino example.

Quote
Type is FAT16
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 723.07 KB/sec
Maximum latency: 21609 usec, Minimum Latency: 4929 usec, Avg Latency: 5625 usec

Starting read test.  Please wait up to a minute
Read 1255.56 KB/sec
Maximum latency: 3874 usec, Minimum Latency: 3204 usec, Avg Latency: 3260 usec

I think I know how to speedup writes by using all four bytes of the SPI fifo.

DMA may be required to achieve maximum speed since using the fifo has a bit more overhead than I first assumed.
892  Development / Other Hardware Development / Re: Teensy 3.0 on: October 15, 2012, 10:56:27 am
First the Teensy Kinetis CPU does not have hardware floating point.  Hardware floating point is optional for Cortex M4.  Only the high end K20 processors have floating point http://www.freescale.com/files/microcontrollers/doc/fact_sheet/KNTSK20FMLYFS.pdf.

newlib math function are really old C functions.

newlib execution times depend on the value of the arguments.

Here are two examples for 32-bit sine:
Code:
float sinf(float);

I ran this code
Code:
float sinanswers[401];
float sinarg[401];
  for (int i = 0; i < 400; i++) {
    sinarg[i] = factor*i;
  }
  time1 = micros();
  for (int i = 0; i < 400; i++) {
    sinanswers[i] = sinf(sinarg[i]);
  }
  time2 = micros();
If factor is 0.01 so the range is from 0.0 - 4.0
Quote
time elapsed = 17110 micros


If factor is 1.0 so the range is 0.0 - 400.0
Quote
time elapsed = 105353 micros

The algorithms for 64-bit double are totally different than for 32-bit float.

Much of this dates back to work in the 1980s on BSD Unix at UC Berkeley.  I was at UCB when BSD Unix was developed.

Bill Joy was a key developer of BSD and used it at Sun Microsystems as the base for Solaris.
893  Development / Suggestions for the Arduino Project / Re: Please add the AVcc filter on: October 12, 2012, 11:30:37 am
I agree, an AVcc filter would be great.

Without the filter, accuracy depends a great deal on how you power the Arduino.  I did some tests with a high quality 9V power supply http://arduino.cc/forum/index.php/topic,120004.0.html.

For a perfect ADC, the maximum error is 1/2 LSB and the mean error is 1/4 LSB.

For 5V Vref and a 10-bit perfect ADC the max error is 2.44 mv and the mean error is 1.22 mv.

My result for three Uno Arduinos with the high quality power supply was max error around 4 mv and mean error 1.4 mv.

So you can get very close to 10-bit accuracy with a good grounded 9V power supply and no modifications to a Uno.

The best solution would be a filter.  Good 9V supplies cost a lot and good small three prong supplies are hard to find.

ADC calibration is absolutely necessary to achieve good results.  The above link describes my calibration setup.
894  Using Arduino / Storage / Re: SdFat - tuned for 1284p on: October 12, 2012, 07:13:12 am
I have have modified SdFat so large reads/writes will go faster and I will soon post a beta.

I changed the way cache is handled and the SD commands used for large reads and writes.

Increasing internal buffering on AVR is not effective since the CPU cycles to copy data cost so much.

I still use a single block cache to hold partial blocks.  Most of a large write is sent to SPI directly from user memory.

If you use a size that is a power of two for a write, all data will be written directly to the SPI bus. Writes will not cross cluster boundaries if you correctly format the card.  This allows all data to be written to the SD with a single multi-block write command.

Here is the improvement for a Mega with a 4096 byte write.

Old SdFat:
Quote
Type is FAT16
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 247.37 KB/sec
Maximum latency: 100604 usec, Minimum Latency: 15084 usec, Avg Latency: 16447 usec

Starting read test.  Please wait up to a minute
Read 451.70 KB/sec
Maximum latency: 10168 usec, Minimum Latency: 8844 usec, Avg Latency: 9061 usec

New SdFat:
Quote
Type is FAT16
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 535.71 KB/sec
Maximum latency: 21940 usec, Minimum Latency: 6912 usec, Avg Latency: 7601 usec

Starting read test.  Please wait up to a minute
Read 595.53 KB/sec
Maximum latency: 7992 usec, Minimum Latency: 6808 usec, Avg Latency: 6872 usec
Write speed has doubled.

This C function is used to send data to the SPI bus:
Code:
void spiSendBlock(uint8_t token, const uint8_t* buf) {
  SPDR = token;
  for (uint16_t i = 0; i < 512; i++) {
    uint8_t b = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = b;
  }
  while (!(SPSR & (1 << SPIF)));
}
It takes about 820 microseconds to execute so the limit for a write is about 625 KB/sec.  Converting it to assembly would probably improve performance a lot.  Currently I am more interested in Cortex M ports of SdFat.

This will not help 328 Arduinos with small writes.  Here is the result for a Uno with 100 byte writes:
Quote
Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 179.75 KB/sec
Maximum latency: 109332 usec, Minimum Latency: 84 usec, Avg Latency: 550 usec

Starting read test.  Please wait up to a minute
Read 311.84 KB/sec
Maximum latency: 3500 usec, Minimum Latency: 88 usec, Avg Latency: 314 usec
895  Development / Other Hardware Development / Re: Teensy 3.0 on: October 11, 2012, 03:13:07 pm
Paul,

I mailed you a version with mods for faster reads/writes.  It compiles for Teensy 3.0 but is not tested on Teensy 3.0 since I have not received the replacement for the missing Teensy 3.0 that Robin sent yesterday.

The results for a AVR Mega with 4096 byte writes and reads are promising.

Quote
Type any character to start
Free RAM: 2666
Type is FAT16
File size 5MB
Buffer size 4096 bytes
Starting write test.  Please wait up to a minute
Write 536.40 KB/sec
Maximum latency: 10336 usec, Minimum Latency: 6908 usec, Avg Latency: 7592 usec

Starting read test.  Please wait up to a minute
Read 595.04 KB/sec
Maximum latency: 7984 usec, Minimum Latency: 6804 usec, Avg Latency: 6877 usec
896  Using Arduino / Storage / Re: Problem with directory listings after other file IO on: October 10, 2012, 02:43:17 pm
There is a bug in the SD library for opening root that causes the position of root to not be at the beginning.

Add a dir.seek(0) to your printDirectory function like this:
Code:
void printDirectory(File dir, int numTabs)
{
  int nEntries = 0;
  char buf[32];
  dir.seek(0);
   while(true) {
     
     File entry =  dir.openNextFile();
     if (! entry) {
       // no more files
       //Serial.println("**nomorefiles**");
       entry.close(); // Added
       break;
     }
     for (uint8_t i=0; i<numTabs; i++) {
       Serial.print('\t');
     }
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       //Serial.println("/");
       //printDirectory(entry, numTabs+1); // no recursion
     } else {
       // files have sizes, directories do not
       Serial.print("\t\t");
       Serial.println(entry.size(), DEC);
       nEntries++;
     }
     entry.close(); // Added
   }
   
   sprintf( buf, "%d", nEntries );
   
   Serial.println( "" );
   Serial.print( buf );
   Serial.println( " entries found." );
   
}
897  Using Arduino / Storage / Re: MEGA with cardinfo.h and "LC studio" SD card reader dont work on: October 08, 2012, 02:43:58 pm
A new card seems like a good idea.   

I worry that the problem may be noise on the SPI bus.  There are often SPI problems with modules like the LC Studio.  Most reliable modules have a 5V to 3.3V level shifter based on a buffer chip.   Level shifters based on resistors and diodes work poorly with some cards.

Some cards use edge detectors and the sharp signals of a buffer chip work better.

If you new card does not fix the problem, make sure the SPI wires are as short as possible.
898  Development / Other Hardware Development / Re: Teensy 3.0 on: October 08, 2012, 12:32:31 pm
Sorry, I forgot to mention the packed attribute. for FAT structs.

I replaced the name fpos_t with FatPos_t

I had problems with types like uint16_t in extractors and inserters so I changed them to C types int, long...

The diff file that makes SdFat compile is:
Code:
diff -rb ArduinoOldVer/SdFat/SdBaseFile.cpp Arduino/libraries/SdFat/SdBaseFile.cpp
335c335
< void SdBaseFile::getpos(fpos_t* pos) {
---
> void SdBaseFile::getpos(FatPos_t* pos) {
1110c1110
<   fpos_t pos;
---
>   FatPos_t pos;
1794c1794
< void SdBaseFile::setpos(fpos_t* pos) {
---
> void SdBaseFile::setpos(FatPos_t* pos) {
diff -rb ArduinoOldVer/SdFat/SdBaseFile.h Arduino/libraries/SdFat/SdBaseFile.h
32c32
<  * \struct fpos_t
---
>  * \struct FatPos_t
36c36
< struct fpos_t {
---
> struct FatPos_t {
41c41
<   fpos_t() : position(0), cluster(0) {}
---
>   FatPos_t() : position(0), cluster(0) {}
201c201
<   void getpos(fpos_t* pos);
---
>   void getpos(FatPos_t* pos);
205c205
<   void setpos(fpos_t* pos);
---
>   void setpos(FatPos_t* pos);
diff -rb ArduinoOldVer/SdFat/SdStream.h Arduino/libraries/SdFat/SdStream.h
122c122
<   void getpos(fpos_t* pos) {SdBaseFile::getpos(pos);}
---
>   void getpos(FatPos_t* pos) {SdBaseFile::getpos(pos);}
138c138
<   void setpos(fpos_t* pos) {SdBaseFile::setpos(pos);}
---
>   void setpos(FatPos_t* pos) {SdBaseFile::setpos(pos);}
185c185
<   void getpos(fpos_t* pos) {SdBaseFile::getpos(pos);}
---
>   void getpos(FatPos_t* pos) {SdBaseFile::getpos(pos);}
193c193
<   void setpos(fpos_t* pos) {SdBaseFile::setpos(pos);}
---
>   void setpos(FatPos_t* pos) {SdBaseFile::setpos(pos);}
diff -rb ArduinoOldVer/SdFat/bufstream.h Arduino/libraries/SdFat/bufstream.h
61c61
<   void getpos(fpos_t *pos) {
---
>   void getpos(FatPos_t *pos) {
72c72
<   void setpos(fpos_t *pos) {
---
>   void setpos(FatPos_t *pos) {
diff -rb ArduinoOldVer/SdFat/istream.cpp Arduino/libraries/SdFat/istream.cpp
70c70
<   fpos_t pos;
---
>   FatPos_t pos;
143c143
<   fpos_t endPos;
---
>   FatPos_t endPos;
231c231
<   fpos_t pos;
---
>   FatPos_t pos;
264c264
<   fpos_t endPos;
---
>   FatPos_t endPos;
323c323
<   fpos_t pos;
---
>   FatPos_t pos;
384c384
<   fpos_t pos;
---
>   FatPos_t pos;
407c407
<   fpos_t pos;
---
>   FatPos_t pos;
diff -rb ArduinoOldVer/SdFat/istream.h Arduino/libraries/SdFat/istream.h
138a139
>
144c145
<   istream &operator>>(int16_t& arg) {
---
>   istream &operator>>(int& arg) {
153c154
<   istream &operator>>(uint16_t& arg) {
---
>   istream &operator>>(unsigned int& arg) {
162c163
<   istream &operator>>(int32_t& arg) {
---
>   istream &operator>>(long& arg) {
171c172
<   istream &operator>>(uint32_t& arg) {
---
>   istream &operator>>(unsigned long& arg) {
256c257
<   int16_t getch(fpos_t* pos) {
---
>   int16_t getch(FatPos_t* pos) {
264c265
<   virtual void getpos(fpos_t* pos) = 0;
---
>   virtual void getpos(FatPos_t* pos) = 0;
271c272
<   virtual void setpos(fpos_t* pos) = 0;
---
>   virtual void setpos(FatPos_t* pos) = 0;
diff -rb ArduinoOldVer/SdFat/ostream.cpp Arduino/libraries/SdFat/ostream.cpp
158a159,160
>   putStr(str);
>   /*
164a167
>   */
diff -rb ArduinoOldVer/SdFat/ostream.h Arduino/libraries/SdFat/ostream.h
137c137
<   ostream &operator<< (int16_t arg) {
---
>   ostream &operator<< (int arg) {
145c145
<   ostream &operator<< (uint16_t arg) {
---
>   ostream &operator<< (unsigned int arg) {
153c153
<   ostream &operator<< (int32_t arg) {
---
>   ostream &operator<< (long arg) {
161c161
<   ostream &operator<< (uint32_t arg) {
---
>   ostream &operator<< (unsigned long arg) {

I suggest you try the bench example.  Remove this line to get it to compile:
Code:
  cout << pstr("Free RAM: ") << FreeRam() << endl;
899  Development / Other Hardware Development / Re: Teensy 3.0 on: October 08, 2012, 09:45:14 am
This beta of SdFat uses the above functions: SdFatBeta20120825.zip http://code.google.com/p/beta-lib/downloads/list.

The functions are at the top of Sd2Card.cpp

It also has the following function to initialize AVR SPI pins:
Code:
/**
 * initialize SPI pins
 */
static void spiBegin() {
  pinMode(MISO, INPUT);
  pinMode(MOSI, OUTPUT);
  pinMode(SCK, OUTPUT);
  // SS must be in output mode even it is not chip select
  pinMode(SS, OUTPUT);
  // set SS high - may be chip select for another SPI device
#if SET_SPI_SS_HIGH
  digitalWrite(SS, HIGH);
#endif  // SET_SPI_SS_HIGH
}

This version of SdFat does not have the new stuff to speed up large reads and writes.  That involves changes to use multi-block SD commands.
900  Development / Other Hardware Development / Re: Teensy 3.0 on: October 08, 2012, 08:45:16 am
Paul, I am doing a major redesign of SdFat to use better caching and faster SD commands so large writes/reads will be much faster.

I plan to support SPI and 4-bit SDIO on various Cortex M chips.  I also want to make SdFat RTOS friendly when using DMA.

I would love to have a better low level SPI library for each chip.

I need a way to restore the SPI speed and mode each time I access the SD.  I need single byte read and write for sending commands, receiving status, and polling for busy.

I need fast block read and write routines.  These could use a fifo or DMA.

I am ready to start testing with some prototype SPI functions I have done but for some reason my Teensy 3.0 has not arrived in California yet.

Edit:  I need the equivalent of these AVR functions.
Code:
//------------------------------------------------------------------------------
/**
 * Initialize hardware SPI
 * Set SCK rate to F_CPU/pow(2, 1 + spiRate) for spiRate [0,6]
 */
static void spiInit(uint8_t spiRate) {
  // See avr processor documentation
  SPCR = (1 << SPE) | (1 << MSTR) | (spiRate >> 1);
  SPSR = spiRate & 1 || spiRate == 6 ? 0 : 1 << SPI2X;
}
//------------------------------------------------------------------------------
/** SPI receive a byte */
static uint8_t spiRec() {
  SPDR = 0XFF;
  while (!(SPSR & (1 << SPIF)));
  return SPDR;
}
//------------------------------------------------------------------------------
/** SPI read data - only one call so force inline */
static inline __attribute__((always_inline))
  void spiRead(uint8_t* buf, uint16_t nbyte) {
  if (nbyte-- == 0) return;
  SPDR = 0XFF;
  for (uint16_t i = 0; i < nbyte; i++) {
    while (!(SPSR & (1 << SPIF)));
    uint8_t b = SPDR;
    SPDR = 0XFF;
    buf[i] = b;
  }
  while (!(SPSR & (1 << SPIF)));
  buf[nbyte] = SPDR;
}
//------------------------------------------------------------------------------
/** SPI send a byte */
static void spiSend(uint8_t b) {
  SPDR = b;
  while (!(SPSR & (1 << SPIF)));
}
//------------------------------------------------------------------------------
/** SPI send block - only one call so force inline */
static inline __attribute__((always_inline))
  void spiSendBlock(uint8_t token, const uint8_t* buf) {
  SPDR = token;
  for (uint16_t i = 0; i < 512; i++) {
    uint8_t b = buf[i];
    while (!(SPSR & (1 << SPIF)));
    SPDR = b;
  }
  while (!(SPSR & (1 << SPIF)));
}
Pages: 1 ... 58 59 [60] 61 62 ... 112