Show Posts
Pages: [1] 2 3 ... 111
1  Using Arduino / Storage / Re: Changing pins on SD card library? on: Today at 03:06:43 pm
The LC studio card is not recommended for use with Uno.  This SD module has no 5V -> 3.3V level shifters.  It works with some SD cards but stresses these cards and the Uno by attempting to apply 5V to SD pins.  This causes the SD card to draw high currents through over-voltage protection circuits in the SD.

You can share the SPI bus with multiple devices so it's not clear you need to use other pins. 

The Uno has only one SPI controller but it is possible to use software SPI on other pins with either SdFat https://github.com/greiman/SdFat or the Adafruit SD.h library.  SdFat is faster since software SPI pins must be defined as constants.  The Adafruit SD.h library allows the pin to be defined at run time but is slower.
2  Using Arduino / Storage / Re: Do you want Long File Names in SdFat? on: Today at 02:52:21 pm
Quote
I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser

This is not true. You can not guess how diverse the apps are that use SD cards on Arduino.  Over 50,000 users have downloaded SdFat and I often hear from users with requests for new features.  SD.h is a wrapper for a simple early version of SdFat.  SdFat has evolved a great deal on the basis of these user requests.

I looked at you app and it is nice but it makes very simple use of files so I agree you don't need more.  I got your message you don't need LFN!

I have helped a number of users with minimal access to long file names and some users need more.   I want to hear from more users that need LFN so I can decide what to implement.

I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems.  No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.

SdFat is used in very sophisticated apps on Teensy 3.1 and Due.  These boards have lots of memory and developers want more features.

The most popular embedded File System, FatFs, provides full Long File Name Support in ANSI/OEM code pages or Unicode.  This includes Japanese Shift-JIS, Simplified Chinese, Korean EUC-KR, and Traditional Chinese.  I plan a far simpler implementation.
3  Using Arduino / Storage / Re: Do you want Long File Names in SdFat? on: September 14, 2014, 12:04:30 pm
Quote
If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.

Yes, your app has very simple filename usage. 8.3 file names have been good enough for a long time.  I think CP/M was the first OS with 8.3 filenames in 1973-1974.  The first FAT long filenames  were in Windows NT 3.5 in 1994.

CP/M 8.3 was an improvement over 6.3 in DEC TOPS-10,  RSX-11/M and early VAX VMS.  Valid 6.3 characters were 0-9 A-Z $ and %. filenames were encoded in RADIX-50, which packed those nine characters into only three 16-bit words (six bytes) http://en.wikipedia.org/wiki/DEC_Radix-50.   I thought 6.3 was good enough with DEC machines.

I am looking for examples where LFN support is needed or desired and the reason.

Edit: Here are the ugly details of encoding DEC filenames.
Quote
Encoding

All Radix-50 strings must be a multiple of three characters long and should be padded by spaces at the end if necessary. To encode the character sequence “xyz” into the 16-bit word e the following formula should be used:

    e = (((x × 40) + y) × 40) + z
4  Using Arduino / Storage / Re: Do you want Long File Names in SdFat? on: September 13, 2014, 12:02:03 pm
Quote
Files created by Arduino : LFN not needed.
But I would like to open files created by a computer with a long name.
This sounds like a good first step.  I already have a prototype working.  The prototype supports 7-bit characters, the original ASCII http://en.wikipedia.org/wiki/ASCII.  Extending to 8-bit SBCS code pages or UTF-16 adds a huge amount of complexity.

The the extra characters in standard U.S. Microsoft code page 437 are now anarchic http://en.wikipedia.org/wiki/Code_page_437.

UTF-16 implementations take over 100 KB of flash http://elm-chan.org/fsw/ff/en/appnote.html.

Quote
are there other file formats for SD cards e.g. those from linux that could be applied to an SD that are faster in general?
Can an SDcard have partitions?
Long File Names do not change the speed of I/O.  Just the time to open/create files due to the directory complexity.

Linux/Unix file systems are way too complex for Arduino and would not be faster on SD cards.  The flash structure of SD cards is designed for FAT file systems.

The fastest way to access a file on an SD is to create a contiguous file and access it using multi-block raw SD reads and writes.  SD cards are fast only if you use very large multi-block transfers since the flash pages are now some multiple of 16 KB.

See the SdFat LowLatencyLogger example for an implementation https://github.com/greiman/SdFat/blob/master/SdFat/examples/LowLatencyLogger/LowLatencyLogger.ino.

Quote
As far as I know, today most memory sticks/cards have a partition table with a single partition. So you you can have many partitions, just as with a harddisk.
The base code in SdFat allows four partitions on an SD card or a single floppy style volume.  The SD standard now only permits a single partition with a MBR so this feature is hidden.

5  Using Arduino / Storage / Do you want Long File Names in SdFat? on: September 13, 2014, 10:18:40 am
I would appreciate comments on the need for LFN, Long File Names, in SdFat.

I have not implemented LFN because of the complexity, amount of flash, and RAM required.  I am now looking at making LFN a configuration option so the default size of SdFat would not increase.

LFN uses UTF-16 characters but I only plan on supporting an 8-bit character set.

LFN is slow and uses lots of memory.
Quote
The long file name system allows a maximum length of 255 UTF-16 characters, including spaces and non-alphanumeric characters.  This is achieved by chaining up to 20 directory entries of 13 2-byte unicode characters each.

Allowing each portion of a path name to be 255 characters long requires lots of memory.  Some implementations of FAT for embedded systems limit the maximum length of a file/directory name.  I plan to provide a configuration parameter so small memory Arduinos like Uno can use limited LFN.

Please comment on on this plan.
6  Using Arduino / Storage / Re: How can I dump a large serial stream strait to an SD card? on: September 12, 2014, 10:32:30 pm
Quote
that way the receiver program which is running on the Arduino knows when to stop and close the file.

Most Serial logging applications don't know the file size at the beginning.  You generally wrap data packets with a simple protocol to indicate EOF and other info.

There are lots of ways to solve the  EOF problem.

If there is just one file, you can just call flush or sync every time there is a gap in the data.  This is the same as closing the file so you can stop the receiver when ever the sender is done and not lose data.
7  Using Arduino / Storage / Re: How can I dump a large serial stream strait to an SD card? on: September 12, 2014, 05:39:58 pm
Quote
I do agree the variable size was a bit excessive.

It's not the size but why write the size you expect a file to be at the start of a file?  I didn't see a requirement for this and was just curious why it is needed.  If the file is always the same size, you don't need to write the size.  If the size varies, when do you know the size?

You can always get the real size of a file on any OS.

For Arduino SD.h
Code:
 file.size()

With SdFat
Code:
 file.fileSize()

Linux, OS X, and Windows have similar functions.
8  Using Arduino / Storage / Re: Using an SD card with Arduino Mega on: September 11, 2014, 01:31:55 pm
The SPI pins on the Mega are in two location, either the six pin ISP connector in the middle of the Mega or pins 50-53 at the end of the Mega.

Many shields that work on Uno such as the Adafruit data logging shield will only work on Mega with software SPI which is slow.

Arduino shields, such as the Ethernet shield, have a SD socket and connect to the ISP connector.  The Seeed v4.0 also uses the ISP http://www.seeedstudio.com/depot/SD-Card-Shield-V4-p-1381.html

Some users connect the shield's SPI pins to the Mega SPI pins using extra wires/jumpers.

Any breakout module with level shifters should work.  Level shifters convert 5V Arduino SPI signals to 3.3V SD signals.

9  Using Arduino / Storage / Re: How can I dump a large serial stream strait to an SD card? on: September 11, 2014, 10:06:51 am
Quote
First send the file size as an unsigned 32-bit integer

Why?  How do you know how big the file will be?

If the data rate is slow enough you can just read from Serial and write to the file.  The Serial stream has  64 byte input and output buffers.  This will work if the the rate is under about 640 bytes per second.

If the rate is higher, you may overrun the buffer since SD write latencies can be 100 ms or more.

I wrote a special Serial library and logger for higher speeds https://github.com/greiman/SerialPort/tree/master/SerialPortLogger.

10  Using Arduino / Storage / Re: Difference between SD Card and Micro SD Card? on: September 11, 2014, 09:54:23 am
As SuferTim posted while I wrote this:

The resistors on the LC Studio card are not level shifters, just pull to 3.3V.

The full 5V Arduino signals are applied to MOSI, SCK, and SDCS.  This works on some cards because the input protection on the card causes a lot of current to be drawn and the signals never reach 5V.  You may be exceeding the max pin current of 40 ma for arduino pins.

You may damage cards and often the cards will be hot since you are stressing them beyond spec.

So the difference is a random draw of microSD cards that will tolerate the poor design of the LC Studio card and standard cards that fail with the LC Studio card.
11  Using Arduino / Storage / Re: MicroSD Weirdness on: September 06, 2014, 08:58:50 pm
Quote
In this older post, you recommend not using flush(), so I'm even more confused now...

I only suggested calling flush instead of close/open which you proposed.  People sometimes call flush after each data record if they fear a crash or power failure.  These flush calls are slow but not nearly as slow as a close followed by an open.

The fastest way to write a file is to open the file once in setup, write data to the file with any number of print and write statements, and then close the file when you are done logging data.

Quote
What if I'm willing to bet that the program won't crash?  Do I still have to put in a dataFile.close() in there to save data, or does the program automatically do that?

You must call close since there is no way to know when to automatically close a file on Arduino.  

Once again, all data in the file will be lost if you do not close the file.  The close call updates the directory entry for the file to indicate which clusters contain data and the total byte count in the file.

If you do not close the file the file data will be in "lost clusters".  A scan of the file system on a PC will recover these clusters but will not repair the file which was not closed.
12  Using Arduino / Storage / Re: MicroSD Weirdness on: September 06, 2014, 01:20:13 pm
Quote
  I've also read that the SD library has an automatic flush when it reaches 512 bytes, but it shouldn't be relied on because it may fail occasionally, resulting in lost data.

No the "automatic write" does does not fail occasionally.   You will not lose data if you close the file when you are done.  

Do not close and reopen the file.  A call to open can be very costly since a linear search of the entire directory may be required.  A call to flush() is the same as a close followed by an open but minimizes SD accesses.  You only need to call flush() if you fear your program will crash or lose power before you close the file. Data written after the flush call will be lost if the file is not closed.

Close and flush do more than writing the block cache, these calls also update the directory entry which is necessary since the file size is stored in the directory entry. 

Counting 512 byte intervals is of no value in saving data, all data written before a flush/close call is saved.  If you don't close a file, all data written after the last open or flush is lost.
13  Using Arduino / Storage / Re: MicroSD Weirdness on: September 05, 2014, 05:24:24 pm
Opening and closing the file will take a long time and use a lot of power.  Also SD.h may not allow the SD to go into low power sleep mode.  SD.h is based on a very old version of SdFat and has some problems for low power use.

High power use only occurs while a 512 byte block is written to the SD.  If you open a file once and only call flush() when you want to insure data is written to the SD, you will save power.

There are a number of posts in this forum that measure SD power use.

14  Using Arduino / Storage / Re: SdFat update on GitHub on: September 05, 2014, 03:30:02 pm
I posted a beta of SdFat with a faster printField() for the SdFile and SdBaseFile classes https://github.com/greiman/SdFat-beta.

I also added a member function for float:
Code:
  int printField (float value, char term, uint8_t prec = 2);

printField() is about four times faster than println(). Here are results from the PrintBenchmark example comparing printField() with println():
Quote
Test of println(uint16_t)
Time 6.73 sec
File size 128.89 KB
Write 19.15 KB/sec
Maximum latency: 5168 usec, Minimum Latency: 172 usec, Avg Latency: 328 usec

Test of printField(uint16_t, char)
Time 1.59 sec
File size 128.89 KB
Write 80.91 KB/sec
Maximum latency: 72400 usec, Minimum Latency: 44 usec, Avg Latency: 72 usec

Test of println(uint32_t)
Time 9.91 sec
File size 200.00 KB
Write 20.17 KB/sec
Maximum latency: 5532 usec, Minimum Latency: 464 usec, Avg Latency: 488 usec

Test of printField(uint32_t, char)
Time 2.07 sec
File size 200.00 KB
Write 96.67 KB/sec
Maximum latency: 73676 usec, Minimum Latency: 72 usec, Avg Latency: 95 usec

Test of println(float)
Time 12.88 sec
File size 149.00 KB
Write 11.57 KB/sec
Maximum latency: 5652 usec, Minimum Latency: 548 usec, Avg Latency: 636 usec

Test of printField(float, char)
Time 3.41 sec
File size 149.00 KB
Write 43.64 KB/sec
Maximum latency: 5272 usec, Minimum Latency: 136 usec, Avg Latency: 163 usec

Done!

This beta also has software SPI for Teensy 3 and Due.  See the SoftwareSPI.txt file.

SPI transactions have been added. See SPI_Transactions.txt.
15  Using Arduino / Storage / Re: Writing data to csv-file on SD-Card fails on: September 03, 2014, 06:42:11 pm
The old SdFat in SD.h may still have SPI sharing problems.  It should be OK in your tests where you do not access the EEPROM.

To verify you have no SPI problems do the following.

Clone or download SdFat https://github.com/greiman/SdFat and install it.

Edit SdFatConfig.h and change this symbol from 0 to 2.
Code:
/**
 * To enable SD card CRC checking set USE_SD_CRC nonzero.
 *
 * Set USE_SD_CRC to 1 to use a smaller slower CRC-CCITT function.
 *
 * Set USE_SD_CRC to 2 to used a larger faster table driven CRC-CCITT function.
 */
#define USE_SD_CRC 0

Run the bench example https://github.com/greiman/SdFat/tree/master/SdFat/examples/bench.

This will verify SPI data transfers using the standard 16-bit SD CRC.

You can also format SD cards using the SdFormatter example https://github.com/greiman/SdFat/blob/master/SdFat/examples/SdFormatter/SdFormatter.ino.

Linux, OS X, and Windows do not conform to the SD standard.   SdFormatter uses proper cluster sizes and alignment so file system boundaries match flash chip programming and erase boundaries.
Pages: [1] 2 3 ... 111