RamDisk - a file system library for RAM devices like 23LCV1024

fat16lib · March 29, 2014, 6:04pm

Update: An updated version, 20140429, of RamDisk is attached to this post.

I have attached a very early version of a FAT like file system for external RAM devices.

I wish to thank pito for encouraging me to look into a file library for devices like his 8 MB RAM device.

I have been testing with a 23LCV1024 1Mbit SPI Serial SRAM with Battery Backup. This 8 pin DIP with a small coin cell provides 128 KB of file storage.

Three example sketches are included with the library.

Please try the library and post any comments.

Here is the readme file.

This is a very early version of a RamDisk library. It provides a FAT style
file system on an external RAM device.

RamDisk is incomplete and likely has bugs. I am releasing this version so
future users can provide comments and input.

Please post any comments in the Arduino forum topic where this file
was posted.

This library was developed using a Microchip 23LCV1024 1 Mbit SPI
serial SRAM with battery backup.

You must have working 23LCV1024 or 23LC1024 hardware or write your own
RAM device library. You should be able to use other RAM devices by
writing a library with the RamBaseClass API.

Please read the html documentation. Go to the Classes tab and look
at the RamDisk and RamBaseClass.

You must install three libraries to try the examples included with RamDisk.

The RamDisk folder contains file-system code and three examples.

The 23LCV1024 folder contains the raw SRAM driver.

The SdFat folder contains a version of SdFat that can coexist with RamDisk.

There are three examples that demonstrate RamDisk features.

Start with HelloRamDisk.ino. This example writes two lines of text to
a file and then reads the file and displays the text on the serial monitor.

Here is the output from HelloRamDisk:

Format (Y/N): N
1: Hello RamDisk!
2: Line to test fgets

RamDiskLoggerTest.ino will log 12,000 samples to RAM as fast as possible.
each sample consists of value from five analog pins. A total of 60,000
values are written to a binary file. Timing statistics and the first ten
samples in the log file are printed.

Here is output with pins 0, 2, 4 grounded. Pin 1 is 5 V and pin 3 is 3.3 V.

Format (Y/N): Y
clusterCount: 250
Starting test
t: 7.53 sec
minMicros: 60
maxMicros: 216
avgMicros: 66
sync micros: 156
filesize: 120000

pin0,pin1,pin2,pin3,pin4
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
0,1023,0,676,0
Done

RamDiskToSdFat.ino creates a file with 1000 text lines in RAM. It then
copies the text file to an SD file.

Here is the output from RamDiskToSdFat:

FreeRam: 1092
Format (Y/N): Y
Writing ramFile
TEST.CSV 2000-01-01 01:00:00 16746
Copying ramFile to sdFile
BENCH.DAT 2000-01-01 01:00:00 4999680
TEST.TXT 2000-01-01 01:00:00 16743
TEST.CSV 2000-01-01 01:00:00 16746
Done

RamDisk20140329beta.zip (396 KB)

RamDisk20140429beta.zip (552 KB)

pito · March 29, 2014, 7:04pm

Here is my mod for 8MB ramdisk (lib for M8MBRDSK11 and HelloRamdisk_8MB), coping with how to define fast IO within the lib.
Enclosed my files, pls be so kind and advice me how to tackle the templated signal pins (and bus port) there in the ram class
HelloRamdisk_8MB:

// With 8MB Ramdisk, atmega1284p
// Fast IO is required with bitbanging here
#include <DigitalIO.h>
DigitalPin<12> NRD(OUTPUT);    // /RD active LOW
DigitalPin<13> NWR(OUTPUT);    // /WR active LOW
DigitalPin<14> NDATA(OUTPUT);   // /Data active LOW
// PortC is the data port

#include <M8MBRDSK11.h>
#include <RamDisk.h>

M8MBRDSK11 ram;
RamDisk file;
char buf[40];
..

Also, I need to increment the *buf while shooting bytes in/out:

C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp: In member function 'virtual bool M8MBRDSK11::read(uint32_t, void*, size_t)':
C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp:81: error: ISO C++ forbids incrementing a pointer of type 'void*'
C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp:81: error: 'void*' is not a pointer-to-object type
C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp: In member function 'virtual bool M8MBRDSK11::write(uint32_t, const void*, size_t)':
C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp:88: error: ISO C++ forbids incrementing a pointer of type 'const void*'
C:\MyCode\Arduino\libraries\M8MBRDSK11\M8MBRDSK11.cpp:88: error: 'const void*' is not a pointer-to-object type

This is what I need (as discussed offline):

..
bool M8MBRDSK11::read(uint32_t address, unsigned char *buf, unsigned int nbytes) {
..
bool M8MBRDSK11::write(uint32_t address, unsigned char *buf, unsigned int nbytes) {

where
address is the starting address of a "block"
*buf is the pointer to buf[nbytes]
nbytes is the number of bytes in the block, ie 512

M8MBRDSK11.zip (3.27 KB)

fat16lib · March 29, 2014, 8:22pm

This compiles. maybe.zip might be a step forward.

maybe.zip (2.93 KB)

pito · March 29, 2014, 8:34pm

Thanks!
I get now (atmega1284p @16MHz):

Format (Y/N): Y
1: Hello RamDisk!
2: Line to test fgets

Format (Y/N): Y
clusterCount: 250
Starting test
t: 7.32 sec
minMicros: 40
maxMicros: 120
avgMicros: 41
sync micros: 96
filesize: 120000

pin0,pin1,pin2,pin3,pin4
226,208,165,186,153
206,221,206,216,200
213,228,224,230,224
225,237,239,245,244
239,243,242,243,243
238,239,239,240,239
234,234,234,234,234
230,229,227,228,227
224,223,221,218,216
213,211,209,208,207
Done

pito · March 29, 2014, 8:48pm

With

//for (int j = 0; j < 5; j++) data[j] = analogRead(j);

Format (Y/N): Y
clusterCount: 250
Starting test
t: 0.57 sec
minMicros: 40
maxMicros: 124
avgMicros: 42
sync micros: 92
filesize: 120000

pin0,pin1,pin2,pin3,pin4
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
Done

And the same with:

if (RamDisk::format(&ram, 16384, 4, 1)) break;

Format (Y/N): Y
clusterCount: 16315
Starting test
t: 38.98 sec
minMicros: 40
maxMicros: 124
avgMicros: 42
sync micros: 92
filesize: 8000000

pin0,pin1,pin2,pin3,pin4
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
Done

fat16lib · March 29, 2014, 9:01pm

Try running the logger a second time without fomatting. It will just rewrite the existing file and the maxMicros should be less.

It just needs to fetch the location of the next cluster, not allocate it.

In the future I will optimize contiguous files. then all writes will take the same time.

pito · March 29, 2014, 9:12pm

It will just rewrite the existing file and the maxMicros should be less.

Starting test
t: 37.89 sec
minMicros: 40
maxMicros: 80
avgMicros: 41
sync micros: 4
filesize: 8000000

pito · March 29, 2014, 9:22pm

Is it possible to run the "big tests" from SdFat lib?

fat16lib · March 29, 2014, 9:35pm

Just open a file on an SD instead of on the RAM.

SdFat sd;
SdFile file;

....

  if (!sd.begin(SD_CS_PIN)) sd.errorHalt();
// rest the same.
  if (!file.open("TEST.BIN", O_CREAT | O_RDWR)) {
    Serial.println(F("open fail"));
    return;
  }

fat16lib · March 29, 2014, 9:45pm

Here are SdFat times.

Starting test
t: 69.83 sec
minMicros: 40
maxMicros: 61004
avgMicros: 81
sync micros: 2176
filesize: 8000000

minMicros is just the time to copy data to the cache.

fat16lib · March 29, 2014, 9:50pm

SdFat rewrite is bad since you must read a block into the cache update it and write it.

Starting test
t: 107.11 sec
minMicros: 40
maxMicros: 62024
avgMicros: 127
sync micros: 8
filesize: 8000000

pito · March 29, 2014, 9:51pm

RamDiskToSdFat_8MB.ino:

FreeRam: 15256
Format (Y/N): Y
Writing ramFile
TEST.CSV      2000-01-01 01:00:00 16664
Copying ramFile to sdFile
TEST.CSV       2000-01-01 01:00:00 16664
Done

Test.csv from the sdcard:

0,Line,0
336,Line,1
728,Line,2
1128,Line,3
1536,Line,4
..
567696,Line,996
568280,Line,997
568860,Line,998
569444,Line,999

graph.jpg.bmp (1.26 MB)

pito · March 30, 2014, 12:54am

A quick hack of sdfat's bench sketch - bench_8MB.ino:

Type any character to start
Free RAM: 15761
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 745.60 KB/sec
Maximum latency: 212 usec, Minimum Latency: 112 usec, Avg Latency: 129 usec

Starting read test.  Please wait up to a minute
Read 745.41 KB/sec
Maximum latency: 172 usec, Minimum Latency: 120 usec, Avg Latency: 128 usec

Done

Not sure how to interpret the latencies here as they depend on the buffer size.. :~

Type any character to start
Free RAM: 11861
File size 5MB
Buffer size 4000 bytes
Starting write test.  Please wait up to a minute
Write 1029.44 KB/sec
Maximum latency: 4120 usec, Minimum Latency: 3816 usec, Avg Latency: 3880 usec

Starting read test.  Please wait up to a minute
Read 992.26 KB/sec
Maximum latency: 4288 usec, Minimum Latency: 3996 usec, Avg Latency: 4025 usec

Done

It seems those are the times for copying to/from buffer, or something like that..

fat16lib · March 30, 2014, 1:12am

Not sure how to interpret the latencies here as they depend on the buffer size..

Latency may not be the correct term. It is just the time to write a buffer so it depends on the buffer size.

These are great times/speeds. The variation is very small compared to the typical SD in single block mode.

pito · March 30, 2014, 8:48am

FYI - this RamDisk filesystem could work nicely with serial SPI FRAM (ie. FM25V10, 256kBytes per chip, 8pin) or serial SPI MRAM (ie. MR25H40, 512kBytes per chip, 8pin) where FRAM and MRAM are nonvolatile.
The performance will be exactly the same as the one with above serial 23LCV1024 SRAM, the driver is basically the same (small change required probably in the addressing sequence).

robtillaart · March 30, 2014, 11:03am

Congratulations with this impressive work!
(waiting for the RAM-SD shield combination)

fat16lib · March 30, 2014, 4:23pm

(waiting for the RAM-SD shield combination)

I have a 23LCV1024 on an Adafruit Data logging shield. This provides 128 KB of NVRAM by using the shield's CR1220 battery to backup the SRAM.

The 23LCV1024 requires about 1 uA and the DS1307 on the shield uses about 0.5 uA.

A CR1220 is good for about 35 mah. At 1.5 uA this is almost 1,000 days.

The 23LCV1024 costs $2.95 from Digi-Key.

pito · March 30, 2014, 7:13pm

FYI (digikey, rlx):
Small comparison of "serial" SPI SRAM/FRAM/MRAM prices (single quantity and per MByte).

pito · April 1, 2014, 9:45pm

Performance (5MB bench test) of the 8MB ramdisk with RamDisk file system - buffer size vs. rd/wr speed vs. "latency - buffer copy time".
Below picture with atmega1284p @16MHz and bitbanging (slow). Will be much better with DUE and its External Memory Bus (8bit static device) - I would guess 11-12MB/sec with DUE is feasible.

Gerry48 · April 12, 2014, 1:16am

Excellent work! I got some 23LC1024 on order. Will give it a try. The SD Card shield is a perfect place for the ram.

Gerry

Topic		Replies	Views
File System for external EEPROM Storage	3	4954	May 6, 2021
8MB Ramdisk (external RAM) for Arduino.. Other Hardware Development	32	43549	May 6, 2021
Problems getting HelloRamDisk running Storage	3	975	May 6, 2021
Adding a second 23LC1024 Storage	16	9143	May 6, 2021
8MB cache (external RAM) Storage	5	2324	May 6, 2021

RamDisk - a file system library for RAM devices like 23LCV1024

Related Topics