How does one buffer audio Writing to an SD card

Using Arduino Storage
1

I have looked at some the posts here and I can't seem to find what I am looking for. I am using an audio record/playback chip, 4 bit W/R ADPCM type, at 8khz sample with a MEGA to store the audio on an SD card. I have 125 us between samples. I know about the SD write latency to a point. I do not know how to handle the SD card to buffer the data. I saw the reference to SdFat lib and my question is, is there a procedure in SdFat lib that would handle the stream of audio data without the coruption. Or, does any one have another idea on how to do this?

Stan K'

2

Opening and closing the file on the SD card takes a certain minimum amount of time. But you don't normally want to open a file in setup() and leave it open forever because then the card will show as corrupted when you take it out of the Arduino with the file left open.

So buffer your data. Create an array with 100 elements. Store the incoming data in that. This is very fast using the Arduino RAM memory. Then when your buffer is full, open the file and tell it to save all 100 items. This works much faster than saving one at a time.

3

So buffer your data. Create an array with 100 elements. Store the incoming data in that. This is very fast using the Arduino RAM memory. Then when your buffer is full, open the file and tell it to save all 100 items. This works much faster than saving one at a time.

This won't work for writing audio to an SD card if you have the tight latency requirement of 125 μs. Many audio chips have buffering and allow longer latency.

You need an interrupt routine or a high priority RTOS thread to read audio from the chip and buffer it. The Arduino loop thread or a low priority RTOS thread writes audio from buffers to the SD. You need several buffers.

If the audio chip uses SPI, you will need software SPI for the chip or use software SPI for SdFat.

I wrote this audio recorder for Arduino about five years ago. It uses the Arduino ADC, not your chip but demonstrates the principles.

This SdFat example demonstrates recording data from an analog pins at high speed.

You must use muli-block streaming to a pre-allocated contiguous file like the above examples. The standard single block write in SdFat can have latencies as long as 100 ms. SD cards only perform well with multi-block writes.

4

Thanks for the replies. I have a lot of these chips and I have used them before so I am sticking with them. I use a polling of the sync pulse, which could be used for an interrupt but it is the only thing happening during record so I am doing this way, see code attachment.

Now, if I write 512 bytes to a buffer. how long does the MEGA take to write to the SD card buffer and what's a typical write time for 512 bytes to flash? Can I write 512 bytes to the card in 125 usecs?

I have looked at fast analog code on github but I haven't bee able to find the relative parts for this issue yet. I will look at it some more, maybe you, fat16lib, could point me to the particular part of the code that does the sd write.

Stan K'

5

Had to reattach the code for prevoius comment.

SDwritecode.txt (2.11 KB)

6

I'll get this post right sooner or later. The program works ok as written but is is not the correct way to do this. It corupts the flow every 512 bytes, 70ms.

  //RV = Record Voice
  //******************************************************
  else if (modeNum == 15)
  {
    VOICEDDR = B11011111;//Write
    byte vdat = 0xff;
    VOICEBUSS = vdat;
    //   VOICEPIN = vdat;
    SD.remove((char*)"Voices/voiceDat.dat");
    voiceDataFile = SD.open((char*)"Voices/voiceDat.dat", FILE_WRITE);
    if (! voiceDataFile)
    {
      Serial.println("error opening file");
      Serial.println("voices/voiceDat.dat");
      // Wait forever since we cant read data
      while (1) ;
    }
    Serial.println("SD ready");
    Serial.println("Press Button to Record");
    voiceSetUp();   
    while (digitalRead(recPlayButton) == 1);
    //Record command
    byte vMonStat = 0;
    voiceBusy();
    digitalWrite(vCE,LOW);
    digitalWrite(vWR,HIGH); 
    VOICEBUSS = 0x03 | B01010000;//Record command
    digitalWrite(vWR,LOW); 
    digitalWrite(vWR,HIGH);
    //External Command
    delay(1);
    VOICEBUSS = 0x0B | B01010000;//Exernal command
    digitalWrite(vWR,LOW); 
    digitalWrite(vWR,HIGH); 
    digitalWrite(vCE,LOW); 
    delayMicroseconds(125);
    VOICEDDR = B11010000;//read
    while (digitalRead(recPlayButton) == 0)
    {
      while (digitalRead(vMon) == 0);//Mon is the sync pulse, waity till rise
      VOICEDDR = B11010000;//Read inputs, four lower bits
      digitalWrite(vCE,LOW);//Chip enable      
      digitalWrite(vRD,LOW);//Write to low
      vdat = VOICEPIN;//Data to buss
      delayMicroseconds(1);
      digitalWrite(vRD,HIGH);//Read buss
      vdat = vdat & B00001111;
      //      delayMicroseconds(100);
      voiceDataFile.write(vdat);//Write data to SD 
      vMonStat = vMonStat + 1;
      while (digitalRead(vMon) == 1);//wait till pulse completes
    }
    //Stop command
    VOICEDDR = B11011111;
    digitalWrite(vWR,HIGH);
    //   byte temp = VOICEBUSS  
    VOICEBUSS = 0x05 | B01010000;//Stop command
    //    Serial.print(VOICEBUSS,BIN);
    digitalWrite(vWR,LOW);
    delayMicroseconds(1);
    digitalWrite(vWR,HIGH);
    digitalWrite(vCE,HIGH); 
    Serial.println("Voice Record done");
    voiceDataFile.flush();
  }
7

You need to do some measurements instead of asking. Write a simple program to write 512 samples to the card 1000 times. Time how long it takes with the millis() or micros() functions. Try it with 1 sample 512,000 times. Try 2 samples 256,000 times.

Then you will know what your hardware is capable of and you can start looking for optimising your code to suit.

8

You need to do some measurements instead of asking.

The measurements for many cards are here.

A typical write test for a high quality SanDisk card:

File size 5 MB
Buffer size 512 bytes

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
284.82,25620,1512,1791
295.47,10644,1532,1727
295.70,10652,1600,1725
295.35,10636,1600,1727
295.28,22104,1540,1728
295.49,10640,1600,1726
295.52,10640,1596,1726
295.52,10640,1536,1726
295.49,10628,1600,1726
295.26,22008,1600,1728

The minimum time is over 1,500 μs the max is over 10,000 μs.

You have zero probability of achieving 125 μs latency with a simple single threaded Arduino program.

You must either use an audio chip with much more buffering or change the software design to overlap reading from the audio chip and writing a block to the SD card.

This is my last post on your problem since there are dozens of posts on this topic in this forum.

9

Thanks for your help, I'll test the arduino write speed with my virtual buss and see how many times I can write in 125usecs. I suppose I could squeeze 32 writes from a buffer in that period and do it while the interrupt is filling the buffer. I think I need to find more about how the SD card handles the writes to it while it is flashing a block. I'll figure it out.

Stan

10

Hello again,

I have one more question, I've looked at many different sketches and similar pieces of code that deal with this and I can't figure out how you know when to write to the SD card. I've spent many hours trying to resolve this on my own which I normally do, but I am stuck. I'm looking a double buffer function that seems to things the way I need too, but I can't figure when to write the buffer to the SD. The way I understand, when it is flashing the memory it can't read incoming data. How do I know this ? I'm missing something here. Does the SD function deal with this issue?

Stan K

11

The way I understand, when it is flashing the memory it can't read incoming data. How do I know this ? I'm missing something here. Does the SD function deal with this issue?

There are many layers of buffering in the SD library and in the SD card. There is no way you can tell when the flash in the card is written. This is why I don't do file system writes when recording data like audio.

I wrote the first version of SdFat over six years ago and added features for fast data recording. The Arduino SD.h library is just a wrapper for an old version of SdFat so it works the same way.

If you do file system writes, the following happens. If the current write starts a new cluster, a search of the FAT (File Allocation Table) is done to find a free cluster. This search reads 512 byte blocks from the FAT and when a free cluster is found, the block is updated and written back to the FAT. This can take a long time if the SD is fragmented.

The data you are writing is then copied to an internal 512 byte block buffer in the SD library. When the buffer is full, the block is written to the SD.

The SD has a very complex RISC controller that buffers data and emulates 512 byte flash pages. The actual flash pages are usually some multiple of 16 KiB. Unfortunately the single block writes used for file system writes cause an entire 16 KiB block to be programed so there is lots of data buffering and copying in the SD. This can take a long time.

The the controller also maintains a pool of erased flash. Flash erase groups are usually larger than 128 KiB so this can also cause random delays.

Finally the controller does flash wear leveling so it may decide to move and remap a internal flash regions which will take a long time.

SD cards have a streaming mode that provides a more predictable access. You can inform the SD card controller that you intend to write a large contiguous block of flash and it will optimize access to this block. You transfer successive 512 byte blocks and the SD controller buffers these in RAM until a full flash page is accumulated and then programs the flash.

This is the method used by the AnalogBinLogger example. I create a large contiguous file and do raw multi-block writes to the file using 512 byte block buffers. When I finish logging data, I truncate the file to the correct size.

You might want to look at the RawWrite example. It shows how to create and write to a contiguous file with multi-block writes but does not demonstrate the buffering technique used in the AnalogBinLogger example.

If you have sufficient buffer you may get file writes to work. I think you are using a Mega 2560 or a 2561. The SD spec limits write latency for 512 byte writes to 250 ms. Unfortunately this is 2000 times the 125 μs you were looking for. Many modern cards do better than the 250 ms max.

12

Thanks fat16lib,

You comment was very useful, I appreciate your detailed response. I have already down loaded your examples and will use them as a guide line. I think I will use 512 byte buffers. I will post the results when I get it working. There is a lot more to SD cards than meets the eye. I took a look at them technically years ago, more complicated than I needed to know at the time, but I didn't need to write anything fast so I didn't dig very deep into their operation.

Thanks again,

Stan K'

13

I now realize you really shouldn't have a problem since 4 bit W/R ADPCM at 8 kHz is really slow.

I am curious why you are doing this? Just a voice recorder?

Does your chip do something that popular chips like the VS1003 won't do? These chips have huge FIFOs.

Popular chips claim to record 8 kHz IMA ADPCM audio but I don't know what this means.

I just know people use these chip to record audio with SdFat, probably in some other format.

Here are typical modules people use:

Adafruit

WaveShare

14

Hi Again,

Well, I am using these chips because I have a bunch of them, like 500. I is an old OKI semi part M6588. It is mono an I have used it before and it works just fine for plain Jane audio. I have thought of using a different part and I just may do that but I thought I'd try this with the SD card. I want more storage so I figured an SD card would work. I just haven't dug very deep into SD cards before so this is where my problems have risen. I am building a project that uses a GSM modem and and other stuff for signaling, voice and SMS. That's pretty much it.

Stan K'