MicroSD stops writing after 2^16 records

Projects Programming
1

This code works fine writing and reading back uint16_t data but it stops writing after 2^16 records. Arduino accepts the pre_allocate only after the file is opened which suggests a problem that I can't correct. It also issues a warning " 448 | #warning File not defined because __has_include(FS.h)". Using Pico RP2040 on Philhower core with Win11 Arduino IDE and terabyte card. Any suggestions?

// sample2uSD.ino
// Demonstrates two buffers and two processors alternately sampling and storing to uSD
// 12 bit sampling A0 at 210,000 samples/sec, storing to uSD much faster.  
// Larry Gorham
// This code is public domain
//
#include <SPI.h>

#define BUFFSIZE 512 //  Size of single buffer.
#define UPDATES 129     //5
#define invert(r)  (r+1)%2

int buff0[BUFFSIZE] = {0};
int buff1[BUFFSIZE] = {0};
uint16_t flip = 0;
uint16_t flop = 0;
uint32_t last;
uint32_t tim;
uint32_t fl = 0;
float approxBuffRate = (float) BUFFSIZE/.22; // sampling rate microseconds
uint16_t printBuff[UPDATES*BUFFSIZE] = {0};
uint16_t writeBuff[BUFFSIZE] = {0};

//----------  SD  Configs  --------------------------
#include "SdFat.h"
#include "sdios.h"
#include "FreeStack.h"
#include <stdio.h>
#include "pico/stdlib.h"
#include "hardware/gpio.h"

//  SD_FAT_TYPE = 0 for SdFat/File as defined in SdFatConfig.h,
//  1 for FAT16/FAT32, 2 for exFAT, 3 for FAT16/FAT32 and exFAT.
#define SD_FAT_TYPE 2

const uint8_t SD_CS_PIN = SS;

#define SPI_CLOCK SD_SCK_MHZ(25) // 50
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SPI_CLOCK)
#define error(dum)   Serial.printf("%s\n", dum);
#define GPIO16pad ((uint32_t*)0x4001c044)   //  pin 21
#define GPIO19pad ((uint32_t*)0x4001c050)   //  pin 25

// Set PRE_ALLOCATE true to pre-allocate file clusters.
const bool PRE_ALLOCATE = true;

const uint64_t MAX_FILE_SIZE = (pow(2,32) - 1);  

//========================  End SD Configs  =================================================

SdExFat sd;
ExFile file;

//**********************************************************************

void setup() {

Serial.begin(115200);
 // Wait for USB Serial
while (!Serial) {
  yield();
}

  delay(1); 
 

  if (!ENABLE_DEDICATED_SPI) {
    Serial.printf("\nSet ENABLE_DEDICATED_SPI nonzero\n");
  }


  if (!sd.begin(SD_CONFIG)) {
    sd.initErrorHalt(&Serial);
  }



  // open or create file - truncate existing file.
  if (!file.open("bench.dat", O_RDWR | O_CREAT | O_TRUNC)) {
    error("open failed");
  }


  if (PRE_ALLOCATE) {
    if (!file.preAllocate((uint64_t)MAX_FILE_SIZE)) {
      error("preAllocate failed");
    }
  }

  *GPIO16pad = 0x46;  //   sets drive strength to 2mA
  *GPIO19pad = 0x46;

    uint16_t temp = 0;
  for (uint32_t iUp = 0 ; iUp <  UPDATES ; iUp++){


    while (flip == flop){
      asm volatile (" nop\n"); //  smallest delay
    }


  for (uint32_t j=0;j<BUFFSIZE;j++){
    temp = iUp*BUFFSIZE + j;

    if (flip == 1){
      //printBuff1[temp] = (uint16_t) buff0[j];
      writeBuff[j] = (uint16_t) buff0[j];
    }else{
      //printBuff1[temp] = (uint16_t) buff1[j];
      writeBuff[j] = (uint16_t) buff1[j];
    }

  }


  //   ********************  writing to uSD ***************************
  if (file.write(writeBuff, BUFFSIZE*2) != BUFFSIZE*2) {  //  writes in bytes
    error("write buff failed");
  }
    flop = invert(flop);
  }


    //  *********************    read what was written to  uSD **********************
  file.rewind();

  for (uint32_t iUp = 0 ; iUp <  UPDATES ; iUp++){  
    int32_t nr = file.read(writeBuff, BUFFSIZE*2);   // reads in bytes
    if (nr != BUFFSIZE*2) {
     error("read failed");
    }

    for (uint32_t iOut = 0; iOut < BUFFSIZE ; iOut++){
      temp = iUp*BUFFSIZE + iOut;
      printBuff[temp] = writeBuff[iOut];
    }

  }


  Serial.printf("Read it back\n");
  for (uint32_t i = 0; i < UPDATES*BUFFSIZE ; i++){
    //Serial.printf("%d ,  %d\n",i,printBuff[i]);
    Serial.printf("%8d  %4d\n\r",i,printBuff[i]);
  }



}

void loop() {}

// ***************************Running on Core1*****************************************************
void setup1() {


Serial.begin(115200);


  delay(3000);

  flip = 0;
  flop = 0;

  //****************   Sampling   ***********************

  analogReadResolution(12);

  for (uint32_t iUp = 0 ; iUp <  UPDATES; iUp++){

    if (flip == 0){
      for (uint32_t iTo = 0; iTo < BUFFSIZE; iTo++) {
        buff0[iTo] = analogRead(A0);  
      }
    }else{
      for (uint32_t iTo = 0; iTo < BUFFSIZE; iTo++) {
        buff1[iTo] = analogRead(A0);
      }     
    }


    delay(2);
    //flag = flip + 1;
    flip = invert(flip);

  }

}

void loop1() {}
2

What is the requirement to read and write more than 2^16 records at once? Isn't it possible to write first one 2^16 buffer, than allocate the next and write it?

3

The commercial product, at ~1000 USD, SM4BAT FS claims continuous recording to 2 gigabytes and published displays show apparent continuous recording to 2 seconds. Two seconds is my goal. Sampling at 200 KSps requires 6 times what I am getting. Are you suggesting the Arduino software doesn’t support continuous recording beyond 2^16?

4

Have you examined the library's source code to determine what might be limiting it such as datatypes, defined max values, etc?

5

64 bit math is not well supported on the older Arduinos, and since the SD libraries were among the earlier to be developed, probably have some built in limitations.

In addition, the SD libraries on Arduino are very unreliable compared to commercial examples, and often fail for inexplicable reasons. Many people on this forum report similar observations.

I find Arduinos with SD modules to be generally unusable for serious, long term or high speed data collection. I would be surprised if what you are trying to do would work. Try this on an RPi instead, as the SD libraries are orders of magnitude more reliable.

1 Like
6

Sorry. Another coding error. SD was writing fine.
Line 93 reads: uint16_t temp = 0;
Should be uint32_t temp = 0;

Thanks jremington. Yes, either soft or hardware causing some writes to drop below my threshold of 450 kBps. If this continues, I'll have to switch to an RPi.

7

I have a collection of them, and find the Pi Zero W or 2 W to be absolutely superb for small projects. The SD library is rock solid. They are power hungry, though!

8

Jremington, yes, RPi 10 times the power consumed. Are the pico SD libraries more stable using VS to build vrs arduino?

9

I haven't used the Pico, although I would be surprised if that the SDK libraries aren't much more professional and reliable than Arduino libraries.

You might try this example: How to write to an SD card with the Raspberry Pi Pico

Based on this library, so it is not part of the Pico SDK:

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.