SdFat: do not overwrite

Hello guys!

I have a rather stupid question.
I'm using the SdFat library to talk to a catalex v1.0 SD card module.

I want to log some data continously, but unfortunataly the library overwrites a files content with every open() command.

I could now read all the data from the card, store it and then just add the new ones and finally print them on the card, but isn't there a way where I don't have to do this?

Thank's for any suggestions!
Best regards!

You need to open the existing file for append or position to the end of file.

Thanks for your help!

I currently open the file using file.open(O_WRITE);
Do you mabe have an example how to do it better?

You should read the SdFat documentation in the html folder.

bool FatFile::open(const char* path, uint8_t oflag)

Open a file or directory by name.

Parameters
[in] path A path with a valid name for a file to be opened.
[in] oflag Values for oflag are constructed by a bitwise-inclusive OR of flags from the following list

O_READ - Open for reading.

O_RDONLY - Same as O_READ.

O_WRITE - Open for writing.

O_WRONLY - Same as O_WRITE.

O_RDWR - Open for reading and writing.

O_APPEND - If set, the file offset shall be set to the end of the file prior to each write.

O_AT_END - Set the initial position at the end of the file.

O_CREAT - If the file exists, this flag has no effect except as noted under O_EXCL below. Otherwise, the file shall be created

O_EXCL - If O_CREAT and O_EXCL are set, open() shall fail if the file exists.

O_SYNC - Call sync() after each write. This flag should not be used with write(uint8_t) or any functions do character at a time writes since sync() will be called after each byte.

O_TRUNC - If the file exists and is a regular file, and the file is successfully opened and is not read only, its length shall be truncated to 0.

use open(fileName, O_WRITE | O_AT_END)

If the file may not exist, use open(fileName, O_WRITE | O_CREAT | O_AT_END)

You can also use the SD.h style flags

/** Arduino SD.h style flag for open for read. */
#define FILE_READ O_READ
/** Arduino SD.h style flag for open at EOF for read/write with create. */
#define FILE_WRITE (O_RDWR | O_CREAT | O_AT_END)
2 Likes

Many thanks for your help!

I've totally not seen the html docu!

Have a nice day,
best regards!

Hello,

I am having a problem to store data on my SD card.
Whenever I give series of command one after the other from the Serial Port, my program executes the corresponding function. But everytime a new command is given the old one is overwritten. At the end I need to save this commands given from the Serial port in this SD card file .
My program is as follows: '

#include <scpiparser.h>
#include <Arduino.h>
#include <SPI.h>
#include "SdFat.h"
SdFat SD; 

struct scpi_parser_context ctx;
String value;
const unsigned int MAX_INPUT = 50;



scpi_error_t identify(struct scpi_parser_context* context, struct scpi_token* command);
scpi_error_t print_value(struct scpi_parser_context* context, struct scpi_token* command);
File myFile;

void setup()
{

  struct scpi_command* measure;
  scpi_init(&ctx);  
  pinMode(SDCARD_SS_PIN, OUTPUT);
  
  scpi_register_command(ctx.command_tree, SCPI_CL_SAMELEVEL, "*IDN?", 5, "*IDN?", 5, identify);     //Initialise SameLevel function of Identify
   measure = scpi_register_command(ctx.command_tree, SCPI_CL_CHILD, "MEASURE", 7, "MEAS", 4, NULL);  // Command tree first child with main header name
   scpi_register_command(measure, SCPI_CL_CHILD, "PRINT", 5, "PRI", 3, print_value);            // Command tree subchild  to the first child
   
  Serial.begin(9600);                                 // Open serial communications and wait for port to open:
  while (!Serial) {
    ; }
  
  Serial.print("Initializing SD card...");
  if (!SD.begin(SDCARD_SS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  
}
 
void loop()
{
  // nothing happens after setup

  
   myFile = SD.open("test.txt", O_WRITE | O_CREAT |O_TRUNC );   // open the file. note that only one file can be open at a time
 // myFile = SD.open("NEW.txt", FILE_WRITE );   // open the file. note that only one file can be open at a time
  if (myFile) {                                 // if the file opened okay, write to it:
    Serial.print("Writing to test.txt...");
    
    while(!Serial.available()){}
    
     processIncomingByte (Serial.read ());
   
    myFile.print(value);     // Data from serial monitor 
  
    //myFile.println("*IDN?");        // Data provided internally (works fine)
   
    myFile.close();
    Serial.println("done.");
  }
  else {
    
    Serial.println("error opening test.txt");   // if the file didn't open, print an error:
  }
 
 
  myFile = SD.open("test.txt");                  // re-open the file for reading:
  if (myFile) {   
    for(int i = 0; i<1; i++){
     String line= myFile.readStringUntil('\n');                              //Read from the SD card
     line.trim();
     Serial.print("Line read from the sd card is :"); Serial.print(line); Serial.println("");
     int llength = line.length();
     Serial.println(llength);
     char my_buffer[256];
     line.toCharArray(my_buffer,256);
     Serial.println("my_buffer:");
     Serial.println(my_buffer);

     if(llength > 0)
    {
      scpi_execute_command(&ctx, my_buffer, llength);                  // function call for execution of command
    }
    }
       

 myFile.close();
}
  
}  



/*
 * Respond to *IDN?
 */
 
scpi_error_t identify(struct scpi_parser_context* context, struct scpi_token*  command)
{
  scpi_free_tokens(command);

  Serial.println("MKR ZERO");
  return SCPI_SUCCESS;                                                   
}

/*
 * Respond to the command :MEASURE:PRINT
 */
scpi_error_t print_value(struct scpi_parser_context* context, struct scpi_token* command)
{
  scpi_free_tokens(command);
  Serial.println("Level 1 reached");
  return SCPI_SUCCESS;
}

void process_data (const char * data)
  {
  // for now just display it
  // (but you could compare it to some value, convert to an integer, etc.)
  Serial.println (data);
  value = data;
  }  // end of process_data
  
void processIncomingByte (const byte inByte)
  {
  static char input_line [MAX_INPUT];
  static unsigned int input_pos = 0;

  switch (inByte)
    {

    case '\n':   // end of text
      input_line [input_pos] = 0;  // terminating null byte
      
      // terminator reached! process input_line here ...
      process_data (input_line);
      
      // reset buffer for next time
      input_pos = 0;  
      break;

    case '\r':   // discard carriage return
      break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (input_pos < (MAX_INPUT - 1))
        input_line [input_pos++] = inByte;
      break;

    }  // end of switch
   
  } // end of processIncomingByte

Is the problem happening because I always perform this step:

myFile = SD.open("test.txt", O_WRITE | O_CREAT |O_TRUNC );

What combination could be used , for a proper line to line saving on the SD CARD file.
Because if I use only O_WRITE . The output is weird
and If i use O_WRITE and O_CREAT then It overwrites the old command.

O_TRUNC truncates the file, if it exists, resulting in an empty file. If that is not what you want, don't use O_TRUNC.

  // nothing happens after setup

Bullshit. loop() happens, and you do a whole lot of stuff!

Feel free to delete stupid comments from your code.

Feel free to delete stupid comments from your code.

Why not if it bothers you so much.

Like I said before:

and If i use O_WRITE and O_CREAT then It overwrites the old command.

The aim is to save all the commands provided from the Serial Port.
and if use anything like:

myFile = SD.open("test.txt", O_WRITE | O_CREAT  );

or

myFile = SD.open("test.txt", O_WRITE );

works absolutely fine.. except it overwrites the earlier command and i could see only the last executed command on the SD CARD.

    for(int i = 0; i<1; i++){

How useful. A for loop that iterates exactly once.

You open the file, wait for ONE character, and then close the file. Then, you open the file again, and expect the file to magically contain a complete command. Is that a reasonable thing to do?

For now, ditch the code that opens the file for reading. Just open the file, wait for serial data, read one character and close the file, over and over as you send data.

When you get tired of that game, take the card out of the Arduino, and open it on the PC. What is in the file then?

Only storing part works fine :

Program :

#include <scpiparser.h>
#include <Arduino.h>
#include <SPI.h>
#include "SdFat.h"
SdFat SD; 

struct scpi_parser_context ctx;
String value;
const unsigned int MAX_INPUT = 50;

File myFile;

void setup()
{
  
  pinMode(SDCARD_SS_PIN, OUTPUT);
   
  Serial.begin(9600);                                 // Open serial communications and wait for port to open:
  while (!Serial) {
    ; }
  
  Serial.print("Initializing SD card...");
  if (!SD.begin(SDCARD_SS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  
}
 
void loop()
{

   myFile = SD.open("newfile.txt",FILE_WRITE );   // open the file. note that only one file can be open at a time

  if (myFile) {                                 // if the file opened okay, write to it:
    Serial.print("Writing to test.txt...");
    
    while(!Serial.available()){}
    processIncomingByte (Serial.read ());
    myFile.println(value);     // Data from serial monitor   
    myFile.close();
    Serial.println("done.");
  }
  else {
    
    Serial.println("error opening test.txt");   // if the file didn't open, print an error:
  }
  
}  


void process_data (const char * data)
  {
  // for now just display it
  // (but you could compare it to some value, convert to an integer, etc.)
  Serial.println (data);
  value = data;
  }  // end of process_data
  
void processIncomingByte (const byte inByte)
  {
  static char input_line [MAX_INPUT];
  static unsigned int input_pos = 0;

  switch (inByte)
    {

    case '\n':   // end of text
      input_line [input_pos] = 0;  // terminating null byte
      
      // terminator reached! process input_line here ...
      process_data (input_line);
      
      // reset buffer for next time
      input_pos = 0;  
      break;

    case '\r':   // discard carriage return
      break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (input_pos < (MAX_INPUT - 1))
        input_line [input_pos++] = inByte;
      break;

    }  // end of switch
   
  } // end of processIncomingByte

Serial Monitor Output:

Initializing SD card...initialization done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...WRITE
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...ANY
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...CHARACTERS
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...FROM
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...A TO
done.
Writing to test.txt...done.
Writing to test.txt...Z
done.
Writing to test.txt...done.
Writing to test.txt...A
done.
Writing to test.txt...done.
Writing to test.txt...D
done.
Writing to test.txt...
done.
Writing to test.txt...done.
Writing to test.txt...F
done.
Writing to test.txt...done.
Writing to test.txt...G
done.
Writing to test.txt...done.
Writing to test.txt...H
done.
Writing to test.txt...
done.
Writing to test.txt...done.
Writing to test.txt...H
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...ZJ
done.
Writing to test.txt...
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...RU
done.
Writing to test.txt...
done.
Writing to test.txt...
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...ET
done.
Writing to test.txt...done.
Writing to test.txt...done.
Writing to test.txt...RZ
done.
Writing to test.txt...done.
Writing to test.txt...U
done.
Writing to test.txt...done.
Writing to test.txt...J
done.
Writing to test.txt...
done.
Writing to test.txt...done.
Writing to test.txt...R
done.
Writing to test.txt...done.
Writing to test.txt...D
done.
Writing to test.txt...done.
Writing to test.txt...H
done.
Writing to test.txt...done.
Writing to test.txt...J
done.
Writing to test.txt...done.
Writing to test.txt...J
done.
Writing to test.txt...

Data stored in SD Card File :

WRITE
WRITE
WRITE
WRITE
ANY
ANY
ANY
ANY
ANY
ANY
ANY
ANY
ANY
ANY
ANY
CHARACTERS
CHARACTERS
CHARACTERS
CHARACTERS
CHARACTERS
FROM
FROM
FROM
FROM
FROM
A TO
A TO
Z
Z
A
A
D


F
F
G
G
H


H
H
H
ZJ



RU




ET
ET
ET
RZ
RZ
U
U
J


R
R
D
D
H
H
J
J
J

Well, it seems obvious that you are writing to the file when there is nothing new to write. Which looks like a problem.

In any case, your problem had nothing to do with the way that you opened the file for writing, or with how you wrote to the file, or how you closed it.

The code that was reading from the file was somehow truncating the file (unlikely) or simply not reading all of the file (highly likely).

processIncomingByte() KNOWS when it is appropriate to save the data to the SD card, because it calls process_data() at that time. I think you should move the code to open the file for write, to write to the file, and to close the file to process_data().

When you want to read the file is a bit of a mystery. You'll need to shed some light on that, next.

Hey Paul,

Following is the code , in which all the modes which I want to execute to/from the SD Card are mentioned.
I am facing the problem in the Recording mode. where I have to write the data to the SD Card (data coming from Serial Port)

#include <scpiparser.h>
#include <Arduino.h>
#include <SPI.h>
#include "SdFat.h"
SdFat SD; 

/* Defining Global Variables here */

struct scpi_parser_context ctx;
String value;
const unsigned int MAX_INPUT = 50;


/* Defining the Command tree structure with its call function here */

scpi_error_t identify(struct scpi_parser_context* context, struct scpi_token* command);
scpi_error_t call_mode(struct scpi_parser_context* context, struct scpi_token* command);
scpi_error_t print_value(struct scpi_parser_context* context, struct scpi_token* command);
File myFile;

/* Modes are defined here */

bool recordingMode = true;
bool directExecution = false;
bool recordAndExecute = false;
bool scriptExecution = false;



void setup() {
  
  struct scpi_command* measure;
  scpi_init(&ctx);  
  pinMode(SDCARD_SS_PIN, OUTPUT);

 /* Command Tree *
  *  MODE    -> call_mode
  *  *IDN    ->  identify
  *     :MEASURE
  *         :PRINT  -> print_value
  */
  scpi_register_command(ctx.command_tree, SCPI_CL_SAMELEVEL, "*IDN?", 5, "*IDN?", 5, identify);     //Initialise SameLevel function of Identify
  scpi_register_command(ctx.command_tree, SCPI_CL_SAMELEVEL, "MODE", 4, "MODE", 4, call_mode);     //Initialise SameLevel function of Identify
   measure = scpi_register_command(ctx.command_tree, SCPI_CL_CHILD, "MEASURE", 7, "MEAS", 4, NULL);  // Command tree first child with main header name
   scpi_register_command(measure, SCPI_CL_CHILD, "PRINT", 5, "PRI", 3, print_value);            // Command tree subchild  to the first child
   
  Serial.begin(9600);                                 // Open serial communications and wait for port to open:
  while (!Serial) {
    ; }

 /* Check if SD Card is properly installed */
  
  if (!SD.begin(SDCARD_SS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  
}   // end of Setup 
 

void loop() {
  
  
  
  char line_buffer[256];                                                      // Initialisation to read the command from the user interface from the buffer
  unsigned char read_length;

  while(1)
  {
    /* Read in a line and execute it. */
    read_length = Serial.readBytesUntil('\n', line_buffer, 256);
    if(read_length > 0)
    {
      scpi_execute_command(&ctx, line_buffer, read_length);                  // function call for execution of command
    }
  }


}   // End of Loop


/* Respond to the command MODE */

scpi_error_t call_mode(struct scpi_parser_context* context, struct scpi_token*  command)
{

  /* Mode 1 and 2*/
  if(recordingMode == true || recordAndExecute == true ){
      sd_card();
     }else {
     return SCPI_COMMAND_NOT_FOUND;
      }

  /* Mode 3  */
  if(directExecution == true){
           
  scpi_free_tokens(command);

  Serial.println("Performing Direct Execution");
    }
    
  /* Mode 4 */
  
 if(scriptExecution == true){

myFile = SD.open("commands.txt");      // open existing file 

   if (myFile) {   
   for(int i = 0; i<3; i++){
      char my_buffer[256];
    
     String line = myFile.readStringUntil('\r');                              //Read from the SD card
     line.trim();
     int llength = line.length();
   
     line.toCharArray(my_buffer,256);
     Serial.println("my_buffer:");
     Serial.println(my_buffer);
     
   if(llength > 0)
    {
      scpi_execute_command(&ctx, my_buffer,llength);    
    }
    delay(100);
   }
    
    
}
        }

  return SCPI_SUCCESS;                                                   
}   // End of MODE 



/*
 * Respond to the command *IDN?
 */
scpi_error_t identify(struct scpi_parser_context* context, struct scpi_token*  command)
{
 scpi_free_tokens(command);

  Serial.println("MKR ZERO");
    return SCPI_SUCCESS;  
  }

  
/*
 * Respond to the command :MEASURE:PRINT
 */
scpi_error_t print_value(struct scpi_parser_context* context, struct scpi_token* command)
{
  scpi_free_tokens(command);
  Serial.println("Level 1 reached");
  return SCPI_SUCCESS;
}

/* Function call from recordingMode and recordAndExecution modes */

void sd_card(){
   myFile = SD.open("newfile.txt",O_WRITE| O_CREAT| O_TRUNC );   // open the file. note that only one file can be open at a time

  if (myFile) {                                 // if the file opened okay, write to it:
    Serial.print("Writing to test.txt...");
    
    while(!Serial.available()){}
    for(int j = 0; j< 20; j++){                  // loop written temporarily 
    processIncomingByte(Serial.read());          //Problem : How to read it inside loop ?
    }
    myFile.println(value);     // Data from serial monitor   
    myFile.close();
    Serial.println("done.");
  }
  else {
    
    Serial.println("error opening test.txt");   // if the file didn't open, print an error:
  }
  
  }

void process_data (const char * data)
  {
  Serial.println (data);
  value = data;
  }  // end of process_data
  
void processIncomingByte (const byte inByte)
  {
  static char input_line [MAX_INPUT];
  static unsigned int input_pos = 0;

  switch (inByte)
    {

    case '\n':   // end of text
      input_line [input_pos] = 0;  // terminating null byte
      
      // terminator reached! process input_line here ...
      process_data (input_line);
      
      // reset buffer for next time
      input_pos = 0;  
      break;

    case '\r':   // discard carriage return
      break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (input_pos < (MAX_INPUT - 1))
        input_line [input_pos++] = inByte;
      break;

    }  // end of switch
   
  } // end of processIncomingByte

So, I have marked the problem through comments in the program.
I need a way to somehow read the function processIncomingByte() continuously.
If any suggestions?

I need a way to somehow read the function processIncomingByte() continuously.

You don't "read" a function. You call a function.

You do NOT want to call processIncomingByte() continuously. You want to call it only where there is data to read.

Waiting for 1 or more bytes to be available, and then reading 20 doesn't make sense.

I don't understand what you are trying to do. It appears that you are sending something to the Arduino, via the serial port, that scpi_execute_command() is supposed to do something with, and that that results in a call to call_mode(), which, if recordingMode is true, results in a call to sd_card(), without scpi_execute_command() actually executing any command. How is that supposed to work?

Assuming that it does, the rest of the serial data that is sent is mis-handled, and crap written to the SD card.

Frankly, I think you'd be far better off having 4 sketches, for the 4 different modes, instead of trying to do everything in one sketch.