send buffer over serial until marker using petitFS and petitSerial

Hello,

I would like to ask for some guidance since I got stuck here, I am reading a txt file from a Sd card using petitFS and send over serial line by line using petitSerial (In the past, I managed to make it work using Strings and SD library, the old post is here: https://forum.arduino.cc/index.php?topic=552674.0) the whole sketch got so fat quickly that I must to upgrade it using slim libraries.

So every time I send a line I wait for a respond over serial (character k) before sending the next one,until the entire file is sent. Right now I can read both the serial port and the sd card, an send the entire content of the buffer every time I type “k” :

// Petit FS test.
// For minimum flash use edit pffconfig.h and only enable
// _USE_READ and either _FS_FAT16 or _FS_FAT32

#include "PetitFS.h"
#include "PetitSerial.h"

PetitSerial PS;
// Use PetitSerial instead of Serial.
#define Serial PS

// The SD chip select pin is currently defined as 10
// in pffArduino.h.  Edit pffArduino.h to change the CS pin.

FATFS fs;     /* File system object */
//------------------------------------------------------------------------------
char buf[80];
bool okToSendGcode = false;

//File myFile;
int readline(int readch, char *buffer, int len) {
  static int pos = 0;
  int rpos;

  if (readch > 0) {
    switch (readch) {
      case '\r': // Ignore CR
        break;
      case 'k': // Return on new-line

        rpos = pos;
        pos = 0;  // Reset position index ready for next time
        return rpos;
      default:
        if (pos < len - 1) {
          buffer[pos++] = readch;
          buffer[pos] = 0;
        }
    }
  }
  return 0;
}
void errorHalt(char* msg) {
  Serial.print("Error: ");
  Serial.println(msg);
  while (1);
}
void test() {
  uint8_t buf[32];

  // Initialize SD and file system.
  if (pf_mount(&fs)) errorHalt("pf_mount");

  // Open test file.
  if (pf_open("TEST.TXT")) errorHalt("pf_open");

  // Dump test file to Serial every time I get a "k".
  while (1) {
    UINT nr;
    if (pf_read(buf, sizeof(buf), &nr)) errorHalt("pf_read");//the value of nr will be altered later cos &
    if (nr == 0) break;
    if (okToSendGcode == true) {
      Serial.write(buf, nr);
      okToSendGcode = false;
    }
  }
}

void setup() {
  Serial.begin(115200);

}
void loop() {

  if (readline(Serial.read(), buf, 80) > 0) {
    okToSendGcode = true;
  }
  test();
}

but I have no idea how to move on and send line by line the content of buf, the documentation said buf is a pointer to the buffer to store the read data. so is the buffer an array? I tried use my previous implementation and simply used :

 if (okToSendGcode == true) {
                
                String l_line = "";//create an empty string
                l_line = buf.readStringUntil('\n'); //it looks for end of the line in my file
                Serial.println(l_line);//Yes you can send this line
                okToSendGcode = false;
              }

but of course didn’t work :frowning: Do you think my working code is the right approach?
any help would be greatly appreciated.

From a glance you're using a good approach in the first code. No String in sight, that's good thing. String will come to bite you in the behind sooner or later.

Basic approach should be: read chunk of bytes from file, write it to the serial console (preferably the same number as your serial buffer - by default 64 bytes on an Uno), continue until eof.

thanks for your help! The thing is I do not how to read the chunk of file, the first line. I just know how to read the entire file and put it in "buf" that I am guessing is an array, the documentation for pf_read says:

Parameters
buff : Pointer to the buffer to store the read data. A null pointer specifies the streaming read mode.
btr: Number of bytes to read.
br: Pointer to the variable to return number of bytes read.

The file read pointer in the file system object advances in number of bytes read. After the function succeeded, *br should be checked to detect end of file. In case of *br is less than btr, it means the read pointer has reached end of the file during read operation.

If a null pointer is given to the buff, the read data bytes are forwarded to the outgoing stream instead of the memory. The streaming function depends on each project will be typically built-in the disk_readp() function.

I am pretty sure I need to loop to read the file, find the marker(eol),wait for a serial respond (k) and then do the same for the next lines until eof, it is just I do not how to manipulate "buf" being I was unable to find more examples.

btw I am not programmer, I learned a bit of programming arduino the hard way, by bitting the dust :slight_smile:

file.seek() lets you set the pointer to a specific point in the file.
file.readBytes() lets you read a specific number of bytes.
file.size() gives you the total size of the file.
Between those three it's easy to write a program that reads a number of bytes from a file, sends it to the Serial console, then reads the next set of bytes. seek() isn't even needed really as you normally start reading where you left off if you don't close the file.

I always thought those file class were for the standard SD library only! :o may you please confirm? In the meanwhile, I kept the arrays, this is almost what I want, the only issue is that is sending the next line of the file every two characters “k” instead of one, as usual , I am not sure why :slight_smile:

// Petit FS test.
// For minimum flash use edit pffconfig.h and only enable
// _USE_READ and either _FS_FAT16 or _FS_FAT32

#include "PetitFS.h"
#include "PetitSerial.h"

PetitSerial PS;
// Use PetitSerial instead of Serial.
#define Serial PS

// The SD chip select pin is currently defined as 10
// in pffArduino.h.  Edit pffArduino.h to change the CS pin.

FATFS fs;     /* File system object */
//------------------------------------------------------------------------------
uint8_t bufSD[32];//for Sd file content
char buf[64];//for serial
bool okToSendGcode = true;
int marker = 0;
int readline(int readch, char *buffer, int len) {
  static int pos = 0;
  int rpos;

  if (readch > 0) {
    switch (readch) {
      case '\r': // Ignore CR
        break;
      case 'k': // Return on new-line

        rpos = pos;
        pos = 0;  // Reset position index ready for next time
        return rpos;
      default:
        if (pos < len - 1) {
          buffer[pos++] = readch;
          buffer[pos] = 0;
        }
    }
  }
  return 0;
}
void errorHalt(char* msg) {
  Serial.print("Error: ");
  Serial.println(msg);
  while (1);
}
void test() {
  // Initialize SD and file system.
  if (pf_mount(&fs)) errorHalt("pf_mount");
  // Open test file.
  if (pf_open("TEST.TXT")) errorHalt("pf_open");

  // Dump test file to Serial every time I get a "k".
  if (okToSendGcode == true) {
    //Serial.print("test");
    while (1) {
      UINT nr;

      if (pf_read(bufSD, sizeof(bufSD), &nr)) errorHalt("pf_read");//the value of nr will be altered later cos &
      if (nr == 0) break;

      for (uint8_t i = marker; i < nr; i++) {

        if (bufSD[i] == '\n') {
          marker++;
          break;
        } else {

          Serial.write(bufSD[i]);
          marker = i;
        }
      }
    }
    okToSendGcode = false;
  }
}
void setup() {
  Serial.begin(115200);
  if (pf_mount(&fs)) errorHalt("pf_mount");
}
void loop() {
  if (readline(Serial.read(), buf, 80) > 0) {
    
    okToSendGcode = true;
  }
  test();
}

I am running out of ideas, more coffee more sleep.

It also works for the SPIFFS library, and I expect that your special SD library will also give you a standard File class object to work with.

Thank you wvmarle!
ok here what I did, the code below will read a file from a Sd card and will send line by line after a character “k” is received over serial port. Using this code you can connect to your CNC machine running GRBL 1.1 and send your favorite gcode file with txt extension.

the cool part is that sketch uses 3,896 bytes (12%) of program storage space (under linux) and global variables use 328 bytes (16%) of dynamic memory, leaving 1,720 bytes for local variables. These numbers can be reduced if you use smaller arrays for another purpose.

The hard part for me was to understand the libraries; petitFS is a very abstract one and it took me ages to understand the behavior of the read pointer and the usage of pf_lseek()

So maybe you guys can take a look and check if the code can be shrunk.

// For minimum flash use edit pffconfig.h and only enable
// _USE_READ and either _FS_FAT16 or _FS_FAT32

#include "PetitFS.h"
#include "PetitSerial.h"

PetitSerial PS;// Use PetitSerial instead of Serial.

#define Serial PS

// The SD chip select pin is currently defined as 10
// in pffArduino.h.  Edit pffArduino.h to change the CS pin.

FATFS fs;     /* File system object */
//------------------------------------------------------------------------------
char bufSD[127];//for Sd file content,regardless of file size.Cool!
char buf[127];//for GRBL serial
bool okToSendGcode = true;
bool flag = true;

FILE *fptr;
int readline(int readch, char *buffer, int len) {
  static int pos = 0;
  int rpos;

  if (readch > 0) {
    switch (readch) {
      case '\r': // Ignore CR
        break;

      case 'k': // trigger action when we got this character.

        rpos = pos;
        pos = 0;  // Reset position index ready for next time
        return rpos;

      default:
        if (pos < len - 1) {
          buffer[pos++] = readch;
          buffer[pos] = 0;
        }
    }
  }
  return 0;
}

void test() {
  UINT nr;
  pf_read(bufSD, sizeof(bufSD), &nr);
  if (nr < sizeof(bufSD)) {               //if we got the end of file print the remain characters and stop there
    for (int i = 0; i < nr; i++) {
      Serial.write(bufSD[i]);
    }
    flag = false;
  }
  if (flag == true) {
    for (int i = 0; i < sizeof(bufSD); i++) {
      if (bufSD[i] == '\n') {
        Serial.write(bufSD[i]);
        Serial.write('\r');
        pf_lseek(fs.fptr - (sizeof(bufSD) - (i + 1)));
        break;
      }

      Serial.write(bufSD[i]);
    }
  }
}//end of void test()

void setup() {
  Serial.begin(115200);
  pf_mount(&fs);// errorHalt("pf_mount");// Initialize SD and file system.

  // Open test file.
  pf_open("TEST.TXT"); //errorHalt("pf_open");
}

void loop() {
  if (readline(Serial.read(), buf, sizeof(buf)) > 0) {
    okToSendGcode = true;
  }
  if (okToSendGcode == true) {
    test();
    okToSendGcode = false;
  }
}

Here some helpful links and the wiring diagram I used for the SD card, for noobies like me:

mp3 player using petitFS on Attiny85:

files manipulation in C:
https://www.programiz.com/c-programming/c-file-input-output#opening

crash course of C:
https://www.tutorialspoint.com/cprogramming/index.htm

documentation of the libraries:
http://www.elm-chan.org/fsw/ff/00index_p.html

sd-card_thumb.jpg

If it fits, why bother trying to shrink it?

cos this is only a part of a bigger sketch, there are a lot of features in the device;lcd,jogging,remote control,sound alarm,abort,so any bit of flash/sram gained counts :slight_smile:

Make sure all your string literals and other constants are in program memory - use the F() or PSTR() macros and appropriate functions, usually the _P version.

Use the smallest data type possible - if a number never goes over 255 use a byte instead of an int. Especially when using arrays that can have huge savings.

Keep arrays at minimum size.

Same for buffers. Using a single global buffer for all your temporary storage needs rather than producing local buffers time and again also helps keeping memory under control.

Look for repetitive pieces of code, and put those in a function (you should do that anyway for easier maintenance of the code).

Avoid floating point calculations where you can. Those eat up lots of memory.

wvmarle:
Avoid floating point calculations where you can. Those eat up lots of memory.

And all I thought about those was the cycles it takes! Of course there's more code, AVR has no FPU, DIV and MUL are integer opcodes!

Tks WVM!