SD card .txt file values to Serial.print without using String

I am using a MKR Zero with an SD card loaded, I have a timer that wakes up the MKR every 10 seconds to take readings, and store them on the SD card, then cut power to the MKR.
Every 5 minutes I plan on sending these values from the SD card over radio (UART).
My main roadblock right now is reading the values from the .txt file on the SD and converting them to a suitable format for sending over UART to the radio.

I originally was using String but have since read about all the dangers of using String due to memory fragmentation (Evils of Strings

So for example, if I have 3 .txt files that contain this text:
ch1.txt = 100,100,100,100,
ch2.txt = 200,200,200,200,
ch3.txt = 300,300,300,300,
I want to read each line, combine them, then send over UART like this:
Serial1.print("AT+SEND=1:12: 100,100,100,100,200,200,200,200,300,300,300,300,300,")
Where Serial1 is the radio UART port.

This code will read the .txt file and print easily:

#include <SPI.h>
#include <SD.h>

const int chipSelect = SDCARD_SS_PIN;

File ch1_file;

void setup() {
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  if (!SD.begin(chipSelect)) {
    Serial.println("SD initialization failed.");
    while (true);
  }
  Serial.println("initialization done.");
}

void readData()
{
  ch1_file = SD.open("ch1.txt");
  while (ch1_file.available()) 
  {
    char c = ch1_file.read();
    Serial.print(c);
  }
  ch1_file.close();
}

void loop() {
  delay(5000);
  readData();
}

This outputs in the console 100,100,100,100, as expected.

How can I read all three .txt files, append them into one object, then send without converting anything to a String.

I tried to create arrays like this:

#include <SPI.h>
#include <SD.h>

const int chipSelect = SDCARD_SS_PIN;

File ch1_file;
File ch2_file;
File ch3_file;
char arr1[13];
char arr2[13];
char arr3[13];
char arrCombined[39];

void setup() {
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  if (!SD.begin(chipSelect)) {
    Serial.println("SD initialization failed.");
    while (true);
  }
  Serial.println("initialization done.");
}

void readData()
{
  int j=0;
  ch1_file = SD.open("ch1.txt");
  while (ch1_file.available()) 
  {
    char c = ch1_file.read();
    arr1[j] = c;
    j += 1;
  }
  ch1_file.close();
  
  ch2_file = SD.open("ch2.txt");
  while (ch2_file.available()) 
  {
    char c = ch2_file.read();
    arr2[j] = c;
    j += 1;
  }
  ch2_file.close();
  
  ch3_file = SD.open("ch3.txt");
  while (ch3_file.available()) 
  {
    char c = ch3_file.read();
    arr3[j] = c;
    j += 1;
  }
  ch3_file.close();
  strcat(arrCombined, arr1);
  strcat(arrCombined, arr2);
  strcat(arrCombined, arr3);
  Serial.println(arrCombined);
}

void loop() {
  delay(5000);
  readData();
}

This printed nothing to the console.

Even if this did work and printed 100,100,100,100,200,200,200,200,300,300,300,300,300,, I am unsure how to create the required String (assuming that is a string?) to send the radio command.
I just wish I could use Strings!

If you only want to print, I would not store it in an intetmmediate array. Take your first code, repeat (for each file) the opening of the file, printing of the contents and closing of the file.

To differentiate between the three files, you can add a diffetent separator after the printing of each file.

Note:
You should check if opening of a file succeeds.

Handling these on a file by file, line by line or character by character basis has a big advantage,.

Minimising RAM utilisation for a buffer,

If you’re looking for particular words or values, maybe read & parse line by line, but big blocks of data are dangerous in a small memory system.

Your strings are 16 characters each and need a null terminator so your arrays should be AT LEAST 17 characters.

Your combined string will be 48 characters so this array must be AT LEAST 49 characters.

This will fail the second time because 'arrCombined' will no longer be empty to start with. Better to use:

  strcpy(arrCombined, arr1);
  strcat(arrCombined, arr2);
  strcat(arrCombined, arr3);

The referenced article is "technically" correct but if you look at the printing date, 2016.02.04, that represents a light-year in compiler sophistication. The uC you are using uses the same compiler as the ARM-based Arduino Due which is the same as the Espressif ESP32 and ESP8266.

Consider, HTML code is heavily String based and you can write code without Strings but it certainly is more complex. One hardly hear about serious issues, but the String poopooing is still a major forum discussion.

My thoughts: 'Unless you are having known issues with Strings, use them if you wish.' The Arduino 'core' writers still include "String" in the language as a supported class; much effort has been put into this native C++ methodology. True, C++ uses more memory for String manipulation, but the GNU documentation says:

It’s fairly common for beginning C programmers to “reinvent the wheel” by duplicating this functionality in their own code, but it pays to become familiar with the library functions and to make use of them, since this offers benefits in maintenance, efficiency, and portability.

So, unless you are using an extremely small-SRAM uC or unless you are exceeding SRAM 80'ish % target usage, you should be fine using Strings. Do please read/review this reference:
reserve() - Arduino Reference

Happy Programming: "... if it works and does not hurt, do it."

Ray

2 Likes

@mrburnette

did you do some million-iteration-tests on this?

I mean writing a code that assigns new values to a string-variable of different length over and over again for many many times and printout how free RAM is changing over the many many iterations?

Are there any differencies between assigning Strings with always the exact same length
compared to assigning Strings of different length
always becoming shorter (less characters than before)
always becoming longer (more characters than before)
randomised number of characters at each new assignment

variables of type String with global scope
variables of type String with scope local to a function that is called over and over again
difference that it makes if you declare a variable of type String that is local to a function as static

Yes.
I have an ESP8266 that has run for almost 8 years that parses GPS serial, extracts time+date, and broadcasts UDP to a couple of ESP8266 receivers.

It would seem to me you should address this concern to the GNU compiler braintrust.

My point was, GNU supports and even recommends the String class. I may be brilliant but I am no match for compiler gurus.

After thought
If we C++ folks use the String class as documented and it breaks then we can direct the error to GNU team to research and hopefully correct.

If there is some Arduino-centric limitation, I am unaware of same.

This is a very generalised statement. On a computer with multi GBs of RAM there will be only one in a billion cases where the String is has sooo many characters that a new memory-hole can not be found.

The situation is very different on a microcontroller like the Atmega328P with 0.00002 GB of RAM
Even the ESP8266 has 40 kB of RAM for the user. This number of 40 kB is taken from here
https://bbs.espressif.com/viewtopic.php?t=129

@mgale31

you can use the SafeString-library which offers almost the same comfort as String but without the sometimes weirdness of class String.

SafeStrings have a well defined limited length.
I have tested the SafeString-library and tried to store more characters into a SafeString than the defined length. Reaction is safe. The too many characters just get dropped and that's all what is happening. This means if you try to concanate too many characters to a SafeString you loose some data but the code is not crashing by RAM-overwriting.

I do not believe I suggested AVR 8-bit toys. The Op stated:

I stated:

I personally would not use String on UNO except to play around on the bench. But I have used String on the Mega without issue, but I also reserved() a nice little buffer.

Context... it is all about context. ESP32, ESP8266, STM32, are all ARM platforms and denying a script writer (programmer) String is rather hampering IMO.

Oh, my... this tingles my spider-senses. The 'ol protect the programmer from herself :upside_down_face:

Any computer, big or small, can be broken by a careless programmer. Misuse of "convenience" classes happens all the time with novices and often with more mature programmers. 'tis the nature of the beast.

I am a very strong proponent of lengthly test cycles. If one uses something correctly in code but the code breaks in testing; my approach is to recommend an alternate solution. Too darn much time is taken in hitting one's head against a brick wall: the most common symptom is a headache.

So, my final recommendation:

  • If String works for you, test with real-world data and if you are having breakdowns in your code consider rewriting or move up the SRAM ladder.

The above works for me but application universally is not guaranteed.

I did some research (relaxing reading with a diet cola) and I like the overall idea within the context of what the author states:

Both Sparkfun and Adafruit advise against using the Arduino String class but Arduino Strings are very safe and usable, if you follow the guidelines in Taming Arduino Strings.

I did not go back in time to date the remarks of Ladyada or the Sparkfun dudes, but let's just take this at face-value.

The library author states also:

This SafeString library is designed for beginners to be a safe, robust and debuggable replacement for string processing in Arduino.

To which I say, "Hallelujah." Anything to assist newbies/beginners/novices/the-inexperienced/...
Caveat:
-- While the author states the library "works" with All Arduino architectures there is no guarantee that the library is perpetuality supported; things evolve quickly in the uC corner of computing.
-- Learning SafeString will not translate into raw C++ knowledge, so it appears to be a crutch and eventually out-grown as your programming talents become mature.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.