Using binary format to store for an SD card

I want to have two variables, “x” and “y” stored in an SD card. But here’s the catch, I am using binary data because it is much faster.

I want to store a lot of data using the two variables. For example, if I want 100 blocks of data, I will have 100 values of x and 100 values of y.

I don’t know how to do this with the binary format because you need to have a pre-made array, in this case, an array of 100. I don’t want to make big arrays, instead, I want to add data on top of what is already on the SD card in order to save SRAM on the Arduino.

Here is my code, and it works, but is a memory hogger :frowning:

#include <SD.h>
#include <SPI.h>

const int chipSelect = 10;
File dataFile;

struct Datastore {
  uint16_t x;
  uint16_t y;
};
String filename = "datalog.dat";
Datastore myData[100];
Datastore Data[100];
void setup() {
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  pinMode(chipSelect, OUTPUT);

  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    return;
  }
  Serial.println("card initialized.");
  if (SD.exists(filename)) {
    SD.remove(filename);
  }

  makeData();
  readIt();
}

void loop() {
  if (Serial.available()) {
    switch (Serial.read()) {
      case 'd':
        readIt();
        break;
    }
  }
}

void makeData() {
  Serial.println("Fetching pre-made data");
  dataFile = SD.open(filename, FILE_WRITE);
  //Serial.print("Length: "); Serial.println(sizeof(myData) / 4);
  for (int i = 0; i < sizeof(myData) / 4; i++) {
    myData[i].x = i + 1;
    myData[i].y = (i + 1) * 2;
  }
  dataFile.write((const uint8_t *)&myData, sizeof(myData));
  dataFile.close();
}

void readIt() {
  Serial.println("Reading Data");
  dataFile = SD.open(filename, FILE_READ);
  dataFile.read((uint8_t *)&Data, sizeof(Data));
  Serial.print("x"); Serial.print("\t"); Serial.println("y");
  for (int i = 0; i < sizeof(Data) / 4; i++) {
    Serial.print(Data[i].x); Serial.print("\t"); Serial.println(Data[i].y);
  }
  dataFile.close();
}

You can write and read back from an SD card without storing the entire contents in an intermediate array.
You can also append data to an existing SD card file with the appropriate SD.open() command so it is not necessary to read the whole SD card into memory to add to it.
You can't influence the storage format on the SD card so what do you mean by binary format here ?

I know the text format is popular and more widely used, but it is also known to be slow compared to binary format. I learnt this binary format from Fast, Efficient Data Storage on an Arduino | Majenko's Hardware Hacking Blog

This was old knowledge in 1972, have you ever read a book? Or does all your computer knowledge come from You Tube?

I think the main focus of the link you referenced was not writing small data elements individually to the SD card, but to group them and write the group.

If you have text data, for example the names of all the people in a class at school, and you need to store these on an SD card, then there is no alternative but to store text data. Ultimately, it will have a binary representation on the SD card.

With this structure, you have partly solved the problem of consolidating data before writing it. Your problem is now that you are creating huge intermediate arrays which will break the RAM limit on your arduino. However, it is true that, up to a limit, the more data you can write at once to the SD card, the quicker it is. Better might be to use the same buffer for both writing and reading, since you won't be doing these operations simultaneously.

struct Datastore {
  uint16_t x;
  uint16_t y;
};

:), yup.

The SRAM is suffering. I suppose text files are the way. I think converting each incoming char into a number would be fast. So I got the reading part down, but I’m not sure if writing to the SD card is slow with the current SD library.

Forget text or binary or anything else...
You are storing DATA.
It’s always binary until you convert it to something else for user entry or display.

While it won’t affect your sketch, you should get into the habit of getting away from Strings, anger into the use of c-strings.

lastchancename:
Forget text or binary or anything else...
You are storing DATA.
It’s always binary until you convert it to something else for user entry or display.

So what do I do with my sketch? Do I stick to txt files?

What is the range of values for x and y?
Choose a variable type that fits your requirements, then build on a strategy that uses arrays of that type.

Numeric and text ‘entry’ or serial data streams are the slowest part of your problem - finger or comms speed - so any processing can be done during entry.
Same for output rendering to whatever you want to display or control.

In the middle, once you have chosen your priorities you can just bounce values around in their native format. EEPROM can slow things a little bit, but that can be optimised as noted above.

Where is the dataset coming from, and what is it being used for?
What’s the need for speed?