Error checking protocol for file on SD card (Using Excel & ESP32)

Projects General Guidance
1

For my ESP32-based ramp/soak temperature controller project, I write out multiple firing schedules on an excel spreadsheet, and then save each of these schedule's to a separate comma delimited text file (.txt).

The text file is stored on an SD card. The ESP32 will then read this file into a temporary char buffer, and then the data is parsed into variables.

Currently, each text file is in human-readable ascii, is contained within start/end markers, and looks something like this:

“<program name, program description, 300,1000,10, 300,1350,5, 9999,900,20>”

The length of these text files could be something like 80 characters minimum to 400 characters max (ballpark).

I am confident in the basic techniques to get all this data parsed by the esp32, and how I could split a large program into multiple packets to save on SRAM.

I am now interested in how I can add a fairly robust error checking protocol to this data, and how to use one in both MS Excel and the ESP32, which will prevent any erroneous transmissions from the SD card to the ESP32, through either noise, fault in the SD card, etc.

I have some basic understanding of how CRC’s and checksums work, where you calculate a number and add it to the end of a transmission, and then you recalculate the number on the receiver side and check if both numbers match, but I’m not too sure which type of error checking protocol would be sufficient for this use case, and how to use one in both excel and the ESP32.

I’d appreciate any guidance to get started on this.

2

Best to check with the Excel experts, first.

3

I did a bit more reading and found out that the most basic of checksums are simply the sum of each byte in the transmission. Therefore I got my example char array:

" program name, program description, 300,1000,10, 300,1350,5, 9999,900,20"

And added together the decimal ascii code of each character in the array. I added this together into a larger 16-bit integer.

// Data read from text file
char dataArray[] = "program name, program description, 300,1000,10, 300,1350,5, 9999,900,20";

// Calculate the size of the dataArray
const int arraySize = sizeof(dataArray);

// Function to calculate the 16-bit checksum
uint16_t calculateChecksum() {

  uint16_t sum = 0;

  // Calculate the sum of all signed 8-bit chars in the array
  for (int i = 0; i < arraySize; i++) {
    sum += dataArray[i];
  }
  return sum;
}

void setup() {
  Serial.begin(9600);

  // Calculate the 16-bit checksum
  uint16_t checksum = calculateChecksum();

  Serial.println(checksum);
}

void loop() {

}

The checksum is calculated to be: 5074 in the code.

Using some poorly optimized excel function, I placed each character of the string into a separate cell, and then got the 8-bit ascii code of each character with the function "=CODE()", I then used the =SUM() function to add all those codes together to get the same result of 5074.

image

My thinking is that I would add this checksum to the end of the string in the .txt file, and then recalculate it when the file is opened up on the MCU. I found some Macro's online that can get some of this process done automatically.

Just from my basic intuition, it would seem pretty unlikely that this checksum test would pass with corrupted data, as if just one byte of the payload gets corrupted, you would also need to have the checksum value get corrupted too. With this checksum being a 16-bit unsigned integer you will have a 1 in 65,535 chance that the checksum will pass with corrupted data. does that sound right?

These are just be beginner thoughts. Further clarification would be appreciated. Making the protocol even less prone to failure, or even adding error-correction, would be interesting too.

thanks

4

I've been trying to wrap my head around how the CRC32 algorithm works in this EEPROM example.

In particular, how all the bitwise operators work in this for() loop:

  unsigned long crc = ~0L;

  for (int index = 0 ; index < EEPROM.length()  ; ++index) {

    crc = crc_table[(crc ^ EEPROM[index]) & 0x0f] ^ (crc >> 4);

    crc = crc_table[(crc ^ (EEPROM[index] >> 4)) & 0x0f] ^ (crc >> 4);

    crc = ~crc;

And I've been able to fully replicate this algorithm into google sheets, using some of the excel-based bitwise functions. A lot of the hex values are shown as decimal in the spreadsheet.

I don't know how to write for() loops into spreadsheets yet, so for each character in the string, I've had to print out the result of all 3 lines of the for() loop. This obviously wastes a lot more space in the spreadsheet. The code in some of the cells is also very hard to read.

Here is the link to the spreadsheet, as well as a wokwi simulation demoing the algorithm. Both of these use the same ascii string, and end up calculating the exact same CRC32 value.

So I'm pretty happy with this result so far, but let me know if there's a more suitable check I could use for this particular use case.

thanks

5

MY background has been in data communications with lots of error checking. Both vertical errors which are caught with parity checking and horizontal bit errors which are caught with various schemes. With single bit errors, some may be corrected.
In order to error check, you MUST know what possible errors can occur. Have you studied and found the possible errors that can happen with your data? If so then you can prepare test to discover the error.
Seems you are just playing around and do not actually know if errors can occur or not.

6

That's part of what my question is, basically. Don't know where to start. Do you have any insights on how I can figure out what kinds of errors can occur with SD cards (spi protocol), with my files being variable sized ascii strings (80-400ish bytes in length)?

Thanks.

7

Error rates are essentially zero because of dos emulation. crc etc is done in both the hw and driver level, and will return clean data or error out. This is a non issue.
Now serial comms is noise sensitive, but simple parity check will catch most of that, with message checksum catching the rest.

8

So, in effect, you are chasing phantoms!

9

are you saying there's already an inbuilt CRC that prevents the esp32 from parsing a corrupted file (or some other type of error?) from the SD card?

10

Yep. The dos emulation gives the same file reliability a pc based disk system does. That one of the reasons the data rate is down ~10kb/S. Dos disk systems have error detection built in an the sd card operates the same way. You basically never get bad data without detection. You get clean data or an error, never corrupted data without an error.

11

Thanks. I hadn't heard of this before, but makes a lot of sense.

Thank you for pointing out the flaw in my logic, I will take this into account when I try to research other questions I have. Seriously, no sarcasm. I've inadvertently fallen into a lot of X.Y. problems and I don't want other beginners to find my posts and waste their time too.

12

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.