Data too long for SD to Read?

My Arduino project is to read data from an SD created by a non-arduino device and to play that data out through an attached CANBus interface.

The whole process is working except for the handling of a header line on the external SD file. The main data is 33chr of data in 10 fields CSV format. I am reading this data ok using 'readStringUntil ('\n').

The problem is handling the header. This is 55char of text followed by about 500 spaces before the '\n'. It screws up the following read cycles with incomplete data recovery, sometimes only the 1st char. I have used 'Trim()' on the data stored in the String from the SD read but this is to no effect, the problem appears to be in the actual read process and perhaps lack of buffer/memory.

I can use a text editor to remove the header from the file before reading it in the Arduino. I have experimented with different length records. As mentioned above, 500+ fails but 225 and 55 are OK. Unfortunately I have no control over the file I am provided. Editing the file is simple but inconvenient. I would prefer to handle the raw file without editing it.

Has anybody found this problem before? I considered limiting the length of the String used to receive the SD data but can't find an option for that. Is there one?

All inputs and ideas welcome.
Thanks

Please post the code that you are using to read the file

There is, of course, no need to save the 500 spaces as they are read, they can simply be discarded

read byte wise,
process your first 55 bytes
than continue your byte wise read until ' \n ' (and do nothing with your blanks)
Start again till end of file

Well, yes, but I don't know how to do that. The line is read from the SD into a String and by that time the damage seems to be done. I can immediately null the String but the next read is already corrupted. My playback code is included...I hope. Haven't done this before.
//********** Play Section below ********************

playSection:

// open the file. note that only one file can be open at a time,
myFile.close();
// Check to see if the file exists:
if (SD.exists(logPlayFile))
{
Serial.print("Found file, loading....");
Serial.println(logPlayFile);
Serial.print("Playing ");
Serial.println(logPlayFile);
}
else {
Serial.println("Can't find file.");
delay(500);
goto start;
}
myFile = SD.open(logPlayFile);
if (myFile)
{
// read from the file until there's nothing else in it:
while (myFile.available()) //check for bytes available from SD file
{
ReadSd = myFile.readStringUntil('\n');//stores data into 'ReadSd' until \n is reached
//Serial.print("This is ReadSD raw data..."); //debug use
//Serial.println(ReadSd); //debug use

   int dataLength = 0; //variable for length of line read from uSD
   dataLength = ReadSd.length(); //check length of data record

//part error trap loop but not complete
if(dataLength < 34)
{
if(dataLength == 33) //this is a full length record excluding the 'Len' paramenter
{
ReadSd.setCharAt(0, '9'); //replace leading char of ID with '9'
NewReadSd = ReadSd; // pass record to NewReadSd variable
//Serial.print("modified 33char line: "); // debug use
//Serial.println(NewReadSd); // debug use
}
else
{
if(dataLength == 32)
{
NewReadSd = '8' + ReadSd; //if record is not 33 char then add '8'
//Serial.print("modified 32char line: ");//debug use
//Serial.println(NewReadSd);//debug use
}
}
String lineToParse = NewReadSd; //passes uSD data to lineToParse String

   can_frame values;//use 'values' to hold data because the function needs a var not a struct

    if (parseLine(lineToParse.c_str(), values)) //calls 'parseLine' ROUTINE and converts Strings to strings
    { 
     //printData(values); //calls printData ROUTINE if parseLine ROUTINE value is true.
     // commented out because serial print not required, can be used for debugging
     ; 
    }                                                        
    else {
          //Serial.print(F("Parsing error for "));//debug use
          //Serial.println(lineToParse);//debug use
          }
          if (resultButton == 1 || resultButton == 2)
            {
             myFile.close();
            // Serial.print("Button pressed returning to Stop");
          goto start;
          }
    
    mcp2515.sendMessage(&canMsg); // send data loaded in parsing routine to MCP2515 
    //Serial.println("CAN Msg sent...");         
    delay(readDelay);//throttles read rate
    }
   } //end of while available read SD 
  
myFile.close();
Serial.println();
Serial.println("End of Data, file closed.");
} //end of if(myFile)  

goto playSection;//go to 'playSection' to repeat play unless button is pressed

}//end of VOID loop()

// ROUTINES BELOW
//*************************************************************************
//routine below works with comma delimiter and can have leading zeros or not on single chr bytes
//this routine parses each line and loads the result directly into canMsg struct.

bool parseLine(const char* line, can_frame& data) //parseLine is a ROUTINE, * indicates var is a pointer
{
//Serial.print(F("\nParsing, ")); //debug use
//Serial.println(line); //debug use

char * tmpPtr = nullptr;

canMsg.can_id = strtoul(line, &tmpPtr, 16); //converts id section of line to unsigned long and stores it to canMsg.can_id
canMsg.can_dlc = 8; //forces Len to be 8 specifically for files from CanBusLogger which does not include this field
// canMsg.can_dlc = strtoul(tmpPtr + 1, &tmpPtr, 16); // +1 to skip the comma, stores 2nd field to data.len
for (byte i = 0; i < min(canMsg.can_dlc, sizeof(canMsg.data)); i++) //counts steps to read payload data
{
canMsg.data[i] = strtoul(tmpPtr + 1, &tmpPtr, 16); //stores 3rd to 10 field to data.payload
}
return true; // when this is done it returns true

} // end of parsLine routine

void printData(can_frame& data) //ROUTINE to print data held in t_data
{
Serial.print(F("ID = 0x")); Serial.println(canMsg.can_id, HEX);//debug use
Serial.print(F("Length = ")); Serial.println(canMsg.can_dlc, HEX);//debug use
Serial.print(F("Payload = "));
for (byte i = 0; i < canMsg.can_dlc; i++)
{
if (canMsg.data[i] < 0x10) Serial.write('0'); // adds leading zero to single chr byte
Serial.print(canMsg.data[i], HEX);//debug use
Serial.write(' ');//debug use
}
Serial.println();//debug use
} //end of print data routine

Bob you have saved me on this project before, so maybe again.
Tks

Actually I would be quite happy to discard the header or even the first 2 or 3 lines of data. Would that be easier to do?
Ray

Please follow the advice given in the link below when posting code . Use code tags when posting code here to make it easier to read and copy for examination

Looking at your code I frankly have no idea what you are doing and why. Please explain what is in the file and what you want to extract from it. Ideally, attach a sample file that you can refer to in your explanation

Thanks for your patience. I understand the need for more clarity and now I know the correct way to upload my code.

The good news is your 'prompts' about posting have led to my solving the problem.

I only loaded the playback part of my code because I knew the rest worked from other versions I have been working on and using, but on reflection I can understand that if I need help I have to focus in on the problem and not expect you/others to understand what I am doing in my code, particularly since I am sure it is a mess and very inefficient given my 'kindergarten' status.

So, I extracted the SD read code into a small focused sketch with a view of loading it to the forum with a short file of the code to read. However, on testing it was clear that there was no problem with the SD read of 500+ spaces on this small sketch. Bit by bit I loaded up the sketch with other functions that are in the original and eventually I made it fail!

There is no one line of code to cause it to fail, I can include one or other screen or MISO output lines when I get to the threshold to make it fail. Prog mem is only 50% and dynamic mem is 55% at the fail point. No matter, this threshold is real. The playback of the file with a header of 50chr is good, but with 511chr it is bad. Once I thinned out the rest of my code to 45% prog and 54% dynamic, it works well with the long header data. I guess it just gets to the point where the remaining memory can't be allocated to do the job, or just gets scrambled.

So thanks for those who offered an initial response. I will be sure to offer up a more focused query and tagged code if the need arises again....no doubt it will.
Ray

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.