Before writing a 'set' of measurements, I write '\n' to create a new line. I chose this because if my Arduino loses power, when it is powered up again, this next set of measurements will be separated from the previous set (which might have been incomplete because of the power loss). I can then later 'discard' these incomplete sets when reading back the data.
The only issue I'm having with this approach is with the very last line! Because I've chosen to place a new line at the beginning of every set of measurements, this leaves the last line 'unterminated'. This is making it difficult for me to find a way to read back line by line.
This is what I want to do:
loop:
String line = myFile.readStringUntil('\n');
I would appreciate any ideas of what I could do instead, or how to deal with that 'last line'.
Actually, when power is lost, you will completely loose the last partial block of bytes of the file, since the data is only written to the SD card in blocks of 512.
Did you remember to close the file before turning off the power to the Arduino?
Hi! It's my understanding that file.available() tells me if there's any data actually in the file, and file.read() reads a certain number of bytes. I'm still not sure how I would apply this to solve my problem? I think I'd still have an issue where I could be asking for a byte that isn't there. For example:
while(file.Available()):
voltage = file.read() # reads one byte for example
temp = file.read() # another byte
pressure = file.read()
...
The pressure reading might not exist (due to the aforementioned power loss). Would I need to check file availability before every single read?
Hi! I didn't know this actually. The Arduino is supposed to be a datalogger in a remote location. The 'power loss' scenario was a hypothetical, that I assume could happen to the device while in operation. In that case, I wouldn't be able to close the file. It sounds like I'll incur a whopping data loss unless I have some kind of power backup!
In the case of device testing, if the Arduino is continually writing data to the SD card, how would I then go about closing the file before unplugging it? Because surely the Arduino doesn't know when I want to unplug it or stop the sketch?
Thank you for the incredibly fast response. I think this should do the trick. But I would need precise knowledge of how many 'chars' my data values are i.e. a float or int. The readStringUntil('\n') sort of hides that from you, which was nice. Is there any easy way to know this specifically for the floats and ints I'm storing?
Make the line variable as big as the longest line you will have in the file. readStringUntil uses a String, which uses dynamic memory (re)allocation, which is not recommended on microcontrollers
Or, don't store the whole line in a char array, and process characters as needed
What intrigues me is why you would want to have varying line lengths in the first place. It seems to me that you are just going out of your way to make life difficult. It's just a CSV file you are making isn't it? If that is indeed the case, you might wonder why everybody else has the new line command at the end of the line.
Hi - it's not so much that I 'want' varying line lengths, but that I thought these cases could arise if a set of measurements doesn't get fully written (for whatever reason), and am trying to find a way to account for that. I think @Paul_KD7HB 's suggestion with the push button is a good one - and in that case, it would be easy to append the '\n' to the end as I'd hoped because every set has time to complete. Thanks!
I should have stated that the button would be used 'for testing'. As @Paul_KD7HB also brought to my attention, I'll lose a chunk of data unless I have a power backup out in the field (even if it's a supercapacitor that only buys me tens of seconds). This should be enough time to finish recording a set of measurements. So no partial sets after all
That sounds very much like you are simply moving the problem rather than fixing it.
So how fatal is it to have a partially completed line of data anyway? It is probably something fixed by wipe and delete in Excel rather than flogging a dead horse in Arduino.
You are still not getting the picture that you need to completely fill the current 512 byte buffer so the library code will actually write that 512 byte buffer to the SD card, then update the system directory record with the NEW actual file length and rewrite that director record. Only then is your data complete and available on the SD card.
When you execute a file close instruction, all that is taken care of behind your back.
Yes in the few seconds my power backup provides me, I would complete the set and perform a file.close() - surely that shouldn't take very long? That's my mistake for not being explicit about closing the file. Thanks
