Header for SD card .txt file

Hello,

I am working with an Arduino Uno with WIFI shield. I've been using the SD port to write sensor data to a .txt file and ran into a problem with the header.

Initially, I was only writing to the .txt file in the loop portion. This worked fine but I wanted a header that would provide names for the columns I was creating. I added code similar to the sensor data write into the setup to open a .txt, write a single line and close. The loop then opens the same file, writes to it during each iteration, and closes.

The issue I am having is that the code includes a serial.print and the setup gets repeated when I access the serial port. When the device is activated, it starts logging sensor data after having written a header line. If I then open my serial port, the header is written again and the time is reset (time is also initialized in setup).

I realize this is part of a higher level question regarding why setup is accessed when the serial port is opened but right now I am trying to focus on the SD card functionality. Is this a programming error on my part or is there a correct way to create a header with the SD library that I am not aware of.

Hope to hear from you,
Yusif Nurizade

SD_Logger_Spreadsheet_10_10a_2014.ino (3.21 KB)

I realize this is part of a higher level question regarding why setup is accessed when the serial port is opened

Because opening the serial port resets the Arduino. The setup() function is called as soon as init() is done.

Is this a programming error on my part or is there a correct way to create a header with the SD library that I am not aware of.

Since I can't see your code, I'd have to say yes. It's certainly something that your code COULD do right. After all, you can tell if the file already exists (no header needed) or not (need to create the file and write the header). The SD library manages creating files, reading from them, and writing to them. It doesn't know a header from a movie star. That is your responsibility.

PaulS,

Thank you very much for your reply. I took the week to format the code based on your guidelines and had some followup questions I was hoping you could help me with.

I started out using the SD.exists(filename) command. The idea was that if the file name did not exist, it would create the file, write and header and the first sensor readings. If the file name did exist, it would only append the sensor readings. This worked pretty well but, since I am creating my file name based on the date, running for multiple days would lead to a new file being created.

My next thought was to try the File.available() command. The idea here was to check if anything is written in the file and, if so, append the data rather than include a header. The problem I encountered is that I can do either FILE_WRITE or FILE_READ, not both. I tried following the ReadWrite example on the Arduino site but can't get it to work in the context of what I'm trying. I either get 0 bytes back on file.available() in write mode or nothing gets written in read mode. I've tried doing an SD open with neither write or read specified, I've tried opening and closing in different modes, I've tried just using write which the site says can do both, but nothing gets them working simultaneously.

The trouble I'm having may very well be related to my OO skills, which I am working on, but I would really appreciate your input. The latest code, of many, many attempts, is attached.

Hope to hear from you,
Yusif Nurizade

Off_Read_Write_10_17c_2014.ino (1.62 KB)

What I'd do is open the file for read. Get it's size. Close it.
Open it for write. If the size is not 0, write the header.

Here is a slight variation on PaulS's suggestion.

You can just open the file for write and check the size. I added a size test to the ReadWrite.ino example after the open for write.

  myFile = SD.open("test.txt", FILE_WRITE);
  Serial.print("Size: ");
  Serial.println(myFile.size());

Here is the output when the file is created:

Initializing SD card...initialization done.
Size: 0
Writing to test.txt...done.
test.txt:
testing 1, 2, 3.

Here the output on the next run

Initializing SD card...initialization done.
Size: 18
Writing to test.txt...done.
test.txt:
testing 1, 2, 3.
testing 1, 2, 3.

open() with FILE_WRITE opens for read and write but positions the file at EOF so available() return zero.

My apologies for not responding sooner. I thought I'd enabled the setting to email me when a response is posted but something went wrong.

Paul S - I realize my code does not reflect your suggestion but it is one of many variations I created to try to get this thing to work. I did try what you are suggesting and was not successful in implementing it; I would get a 0 back even though the file should have been written to. I will run it again and post my code along with my results.

fat16lib - Thank you so much for the suggestion! The placement at EOF would explain a lot! Is there any way to reposition for available()? I will certainly try size() and get back with the results.

Thanks again,
Yusif Nurizade

Is there any way to reposition for available()? I will certainly try size() and get back with the results.

You can use file.seek(0) but there is no point since file.size() is independent of file position. Also, available() is an int so it will return the smaller of file.size() and 32767, the largest 16-bit int.

fat16lib,

I tried your suggestion, using size() instead of available(), and it worked perfectly! I am now able to either create a file with a header or write to it if it already exists.

This is great progress but only works for the day in question. Since the file name is contingent on the date, a new file name will be generated in at most 24 hours. This will cause the microcontroller to create a new .txt file on the SD card and, though functional, it feels like bad programming practice.

I am wondering what I could do to create a single .txt file that could be updated over several days. I could, of course, opt for a file that doesn't depend on the date but I am hoping to avoid having to interact with the device. Any prompt to input a file name or having to put it in the source code makes it more complicated for the user.

My attempt to fix this issue rests on that fact that looping back to file name generation being the root cause. I tried adding a While Loop after the file name is created so that it can never be updated but am now running into the same problem I had with available(). The file.size() returns a 0 and only the header is written. Even if I do get this fix working, however, it feels wrong style-wise since I am nesting a loop in a loop and would be back where I started if the device reset for any reason.

Do you have any suggestions for creating a file name that cannot be change?

Thanks again for the help,
Yusif Nurizade

I am wondering what I could do to create a single .txt file that could be updated over several days.

You random()ly pack a size for the file. When it gets that big, create a new one.

In general, when discussing requirements, it is better to put your hands in your pockets. Hand-waving is SO hard to translate to functioning code.

PaulS,

Thanks very much for the reply. Given the time interval that I will be logging at, the likelihood is higher that I will hit a new date than reach a size limit. In any case, the simulations I've done have the device creating a new file when the date flips and updating the same as the first. The downside is that the end user would have to string together multiple files and I am wondering if it is possible to get all the data into one file whose file name is based on the date the test started. What I have now is functional and I may be able to get what I want by placing a While Loop into the loop(), thought preliminary attempts have not succeeded, while keeping with good programming practice.

Forgive me but I did not understand the second part of your message. English is not my first language (and here I thought I had a good handle on it) and I couldn't make sense of the terms "put your hands in your pockets" and "hand-waving". I looked them up online and the definition I am seeing for the former is "to give money to someone or to charity". Not sure how that applies.

Best,
Yusif Nurizade

Have you ever watched anyone talk who was excited about the topic? They move about. They wave their hands around. The hand waving conveys no information. The words do.

But, if you constrain the person, so that they can't wave their hands around, they calm down, and the words change. They become more coherent, and easier to understand, because they are forced to think about what they want to convey.

My point was that it was not at all clear about when you want to create a new file, as opposed to continuing to write to the same file.

PaulS,

My apologies, let me try to clarify.

I am currently at a functional point. The file is created on start up and named according to the date (11_25_14.txt). This ensures that if the Arduino restarts or the COM port is opened, the data will be written to the same, intended file.

My issue with this is that if the program runs for more than a day, multiple files will be created. This is easily fixed - I've already written a program that can concatenate these files into one. I can also change the naming code to a user input that will be unique and not affected by the date change (11_25A_2014.txt).

On a purely "it's interesting to me" level, I am curious how I can fix this without the work-around. That is, could I keep my date-based naming convention and not have a new file created when the date changes (while avoiding user input).

Hope that was more concise, this is a good exercise in engineering-writing!

Best,
Yusif