looking for a long-term data logger

Hi makers,

I am currently planning a long-term data logger based on the adafruit data logger shield with SD and RTC functionality.
This data logger should record one byte every second in a file named yymmdd.log (i.e. a new file every day) and ensure that every byte of that file is populated at the place corresponding to the current time.
That means every file is 86400 bytes long at the end of the day. That would also mean either to wait to start exactly at 00:00:00 or, better to prepare a file on the fly populated with enough zeroes to start recording at the right place during the day.

Has someone already done something similar?
Regards

RIN67630:
or, better to prepare a file on the fly populated with enough zeroes to start recording at the right place during the day.

I don't see what's better about it, indeed I suspect it is a bad idea. You might consider logging to a new file at midnight, i.e. when the clock says it's a new day.

void loop() {
GetClock();
  if (today != day)
  {
   today = day;
   getFileName(); 
  }
  

void getFileName(){
sprintf(filename, "%02d%02d%02d.csv", year, month, day);
}

Nick_Pyner:
I don't see what's better about it, indeed I suspect it is a bad idea.

The main disadvantage of the wait-until-midnight method is that you can't record partial days, so every maintenance is a full lost day, and that makes it additionally hard to debug.

Filling zeroes in a partial-day-file should not be that complicated, I just must ensure that it would not take too long.
The other aspect to probably consider is that it would be wiser not to flush the data every second to the SD card.

So I am surely forgetting some other important things, that was the very reason to ask first if someone has done it before...

Regards
Laszlo

RIN67630:
The main disadvantage of the wait-until-midnight method is that you can't record partial days, so every maintenance is a full lost day

Nonsense. The current date is available all day, so you set the file name in Setup using the same subroutine, and start when you like. Filling in zeros is not only absurd, but also a waste of time, and then you faced with removing them later.

Nick_Pyner:
Nonsense. The current date is available all day, so you set the file name in Setup using the same subroutine, and start when you like. Filling in zeros is not only absurd, but also a waste of time, and then you faced with removing them later.

Would you please explain to me how to start writing e.g. at position 45230 of an empty file?

No, I think it would be much better if you explain:

  1. what position 45230 is (actually, I’m not sure you need bother about doing that)
  2. why you would want to start there, rather than start writing at the beginning of a file like everybody else does.

It is pretty clear that you are looking for problems that don’t exist, and letting some dodgy theory cloud sensible practice…

Nick_Pyner:
No, I think it would be much better if you explain:

  1. what position 45230 is (actually, I'm not sure you need bother about doing that)
  2. why you would want to start there, rather than start writing at the beginning of a file like everybody else does.

The data must be recorded in one new file every day and be FTP'ed to a server later.
That file is always 86400 bytes long +overhead +checksum (86400=one byte per second)
The place in the file determines exactly the time of the event (position 45230 should approximately correspond to 12:xx:xx: i did not compute that example exactly)
So if you (re)start during the day you must fill the missing bytes with 0s.

The concept behind this file format is not mine.
It is the standard shared by many vendors in a network of measurement stations.

RIN67630:
The concept behind this file format is not mine.

Well, that's a relief. It sounds rather like the "many vendors" are complete idiots, to the point where there probably aren't "many" of them or, to be charitable, perhaps there used to be a few but they have all gone out of business because they stuck to "the standard" conceived when 1Kb of memory cost serious money.

So, secure in the knowledge that you have an on-board clock, allow me to ask the bleeding obvious:
Why not record the time to the file along with the data?

It is a rather good way to ensure time and data stay together.

That's the way everybody but the abovementioned idiots would do it, and there would be no need to count off bytes, and no need to underline any alwayses or musts. You haven't actually said that you must integrate your project with such a moronic "standard" but, if you feel you need to do that, I can only suggest you take a step back and reconsider how good an idea that is. You could be on the wrong tram.

Nick_Pyner:
...It sounds rather like the "many vendors" are complete idiots...

Let's close that silly discussion.

Indeed it is. And about the silliest I have ever encountered.

If you must follow this "standard" format, when your logger starts, calculate the desired file position and write zero bytes to the end of the file until you reach that position. If you're using the SD library, use size() and seek() to get to the end of the file. Some other SD card libraries allow you to "seek" directly to your desired file position and will automatically fill in the missing bytes with zeroes.

Then write one sample per second afterwards.

christop:
Some other SD card libraries allow you to "seek" directly to your desired file position and will automatically fill in the missing bytes with zeroes.

Wow, that sounds cool!
Can you give a bit more info about that "other SD card libraries"?

But this library should not be too demanding on RAM, since I have other tasks to run as well.

Maybe I could just prepare the file with 86400 zero bytes anyway and just write seek() indexed to it afterwards?
At midnight, preparing the new one should be finished until the one minute buffer would overflow.
That should not be a huge challenge.
I have got a scheduler to do several things concurrently.