Go Down

Topic: Using 'findUntil' on a file (Read 654 times) previous topic - next topic

Before I start barking up a gum tree can some one confirm that 'findUntil' works with files on the SD card.

In my project I have logged data written to the SD card in .csv format which I can then down load to the PC via the ethernet, however I dont always want the whole file (a years worth is 52,000+ lines). So I would like to be able to read off the last 2 or three days worth. I'm thinking that :

while (datafile. available()){
   datafile.findUntil(marker,EoF).....

will scan the file untill it finds the marker, being a date/time string that comes in from the web client that is a couple of days previously. hopefully this will run through the file in quick time and then once found the rest of the code will down load the remaining data. Also I dont want to use '/n' or similar for the terminator as the findUntil will then stop at the end of every line in the CSV file so 'end of file' is more appropriate but what character is that?

PaulS

Quote
hopefully this will run through the file in quick time and then once found the rest of the code will down load the remaining data.

The Stream::findUntil() method still needs to read one byte at a time. It just does it for you. But, that doesn't make it any faster.

Quote
In my project I have logged data written to the SD card in .csv format which I can then down load to the PC via the ethernet, however I dont always want the whole file (a years worth is 52,000+ lines).

Having created separate files for each day/week/month might have been smarter.

Quote
so 'end of file' is more appropriate but what character is that?

What character did you write as the end-of-file marker? There is no specific end-of-file character.

I'm not sure that you are going about this right. Reading one gigantic string until the data for some specific day is found isn't the way to do it. Read each record. If it's earlier than the requested date, send it out. Read the next.

If the date is not in the range desired, and is after the end date, stop reading.

Of course, this assumes that the records have time/date information, and are in increasing order by date.


PeterH


I would like to be able to read off the last 2 or three days worth.


If you used a fixed length record structure you could seek to arbitrary locations within the file. However, PaulS's suggestion to use a separate file per day is a far more sensible solution. You just need to record the day of the currently-open file, and when appending to the file check whether the file matches the current day; if not, close the current file and open a new one. Name each file according to the day it represents so that you can identify them afterwards.
I only provide help via the forum - please do not contact me for private consultancy.

The reason for a single file per year is to be able to look for long term trends. Ultimately it gets down loaded to a database and is quired with SQL, graphed out etc Splicing 365 individual files together would require a lot of post processing and a lot of SQL join type statements. However whilst I'm setting things up to be able to pull off the latest 2/7 days worth would give a sensible amount of data to try the querys on. The file has a fixed record structure and a variable number of fields to each record--- DateTime, temp0, temp1, temp2..... etc. DateTime should be easyenough to find.
I don't want to pull the Arduino/Ethernet shield off line for too long as I will lose data aquisition. I want to be reasonably certain that I have a valid code structure before I upload a modified sketch so will

File dataFile = SD.open(filename.FILE_READ);
if (dataFile){
      datafile.findUntil(marker,....

or

    datafile.find(marker)

work the same way as client.find etc
Yes ther will be a lot of records read but I won't be wasting time sending them to the client untill I get to the ones I want.

PaulS

Quote
Ultimately it gets down loaded to a database and is quired with SQL, graphed out etc Splicing 365 individual files together would require a lot of post processing and a lot of SQL join type statements.

Why? The process for inserting the data in the database doesn't need to know how the Arduino organized the data.

Getting the data out of one database table does not require a lot of joins.

PeterH


Splicing 365 individual files together would require a lot of post processing and a lot of SQL join type statements.


You are confusing the requirements of your database storage with those of your Arduino storage. Design the Arduino storage to suit the data structure and the way it's used at the Arduino. Design the database storage to suit the data structure and the way it's used in the database. These are different problems with different requirements.
I only provide help via the forum - please do not contact me for private consultancy.

Admirable though the ideas are about how to improve my data structures and analyse them later,  can we please stick to the point. As far as the Arduino and its SD card are concerned I want to be able to read through a CSV file until I find a given marker/data string and to then transmit the rest of the file over the ethernet to a PC. Sending the whole file is simple and will happen occasionally with this application but for the moment I need to be able to access the later part of the data. So:-

will 'find' or 'findUntil' automatically work their way through an open file to the requiste point where I want to start the down load?

PaulS

Quote
will 'find' or 'findUntil' automatically work their way through an open file to the requiste point where I want to start the down load?

Why don't you just try it and see?

Go Up