I'm working on a device which will be writing data on a SD Card, this device will have to write different dataset with different timescale (using a rtc). What I want is that all is output on the same CSV file, so I can sketch easily their curve. What I was thinking is 2 columns by dataset. The first dataset will be every minute and uninterrupted (for probably hundreds of hours) and the other dataset will be output periodically and will last more than a minute.
I am using the SD Library to write on the SD card and I can't only write line by line, so I will probably have to use seek() and position() but I'm struggling
Hi, welcome.
I see this is your first time posting here on the forum.
Your post describes your difficulty, but does not give us any solid information to help you.
I don't know if you read the recommended topic when subscribing to the forum:
If you haven't read it, I strongly recommend that you do and that after reading it, redo your post giving clear and useful information
so that we can help you and solve your difficulty.
Also, using </> tags, post your code.
I'd move to multiple CSVs, one for each timescale, then if you need to have them in the same Excel sheet, you can just link columns A and B to the first CSV file, C and D to the second, and so on.
Hope you know how to link to external CSV/text data source on Excel...
i though about that, as it would make everything easier. But unfortunately im not doing it for myself and he want all in one file, this device will be used for measurement campaign so it might create a lot of file (and take a lot of storage ?). I could talk about him of creating one folder by campaign with multiple csv file and one universal macro to plot all the curves
And likely to continue - in the absence of better organisation. No matter what "he" wants, I submit you need to to put it in two separate files - particularly if the data is event driven. Further, opening and closing each file
is considered bad practice and may well lead you to grief with both. This implies using two separate SD cards and possibly two Arduinos. It may be just a matter of separate CS pins, and you can run two cards on one Arduino.
If he starts getting precious about it, tell him to fix it in Excel. Sort by timestamp.
After discussing with one of my supervisors we also conclude about making 2 different files, one for the continuous part and one for the periodic. We were thinking about doing something like :
Header1
dataX | dataY
dataX | dataY
Header2
dataX | dataY
dataX | dataY
etc.. so all in 2 columns so no problem with writing everything.
So you tell me opening/closing is a bad practice, I'm curious to know why
This device will be monitoring solar cell, only one for the beginning but eventually up to 32, each one would have the continuous monitoring and the periodical measurement, including writing on their 2 each files
I don't know what exactly is the final purpose of this project and data extraction, but if the intended goal is to have all data under Excel and be able to perform calculations and/or graphs, you could just add everything in a single CSV file (add lines in "append" mode), but each line will have a prefix to indicate the kind of data (e.g. a conventional code for the corresponding kind of reading or time scale, like 0 for continuous, 1 for first scale, 2 for the second, and so on), then a timestamp.
Once you have it, you can (again) link that CSV to Excel as a data source, and using Excel functionalities add all appropriate filtering, computations, summarize, pivot tables, graphs, etc.
As I don't know excel or other similar software very well, I didn't really know how to organize properly my CSV file. After more research and discussing i came to the conclusion to do something similar (ty chatgpt) : one column as an identifier, and 2 columns for the datas. The file will be unreadable without the proper processing (which i won't be in charge), but at least I won't struggle to create the file.
Thank you anyway for your answer
To organize the CSV file as a single file containing all the kind of recordings is what I suggested before and you're saying now: use a column as identifier but I suggest to add a timestamp just to keep correct event/measure sequence.
If your supervisor requires a single CSV file, that's the solution. After that, Excel (or whatever else, e.g. a Java or C# program) could be used to parse the data and split the information.
By the different "datasets" you mean values from different sensors? Some sensors will be reported every minute but other sensors will be reported less frequently?
If I have understood, then I recommend you use a different data format compared to the one you showed. Like this:
Column A: Time
Column B: dataset/sensor name
Column C: Value
You can then use the pivot table/chart feature of excel to review and plot charts of each sensor over time.
Well not exactly, i will monitor photovoltaic cells, so one sensor by cell and measure Power time current and Voltage and i want to have :
Column A: Identifier (cell + type of measurement)
Column B: DataX
Column C : DataY
The "why" is because each time your code opens or closes the file, your program will be BLOCKED from doing anything else until that function has completed. The same applies to each "write" you do.
Yes, i have been working on the plan since january 3rd with engineers to help me, it is well in place. But i want to produce a clean code so i rack my brain
