Keep the basic format of the debug utilities do something like mylooger.log(Keyword, "%<>", vble);
But extend it so that one could specify several of these in one statement.
a. That could be regular text like: "ColumnName : , ColumnName : ...
b. This could expanded into a string using vsprint or similar.
c. Then do substring parsing, and then somehow match to column names.
3.My question is whether there's a better way, namely to use keywords like DBG_INFO, etc but that would correspond to column names and make the parsing a switch statement.
Is there a better way that I'm just not considering.
Your post is an incomprehensible stream of technobabble but, if item 4 is actually a question, then, yes, there probably is. You might try using or adapting a pre-existing datalogging programme, of which there must be a swag around in various tutorials from Adafruit etc. As things are, I submit you are well on the way to re-inventing the wheel, and getting a square one.
A CSV record is a stream of variables separated by commas. All you need do to create one is print to file a row of variables separated by commas each time you go round the loop. The last value gets a println instead.
That is really about all there is to it - Arduino gets the data and sends it to SD, followed by a comma. This is really much the same as sending the same data to the serial monitor, which I imagine you have already done.
Note that one example I suggest you avoid, is that included in later versions on the Arduino IDE, as I believe it has been changed in recent years and now uses Strings, which may therefore cause you grief.
Hi guys, sorry for not being clear. I know what a CSV file is. What I'm doing is getting data for the columns from different modules and then in the logger class, combining and ordering them into the proper order for the csv file. So the question is how to the various modules send their data in a way that can easily be categorized by the logger.
A few decision I've made are to send the data in a printf like manner rather than some other way to handle various types from float to int to string. The question then becomes how to send the column tags. In particular is there a way to have the tags be like the DBG_TYPE in the debug utilities and to have multiple of them in one statement? Would a vector with Tuples be a better way than a print statement where I'd have to parse out the tags and then do a comparison search. If there were tags, parsing and sorting would be a little easier. And if a vector of tuples would be better, this is an area where my knowledge of C++ is a bit weak and an example would be really helpful of what the method syntax looked like.
are you capturing debug information locally and then uploading it much later?
do you expect the format of the data stored locally and the format when uploaded to be the same?
my experience with logging is it depends on the application. when i worked on an ethernet driver, i couldn't afford the delays of our logging system which i used extensively in other cases. the circular "trace buffer" used for ethernet was a very limited number of bytes for each log entry which included a timestamp and usually just an event marker. the buffer was uploaded along with its current position and decoded. i've done similar for real-time applications on an arduino.
i probably just did hex dumps of the buffers and used an awk script on a laptop to decode the logs and probably plotted them
if you're logging from multiple sources and it's less than realtime, put an ascii string into a file for each log entry, probably with a timestamp and a name identifying the entry. the format/content of each type of named log entry can be unique
when we received logs from customers, they were already translated into ascii and we searched thru the logs from our modules and went from there. in may cases the log entry was simply the function name it was generated from and a string w/ or w/o some values
Two issues with this -- firstly, that's quite a chunk of memory to dedicate to this cause; and secondly if you want to log > 120 bytes then you're out of luck. And there's actually a devilish third issue which isn't immediately obvious: the avr libc for Arduino doesn't support floating point conversions in vprintf().
First thanks for the responses. I've started to digest the stack overlfow post. As for post-processing, it may make sense to do the processing outside of the device. For now, I'm trying to produce a proper csv file record by record. IF memory becomes a problem, I will have to go another route.
I'm using a 8266 so memory is less of an issue for me.
works just fine.
3. One of the approaches I'm considering is to first render the input text into its formatted version into that 120byte buffer (or smaller, depending on my needs). I might expect that buffer to contain something like Temp:22.3, Hum:45.6, AoIn:345\n/ Another non-sensor buffer might look like Fan:1, LED:0\n
4. Then I parse these buffers and assign 22.3 to the Temp column variable, 45.6 to the Hum column variable, etc And blank out any columns that don't have data at that timestamp.
5. Then write out the record column by comma-separated column.
6. The alternatives seem to be some kind of vector approach as solution 13 in stackoverflow suggests. And I wonder about a touple. The variables need to be text representations so formatting has to be done somewhere.
7. Concerns about safety. etc are mitigated in this case because it's a closed system and I control the sender and receiver. It's a matter of which way is "best" by some metrics of performance, coding effort, etc.
this looks similar to something i did recently to better understand a large body of code written by someone else using Borland C++ that i ported to gcc and java
but instead of using those presumably standard debug levels (e.g DEBUG, INFO, WARN, ERROR, ...) we used flags that were specific to the module we were interested in debugging. using a 32-bit value allows 32 different flag bits and multiple bits can be set at the same time.
using module debugs instead of simply levels seems common
dduehren:
4. Then I parse these buffers and assign 22.3 to the Temp column variable, 45.6 to the Hum column variable, etc And blank out any columns that don't have data at that timestamp.
5. Then write out the record column by comma-separated column.
So, is the goal to have a "csv logger" type object, which different bits of code can stuff readings into, and which eventually produces some kind of composite CSV line for all the values reported in a given time interval?
Are there lots and lots of different columns to be collated?
I can totally see wanting to put all the "log a CSV line" logic in one place rather than free-coding it all over your sketch. I think in your position I'd probably have a simple function or macro for it though: anything more feels like overkill (to me).
Maybe you have a lot of code in your project though -- I'm thinking more from the perspective of an ATMega328P device where my larger sketches might reach a thousand lines or so.
Hi,
I have read the code you posted here. I don't get which is the problem your are facing. Can't you use a snprintf() to format your data and then save on the log file?