So far another topic poped up in my ongoing project.
The key facts:
I analyze a manchester decoded current modulated signal with 10 bits (t_pulsewidth = 50uS) repeating periodically with different time inbetween the protocols (several mS up to hundreds of mS)
to save RAM I chose to save the bits of my protocol as byte type (values of the protocol bits can be 0, 1, 3 and 4)
the time is saved as an unsigned long (using micros())
I use the #include <CircularBuffer.h> library to save the protocol in a circular buffer
later on I want to save the last protocols to an SD-card
Now I want to buffer some of the latest protocols including the content of the protocol as well as the time of the protocol starting bit. This gives me the ability to run my tester as long as I want to and still have the possibility to save some interesting behaviours if needed by pushing a 'SAVE'-Button. This reduces the amount of generated data and gives the chance to save the data on a SD-card which is too slow to log all data, even if I wanted to.
My big question now is how to save those data for an easy possibility of analyzing them later on. My first guess was to create only one buffer which saves the micros time, then adds '7777', following with the protocol bits and ending with a '8888'. The four numbers should give the ability to identify the beginning of the protocol as well as the ending. Using a type cast to match the byte numbers in the unsigned long buffer. This would look like this:
// saving the actual protocol in the ring buffer
bV_ProtocolBuffer_g.push(ulV_Time_g[0]); // stores the micros value of the first high AD value in the buffer, giving the start time of the protocol
bV_ProtocolBuffer_g.push(7777); // adding the number '7777' to announce the following bits are protocol bits
for (int i = 0; i < sizeof(bV_actualDataProtocol_g); i++)
bV_ProtocolBuffer_g.push((unsigned long)bV_actualDataProtocol_g[i]); // pushes all protocol bits in the circular buffer
bV_ProtocolBuffer_g.push(8888); // adding the number '8888' to announce the ending of the protocol bits
The problem on this is: I planned to be able to import the long printed buffer in excel, using the data import function to state that there is a special sign which indicates a new colum. Unfortunately in excel there can only be one specific character or number to divide the columns. Due to this, I could not seperate by using my 7777 or 8888 identifications. So a question came up: is there any (special) character which can be stored within a unsigned long other than numbers? So far I could not find any. This would help to identify the beginning and ending of my protocols.
Another possible solution could be to generate two different buffers, one as a unsigned long for the time and one for the data protocols in byte format. But I guess this increases the possibility of mistakes as there is no guarantee that over a longer period the times match 100% to the data values saved?!
Do you guys have any further ideas how to save the time beside the protocol content?
Not really. You can assign a special significance to some number that is outside the range of numbers that you expect it to represent, for example the largest possible positive integer.
That is my try with using the 7777 and 8888 --> both numbers should not appear in my code as the protocol bits can only have values of 0, 1, 3 or 4. Only the time of micros() makes it possible but so far unrealistic that I will face a 7777 or 8888 in nature.
My problem on this is that I cannot interprete those data using excel later on. I can only seperate by
a tab stop
semicolon
comma
space
others --> here comes the problem: I can only define one single number or character to divide the data, for example '7'.
Did you investigate in Python language? A Python script could read your data from SD card and convert it, e.g. to a text file while handling your identification numbers. I personally would use Freepascal/Lazarus to create a converter program on PC.
Could you probably supply an example of data in exactly the format you expect, including your identificators, and probably some explanantions to it e.g. as a zip-file?
To be honest, I cannot programm Python so far. Would be my next bigger project to come up with.
Give me a second
Edit: here we go. For a better readability I seperated the time and the protocol content by using a ': '
The txt shows the output of the serial monitor. example of serial output.zip (2,4 KB)
Thats right. If I press the save button, I take as long as it needs to save the data to the SD-card. After that my FSM will return in the state where the protocol is decoded and stored in the buffer. But in the time inbetween, I am aware of loosing some protocols.
Sounds good. But as far as I understand I already have to store the data in seperated variables/array instances before saving them to the SD? During my buffering process, I do not have the time to save to an SD - the next protocol can follow the previous within for example 10mS. My research showed way longer times to access a SD-card.
If I insert a tab to seperate my values, I cannot use the CircularBuffer anymore as this only allows to store instances of the defined type. And if I insert the tab after saving in the buffer, I don't know at which instance to start with counting the 10 bits. Additionally I must admit that my code looses the last bit from time to time - as you may have already noticed in my txt document Thus I cannot only count to 10 and say that a new protocol started.
Edit: but maybe I can use the 3 and 4 which are always the first digit of a new protocol to seperate. Do you have any hints on how to write that kind of text file doing so?
There is always a pair of "time : protocol data" both stored as unsigned long.
Time based on micros() since start of microcontroller
Protocol data, 10 digits, each digit can have a value between 0 and 3
Why not saving exactly what you see on Serial (possibly without except the realtime clock data) to a text file on SD ...? Instead of the ":" you could use a colon or char(9) = tab?
Edit: You can use the ringbuffer as you mentioned (e.g. with delimiters 7777 and 8888) , but do convert it to text when writing to SD card ... (Maybe I do not get your point ... ?!?)
Almost every point correct, just the digits of the protocol are stored in a byte type to save unneccesary waste of RAM. And also the first digit can be a 3 or 4 - my bad, messed up my initial informations.
The time between the protocols could be close to 10 or even 5 mS (depending on the speed of the wheel I am measuring). Thus there is not enough time to save every protocol on the SD, waiting for the next to come up. I have read that it takes several hundreds of mS to start a proper communication and writing data on a SD.
I understand that you want/have to write the buffer content to SD eventually; not in realtime but when you press the "save" button.
So my suggestion is not to write the buffer content directly but to convert it to a text file. I cannot say what influence writing to SD will have on your sketch ... If your controller can handle both in parallel, you could use two ring buffers and switch to the second for measurement while the first is saved (and vice versa again later).
It would require that you write the save-routine as a non-blocking function.
So you mean I can save the effort of using the CircularBuffer and just write to the SD-card in real time, the data are then buffered automatically and written on the SD later? When is the write on the SD then executed? Can I have an influence on that to avoid missing signals?
Only disatvantage I see is then the ammount of data which is produced. In worst case later on I cannot find/identify the spot in the data collection where I pressed my SAVE button as a trigger.
Edit: sorry, have to go for now, will text back tomorrow.
So you suggest to store the time in buffer a and the protocol in buffer b? #1
Or do you mean I can store both in buffer a and then while transferring the data from buffer a I can write in parallel new incoming protocols in buffer b to not loose any informations? #2
In the #2 case: this is not needed, I worry more about the possibility to later on divide the long string of numbers to be able to interprete. I'm totally fine in loosing some protocols why the data is transferred to the SD.
Okay, got it. As I only want to save the latest data and be able to run my code for lets say 5 minutes and in the last 5 seconds something interesting happened, I still need my circular buffer. But this buffer should never exceed the 512 byte in best case so I will be able to transfer the complete buffer by handing it over to the SD buffer which then will be saved with a flush() on the card - is that correct? Optionaly if my circularBuffer is bigger that 512 byte, I have to hand over multiple packages to flush() it on the SD? In that case, do I need to cut the data in two packages or would this be done by the function itself as it recognizes that it will overflow if more data would be saved?
Still my main problem which I did not solve until now is: how could I seperate the data in my loooong array of numbers? I guess for this problem I have to come back to the suggestion @ec2021 made in post #5: using a python script to seperate the numbers and use my initial idea to seperate by using 7777 and 8888
Thus I cannot only count to 10 and say that a new protocol started.