sdFat Stand Alone BIN to CSV converter.

The low latency data logger example is a really great example.
But.
It has one small problem on battery powered devices.
The time spent converting the file on battery could be better spent logging.

If you don't convert the file before the battery runs out .. you can't do it later.

If you try to convert a file after a reboot it says.

No current binary file

At the moment I am measuring the battery voltage and just sending a signal to closefile to start the conversion before the battery runs out.

I also use a pushbutton to start and stop logging.

Like so.

if ((digitalRead(BUTTON_PIN) == LOW) || (measuredvbat < 3.60)) {
			closeFile = true;
		}

If you run out of battery or the arduino restarts, the example as it stands will not convert the latest binary file.

This limits the time you can log because you have to allow battery energy for the conversion before the battery goes too low or brownout protection is activated.

It would be great if the conversion could be done when you plug the logger back into the USB supply later.

I have tried the standalone bin to csv in the AnalogBinLoggerExtras folder.
It gives the error.

Invalid meta data

If anyone could nudge me in the right direction it would be appreciated.
As it is I am quite lost on just how to go about converting a file after a reboot.

I posted a new LowLatencyLogger in SdFat-beta.

It has several new commands.

FreeStack: 1290
Records/block: 42

type:
b - open existing bin file
c - convert file to csv
d - dump data to Serial
e - overrun error details
l - list files
r - record data
t - test without logging

the 'l' command gives a directory listing with the size and file names.

ls:
13312 adc4pin00.bin
11264 adc4pin01.bin
45568 adc4pin02.bin

The 'b' command allows an existing file to be opened by entering the two digit file version. You can then dump the file to Serial or convert it to csv.

Enter two digit version
adc4pin00.bin
File opened

Enter 'c' to convert the file to csv.

FreeStack: 706
Writing: adc4pin00.csv - type any character to stop
92%
Done: 1.20 Seconds

Here is a list of files using the 'l' command that shows the csv file.

ls:
22624 adc4pin00.csv
13312 adc4pin00.bin
11264 adc4pin01.bin
45568 adc4pin02.bin

Finally the 't' command just acquires data records at five samples per second and prints them to serial until a character is entered.

Testing - type any character to stop

micros,adc0,adc1,adc2,adc3
0,745,604,516,512
200000,571,549,510,507
400000,533,522,498,498
600000,515,504,486,488

Fatlib16,
What would we do without you.
Karma added.
You are gold.
Thank you.

Hi fat16lib, first of all, thanks for the fantastic job on creating that lib!
but..i can't find (and understand clearly reading the code) how is formatted the bin file created by LowLatencyLogger, can you explain it in order to create a program, on pc side, to convert bin to csv? (Or, if you have already created it ... :slight_smile: ).
Coverting the bin file, of a long acquisition, on arduino take much time (1h+)

I would like to convert on my PC :slight_smile: because of the speed of conversion. But I have some difficulties to read the data

#include <iostream>
#include <time.h>
#include <sys/time.h>
#include <stdio.h>

using namespace std;
#define millis() tp.tv_sec * 1000 + tp.tv_usec / 1000;
const uint8_t ADC_DIM = 10;

struct data_t
{
    unsigned long time;
    int16_t adc[10];
};
const uint8_t BUFFER_BLOCK_COUNT = 12;
const uint16_t DATA_DIM = (512 - 4) / sizeof(data_t);
const uint16_t FILL_DIM = 512 - 4 - DATA_DIM * sizeof(data_t);
const uint8_t QUEUE_DIM = BUFFER_BLOCK_COUNT + 2;

struct block_t
{
    uint16_t count;
    uint16_t overrun;
    data_t data[DATA_DIM];
    uint8_t fill[FILL_DIM];
};


block_t* emptyQueue[QUEUE_DIM];
uint8_t emptyHead;
uint8_t emptyTail;

block_t* fullQueue[QUEUE_DIM];
uint8_t fullHead;
uint8_t fullTail;


inline uint8_t queueNext(uint8_t ht)
{
    return ht < (QUEUE_DIM - 1) ? ht + 1 : 0;
}


int main()
{

    struct timeval tp;
    gettimeofday(&tp, NULL);

    uint8_t lastPct = 0;
    block_t block;
    uint32_t t0 = millis();
    uint32_t syncCluster = 0;

    FILE* biFile = NULL;
    biFile = fopen("data01.bin", "r");

    if (biFile != NULL)
    {

    }
    else
    {

        printf("Impossible d'ouvrir le fichier data03.bin");
    }

    FILE* csvFile = NULL;
    char csvName[13];

    csvFile = fopen("data0X.csv", "w");

    if (csvFile != NULL)
    {

    }
    else
    {

        printf("Impossible d'ouvrir le fichier data03.bin");
    }

    unsigned long time[512];
    int i = 0;

    unsigned int k  = 0;

    do
    {
        cout<<"SEEK_CUR before readings "<<SEEK_CUR<<endl;
        int resread = fread(&block,1,512,biFile);
        fseek(biFile,512*k,SEEK_SET);
        cout<<"SEEK_CUR after readings "<<SEEK_CUR<<endl;
        cout<<"resread is "<<resread<<endl;
        cout<<"arbitrary readings:"<<endl;
        cout<<"Count"<<block.count<<endl;
        cout<<"time "<<":"<<block.data->time<<endl;
        time[k] = block.data->time;
        for(int z = 0; z<21; z++)
        {
            cout<<block.data[z].time<<", ";
            fprintf (csvFile, "%lu, ",block.data[z].time);
            for(int j = 0; j<ADC_DIM; j++)
            {
                cout<<block.data[z].adc[j]<<", ";
                fprintf (csvFile, "%d, ",block.data[z].adc[j]);
            }
            cout<<"0"<<endl;
            fprintf (csvFile, "%d\n",0);
        }
        cout<<"fill"<<block.fill<<endl;
        cout<<"overrun"<<block.overrun<<endl;
        k++;
    }
    while(k<3);
    cout<<"\n\nk="<<k<<endl;
    fclose(csvFile);
    fclose(biFile);
    return 0;
}

This code does the job, but unfortunately, file position indicator stays at the begins of the file, and next block is the first block.
Please, any help to write good bin2csv convertor on the PC side.

SEEK_CUR before readings 1
SEEK_CUR after readings 1
resread is 512
arbitrary readings:
Count21
time :25261656
25261656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25263656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25265660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25267660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25269656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25271660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25273656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25275660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25277660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25279656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25281656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25283660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25285660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25287656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25289660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25291660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25293656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25295660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25297660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25299660, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
25301656, 0, 101, 202, 303, 404, 505, 606, 707, 808, 909, 0
fill
overrun0

#define millis() tp.tv_sec * 1000 + tp.tv_usec / 1000;

So, everywhere that you call millis(), you want to substitute that string of text. Why?

using namespace std;

Why?

How much memory are you using AT RUN TIME? Each open file uses 512 bytes. You have two open. The arrays of structs look like they use quite a bit of memory. The array of 512 longs uses 2048 bytes.

Which Arduino are you running that code on?