The discontinuity problem of data logging

Other Hardware Storage
1

Hi everyone,

Right now I am using a data logger that uses data logging shield with arduino and it works perfectly fine. It stores the readings from a current sensor (4 channels) in SD card.

My problem is the DISCONTINUITY that happens when getting the data from the SD card, I have to shut down the arduino, insert the memory card into my PC, copy the files and return the SD card again. In this interval I will be using some data which will affect the system, How could I solve this problem?

Is there a way to copy the files without shutting down the arduino? Do I have to use an Ethernet shield or any other technique?

thanks

2

I would do it this way:

Save the data to a circular buffer (long array with wrap around) before you write it to the SD card. When the SD card is in the slot, you will be able to write to it and the buffer will usually be empty (put and get from the buffer will be immediate. When the SD card is not there you will save to the buffer and every time you save a new value you check if the SD card is there. If it is you empty the buffer and then you are back in sync. The buffer should be sized for the time (=number of readings) that you think the card will be removed.

3

Yes. The dumpfile example in the IDE should go a long way to fix this. Doing it while actually logging data would be a bit of a stretch. I simply download today's or yesterday's file to phone via Bluetooth in a quiet period. I guess that depends on the amount of data, and how long the quiet periods are…. For a PC, you should not need anything more than your USB cable and a terminal programme like RealTerm

4

Can you download the data via another interface without taking the card out of the Arduino? Maybe use Serial so you can plug a laptop into the Arduino and request the data? This is a relatively complex solution to program.

How much data? Can you store an array of data in the Arduino's RAM while the SD card is not detected? Arrays can use up the entire RAM very easily, so you may not have enough RAM available.

5

@sabirmgd, do not cross-post. Do not cross-post. Do not cross-post.

Threads merged.

6

@marco_c this is very creative of you!

but the thing is, the maximum buffer size that the SD card can handle is 512 bytes(if I am not mistaken) which is really small . and I am already using a buffer that stores many lines of readings and then send it to the memory card since writing to the memory card will take time if you are sending a line of readings by line, to the memory card.

here is a part of the code:
the dataString variable stores a line of data ( 4 sensors reading and a time stamp), the MemoryString variable is my buffer that stores many lines of dataString and then send it to the memory card

for (noOfReadings=0;noOfReadings<3;noOfReadings++)
{ dataString="";
count++;
dataString=dataString + now.hour()+":"+now.minute()+":"+now.second()+",";
Irms0=emon0.calcIrms(noOfSamples)-0.23;
Irms1=emon1.calcIrms(noOfSamples)-0.22;
Irms2=emon2.calcIrms(noOfSamples)-0.13;
Irms3=emon3.calcIrms(noOfSamples)-0.11;

dataString =dataString+ String(Irms0)+","+String(Irms1)+","+String(Irms2)+","+String(Irms3);
dataString=dataString+"\n";
Serial.print(dataString);
MemoryString+=dataString;
}
dataFile.print(MemoryString);
dataFile.flush();
}

my point is, the buffered data will be too big to be sent to the memory card, and even during sending this data, I won't be able to log new data.

7

@Nick_Pyner

"For a PC, you should not need anything more than your USB cable and a terminal programme like RealTerm"

what do you mean? you mean by connecting to a PC will allow me to copy the files without interrupting the data logging ?

8

@MorganS

"Can you download the data via another interface without taking the card out of the Arduino? Maybe use Serial so you can plug a laptop into the Arduino and request the data? This is a relatively complex solution to program."

as you said, I don't know a way to send the files through the serial port without interrupting the data logging.

I am using ATmega:

The ATmega2560 in the Mega2560 has larger memory space :

Flash 256k bytes (of which 8k is used for the bootloader)
SRAM 8k bytes
EEPROM 4k byte

and I am storing 3.6720 KB/min , assuming the safe time needed to do this is 1.5 mins, then I need
(3.6720 min) * 1.5 = 5.5080 KB of Ram needed , but even thought, if I want to write these data back to the SD card, this will take time, during this time I won't be able to log a new data

9

Hi

You can of course send a data file to any PC web browser on a web page that your Arduino web server can serve up.

I assume you are opening new data files every hour or every day.

During the transmission of the file to the browser web page you could interrupt the process every say 250 milliseconds to execute your data logging procedures so you don't have say a five second gap while the file is transmitted.

My Arduino web server application at http://www.2wg.co.nz has both data file send and receive functionality - I only remove the SD card once every three months or so just to take a backup copy and I am working on a solution to eliminate that as well.

By the way - there are some valuable in memory caching techniques that you can use to buffer data for various reasons. My system holds incoming html data, outgoing emails and outgoing log file records in cached linked lists until my system hits an idle point at which time it will clear its caches/linked lists. This significantly improves application performance. My caching systems also self manages available free ram and will automatically clear the caches early when free ram is running low.

Cheers

Catweazle NZ

10

thank you very much @CatweazleNZ this is very useful ! and great job for the website.
"memory caching techniques that you can use to buffer data for various reasons", could you provide me with a link regarding this matter : )))

11

sabirmgd:
thank you very much @CatweazleNZ this is very useful ! and great job for the website.
"memory caching techniques that you can use to buffer data for various reasons", could you provide me with a link regarding this matter : )))

Download the PDF documentation and source code for my application from its website and ask specific questions from there.

Also Google linked list example is the C language and try to get something going. There is an example application in one of my old posts on here - get that running.

Catweazle NZ

12

Perhaps you could use a bluetooth module to connect it wirelessly to your computer, bypassing the need for an SD card altogether. Or you could have a buffer solution, like marco_c's awesome idea using a circular buffer.

You could use a wifi module as well (ESP8266) to connect to wifi and save the data to a MySQL database.

There are a ton of interesting ways to solve this problem :slight_smile: If you are intent on using an SD card as the storage medium, marco_c's idea is probably best.

13

sabirmgd:
"For a PC, you should not need anything more than your USB cable and a terminal programme like RealTerm"

what do you mean? you mean by connecting to a PC will allow me to copy the files without interrupting the data logging ?

No, I just said what I do, and I said

depends on the amount of data, and how long the quiet periods are

and no, you don't need an ethernet shield.

I can't tell if it would work for you because you are too secretive about the data you are logging. The problem you have is all about time taken to get the data out. I don't think you will ever get a sensible answer from anybody until you are more forthcoming about the time available. This is determined by the time taken to go through the loop, which I assume is regular, and what goes on in that loop, about which you say nothing.

It isn't even clear that you need an SD card. In essence, you only ever need an SD when there is no, or sometimes may not be any, other way, and you wouldn't be the first to suddenly find you don't.

The last par in your reply #7 is not very comprehensible but at least mentions 1.5 mins, which is enough time to do quite a lot of work, and suggests you don't really have a problem.

14

thanks @nightsd01, in fact using an Ethernet shield or the wifi module might be one of the solutions since I am planning to have MySql database. But for the time being I am more concerned with working with the available resources (SD card and data logging shield ) .

15

@Nick_Pyner

actually as I mentioned, the data is the analog readings from 4 analog pins accompanied by the time stamp(using the Real Time Clock) in the data logging shield, it will be saved in a file (.csv file ), attached a portion of the data file as a picture with the time stamp.

my loop is mentioned in reply #5, simply it takes the time stamp as string accompanied with the 4 values of the sensors (one value for each pin) and then it will be stored in dataString variable as string, every 3 dataString s will be added to another string called MemoryString and then this MemoryString will be sent to the memory(because sending every single line of dataString will take more time because of the initialization).

the full code is also attached as well.

It isn't even clear that you need an SD card. In essence, you only ever need an SD when there is no, or sometimes may not be any, other way, and you wouldn't be the first to suddenly find you don't.

how did you get the "I don't need a memory card" thing? I need a memory card because the site I am logging data from is a laundry shop(current sensors for washing machines), and I need to store the data in a memory card until I take them for analysis

The last par in your reply #7 is not very comprehensible but at least mentions 1.5 mins, which is enough time to do quite a lot of work, and suggests you don't really have a problem.

My calculations were based on the size of the data logging file taken within 1 day. and if we say the size of the file is 5 MB for one day , we can simply know what is size for one sec or one min .

from the data file we can see that every cycle of the the loop happens within 1+ second (every 4 rows represent one cycle)

sampless.png

the_full_code.ino (2.98 KB)

16

sabirmgd:
actually as I mentioned, the data is the analog readings from 4 analog pins accompanied by the time stamp(using the Real Time Clock) in the data logging shield, it will be saved in a file (.csv file ), attached a portion of the data file as a picture with the time stamp.

You still make no mention of how often these are read, which has a direct bearing on how easy it is to export the file without interrupting the readings.

my loop is mentioned in reply #5,

the full code is also attached as well.

No it isn't. You specifically say

here is a part of the code:

and time isn't mentioned at all

how did you get the "I don't need a memory card" thing?

I didn't. I very clearly said

It isn't even clear that you need an SD card.

and it wasn't. And it still isn't. "Stand alone" is a term that springs to mind, but nobody has heard. A convenient power supply for an Arduino in a laundry could possibly be the laundry's PC and, in that event, you don't need an SD.

My calculations were based on the size of the data logging file taken within 1 day. and if we say the size of the file is 5 MB for one day , we can simply know what is size for one sec or one min .

from the data file we can see that every cycle of the the loop happens within 1+ second (every 4 rows represent one cycle)

So what's the problem? Well, it looks like it might be - can a 5Mb file be downloaded off Arduino in something less than a second? It doesn't sound likely.

Maybe several things need to be considered

  • does the file really need to be 5Mb? I raise this because I understand you are only recording four figures, same as me, yet my files are 170k. My SD recording intervals are ten seconds. If yours are one second, the file procedure still warrants investigation.

  • what do you do with a 5Mb CSV file once it has been retrieved?

  • are there periods of inactivity when a long download is feasible?

  • if there are, do these readings need to be recorded?

  • would it be a good idea not to record in these periods, even if the is no downloading?

  • does a few seconds of recording missed really matter?

  • what is the download speed?

17

sorry, I just attached the code file and the picture of the readings with the time stamp in my previous reply.

the loop takes around 1 second to complete.

I posted a part of the code in the beginning because my code has more than one mode, it has 2 modes or routines, the first one is to write to the memory card alone, the second mode is to write to the serial port and to the memory card at the same time, I didn't wanna make things complicated so others can focus on just one mode and therefore to pass some advice.

okay to make it clear, we don't have a PC in the laundry shop. and we are not willing to have, it needs to be standalone arduino and later on we take the data from the SD card.

"So what's the problem?"

in fact no problem except that I am trying to reach to a conclusion and a clear methodology of solving my problem.

"does the file really need to be 5Mb? "
yes, the file almost needs to be 5 MB or more

my data looks something like:

0:28:21,27.05,24.29,12.52,13.48
0:28:21,10.45,14.98,4.82,5.58
0:28:21,2.16,8.74,0.96,1.51
0:28:21,3.56,4.79,0.67,0.31

0:28:22,3.63,2.5,1.65,1.42
0:28:22,3.93,1.24,1.77,1.66

in one second (lets say 0:28:21) I have four lines
these four lines each line almost has 30 char which is 30 bytes
30 bytes * 4 is the data size for 1 second = 120 bytes
(120 * 60 sec *60 min *24 hour)/ (1024 to convert to KB *1024 to convert to MB ) = 9+ MB

how yours is just some Kbs ?

"what do you do with a 5Mb CSV file once it has been retrieved?"

I process these data using C#.net and analyse these data to know the healthiness of the machines and some other information like how many cycles of the machines have been done in one day , week ... and other things

"- are there periods of inactivity when a long download is feasible?

  • if there are, do these readings need to be recorded?

  • would it be a good idea not to record in these periods, even if the is no downloading?"

there is no inactivity times because customers may come and insert their coins (money ) under any second to run the machines, we can't stop the data logging. I mean like its all automated and there is no shop keeper.

"- does a few seconds of recording missed really matter?

  • what is the download speed?"

it would be inconvenient if we missed readings for continuous seconds, for example to lose the data from second #0 to #60 is unacceptable, but to record seconds #0 , #2 , #4 , #6 ..... # 60 and lose #1, #3,#5.. is okay, because at least we can have some track of the data ?

what download speed ? do you mean the copying speed ?

the copying speed is almost the time to take out the memory card and insert it into the pc and get it back again. the actual copying process in the pc doesn't take timem the file is too small.

18

OK, I understood you had four sensors, ergo four readings, same as me. I see now that you have sixteen readings within one second, hence a lot more material, and getting rid of some of the redundant 0:28:21s only saves about 15%.

My file is smaller because I just have for numbers plus one time, and I only record at ten second intervals. It is even smaller now, by about 90%, because the whole show is now controlled by a water flow meter - no flow, nothing to record, nothing recorded.

You might be able to do something similar, i.e. implied zeros, but I rather suspect the real problem is still time even if you record every two secs. This is getting difficult but I think an approach might be to have two SD slots, the cards get changed every day, and two Arduinos, the second being just to read yesterday's SD. This means the recording is unimpeded and you have all day to read the data.

By download speed, I meant the serial data rate. I read the card with the phone via bluetooth @ 115400. It takes about three seconds. This is done by sending the date

19

oh the serial port rate is similar, 115200. the difference is in the file size then. so that means even if I used a Bluetooth module, that won't contribute.

I have to check the Ethernet downloading speed also, it may also be the same speed.

using 2 SD cards is a possible solution.

20

I was thinking about this while surfing. I don't think an Ethernet card will help with speed, you are just moving the problem rather than fixing it, but it does provide a path for a shift in thinking.

The objective is to get the data out of the Arduino. The problem is that it is difficult to get it out in bulk, and the solution may be to get it out in small increments instead.

This could be expedited by sending the data to the Internet of Things at the same time as it is sent to SD, i.e. as part of the recording loop. You can then download it at leisure. The SD is only there for backup in the event of Internet failure. I have only done this once, with cosm/xively. With them the maximum upload rate is once every ten seconds, but other systems may have what you want. There is also the possibility of having your own service. This means you not only don't have to winkle the card out, but also you don't even need to go to the laundry.