data logging/processing

Hello,

I am collecting movement data for some animals. I have several openings where I detect the animal pass, each opening uses two IR LED/photodiodes to detect direction. Currently I can successfully count each channel, running total for each channel and the overall movement in and out of the building. At the moment I am limited to 5 channels (10 inputs) but I will be adding further channels maybe up to 30, but that's another problem for later.

My question is about capturing and showing the data in different ways. I would like to show (at the moment on the serial monitor) the data against time intervals. By that I mean showing the data as Total animals "IN" (plus on count) in the last hour, in the last 24 hours, in which hour did maximum "OUT"s occur, that sort of thing. I would like to also collect data against time and retrieve a file to plot a graph against time of "IN"s and "OUT"s but I can leave that for now.

I am very new to Arduino/coding/electronics and it has taken me a fairly long time just to write the code to count and detect direction.....fairly simple stuff for some I am sure....

I am after guidance on how this would normally be achieved, may be some code examples and if applicable pro's and cons for different ways to achieve my goals. As always, thank you for spending time in helping me.
Regards Simon

Currently I can successfully count each channel, running total for each channel and the overall movement in and out of the building.

Are you logging when the transaction occurs, too?

My question is about capturing and showing the data in different ways.

Capturing and showing it where? The Arduino does not have a lot of memory, so it will not be able to hold much data.

You could add an SD card, to increase the amount of data you can store.

Showing that data may or may not be trivial. Depends on where and when you want to show it.

By that I mean showing the data as Total animals "IN" (plus on count) in the last hour, in the last 24 hours, in which hour did maximum "OUT"s occur, that sort of thing.

What would trigger showing this data? Per channel or an aggregate?

I would like to also collect data against time and retrieve a file to plot a graph against time of "IN"s and "OUT"s but I can leave that for now.

Collect it where? Retrieve a file from where?

I am after guidance on how this would normally be achieved, may be some code examples and if applicable pro's and cons for different ways to achieve my goals.

I would simply stuff each animal entry/exit event into a MySQL database, using an ethernet shield. The server hosting the database could timestamp the entries.

Then, use the power of SQL to extract the data that you want - entries and exits in the last hour, the last 24 hours, the busiest channel, etc.

Are you logging when the transaction occurs, too?

No, I have no RTC or Arduino based timing at present

Capturing and showing it where? The Arduino does not have a lot of memory, so it will not be able to hold much data.

Eventually I would like data presented via web, but for now (testing) through the Serial Monitor would be fine

You could add an SD card, to increase the amount of data you can store.

I have bought a W5100 based Ethernet/SD card shield for later development, maybe I could use the SD card to store data now.

Showing that data may or may not be trivial. Depends on where and when you want to show it.

As mentioned, eventually to share formatted results/database queries over the internet. (reference Pachube/Cosm) I am just starting and didn't want to take too bigger steps, get the animals to trigger inputs (done), add direction and count "in"s and "out"s (done), present the data against time intervals on serial monitor...don't know here to start!

What would trigger showing this data? Per channel or an aggregate?

Eventually live streaming to web, but for now a selection of formatted results. "in"s in the last hour, total "in"s today, that sort of thing.

Collect it where? Retrieve a file from where?

At the moment I would like to just print running totals to the serial monitor but next step would be to push it to a file for Excel, then ultimately to present via web a range of live streamed results.

I would simply stuff each animal entry/exit event into a MySQL database, using an ethernet shield. The server hosting the database could timestamp the entries.

Then, use the power of SQL to extract the data that you want - entries and exits in the last hour, the last 24 hours, the busiest channel, etc.

I have no idea about SQL databases or Ethernet shields yet. I think I am after a generalised explanation of how you capture, manipulate and show data such as I am collecting. As an example, how would I deal with working out the total "in"s in the last hour as a rolling total?
Sorry for not asking very specific questions, but I only know what I am trying to achieve and not the tools or techniques I should use.
Cheers Simon

As an example, how would I deal with working out the total "in"s in the last hour as a rolling total?

The first thing you need to know is when the event occurred - either as an absolute time (for which you need an RTC) or relative to some other event (like the Arduino starting).

Then, you need to keep track of when each event occurred. This will be a problem as the Arduino has very little memory. Unless you use an SD card.

Next, you'll need some way of knowing that it is necessary to display some data. Either because a request has come in (how that would happen is up to you) or because enough time has elapsed since the data was last pushed.

Finally, you'll need to look at the stored data, and decide which events occurred inside the window in question (within the last hour, since midnight, since noon last Tuesday, etc.). Sending the number of events that meet the requested criteria is trivial. Determining that number is where the challenge lies.

I like the idea of attacking a large project as a series of tiny steps. I also agree with everyone on logging the data to an sd card. You do need to be able to time stamp each event, however, if you want to be able to do time series analysis. So that is one more step to accomplish.

The place to do the data manipulation is on a PC (or Mac). You have many more tools available.

If you don't want to use the ethernet card to post the data, just remove the card, download the data to an SQL table, erase the file, and reinsert the data card. Or have two cards to swap back and forth.

You may want to use a spreadsheet first as a proof of concept. Then migrate the data to a database later, as another step.

If you are squeemish about the data analysis side there are people here to help you. I would even volunteer.

If you scale down the number of animals 'in' or 'out' as a fraction of your total number of animals, each data would only differ between 0%-100% .
That way you keep the usage of memory low. Probably not more than a few hundred bytes in a two-dimensional array.

Thank you very much for all your advise so far, I am starting to build a picture of what is required. I feel that the conciseness is to not hold data and query it on board the Arduino but to push this to a file.

I am not interested in channel specific results, what I require is total "in"s or total "out"s diced up in specific time periods. To be honest I was hoping to get the Arduino to calculate a few results as rolling totals, but I assume the lack of room to store data makes this difficult on board, especially as there could be hundreds of data points per hour!
So assuming I have to collect and send this data to a file ,due to my lack of knowledge I am afraid I am still at a loss to how I may code this and to add to this I am not sure I could extract my totals given this formatted file!
Vectorges and others - maybe you could help with explaining the limitations of data analysis with Arduino and how to create and analyse data in a file. Thanks Simon
P.S. can I add an RTC easily along with the Ethernet/SD Shield?