Open Source Data Logger Project Using the Arduino?

Forum 2005-2010 (read only) General Frequently-Asked Questions
#1

Hey Guys,

I am heavily involved in the data logger/acquisition industry but I feel that its missing something, and that would be an open source data logger. The issue I come across is pricing, I deal with customers all over the world and find that 90% of my clients (mostly educational) just don't have the budget for an easy to use, compact data logger, any stand alone unit with 8 channels is going to cost a minimum of $500, add the software, cable, etc...soon gets out of hand.

So I was thinking that an open source logger based on the Arduino could be an ideal solution, the biggest issue would be having something easy to use for the end user, a nice front end on the software, but allow the more technical to make modifications for the odd sensor.

I would even be prepared to start some sort of contest to kick start this project, throw in some prizes to generate some interest.

Anyone feel that this would be worthwhile?

#2

Thats a cool and interesting idea. What kind of sensor input are you thinking about, and what should the software on the PC side be able to do ?

#3

Hey,

Glad you like the idea! I was thinking 0-5V, 4-20mA (using a shunt), would be nice to have 30k thermistors (pretty cheap and lots of them), digital pulse counting and maybe also 1-wire?!?! Of course, the 10bit A/D might be a small stumbling block, but to be honest, most applications really don't really require anything with a higher resolution. Also, have the ability to trigger a digital channel on an alarm condition.

As for software on the PC side, I was thinking a simple/basic programming interface, for example, the user could enter the sensor type (i.e. 0.5 to 4.5V), the scaling and logging rate. Then can download the data and export it as a CSV format, all straight forward stuff. I guess my main thing would be packaging, making it as simple as possible for the non-technical user.

What do you all think?

#4

I'd enter the competition if the price was nice. :)

You'd probably have a class for each kind of sensor and store the config in the inbuilt EEPROM (512 bytes). The tricky bit is storing large amounts of data. Either a flash chip or a SD card would do but I never had any luck with the SD.

How do you usually interface with data loggers? Is sending commands via serial the best idea? That makes it reasonably easy to dump data and change the configuration.

#5

I'm aware of a usb memory stick logger solution for the Arduino. Is this different from what you are proposing?

http://www.arduino.cc/playground/Main/UsbMemory

#6

Hey,

Sorry for the delay, so many projects on at the moment. So, yes, memory would be an issue. USB is a great idea, maybe SD, but then is it needed? I guess problems might arise if we wanted to keep this a lower power device. Certain small loggers out there have a small 64k ~ 256k worth of memory, great for small jobs and the majority of the applications. No one be looking at storing GB's of data as no one is really looking at any high speed data acquisition. A sample every minute is enough for most of these applications.

Most units out there still use RS232 for data download and configuration. After all, with 256k of memory, its not a huge problem downloading that over the serial port. All the big players just have a command to download the data.

Give you an idea of applications, I had two this week, one was from a university looking at weather station application. The ideal situation would be to have several low cost stations at remote sites. The one wire station from AAG would be ideal, but they also want to store the data from the sensors. They didn't have the budget for one basic remote weather station, let alone several units. The other was actually from a charity dealing with distributing medical aid to Africa. They have small temperature loggers in place, but the costs add up for an installation with four or five refrigerators, so they would like one central data logger. Again, there budget was $200 and I could not find anything that would do it.

As for prize money, I can certainly put something down, maybe pull in a few other companies as well. Again, I would want to keep this all on an open source platform. Allow other people to copy and improve on the design.

#7

The whole thing should be reasonably easy. I might have a go when I get my new decimelia. You can get a 64k eeprom from futurlec for $2.20 (http://www.futurlec.com/ICEEPROM.shtml) so it shouldn't cost too much, an eeprom, 4051 for more inputs and some kind of shield or circuit board to hold it all. You might be able to put the whole thing together with a real time clock for not much. I'm gonna have a go at interfacing an SD card soon and if that works then it would probably end up cheaper and easier to do that (plus u have the benefit of all that storage).

#8

The tricky bit is storing large amounts of data.

I would have though the tricky part was getting the Arduino to live long enough off a battery to get a reasonable logging period. Or, did you anticipate that people would be using wall-warts to plug these loggers into a socket?

As for the amount of memory you need, I've been playing arond with a design for a simple temperature logger, and have found that, using a simple compression scheme, I can get one to three days worth of per-minute temperature readings into the on-board 512-byte EEPROM. My plan is to use cheap 256kbit eeproms (MUCH larger than the onboard eeprom) for the deployed version of this device, and so memory doesn't look like it's going to be an issue.

My main concern right now is how long the Arduino can run on battery power. Without some tricks, it sounds like it won't go long (less than half a day). I imagine this will require putting the board to sleep in between readings; I know the ATmega168 supports a few sleep modes, but I'm not sure yet how to use them. It seems like the best solution might be just put the board into a full sleep and add an RTC that can send periodic signals to the Arduino to wake it up.

Have you guys thought about this?

#9

Hey, neat idea. I'd like a shot at it.

We'd need to know exactly what the requirements are from the user's point of view. And the criteria for picking a winner. Oh, and the prizes of course!

Thanks,

#10

I would even be prepared to start some sort of contest to kick start this project, throw in some prizes to generate some interest.

Anyone feel that this would be worthwhile?

I think a contest with prizes would discourage people from freely sharing their ideas and collaborating to solve problems. Maybe it's just me, but it seems antithetical to the sprit of the Arduino community.

#11

I have a need for a data acquisition solution and I would be interested to know if it would fit into the scope of this data logger project.

I have a Wood Fired Brick Oven that I built in my back yard. It has several K Type Thermocouples (5 but other ovens could have up to 8) embedded in the mass to measure the temperature of the oven as it heats over several hours. I connected these thermocouples to my PC via a TC-08 Thermocouple Logger from PicoTech (See:http://www.picotech.com/thermocouple.html - I bought the older serial version) and ran a serial cable into the house ( about 50ft). This worked for a while but I the elements got to the logger and it stopped working. The latest version is USB and it's $500 without wireless capabilities. For a replacement I was thinking about a wireless version using Arduino and the XBee shield. To get 8 channels I guess it needs an multiplexer (or is it a demultiplexer?) module, and possibly a thermocouple A/D converter chip, to poll the thermocouples and communicate to the PC.

This real world example seems to highlight some of the common issues with data acquisition/logging applications: - Harsh Environments a the point of acquisition - Remoteness of the data acquisition point - Real Time data is required. In this case it needs to log the data to a real-time display. Logging to a local data store is not an option - Data visualization. It needs a software component to graph the results.

Another logging application I have in mind for Arduino is a GPS logger to geotag digital photos. This would involve logging time and location to a data store (SD for example) and then combining that information with the digital photos based on the timestamp on the photo.

If anyone is interested in a collaborative project for either of these applications. I would be interested. I do not think it needs a competition if there are members of the community that have a common goal. I see advantages for everyone by working on these types of applications under a common data logger/acquisition project. The results could be more modular.

Thanks, Robin

#12

Maxim makes an SPI interfaced K type thermocouple interface, complete with reference junction, amplifier, ADC, etc. The Arduino can easily handle 8 of them via SPI. The arduino can also drive an LCD or LED display.

Bluetooth is another wireless option. The advantage over zigbee is that your computer may already have bluetooth. can't remember the max range off the to of my head, though...

-j

#13

I did a bit of work on this and got an SD card working (reading/writing) no problem which makes it very easy (and cheap!) to store large amounts of data. Using even a small SD card u could capture data at reasonably high sample rates for a long time. I also added a max1307 real time clock to add timestamps. I did a bit of work on an arduino shield for the circuit with all of this on it but haven't sent it off yet.

To get 8 channels I guess it needs an multiplexer You can add an 4051 which will give you 8 channels on one analog input and use 3 digital lines.

Real Time data is required/Data visualization This is actually easier than storing it in a storage device. Just write the results you get to serial. On the pc side read them in and graph them (processing can do this) other options could be python or VBA. There is also a few programs around to do this. A quick google found this one http://www.live-graph.org/

I'll post up my circuits and code once I have it working ok and everything is cleaned up and its not too embarrassing :P

#14

everything is cleaned up and its not too embarrassing

I would like to start a campaign on these forums for people to start releasing code when it's still at the "embarassing" stage. The number of times a thread ends with a post saying "I'll release it when it's tidied/not embarrassing/perfect" and no further code seems disturbingly high.

Embrace your code's imperfections in the face of its utility, release now!

In a month's time untidy, embarrassing code is still going to be a lot more useful to others than no code--it also probably won't impact your future employment/dating opportunities too much. :-)

--Phil.

P.S. This isn't aimed specifically at you, this thread just happened to be active as the thought occurred to me. :-)

#15

everything is cleaned up and its not too embarrassing

I would like to start a campaign on these forums for people to start releasing code when it’s still at the “embarassing” stage. The number of times a thread ends with a post saying “I’ll release it when it’s tidied/not embarrassing/perfect” and no further code seems disturbingly high.

Embrace your code’s imperfections in the face of its utility, release now!

In a month’s time untidy, embarrassing code is still going to be a lot more useful to others than no code–it also probably won’t impact your future employment/dating opportunities too much. :slight_smile:

I agree! I think few of us are real professionals here and we’re all here to learn, even a brain dump of imperfect information thrown back into the cloud is better than nothing at all, and we can all benefit from dissecting and optimizing what you’ve got so far. :slight_smile:

#16

It’s great that you brought this up. I’m actually working on using an Arduino to store data off to a USB thumb drive. I’m logging from one USB port and saving to the thumb drive. Just need to get the closing of files working correctly.

#17

I'm a co-op student working for a company right now and am highly interested in working on this project. I do have some questions about the actual operation of the data logger though. 1. How good is the resolution? I know it's 10 bit, but what is the max value for the analog input? Our main need here is thermocouples and 1 mV can be 20 degrees difference or more. 2. How could we get more inputs? I know you can get more boards or more A/D converters and then put them on the digital ins, I was just wondering if there was an easier way to do this?

My main goal is to get as many thermocouple inputs as possible. Any code I write, no matter how "dirty" would be posted back.

Sorry if most of the questions have been answered elsewhere, I'm just now getting into the arduino board community. Most of my experience is with Linux based microcontrollers.

Thanks. Ryan Tenn. Tech U.

#18

Hi Ryan, by default the arduino uses 5 volts for the ADC reference. As I recall from the datasheet, you can use an internal reference of 1.1 volt giving about 1mv resolution. The guaranteed accuracy is plus/minus 2 bits.

There are six analog inputs and you can add an external multiplexer if you need more.

#19

Has anybody taken on this project? I have a few ideas for a modular system. I think better resualts could be gained by using standalone ADC chips, combined with a modular system, you could just plug in the modules you need!

#20

hey Ryan,

Maxim has a couple of thermocouple interfaces (MAX6674 MAX6675) that would attach to the Arduino via SPI. Since SPI is a bus, requiring a separate chip select for each device, you could easily put a dozen or so on an Arduino as-is. With an IO expander IC you could add even more.

I've got these on my list to play with, but haven't had time yet.

-j

(TTU class of '90)