Arduino Forum

Forum 2005-2010 (read only) => General => Frequently-Asked Questions => Topic started by: eastcoast on Dec 06, 2007, 07:25 pm

Title: Open Source Data Logger Project Using the Arduino?
Post by: eastcoast on Dec 06, 2007, 07:25 pm
Hey Guys,

I am heavily involved in the data logger/acquisition industry but I feel that its missing something, and that would be an open source data logger.  The issue I come across is pricing, I deal with customers all over the world and find that 90% of my clients (mostly educational) just don't have the budget for an easy to use, compact data logger, any stand alone unit with 8 channels is going to cost a minimum of $500, add the software, cable, etc...soon gets out of hand.

So I was thinking that an open source logger based on the Arduino could be an ideal solution, the biggest issue would be having something easy to use for the end user, a nice front end on the software, but allow the more technical to make modifications for the odd sensor.

I would even be prepared to start some sort of contest to kick start this project, throw in some prizes to generate some interest.

Anyone feel that this would be worthwhile?
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: MikMo on Dec 06, 2007, 08:51 pm
Thats a cool and interesting idea.  What kind of sensor input are you thinking about, and what should the software on the PC side be able to do ?
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: eastcoast on Dec 07, 2007, 12:19 am
Hey,

Glad you like the idea!  I was thinking 0-5V, 4-20mA (using a shunt), would be nice to have 30k thermistors (pretty cheap and lots of them), digital pulse counting and maybe also 1-wire?!?!  Of course, the 10bit A/D might be a small stumbling block, but to be honest, most applications really don't really require anything with a higher resolution.  Also, have the ability to trigger a digital channel on an alarm condition.

As for software on the PC side, I was thinking a simple/basic programming interface, for example, the user could enter the sensor type (i.e. 0.5 to 4.5V), the scaling and logging rate.  Then can download the data and export it as a CSV format, all straight forward stuff.  I guess my main thing would be packaging, making it as simple as possible for the non-technical user.

What do you all think?
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Dec 07, 2007, 09:29 am
I'd enter the competition if the price was nice. :)

You'd probably have a class for each kind of sensor and store the config in the inbuilt EEPROM (512 bytes).
The tricky bit is storing large amounts of data. Either a flash chip or a SD card would do but I never had any luck with the SD.

How do you usually interface with data loggers? Is sending commands via serial the best idea?
That makes it reasonably easy to dump data and change the configuration.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: rappa on Dec 10, 2007, 03:10 am
I'm aware of a usb memory stick logger solution for the Arduino.  Is this different from what you are proposing?

http://www.arduino.cc/playground/Main/UsbMemory
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: eastcoast on Dec 13, 2007, 11:06 pm
Hey,

Sorry for the delay, so many projects on at the moment.  So, yes, memory would be an issue.  USB is a great idea, maybe SD, but then is it needed?  I guess problems might arise if we wanted to keep this a lower power device.  Certain small loggers out there have a small 64k ~ 256k worth of memory, great for small jobs and the majority of the applications.  No one be looking at storing GB's of data as no one is really looking at any high speed data acquisition.  A sample every minute is enough for most of these applications.

Most units out there still use RS232 for data download and configuration.  After all, with 256k of memory, its not a huge problem downloading that over the serial port.  All the big players just have a command to download the data.

Give you an idea of applications, I had two this week, one was from a university looking at weather station application.  The ideal situation would be to have several low cost stations at remote sites.  The one wire station from AAG would be ideal, but they also want to store the data from the sensors.  They didn't have the budget for one basic remote weather station, let alone several units.  The other was actually from a charity dealing with distributing medical aid to Africa.  They have small temperature loggers in place, but the costs add up for an installation with four or five refrigerators, so they would like one central data logger.  Again, there budget was $200 and I could not find anything that would do it.

As for prize money, I can certainly put something down, maybe pull in a few other companies as well.  Again, I would want to keep this all on an open source platform.  Allow other people to copy and improve on the design.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: agent_orange on Dec 15, 2007, 11:37 pm
The whole thing should be reasonably easy. I might have a go when I get my new decimelia. You can get a 64k eeprom from futurlec for $2.20 (http://www.futurlec.com/ICEEPROM.shtml) so it shouldn't cost too much, an eeprom, 4051 for more inputs and some kind of shield or circuit board to hold it all. You might be able to put the whole thing together with a real time clock for not much. I'm gonna have a go at interfacing an SD card soon and if that works then it would probably end up cheaper and easier to do that (plus u have the benefit of all that storage).
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: TomP on Jan 26, 2008, 05:04 am
Quote
The tricky bit is storing large amounts of data.

I would have though the tricky part was getting the Arduino to live long enough off a battery to get a reasonable logging period.  Or, did you anticipate that people would be using wall-warts to plug these loggers into a socket?

As for the amount of memory you need, I've been playing arond with a design for a simple temperature logger, and have found that, using a simple compression scheme, I can get one to three days worth of per-minute temperature readings into the on-board 512-byte EEPROM.  My plan is to use cheap 256kbit eeproms (MUCH larger than the onboard eeprom) for the deployed version of this device, and so memory doesn't look like it's going to be an issue.  

My main concern right now is how long the Arduino can run on battery power.  Without some tricks, it sounds like it won't go long (less than half a day).  I imagine this will require putting the board to sleep in between readings; I know the ATmega168 supports a few sleep modes, but I'm not sure yet how to use them.  It seems like the best solution might be just put the board into a full sleep and add an RTC that can send periodic signals to the Arduino to wake it up.  

Have you guys thought about this?
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: ckiick on Jan 26, 2008, 05:51 pm
Hey, neat idea.  I'd like a shot at it.

We'd need to know exactly what the requirements are from the user's point of view.  And the criteria for picking a winner.  Oh, and the prizes of course!

Thanks,
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: TomP on Jan 26, 2008, 06:17 pm
Quote
I would even be prepared to start some sort of contest to kick start this project, throw in some prizes to generate some interest.

Anyone feel that this would be worthwhile?


I think a contest with prizes would discourage people from freely sharing their ideas and collaborating to solve problems.  Maybe it's just me, but it seems antithetical to the sprit of the Arduino community.  
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: rcottiss on Feb 03, 2008, 05:14 pm
I have a need for a data acquisition solution and I would be interested to know if it would fit into the scope of this data logger project.

I have a Wood Fired Brick Oven that I built in my back yard. It has several K Type Thermocouples (5 but other ovens could have up to 8) embedded in the mass to measure the temperature of the oven as it heats over several hours. I connected these thermocouples to my PC via a TC-08 Thermocouple Logger from PicoTech (See:http://www.picotech.com/thermocouple.html - I bought the older serial version) and ran a serial cable into the house ( about 50ft). This worked for a while but I the elements got to the logger and it stopped working. The latest version is USB and it's $500 without wireless capabilities. For a replacement I was thinking about a wireless version using Arduino and the XBee shield. To get 8 channels I guess it needs an multiplexer (or is it a demultiplexer?) module, and possibly a thermocouple A/D converter chip, to poll the thermocouples and communicate to the PC.

This real world example seems to highlight some of the common issues with data acquisition/logging applications:


Another logging application I have in mind for Arduino is a GPS logger to geotag digital photos. This would involve logging time and location to a data store (SD for example) and then combining that information with   the digital photos based on the timestamp on the photo.

If anyone is interested in a collaborative project for either of these applications. I would be interested. I do not think it needs a competition if there are members of the community that have a common goal. I see advantages for everyone by working on these types of applications under a common data logger/acquisition project. The results could be more modular.

Thanks, Robin
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: kg4wsv on Feb 03, 2008, 07:49 pm
Maxim makes an SPI interfaced K type thermocouple interface, complete with reference junction, amplifier,  ADC, etc.  The Arduino can easily handle 8 of them via SPI.  The arduino can also drive an LCD or LED display.

Bluetooth is another wireless option.  The advantage over zigbee is that your computer may already have bluetooth.  can't remember the max range off the to of my head, though...

-j

Title: Re: Open Source Data Logger Project Using the Ardu
Post by: agent_orange on Feb 06, 2008, 07:39 am
I did a bit of work on this and got an SD card working (reading/writing) no problem which makes it very easy (and cheap!) to store large amounts of data. Using even a small SD card u could capture data at reasonably high sample rates for a long time. I also added a max1307 real time clock to add timestamps. I did a bit of work on an arduino shield for the circuit with all of this on it but haven't sent it off yet.

To get 8 channels I guess it needs an multiplexer
You can add an 4051 which will give you 8 channels on one analog input and use 3 digital lines.

Real Time data is required/Data visualization
This is actually easier than storing it in a storage device. Just write the results you get to serial. On the pc side read them in and graph them (processing can do this) other options could be python or VBA. There is also a few programs around to do this. A quick google found this one http://www.live-graph.org/



I'll post up my circuits and code once I have it working ok and everything is cleaned up and its not too embarrassing   :P  
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: follower on Feb 07, 2008, 02:49 pm
Quote
everything is cleaned up and its not too embarrassing

I would like to start a campaign on these forums for people to start releasing code when it's still at the "embarassing" stage. The number of times a thread ends with a post saying "I'll release it when it's tidied/not embarrassing/perfect" and no further code seems disturbingly high.

Embrace your code's imperfections in the face of its utility, release now!

In a month's time untidy, embarrassing code is still going to be a lot more useful to others than no code--it also probably won't impact your future employment/dating opportunities too much. :-)

--Phil.

P.S. This isn't aimed specifically at you, this thread just happened to be active as the thought occurred to me. :-)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: blalor on Feb 07, 2008, 03:01 pm
Quote
Quote
everything is cleaned up and its not too embarrassing

I would like to start a campaign on these forums for people to start releasing code when it's still at the "embarassing" stage. The number of times a thread ends with a post saying "I'll release it when it's tidied/not embarrassing/perfect" and no further code seems disturbingly high.

Embrace your code's imperfections in the face of its utility, release now!

In a month's time untidy, embarrassing code is still going to be a lot more useful to others than no code--it also probably won't impact your future employment/dating opportunities too much. :-)

I agree!  I think few of us are real professionals here and we're all here to learn, even a brain dump of imperfect information thrown back into the cloud is better than nothing at all, and we can all benefit from dissecting and optimizing what you've got so far.  :)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: AVRman on Feb 12, 2008, 06:37 am
It's great that you brought this up. I'm actually working on using an Arduino to store data off to a USB thumb drive. I'm logging from one USB port and saving to the thumb drive. Just need to get the closing of files working correctly.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Zujin on Feb 13, 2008, 06:47 pm
I'm a co-op student working for a company right now and am highly interested in working on this project.  I do have some questions about the actual operation of the data logger though.
1.  How good is the resolution? I know it's 10 bit, but what is the max value for the analog input? Our main need here is thermocouples and 1 mV can be 20 degrees difference or more.
2.  How could we get more inputs?  I know you can get more boards or more A/D converters and then put them on the digital ins, I was just wondering if there was an easier way to do this?

My main goal is to get as many thermocouple inputs as possible.  Any code I write, no matter how "dirty" would be posted back.

Sorry if most of the questions have been answered elsewhere, I'm just now getting into the arduino board community.  Most of my experience is with Linux based microcontrollers.

Thanks.
Ryan
Tenn. Tech U.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: mem on Feb 13, 2008, 07:06 pm
Hi Ryan, by default the arduino uses 5 volts for the ADC reference.  As I recall from the datasheet, you can use an internal reference of 1.1 volt giving about 1mv resolution. The guaranteed accuracy is  plus/minus 2 bits.

There are six analog inputs and you can add an external multiplexer if you need more.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: phill on Feb 14, 2008, 10:32 pm
Has anybody taken on this project? I have a few ideas for a modular system. I think better resualts could be gained by using standalone ADC chips, combined with a modular system, you could just plug in the modules you need!
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: kg4wsv on Feb 15, 2008, 06:29 pm
hey Ryan,

Maxim has a couple of thermocouple interfaces (MAX6674 MAX6675) that would attach to the Arduino via SPI.  Since SPI is a bus, requiring a separate chip select for each device, you could easily put a dozen or so on an Arduino as-is.  With an IO expander IC you could add even more.

I've got these on my list to play with, but haven't had time yet.

-j

(TTU class of '90)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Feb 16, 2008, 08:25 am
Quote
Has anybody taken on this project? I have a few ideas for a modular system. I think better resualts could be gained by using standalone ADC chips, combined with a modular system, you could just plug in the modules you need!

I'd be interested in making one if it was a competition or something.
I just dont have any need for a data logger myself though.

kg4wsv I2C would be a better candidate.
Up to 128 sensors with only two wires.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: kg4wsv on Feb 16, 2008, 02:55 pm
Quote
kg4wsv I2C would be a better candidate.
Up to 128 sensors with only two wires.

SPI has two distinct advantages: SPI is much much faster that I2C (25MHz vs 400kHz, IIRC), and, in this case, the thermocouple interface ICs are only available in SPI.  :)

On the number of nodes on the bus issue, I have noticed many ICs that have fixed addresses, or allow only a few bits of the address to be set by the designer, so practically this limit is usually much lower than 127. For example, the DS1631 that was discussed on the forum recently has 3 pins (3 bits = 8 possible addresses) available.

-j

Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Feb 17, 2008, 01:46 am
SPI can hit 25mhz theoretically but the Arduino doesnt go that fast anyway.

Apparantly you can get I2C hubs which allow more.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Zujin on Feb 18, 2008, 07:14 pm
I have been looking for I2C thermocouple ICs and I just can't seem to find one.  Like K said, only SPI.  Since the MAX chips are more expensive than an analog mux, I think I'm going to use one MAX6675 and then just mux all my inputs through it.  It's easy and dirty but should work.  I have a few other ideas I'm still messing around with before I start ordering samples.

This method also leaves open my analog inputs since we may need to take voltage/current readings at some point.

By the way, speed is not a priority for me since most of our measurements are 2 hours apart, but I will try to make it as fast as possible.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 25, 2008, 09:08 am
Hi everyone,

I am a biologist from a wildlife research group in Australia. I have next-to-no experience with electronics, but I am wondering whether a novice like me could learn to build a simple GPS datalogger based on Arduino boards (preferably the Arduino Mini) for use in wildlife tracking? I would appreciate comments from the community here as to whether this is something a newbie like me could attempt, and whether anyone might be willing to assist? Obviously any progress made will be posted here for others to use. Look forward to comments.

Thanks
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Feb 25, 2008, 12:25 pm
Yes a newbie could do it.

All you need is the processor (Mini), a GPS module and some form of memory (I2C EEPROM is easiest).
Whack them together and you have a basic datalogger.

If you want to monitor birds or small animals, I'd leave it to the experts though.
It wouldnt be the smallest datalogger in existence. ;)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: blalor on Feb 25, 2008, 12:28 pm
Quote
Hi everyone,

I am a biologist from a wildlife research group in Australia. I have next-to-no experience with electronics, but I am wondering whether a novice like me could learn to build a simple GPS datalogger based on Arduino boards (preferably the Arduino Mini) for use in wildlife tracking? I would appreciate comments from the community here as to whether this is something a newbie like me could attempt, and whether anyone might be willing to assist? Obviously any progress made will be posted here for others to use. Look forward to comments.

Thanks


I think that kind of depends on what you're a novice at, but it's not outside the realm of possibility at all.  Start small.  Get an Arduino and start playing with it.  Then find a suitable GPS receiver and start playing with that.  Figure out how to get the position and other data out of it that you want and send it back to the computer over serial.  Then find a way to log your data to some kind of storage device (probably the hardest thing to do right now, from what I've seen) and tie it all together.  I think if you break it down into smaller pieces, you'll be ok.  I'd recommend you find a good tutorial on C and C++ if you're not already conversant in it, however.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: mem on Feb 25, 2008, 01:21 pm
The easy part is connecting the GPS, there is some information on connecting a GPS in the  Arduino playground  (http://www.arduino.cc/playground/Learning/Tutorials), this one is a good place to start: http://www.arduino.cc/playground/Tutorials/GPS

The challenge will packaging the complete system to be small and robust with a suitable power source, a practical means to get the data out, and an antenna layout that has sufficient visibility to GPS satellite data when attached to wildlife.

You may want to start a new thread where you can say a little more about your requirements and encourage contributions on various solutions. I would definitely think its worthwhile pursuing further.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 25, 2008, 02:04 pm
Hi Cheater, blalor and mem,

thanks for the tips and encouragement. I guess it's a case of diving in and giving it a go. A separate post might be a great idea if there is enough interest.

We've been trialling the latest GPS dataloggers from Sparkfun, and as much as they're great little units, the "sleep" mode has a 1.5mA draw, that really takes its toll on battery life. Getting the sleep draw down to the lowest possible level is a major goal of a new logger. A recent post on the Arduino playground discusses sleep draw: (http://www.arduino.cc/playground/Learning/ArduinoSleepCode).

From what we can tell, a major portion of active draw in the SF logger seems to be related to NMEA sentences being written to the SD card. Fast(er) fix acquisition and a less intensive approach to writing/storing fix information is another goal.

As you guys have pointed out, size is also a limitation. We can probably work with a final board measuring ~50mm x 50mm and ~30mm high (although the smaller the better). The Sparkfun board measures 38.1mm x 44.5mm x22.9mm), which is close to ideal. Patch antennas are a bit of drama to work with and ultimately we'd like to mount the GPS logger complete with antenna and battery on the underside of collars. At the moment, either the antenna or both the logger and the antenna have to be top mounted, which makes fitting in teh field a drama.

Thanks very much for the replies guys, and I look forward to more discussion.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: kg4wsv on Feb 25, 2008, 04:53 pm
GPS units, by default, write out a set of sentences once a second, but this default can be changed.  Perhaps the Sparkfun unit would be less power hungry if you reduced the number of NMEA sentences to the minimum that you need, then reduce the rate at which the devices transmits sentences?

Having said that, I would assume the GPS receiver itself is probably the biggest power hog in the system.  The Garmin GPS18 is  50 to 65mA (depending on which model you choose).

-j

Title: Re: Open Source Data Logger Project Using the Ardu
Post by: mem on Feb 25, 2008, 05:49 pm
I would think everything but a timer would be powered down between readings if battery drain is a critical factor. Perhaps WHEB could say at what frequency he would like to log readings.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 25, 2008, 09:40 pm
Hi kg4wsv,

Thanks for the reply. According to the manufacturer's specs (USGlobalsat Inc.) power consumption of the GPS engine board (model EM-406a SiRF III) used in the Sparkfun GPS logger is 44mA. However, Sparkfun list it as 70mA (?). Sparkfun list the total power consumption on the GPS datalogger as 160mA, so I figured that the bulk of the difference was due to logger itself (I could be way off however).

You raise a really interesting idea with the notion of limiting the sentence output. We've discussed the idea here but our utter lack of electronics/programming knowledge meant it was a bit of a pipe dream. Many of the sentences are of little use to us and we really only need date, time, lat/long and HDOP values (possibly satellite number as well to help determine fix quality). Almost all of our projects require reasonably long deployment periods (from two weeks to many months), so fix logging intervals are usually set to between five minutes and one hour, with the unit being in sleep mode the rest of the time. Intermittent operation obviously boosts battery life/deployment (although we're finding some interesting results in terms of the benefits of longer intervals - - i.e. there doesn't seem to be many), but do you think reducing sentence output would have much of a positive impact? Would certainly make things easier when transferring data over at the end of deployment, regardless.

I am wondering how difficult it would be to interface the same GPS engine board above with an Arduino board to play around with sentence output etc? I realise this is all a bit ambitious for a newbie, but I am happy to jump in and try, particularly if there are other people interested.

Cheers
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 25, 2008, 10:13 pm
Hi Mem,

I probably covered fix rates a bit in my last post, but to clarify, we're definitely keen on hourly (or less) fixes. Our static tests are revealing some intriguing battery life projections for five minute fixes, which seem to be the best of both worlds in terms of accuracy (very good average DOP values & Sat #) and operational time (~8secs - 10secs to get a fix with a hold-off value of five). We've run the logger for 7 days at a five minute fix rate using two Duracell DL123 batteries in series (~1550mA capacity) and voltage only dropped from 6.4v to 5.89v. The SF GPS logger tolerates voltages between 4v and 7v, so we were very pleased with these preliminary results, as we might get a couple of weeks of deployment at least out of this set-up. Sleep draw of the SF logger really seems to be the major limiting factor (1.5mA) in deployment projections.

Cheers
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: fornzix on Feb 26, 2008, 01:22 am
WHEB,

Does the thing you're measuring ever stop moving???  If so, you can wire things up to power down with no movement.  Your last GPS reading would be the location that the thing stopped.  If there is constant motion, then this wouldnt help.  I just did something similar and it seems to be working great so far.  
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: kg4wsv on Feb 26, 2008, 04:15 am
If you're talking about report intervals to something on the order of tens of minutes to hours with the GPS asleep between events, reducing GPS sentences won't help enough to be noticeable.  When I mentioned it, I was thinking of a continuous (e.g. a report on the order of once a minute).

I would check out the GPS start time.  Especially with 5 minute intervals, you may be running the GPS a majority of the time, as 5 minutes to first fix from a cold start is not unreasonable to expect.

I like the 123As for physical size, but the Energizer AA lithiums actually have more capacity, if you can stand the four AA form factor to get 6V (compared to the two 123As).

-j

Title: Re: Open Source Data Logger Project Using the Ardu
Post by: follower on Feb 26, 2008, 04:58 am
Quote
I am a biologist from a wildlife research group in Australia.

There are a number of Australians that hang out here, you might want to be more specific about where you're based--you might be living next door to one. (Although I don't think Aussie Arduino users are required to be registered yet... :D )

On the subject of battery life, you might want to look at:

 http://www.faludi.com/projects/arduino-and-xbee-battery-test-results/

Keep in mind that if your processing needs are low you have the option of using lower-power and/or frequency parts.

--Phil.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: fornzix on Feb 26, 2008, 05:00 am
WHEB,
I gave it some more thought and if your project can withstand an hour or even a half hour between gps pings, then powering down might not be a bad option, even if there is constant motion.  Also,  you might want to check out batterys like http://www.megabatteries.com/item_details2.asp?id=14282&cat_id=404&uid=1638.  You can regulate the voltage and get a lot more mAh out of them.  Food for thought.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 26, 2008, 06:36 pm
Hi everyone,

I have to say this is a great forum and thanks for all the responses. I've found little interest in some other non-Arduino forums. I'll try to reply to each point separately.

fornsix
We're tracking dingoes and wild dogs, which are always on the move just about. I guess it would depend on how sensitive the movement "switch" was in terms of extending battery life. For example, if it switches on with head movements, it might never get a rest. If it can be tailored to trigger upon a certain degree of movement, it may well result in some savings. Thoughts?

kg4wsv
I should have qualified my earlier point that the SF logger actually already allows NMEA sentences to be partially restricted to pretty much only those sentences that you want. As far as we can tell, it's not possible to only store parts of sentences though (i.e. it's either a whole sentence or nothing). There are also some issues with the way the data is stored when it comes to importing into something like a spreadsheet app., but this is more of a pain then a major problem. It would be great if we could chop and change individual sentences to get exactly what we want without a lot of messing around, but it is probably easier to do this at the end of a given deployment using existing GPS applications rather than trying to program the board to do it in the field. You're right that the Energizer AAs have more capacity than the 123s, but weight is a bit of an issue for some of our projects. The case we're using in an upcoming project is too small to fit AAs, or even three/four AAAs, but the stocky nature of the 123s, coupled with the fact that we only need two, means we can "cheat" a little and embed the batteries on the side of the case using epoxy and still keep the weight down. The case has to be top-mounted to the collar because the SF loggers have an integrated antenna. Even if they didn't though, I actually like the top-mount, because trying to fit/adjust collars in the field and dealing with a length of coax (or something similar) leading to an antenna is a nightmare. To stop the collar from spinning around, we mount a collar drop-off device and a few other components near the bottom side, which all adds to a little more weight than the GPS case.

We might still end up using AAs for medium term deployments however.

follower
We are based in Brisbane so it would be great if there were a few Arduino aficionados nearby who might be interested in helping. Thanks heaps for those battery life test results as well. Like the SF GPS datalogger I imagine getting the most out of sleep modes could have a lot of impact in terms of battery life.

fornsix
Thanks for the link. We've looked at using similar batteries for long-term deployments, and we still hope to, but there have been some issues relating to the max. current rating of these large capacity D- and C-cells. We were hoping (read pushing our luck) that a single cell would cut it, but the voltage drop/draw when the GPS was operational ended that idea. Using two cells in series seems to work, but we're yet to do proper bench tests to ensure they're going to last in the field. We're just using a diode to drop the voltage of the pack, which seems to work ok.

Cheers everyone, much appreciated.

Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Feb 26, 2008, 09:45 pm
Quote
We are based in Brisbane so it would be great if there were a few Arduino aficionados nearby who might be interested in helping.

Hi
I am your local Arduino expert. ;)
AFAIK I'm the only one in Brisbane.

You can use a single cell and boost it to 5v fairly easily.
Maxim has a chip which does this which I'm currently using in a project.
It gives a stable 3.3v or 5v from as little as 0.8v.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: WHEB on Feb 26, 2008, 11:22 pm
Hi Cheater,

I can't believe you're in Brisbane! Small world huh?

The voltage booster sounds like an interesting idea. Would there still be a potential problem with the max. current of the large capacity lithium batteries, as they're only rated to 200mA continuous (some only 100mA) and 400mA pulse? Given that the GPS logger is supposedly using 160mA when active, I guess it shouldn't be a problem for 200mA+ batteries. Would the Maxim chips be much of drain on batteries? We've also been messing around with some mini solar panels (http://www.powerfilmsolar.com/products/oem_components/modspecs/sp337.htm) in the hope of trickle charging collar batteries and these chips might be perfect for boosting voltage.

Another question I have in relation to GPS and the Arduino boards is whether it might be possible to set up something to trigger a small wireless camera on a collar when a collared animal gets close to waypoints? We tested a prototype wireless camera collar last year with great success, and since then I've been looking to develop the camera collar idea further. I've been interested in GPS controlled devices on-board collars for ages, but my lack of electronics knowledge has left me day dreaming about such ideas.

I am due for some fieldwork within the next couple of weeks, which will see me away for about six weeks, but if you were keen, it'd be great to follow up on some of these ideas upon my return.

Cheers
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: Cheater on Feb 27, 2008, 12:02 am
Quote
I can't believe you're in Brisbane! Small world huh?

Its a very small world. :)

Quote
Would the Maxim chips be much of drain on batteries? We've also been messing around with some mini solar panels (http://www.powerfilmsolar.com/products/oem_components/modspecs/sp337.htm) in the hope of trickle charging collar batteries and these chips might be perfect for boosting voltage.

The Maxim chip doesnt use much power but you do want to be careful.
It boosts the voltage by drawing more amps. They are over 90% efficent but I'd still be looking at using two of those batteries in parallel.

Quote
Another question I have in relation to GPS and the Arduino boards is whether it might be possible to set up something to trigger a small wireless camera on a collar when a collared animal gets close to waypoints? We tested a prototype wireless camera collar last year with great success, and since then I've been looking to develop the camera collar idea further. I've been interested in GPS controlled devices on-board collars for ages, but my lack of electronics knowledge has left me day dreaming about such ideas.

Thats pretty straight forward. Its just a matter of comparing the current position to a list of waypoints.

Quote
I am due for some fieldwork within the next couple of weeks, which will see me away for about six weeks, but if you were keen, it'd be great to follow up on some of these ideas upon my return.

Sure. :)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: agent_orange on Feb 27, 2008, 12:40 pm
In this thread http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1199737337/15#15

one of the arduino members is storing GPS data to an SD card using arduino. It might be helpful to ask him about his project and how its going.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: till on Jul 24, 2008, 11:28 am
I have been following this thread and was delighted when I read about this the other day:

http://www.ladyada.net/make/gpsshield/

Will order one of these soon. Has anyone made anything interesting with it yet? I'd be interested to know how many i/o pins it uses and if there's enough 'code space' to add 1wire temperature stuff or other sensors. One idea I had was to use it for geo-tagging. If I simply added a push-button that would timestamp co-ordinates on the SD card I could just link them to the photos.

Regards,

Till


Title: Re: Open Source Data Logger Project Using the Ardu
Post by: verstapp on Jul 24, 2008, 01:34 pm
More ladyada -
she does some analysis of the uses of MAX756 as a voltage booster as part of the design of her MintyBoost. The whole page is quite fascinating from a 'following the thought processes in getting from problem to solution' point of view.

http://www.ladyada.net/make/mintyboost/process.html

You're in Brisbane?! :)
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: mlindeblom on Aug 16, 2008, 05:19 pm
The DateTime library in the playground keeps time with no additional parts.  I am using Adafruit's resonator based USB boarduino with no time accuracy issues.

A hardware real-time clock to handle power loss would be required if not using batteries.

The new alarm library has the potential to simplify programming of a data acquistion system.

The ethernet library in release 012 would allow long cable lengths 100 meters and web based communication.

In summary, the Arduino platform has lots of potential in this area.
Title: Re: Open Source Data Logger Project Using the Ardu
Post by: joker on Sep 04, 2008, 09:59 am
Hi Robin,

I am exactly in this type of project, trying to make a rugged versatile data logger. After a couple of poor attempts, I focussed on the SD card as a memory device. My strong advise is: SPI! Not much litterature to read on that (except official doc), but not too tough neither. And so much room! I store data in a 512 bytes buffer which matches exactly the size of the SD sectors and leaves you a good additional 512 bytes for the code on the arduino. I placed all the strings in PROGMEM. Even my spare 16Mb SD looks huuuuuuuge!

My data logger can be programmed from a computer and parameters are not volatile. So that you can unplug from usb and plug to DC supply and control runs from one single button (start/stop/clear all runs :P). Runs (and their related headers, containing acquisition parameters) are stored sequencially using a tricky feature which prevents loosing the data in case of unexpected power failure. Rough specifications are: records interval from 10 ms to 1 hour (watch out the rollover of timer), averages from 1 to 16, channels from 1 to 6, programmable nbr of records (or infinite if 0). There is a list and print command for reading headers or data from each run.
Further development could be: LCD display, high speed DAC, programmable ranging, etc.
I do not plan to tease you just for fun, but I say: yes, it is possible! I will probably publish it soon ;)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Sep 18, 2008, 02:22 pm
Just a couple of words to keep you updated. Version 19 is under test. This version includes runs listing, runs printing (selectable), deletion, online help, test mode, upload mode, real time clock (bloody timer roll over  >:( ). But! I cannot shorten the cycle time below 40ms, due writing cycle time (I was looking for 10ms). May be I am a little too ambitious for waht is not a scope, right?
Next development steps are: self probe type recognition, and bulletproof microcode.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Oct 01, 2008, 05:07 pm
Self recognition of probe is running fine (8 # types can be defined). I managed to shorten the write delay to 30ms which is better but does not fully statisfy me. The articles that I found on the net are very encouraging: SD itself says that single block writing is prefered in the case of 512 bytes buffer size, multiple blocks works faster thanks to the fewest "waits for iddle after writting". The prototype fits in a 12x11x4 cm small box. On the other hand, I am reaching the limits of Arduino's memory! From this perspective, we are back to the early 80ies  :D

I am a little bit surprised by the lack of reactions to quite an active topic ;)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: solartoy on Oct 07, 2008, 02:49 pm
Hello Joker
I only just joined the discussion, please do not dispair!
Your project looks very promising.
Currently I am looking for a Arduino logger to monitor the energy used by my electric geyser. I need to simply measure the time it is used over a 1-2 week period, at the same time i also would like to monitor the geyser as well as the water inlet temperatur.
How far are you from publishing?
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Oct 07, 2008, 08:07 pm
Hi Solar toy,

What is an electric geyser  ? :o What is the temperature range you are interested in? In the begining, I will restrict the publication of the whole stuff to the members of my car club. We will see then. But I do not mind releasing portions of the micro code. Thus my questions  ;)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Oct 07, 2008, 08:11 pm
Beraking news: I cut down the writing time to 15ms thanks to, Oh yeah, I know this was trivial... a fast SD card...
The flash scan allows now one kind of scope acquisition (100µs resolution), 512 measurements for one channel, 256... for twice more, etc.
Now comes the question of uploading to a wireless network. Using the 300 bytes left in Flash memory...  :P
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: solartoy on Oct 07, 2008, 10:17 pm
Hello Joker
A geyser is the thing in your attic that usally stores 150l of hot water.
Without it showering is only half the fun.
I do not need a complex logger. One measurement every minute is enough. As long as I have 2 off 0-5V analog inputs and 2 digital inputs I can help myself further to connect the sensors. The unit needs to operate stand alone, its a bit difficult to bring the PC into the attic  ;)
Any help is much appreciated. I use a Diecimila.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Oct 13, 2008, 11:10 pm
I see! Do not ask me why, the trivia name for it in French is balloon (hot water balloon to be accurate) :o
I am in the process of fine tuning the prototype and its micro-code. You will have to use a shield for the SD card: you may consider Adafruit's one (blank GPS shield) or make your own from the Adafruit's proto shield and hack a 1$ USB SD card reader. Then you might not need all the functions that I designed (100 bytes left in Flash...). I will consider publishing a lighweight version.  8-)
Kind regards
Didier
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: guinnessimo on Dec 29, 2008, 07:07 pm
Joker,

Joyeux Noel!

I read with interest about your datalogger.  For many people, a generic datalogger (or datalogger library) could form the basis of their latest project.

SD memory is definitely the cheapest option, and being able to buffer 512 bytes in the Arduino RAM and then write a block to the SD card is probably the most efficient use of memory and power saving.

For many,  a few analogue inputs, which can then be interfaced to a variety of sensors (light, temperature, current, voltage etc) plus some digital pulse-counter channels would satisfy their requirements.

Are you close to publishing your code? I would be most interested.

I have an Arduino and Ethernet Shield and I am serving the output from some analogue sensors to the internet via Pachube - see it here:

http://www.pachube.com/feeds/1257


G

Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Dec 29, 2008, 09:09 pm
Hi Guinnessimo,

Thanks paying attention to my project. Where am I? Well, ahead from my early plans. What was a curiosity to me is slowly turning to a real project... I showed the prototype to the CEO of my group  8-), and he is very enthusiastic...  :)

So that the publication of my works depends on the production of the project which slightly moved towards the concept of an "intelligent sensor" for environmental monitoring. I will let you know  ;)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: guinnessimo on Dec 31, 2008, 01:55 pm
Joker,

Thanks for your quick reply.

I am interested in using the Arduino to monitor home energy consumption with the aim of allowing users to save energy and save money.

Additionally, the Arduino with an ethernet shield is one of the cheapest solutions (< 30 Euros)  to how you can serve real time data to the internet.

With many users sharing their home energy data, it then becomes a competition to use the least energy, and thus lower their domestic carbon footprint.

The Arduino could also be used to measure the output from renewable energy sources such as solar water heating, windpower and photovoltaic.  

If many people shared their renewable energy generation data via the internet, it could be used to build up national statistics about the quantity and distribution of renewable energy installations.  

If we are expected to reach 20% renewables by 2020, then we must find an effective way of monitoring our progress and logging the installations.

Already some people have solar water heating controllers, and electricity meters that output data - and perhaps the <30 Euro Arduino is a good low-cost way to publish this information to the internet.



G
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: joker on Dec 31, 2008, 03:01 pm
Hi Guinnessimo,

I have a strong concern for these energy saving questions. If I had the time... I would investigate the question of the "intelligent" house too. I feel a little sorry not to disclose my works so far.

About the Ethernet connection: As you know, France is part of the "old continent" and most houses are not fitted with networks or what so ever... Mine was built... 2 siecles ago! Thus my concern for wireless...
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: guinnessimo on Dec 31, 2008, 04:52 pm
Joker,

By 2020 the EU will have smart meters for electricity and gas.  Italy already has started fitting them now.

The utility companies are very slow to adopt new technology - they do not want to spend money unless they have to - or are forced by EU Law!

In the meantime, there are a few low cost energy monitors such as Electrisave/Cent-a-Meter, CurrentCost, Wattson and a few new ones just coming to the market.  Some of these have data output which can connect to a PC or serial logging device.

Someone has worked on the Electrisave wireless protocol so that it can be decoded by a PC or Arduino - using just a very low cost 433MHz receiver (taken from wireless doorbell).

For many people, just to see approximately how much energy they are using, and how much it costs, is enough to make them save.  The accuracy of the energy monitors is not too good - perhaps +/- 10%, but that is close enough for the average home user.

For those without ethernet or broadband internet there are several ways in which a logger can get data to a server via a plain old telephone line.  It is possible to implement the DTMF dial and simple modem functions into an ATmega168.

I hope that you will soon be in a position to share your project ideas more.


Bon Annee a Vous!


G

Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: guinnessimo on Dec 31, 2008, 06:59 pm
Joker, List,

Here is an excellent description of a home-built datalogger based around ATmega and SD card.

http://www.turtlesarehere.com/html/data_logger.html

It includes the code and the schematics/layout in Eagle format.

The hardware is simple to adapt from an Arduino.

The "Captain's Universe" is a good source for interfacing SD card to ATmega.

http://www.captain.at/electronic-atmega-sd-card.php


Happy New Year,




G

Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Brad Burleson on Jan 05, 2009, 06:10 am
Nice link G...

I've been working on my own device which boils down to a data logger with a gps and an XBee attached, so I've been following this thread with some interest as well.

I've also found these links to other GPS loggers to be interesting:

http://hackedgadgets.com/2008/11/30/gps-data-logger-using-an-sd-card/

http://www.gedanken.demon.co.uk/gps-sd-logger/ (the completed unit picture seems to be missing now)

At any rate, here's some more food for thought.

Brad.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: eastcoast on Feb 07, 2009, 08:04 pm
Gentlemen,

Its been a while since I last accessed this forum, my interest in a open source data logger based on the Arduino had to be placed on the shelf as I was dealing with a lot of other projects, the paying ones had to come first!

I am still dealing with several calls daily with projects where the Ardunio would be an ideal fit.  Basic research applications where they have a zero budget and an open source solution would solve all their problems.

I am thinking about creating a specific website which will act as a central resource for open source data logger projects, allowing the less technical to communicate with us with there project idea's.  Additionally, It might also work as a place for developers to share code between each other and exchange idea's.

I had plans on competitions, prizes, etc to generate interest and give some incentive for everyone to contribute, I have had both positive and negative comments on implementing this.

What do you you all think?  Good or bad idea?

I noticed that this thread has been viewed over 10,000 times!!  It would appear that there is a lot of interest out there.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: jluciani on Feb 07, 2009, 09:23 pm
I was thinking of developing a general purpose sensor interface peripheral
for my ZB1 board (http://tinyurl.com/5lnhtj). The new board would consist of
four channels of instrument amps or op-amps. I would include a reference,
a negative 5V supply and a little prototyping area. The datasheet
would include a detailed op-amp tutorial.

Since the ZB1 can be connected to a backplane other peripherals like an
SD card could easily be added. Also the ZB1 has an XBee so remote logging
is possible.

I would be interested to hear what types of sensors people use and what difficulties
were encountered.

(* jcl *)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Piez0 on Feb 24, 2009, 07:00 pm
Hi All,

I'm new to the forum.  I just want to say that this is a very interesting project.  Lots of monitoring in the geotechnical engineering industry require small loggers in the field.  Leaving a $500 logger out in the middle of no-where is not ideal.  This would be a great application for this project.

Title: Open Source Serial Data Logger Project
Post by: RyanN on Mar 13, 2009, 09:42 pm
Hello,
I have a need for a different type of data logger. I want to log incoming ASCII serial data to an SD card. I also want to send commands to the attached instrument. I'm thinking that I can use the Adafruit GPS/SD logger shield, but rather than a GPS, I'll attach my device (an ADCP). I haven't found any simpler or cheaper SD card shields.

I'm in the fortunate position of having plenty of (solar) power available, but I have some other applications where this may not be the case.

This could also be used to add memory to instruments where memory upgrades are either unavailable or expensive using the serial output.

There are devices such as the logomatic from sparkfun which comes close, but I'd like to build this around an arduino. I'd also like to use a FAT16 or even FAT32 formatted card. Unless I'm mistaken (which is probable) this shouldn't be too hard to do, especially with an Atmega328.

-RyanN
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: dtdisland on Mar 16, 2009, 04:48 pm
Hi! I'm new to this forum, so I apologize if anything I say is out of context.  As a mechanical engineer at a product design company, I'm very interested in a cheap data logger.

Is there a way that we can all put our minds together to create a "Product Requirements" document?  There seems to be a significant amount of knowledge and ideas, and it would great if we can compile them.  I'm not sure if this is the forum to make a collaborative document, but I'm willing to give it a shot!

From my brief skimming of the posts, I think that as requirements we have the following:

PRODUCT REQUIREMENTS
1) Completely Open-Source, running off of the Arduino Hardware
2) Portable, to record data "in the field"
3) Able to store data to some sort of mass storage (SD, USB, etc.)
4) Read inputs from a variety of instruments, most importantly:
    - temperature
    - GPS
    - fluid pressure
    - air speed
    - light
    - electrical inputs (voltage, current, etc.)
5) Interface with a GUI/graphing software

I know this is a couple steps back from what some people have done, but it would be nice to have a "common goal" in our collaborative efforts!
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: blalor on Mar 20, 2009, 11:32 am
Aside from the GUI requirement, I think that pretty much nails it.  At least for myself, I envision this thing being a headless, disconnected entity that does its sensing autonomously, and the user retrieves the data by grabbing the SD card.

I think a solid, reusable library for reading and writing to SD cards is what's really lacking, right now.  There are a few floating around, but nothing's really been well-packaged for use on the Arduino.  Now that the 328 is available, I think that should be a relatively straightforward endeavor.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Indianas_Back on Mar 29, 2009, 09:44 pm
Having just purchased a gigalog E datalogger for about 260 euros, I would very much welcome a cheaper version. The sort of stuff I need to log is GPS signals along side, side scan sonar and metal detector signals.  

Having never used a datalogger, the PDF manual I have could have been far better. There are too many abbreviations and a lack of clarity. If you are going to aim for a wider market then the manual will be all important.  

If you have the time, one question, If you are logging a signal from a headphone feed, there are three wires, left and right channel plus the ground. I take it only one of the channels needs logging, but does the ground wire need connecting? :)
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: guinnessimo on Jul 04, 2009, 05:26 pm
Hi,

I've managed to introduce the Arduino into my workplace.  We make energy monitoring displays, and we use the Arduino to control our automatic test equipment.

The open-source data logger is of interest to me, and I am keen to see it happen.  We currently use dataloggers from Picotech, but you always need the PC running and they are let down by flaky software and the inevitable crashes of windows.  

I'd just like a box that you could connect up like a multichannel multimeter with lots of 4mm "bananna" plugs that will log to data-flash or SD memory, and only needs a dc power supply to make it work.

We've just built a 6 channel relay board that the Arduino plugs straight into, and it also uses a 4051 analogue mulltiplexer to provide 8 additional analogue input channels.  We also buffer the PWM outputs to give us 4 channels of voltage output, one of which feeds through an opto-isolating op-amp to give a +/-10V swing - entirely isolated from the Arduino ground.  A small +/-12V dc/dc converter supplies the isolated power.

The board is proving useful as a common starting point for the automated test equipment system we are building to test our main product.  We've arranged it so that you can plug additional shields on top of the relay board, such as an ethernet shield for remote web control.

The next step is to provide a shield that carries an SD card for cheap data storage.  I'm looking at a 2 memory system consisting of an Atmel 32Mbit data flash and a separate SD card which can be removed and plugged into a PC for data retrieval.  Flash memory and SD cards are now dirt cheap so there is no real excuse these days not to have huge amounts of storage on a simple datalogger.

My interests include solar pV and solar waterheating - so I'm hoping to use the generic design as a solar pV datalogger and controller.

Other interests include low cost wireless and interfacing micros cheaply to the telephone network.  I'd love to have a generic wireless interface that could connect a $2 micro to the web for under $10. That will truly be one step further towards "the internet of things".

As I'm predominantly a hardware engineer, I'd be looking to partner with anyone who would be willing to colaborate on the firmware aspects of the project.


G.

Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: jumpjack on Oct 21, 2009, 09:29 am
I didn't read all the posts in the thread...
Is this project YOUR project?!?
http://www.openlogger.org/tikiwiki-2.2/tiki-index.php

Or maybe this?
http://openenergymonitor.org/emon/node/28

Or maybe you are working to a totally new project? In that case, which is its status?
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: RyanN on Oct 21, 2009, 02:22 pm
While I feel that an arduino based logger is still a great idea, especially now that the atmega1280 leaves plenty of pins and memory after you've hooked up the SD card, for our purposes, we have decided to go with a PC104 board running linux. Something like the TS-7800 or an Atom based board. This comes at a substantially increased cost and power consumption, but still within our parameters.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: RyanN on Oct 21, 2009, 02:25 pm
While I feel that an arduino based logger is still a great idea, especially now that the atmega1280 leaves plenty of pins and memory after you've hooked up the SD card, for our purposes, we have decided to go with a PC104 board running linux. Something like the TS-7800 or an Atom based board. This comes at a substantially increased cost and power consumption, but still within our parameters.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: akumar5 on Nov 02, 2009, 11:35 pm
Hi,

I am new to this forum and found this thread while searching for a way to build a data logger. I am very interested in building a data logger out of an arduino board but I am unsure if it is fast enough for the application.

I will be using it to sample and log GPS, accelerometer, and analog data from sensors on a race car. It will have to be able to log these at 5Hz minimum (10Hz prefered). And also possibly have an RS232 serial interface to log serial data from the engine control unit.

Do you that this is possible? Any help would be appreciated.

Thanks.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Phlogi on Feb 18, 2010, 01:23 pm
I'm having a similar question like akumbar.

The project I need to work on has specs as following:

- about 30 analog inputs
- resolution of adc: 16bit
- min. 20Hz log rate (not for all the signals, about 15, others slower)
- gps signal over rs-232
- saving to sd memory card

I'm wondering if arduino can handle all this. What would be the limiting elements? Atmega speed? Bus capabilities?

An idea was to connect some adc on the i2c bus and use analog multiplexer, what do you think?

Thanks a lot for your answers.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: PaulS on Feb 18, 2010, 01:40 pm
Quote
- about 30 analog inputs

With multi-plexing, yes. The Duemilanove only has 6 analog ports. The Mega has more, but not that many more.

Quote
- resolution of adc: 16bit

No. 10bit adc on the Arduinos.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Phlogi on Feb 18, 2010, 01:49 pm
Yes internally it has 10bit, but I could just attach some adc via the i2c bus, right?
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: PaulS on Feb 18, 2010, 02:18 pm
Attach what you like to the bus. But, then, it isn't inherently part of the Arduino.

The Arduino ADC is 10 bit.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Phlogi on Feb 18, 2010, 02:27 pm
We need additional electronic anyway to handle all of our inputs. So maybe we are designing a shield for that.

As it looks the arduino mega fits pretty well.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: imgonna on Feb 27, 2010, 04:44 pm
Hi

New to the forum, new to Arduino

As the Science director in a UK High School, what we need is not another data logger that is either too expensive or too complex -- but a data acquisition tool that "just works".

A basic box that had say:

a) 4 channels
b) RTC
c) Sample frequency from ms to daily
d) Basic LCD feedback
e) Minimal memory (cheapest possible) USB or SD Card

As far as schools are concerned, the logger is just a "black box" and it's the sensors that you plug in that make them useful.

A version that I decribed above should only cost $40 (ish) to make -- If such a device existed, they whould sell bucket loads in the UK.

This is the basis of my interest and research into the Arduino.

Cheers
G
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: RyanN on Feb 27, 2010, 06:51 pm
Glen,

Have you seen the Logomatic II?
http://www.sparkfun.com/commerce/product_info.php?products_id=8627
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: imgonna on Feb 27, 2010, 11:20 pm
No....

x8 Analog / Digital inputs -- wrap it into a neat box with some control circuitry -- might be just the job.

Still -- I'll press on with the Arduino -- makes an interesting project.

Good feedback - thanks
G
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: florinc on Feb 28, 2010, 04:38 pm
Glen, please check out the Duino644 board, Arduino-compatible:
http://timewitharduino.blogspot.com/2009/09/introducing-duino644.html
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: bsautner on Mar 11, 2010, 05:58 pm
Hey Guys

i've been working on an online data logging system built on google's app engine and i've pretty much decided to go open source with it. I'd really appriciate some feedback from Arduino users and perhaps even some help.

I've worked with data historians for 12 years and i wanted to write my own.

The system i built allows you do log time series data over a rest web service and your data is stored on google's infrastrucutre. I wrote a .net sdk and an way to view/record data on an android phone.  

i'm release the code soon under the MIT licence.  

I posted some code in my blog on how to wire devices into the web service - it works great, you can see values change in spread sheets and diagrams.

Anyway - i would really appriciate any feedback - you can contact me at bsautner@ (google's free email server)

thanks!

Ben
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: bsautner on Mar 11, 2010, 06:01 pm
Sorry, the web site for my data logging system is  http://www.nimbits.com

The source code will be posted on Google Code:

http://code.google.com/p/nimbits/

I spent 6 years working on this thing, so i hope it can be a good starting point for a cloud based data logger.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: tspringer on Mar 14, 2010, 06:37 pm
Getting back to issues involved in developing an open source data logger based on the arduino:

It seems to me that the development of a shield to handle analog signal conditioning is a key step in making the data logger a broadly appicablel tool.  The ten bit resolution of the Arduino A/D converter (ADC) is not an issue as long the signal from sensors is relatively linear and varies so that the voltage change covers a large fraction of the 0 to 5 volt scale.  However, in many cases, the range of sensor outputs will vary over a much narrower range, and programmable gain amplifiers will be needed to adjust the dynamic range of voltages that are presented to the ADC. (e.g. resolution of the ADC will not be adequate if sensor output voltages range from 0 to 0.5 V).  Also, as many sensor's output varies logarithmically or exponentially with respect to the  stimulus, an exponential or logarithmic conversion will be needed to prevent loss of resolution at the low end of sensor input.   Based on my limited knowledge of OP amps, it seems that a shield to handle this kind of basic signal conditioning might not be too expensive to make. Of course this leaves out some more specialized signal conditioning needs (e.g. cold junction compensation for thermocouples), but if the shield could handle simple variable gain amplification and logarithmic transformation, it would allow the Arduino's 10 bit ADC to serve adequately for many, many data collection situations.

T
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: paolometeo on Apr 21, 2010, 04:44 pm
Hi, I am new in this forum. I am very interested to invest my time and some money in a data-logger development based on Arduino. I think that the problem must be divided into some steps:  how to interface sensors to arduino input, that can be solved with easy electronic circuits;
how to get arduino writing data on a SD card or USB pen drive;
how to get enough power autonomy for long data logging (months).

In fact I think to use this sort of device to monitor slow environmental phenomena (i.e. wine cellar temperature  :)).

Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Phlogi on Apr 21, 2010, 07:49 pm
I can recommend the GHI uALFAT Module for data storing. Its working pretty well.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: Siegfried Loeffler on Jul 01, 2010, 05:28 pm
Ben,

had a look at your nimbits stuff - I like the idea of using google as a data store.

Do you have some sample arduino code that you would use ? I'd like to test the idea with some temperature logging...

Thanks
Siegfried
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: shodan on Jul 19, 2010, 01:35 am
instead of trying to make a "one size fits unknown" datalogger
I think the first thing would be to list all the sensor types likely to be hooked to this datalogger and from that determine what input conditionning will be required at minimum for more kinds of sensors
so at least it's known which input conditionning facilities should be on a standard datalogger and which type can be left off a add-on adapter board

also someone should do an inventory of all open source datalogger project that can be found and enumarate their features and scan them for "good ideas" to incorporate in this design


datalogger are often needed but they come in so many brands and model and industries because the people who measure temperatures over weeks don't know about the people who measure fuel/oxygen ratios during minutes and they don't know about the people who measure viscosity over months but they can all benefits from a single common datalogging core with just add-on for their specific application

datalogging for most is an expensive and R&D intensive affair that has no re-use value that's why people pay 500/1000/2000$ for a datalogger that have specs all over the place and there's a reason for that , there's big money to be made in keeping each project unique and personnalised and impossible to compare/shop around for

and lastly the most important of all, this project needs a good name that clearly indicate what it does to people who need it yet don't know what it is yet and it should be easy to find so that the next time someone needs to log the humidity of their bathroom over the next year they don't re-invent the wheel again !!!
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: tubular on Aug 22, 2010, 02:19 am
in the event of the Internet connection going down is there anyway to save the log updates and sync them when the connection is restored?
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: bsautner on Sep 06, 2010, 04:16 pm
Siegfried, i think you found these already - but here is the blog posting on how to connet arduino to nimbits.

http://nimbits.blogspot.com/2010/07/data-in-connect-arduino-to-nimbits-data.html

tublar - check out www.nimbits.com, there is a lot of client code both .net and java - it'd be pretty easy to make a buffer in your code on the lan if the internet connection dropped. I may just add that to the code library since it's a pretty basic need.

- Ben
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: kurofune on Oct 14, 2010, 06:41 pm
Hi, I'm very new and a scientist interested in dataloggers.

I'm mainly interested in one that can work by sending data to a computer as it analyzes a signal. Does something like that exist? This wouldn't require battery power or very much data storage...
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: slash11 on Oct 14, 2010, 06:53 pm
Quote
I am thinking about creating a specific website which will act as a central resource for open source data logger projects, allowing the less technical to communicate with us with there project idea's.  Additionally, It might also work as a place for developers to share code between each other and exchange idea's.

I had plans on competitions, prizes, etc to generate interest and give some incentive for everyone to contribute, I have had both positive and negative comments on implementing this.

What do you you all think?  Good or bad idea?

I noticed that this thread has been viewed over 10,000 times!!  It would appear that there is a lot of interest out there.


I think it a brilliant idea! The views of this topic is nearly hitting 50,000! :o
So was the central depositry site for data logger projects created yet? ;D
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: bsautner on Oct 14, 2010, 07:31 pm
Guys,

I will do this / collaborate on this. I already manage an open source data logger that connects arduino to google's cloud computing infrastrucure. I have the time, resources and interest to keep something like this going.  I have the code to put up a full text search web site with user authentication

Please contact me if there already is a site that's been started if you just need my help.

I own the domain opensourceloggers.com which i will donate to this cause
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: bsautner on Oct 14, 2010, 07:50 pm
Here is an under construction web site for Posting and searching for open source data logger projects. I'll get the data structures security set up over the weekend and will keep posting progress to this thread:

http://www.opensourceloggers.com/

It'd be very helpful to hear about what sort of catagories data loggers should fall under - such as Arduino, SQL, JDO etc - maybe the language, licences type etc. Posting a description, repository location...anything else?
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: novice on Oct 15, 2010, 05:34 am
Hi kurofune,
"sending data to a computer" over the USB port is easy on Arduino.

It can be captured on the computer using Processing, and plotted or saved.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: slash11 on Oct 15, 2010, 05:58 pm
I am very new to this field and I am building Data logger for a racing kart or a car. I think there should be a category for this type of loggers.
Although, they are similar to any other type of loggers, there are specific aspects of them being exposed to a very hostile environment i.e Rain/water/oil, vibration, EMI. I have faced this problems and finding ways to tackle them, as I am sure many others too. For example SD card slot doesn't take vibration very well and the card tend to move and eventually disconnects, which is not very healthy for logger in a middle of a busy logging session  ;D
It would be nice to share other ideas about the racing data loggers with fellow motorracing enthusiasts.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: billcramer07 on Oct 15, 2010, 07:23 pm
I like this idea also. Here are 2 links to parts I use for data logging and data acquisition on my Arduino weather station.

http://www.sparkfun.com/commerce/product_info.php?products_id=9530

This little guy connects to Arduino's TX line and just logs anything sent.
The next part is a MAX6818 octal debounce chip with interrupt output. I use it for switch inputs and to count pulses from the mechanical rain gauge.

http://pdfserv.maxim-ic.com/en/ds/MAX6816-MAX6818.pdf

Just a couple suggestions.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: jbeale on Nov 03, 2010, 09:38 pm
Can anyone recommend an open-source project focused on data acquisition, display, and control?  The commercial standard is LabWindows from National Instruments. There is something called "Liberlab" http://sites.google.com/site/liberlabsite/ written in Python back in 2006, and adapted for Arduino in 2008
http://www.uchobby.com/index.php/2008/01/19/arduino-liberlab-liberino/
but it does not appear to be an active project now. Is there anything else?

What I really want is just an easy way to do various real-time plots and status indicators on my Windows PC screen to show data coming in on the serial port, and some control buttons to send commands back on the serial port.  "Processing" is nice and user-friendly, but doesn't seem to be really designed for this. It's based on Java, so I looked at just using the Java Swing GUI setup. But from a quick look, that's not easy to jump into with no experience in that area.
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: novice on Nov 04, 2010, 05:35 am
Quote
What I really want is just an easy way to do various real-time plots and status indicators on my Windows PC screen to show data coming in on the serial port, and some control buttons to send commands back on the serial port

Look at how Processing was used to provide a UI for the PID library
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1243714052

Quote
"Processing" is nice and user-friendly, but doesn't seem to be really designed for this.

Wrong.

Quote
It's based on Java, so I looked at just using the Java Swing GUI setup. But from a quick look, that's not easy to jump into with no experience in that area.

Processing uses Java, but UIs developed in Processing do not use Swing to build the GUI.
Read the source code for the example above.

Because Processing is written using Java, it runs on OSX, Windows, and even Linux boxes.
Go to http://www.processing.org/ and follow the Download link to download Processing for your "Windows PC".
Title: Re: Open Source Data Logger Project Using the Arduino?
Post by: paasselin on Jan 23, 2011, 10:16 am
I don't know if it's been mentioned but the Openlog is a good datalogger.
It accepts micro-Sd cards of up to 16 gb.  
Oh and the contest is a great idea ;)

http://www.sparkfun.com/products/9530