I have an idea for a project, but not sure of the best approach. Actually, I have already begun a project and want to expand it incrementally, but not sure of the best approach. I have created an automatic transfer switch, to have a device that runs on 120 VAC, run on grid power normally, while solar panels charge batteries, that are monitored with a Battery Monitor. The monitor sends out a series of ASCI code signals. One of the pieces of data, is a Battery % of charge. When the batteries are nearly charged, monitored by an Arduino UNO, I have the UNO send a signal to turn on an Inverter. When the inverter turns on, I have a 5 vdc output, wall wort plugged into the inverter output and is another input to the UNO. When this signal goes high, I have another UNO output send a signal to energize a 30 Amp contact DPDT relay to transfer power from the grid to the inverter. In my UNO sketch I set the inverter to turn on at 95% and turn off at 90%. This project is still in the breadboard state. This has worked very well for over a year, however I want it to do more. I would like to be able to:
monitor power consumption on both the grid powered default circuit and the inverter powered circuit, but separately, so that I can find out exactly how many Kilowatt hours my batteries contribute to the total consumption.
have all of this data saved to a file (or other, web page, cloud, ?) that is accessible on my home network for easy access to import into Excel and manipulate.
But herein lies my holdup point.
I'm not sure of many things in this area. I have read much about the ESP8266 wireless device. I'm having trouble with getting it to cooperate and play nice. I'm also not sure of how to handle a constant stream of data. I mean how should I handle file size. One full day of daylight hours fill over 3 gig in a single file.
Should I be saving this data to a file in the first place.
I have been able to use CoolTerm to save a text file to my laptop and have been able to save those files to my network, but I don't want to tie up my laptop to do this. I have read about people sending data to a web page. I’m open to that, but I am reluctant to commit to a monthly or yearly fee.
Some advice would most surely help.
I was going to suggest an SD card and use the laptop occasionally to download the data via the serial port. The question is at 3GB per day, how often are you logging data and do you need all of that data?
Thank you Robin2 and adsystems for responding so quickly. As I said in my post, The data that comes out of my battery monitor is ASCI at 2400 baud and it consists of:
A date and time stamp is provided by CoolTerm and is tacked on to the beginning of each row of data.
%=, is the battery % of charge.
W=, is Watts, positive values are watts into the battery, negative values are watts coming out of the battery.
DSC=, is Days since charged,
DSE=, is Days since empty
X=, is code that represent the state of condition of the battery monitor.
V=, is the primary battery voltage.
FV=, is the Float Voltage. It is actually the average of the primary battery.
V2=, is the voltage of a secondary battery, if present.
A=, is Amps. A positive value is Amps into the battery and negative value is Amps out of the battery.
FA=, is Floating Amps. This is really the average Amps.
AH=, is Amp Hours. A negative value here is the number of Amp-Hours that the battery is discharged. As this value increases and approaches 0, the battery is charging. When it reaches 0, the battery is charged to 100%.
This data is a constant stream at about one set a second. You are correct, I may be able to capture this data once a second , but print out only once every 10 seconds. I could also eliminate X, for instance, but most of the other data I do use in Excel to create plots of various things. I use the DSC in my UNO sketch to charge the battery to 100%, once every 2 weeks. The DSC value resets to 0 when the battery is charged to 100%.
So yes, I can do some whittling, and that would decrease my file size. Is there a standard for how people handle a constant stream of data. At what point should the file size be cut off and a new data file started. Or is there a way to stream this data to my network. Thanks Steve
adsystems, Thank you for the link to the power monitor project. That is something that my project needs. So I could have two of these on one UNO, one for the grid circuit and another to monitor the output of my Automatic Transfer Switch. Then I'll capture the total power consumed and then subtract the grid power to find exactly how much the solar is contributing to the total. Steve
The data stream produces one set per second. Why even every 10 seconds? This is solar, it doesn't change that fast. How about once per minute? I hate timers. How about when certain critical (in this case nothing is critical, but the most important) change by more than a certain amount and at least Y seconds but not more than X seconds. These are methods found in common industrial historian applications (OSiSoft Pi and others).
Store this on an SD card using an Arduino with two hardware serial ports, one for the ASCII input and one for the computer. Then program the computer to transfer the data when you need to.
I would really like to find a way to transmit this data wirelessly to a file in my network and not have to bring a laptop to the garage where my project is located. That's why I mention the ESP8266. But I'm not sure if that is the right way to go with this.
I am now wondering about the possibility to incorporate the Power Monitor project on the same UNO as my Automatic Transfer Switch, since I'm not using any of the analog inputs currently.
Steve
sdingman:
This data is a constant stream at about one set a second.
You have listed the data that is available but you have not told us what information is useful to you. There is a big difference between data and information.
I can see that there may be occasions when it would be interesting to see all the fine detail, but as you can do that with Coolterm I can't see any point in anything more complex. Believe me, you will quickly get tired of looking at 3GB of data per day - probably after the 3rd day. Think about the information that actually has value for you.
Some the data will be cumulative. If so you don't need the data from every second, maybe once an hour is sufficient. Maybe even once per day is sufficient so that you can relate the energy to each daily cycle of the sun (or the earth, for the pedants).
I have some solar panels to reduce the run-time of a diesel generator. Once I was satisfied that the panels were producing the power I expected them (the economic rationale for buying them) I only very occasionally check the current. As long as the panels are kept clean there is nothing more I can do to improve the output. It is much more important for me to reduce my electricity consumption.
While I remain convinced that the sensible thing is to reduce the data volume, there should be no difficulty transmitting any or all of the data. Bluetooth would be the simplest option if the range is adequate. Bluetooth is simply serial-by-wireless.
I reckon it would also be practical to read the raw data with a serial port on an ESP8266 and send it by wifi.
I don't know whether you could rely on an intensive wireless link operating 24/7.
The complexity will be the need to deal with transmission failures. An Arduino (preferrably a Mega) or an ESP8266 could buffer several incoming messages in the event that there was a temporary glitch in the wireless communication. However for a longer outage - say one that requires human intervention to restore the system - it may make sense to save the data to an SD Card.
Robin2:
The complexity will be the need to deal with transmission failures. An Arduino (preferrably a Mega) or an ESP8266 could buffer several incoming messages in the event that there was a temporary glitch in the wireless communication. However for a longer outage - say one that requires human intervention to restore the system - it may make sense to save the data to an SD Card.
...R
How about combine the SD card and the wireless? Save to the card and transmit from the card when the wireless link is available. I'm working on a system that will include a logger saving to SD card and will walk down to retrieve the card or transfer the data via wired serial. Eventually I would like to transfer wirelessly but there are a few features of higher priority to do first.
adwsystems:
How about combine the SD card and the wireless?
Surely, the way to go.
If you are working to put 3Gig into Excel every day, you are just kidding yourself. This something you will conclude on the morning of the second day.
Think sensibly about rate of change. In your case, I doubt there is anything that merits recording every second
Determine a criterion for doing anything. That may be enough to make the difference between data and information. In my case it is water flow, and that alone was enough to reduce the daily file size by over 90%.
Check if there is a difference between what you would like to know now, and what you really need next month. That may mean a live transmission of volatile data every second, but record to SD once every ten seconds - another 90% reduction.
Relying on a continuous wireless link can't possibly be a good idea. It is sure to break when you need it - and don't know about it.
Note that the above does mean you will need an onboard RTC, or maybe an internet time source.
You might not be aware of God's great gift to Arduino datalogging. It is called Bluetooth Graphics terminal. This enables you to see three simultaneous live graphs on your phone.
In my case, I send data out at one second intervals via Bluetooth and record every ten seconds to SD. A new file is created at midnight, using data as filename. This means I can stand nearby and watch a live event on the phone, and then send MMDD and download the relevant file to the phone, for subsequent transfer to Excel. You can download several files to the one phone file, as the outgoing commands can be used as separators. If you don't need the graphs, pretty well any Bluetooth terminal can do this. I'm not aware of any Bluetooth terminal that provides timestamping.
It should be possible to use the PLX macro to download direct to Excel, but I think the phone is fine.
Thank you Nick_Pryner, this is great input for me to think about. I think that I may decrease my print cycle, but since I never had any experience with either Solar or the Arduino before starting this project, observing and manipulating the numbers in the fashion that I have been doing has taught me a lot about several things that I otherwise, I believe, would have missed by sending the print command so often.
I learned for instance that I really shouldn't let my batteries go below the 90%, which in my case brings the batteries down to around 12.3 Volts. Below that and it takes several hours to charge to 95%, at which point I turn the inverter on and transfer power. I also found that turning the system on at 95% causes my charge controller to send the most current to my battery (4-100 ah deep cycle Glass Mat wired in parallel). On good sunny days it takes only 2 to 3 hours of charge to reach 95%, cloudy days, 4 to 5 hours.
I was amazed that once the system turned-on, on a sunny day, that it would stay on for 6 to 8 hours a day, running a refrigerator and a deep freeze.
One disappointment though, was when saving this data to a Notepad text file through CoolTerm, sometimes I would go to the garage to check everything only to find that either CoolTerm quit saving or the UNO quit sending, but when I looked into it found that the data was still coming, I mean, the Arduino sketch continued to run. I would then re-start the saving of the data text file. So my question is, what made it stop, the UNO or CoolTerm. I spent a lot of time writing some vba code to stitch together files recorded on the same day in consecutive order and to create the different plots that I wanted to see and calculations that I wanted to know.
So far my file size has not presented itself to be a problem. My main focus now is how to get the data transmitted wirelessly to my home network. You mentioned
"> Relying on a continuous wireless link can't possibly be a good idea. It is sure to break when you need it - and don't know about it."
I though people do this all the time. I don't know how successful they are though. Steve
sdingman:
One disappointment though, was when saving this data to a Notepad text file through CoolTerm, sometimes I would go to the garage to check everything only to find that either CoolTerm quit saving or the UNO quit sending, but when I looked into it found that the data was still coming, I mean, the Arduino sketch continued to run.
That is always going to be a risk and you need to build strategies into the code on the sending and on the receiving side to deal with it.
"That is always going to be a risk and you need to build strategies into the code on the sending and on the receiving side to deal with it."
To save to a text file, or to stop saving, is turned on and off inside of CoolTerm. How might I "build strategies into the code on the sending and on the receiving side to deal with it." I've been to:
and didn't see anything that I might use to be able to tell if CoolTerm quit, so I would think that any, "dealing with it", would have to be some piece of code inside the Arduino IDE that would throw up a flag to at least let me know that it was the sketch that skipped beat and reset. Then at least, I would know if the sketch needs tweaking in some way. If the sketch in the Arduino IDE isn't being reset, then this would mean there must be an issue with CoolTerm. Steve
sdingman:
To save to a text file, or to stop saving, is turned on and off inside of CoolTerm. How might I "build strategies into the code on the sending and on the receiving side to deal with it." I've been to:
Thank you Robin2, but I'm not so sure I'm savvy enough for that. Evan if I can't do what I'm dreaming of, I still have an automatic system that works. I'm saving a little on my electric bill, and with my current charge controller and Inverter, I can expand a little and save even more, whether I save the data like I want or not. Thanks again, you all have given me a lot to work on. Steve
sdingman:
I would go to the garage to check everything only to find that either CoolTerm quit saving or the UNO quit sending, but when I looked into it found that the data was still coming, I mean, the Arduino sketch continued to run...........
"> Relying on a continuous wireless link can't possibly be a good idea. It is sure to break when you need it - and don't know about it."
I though people do this all the time.
The people who do this all the time are the people who are prepared to have a reliable dedicated link at the other end to receive it, and the smart ones probably still have local backup - for reasons you apparently have already discovered. A card slot for Arduino can cost as little as $0.69, free delivery, and is essentially as reliable as the Arduino it is attached to. They are about 19 x 30.
I don't use it, but CoolTerm should be the least of your problems. Notification of a reset need only be some sort of flag in the setup that you can manually cancel. I don't think you should need it.