Go Down

Topic: Automated Reptile Control System(webserver, Data Logging, RTC and much more) (Read 41733 times) previous topic - next topic

mkcinek

What I figured out is fact that Firefox on OSX can communicate with Arduino, however Safari can not. I'd say or arduino can only recognize Firefox or this is the way other browsers talk to Arduino, but it was mentioned that it also works with IE ? Perhaps a line with browser detection - auto detection would help ?

wallaceb


What I figured out is fact that Firefox on OSX can communicate with Arduino, however Safari can not. I'd say or arduino can only recognize Firefox or this is the way other browsers talk to Arduino, but it was mentioned that it also works with IE ? Perhaps a line with browser detection - auto detection would help ?


if i could more easily understand the exact differences in how IE/Firefox Versus Chrome or Safari interact with the arduino, then yes i could use browser detection. i have looked into the differences between some of the browsers, and i think the reason for the "incompatibility" is the length/amount of data that is sent during the HTTP communications. i think that it is causing a buffer over flow since the buffer is only 128 bytes in size.

if you can, perhaps make the line

Code: [Select]
#define STRING_BUFFER_SIZE 128 in the webserver.h file 256 or possibly even bigger, and see if that possibly fixes the issue?

pico

I've been adapting your ARECS code as a demo to connect to the Internet using my wireless RFXduino system, and you'll be pleased to hear that in the process I found your browser bug!

It's a doozy, actually. Basically, you were forgetting to send response headers before your html  in your sendPage() function! So instead of getting

Code: [Select]

HTTP/1.1 200 OK
Server: arduino
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Content-type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><title>Automated Reptile Environment Control System</title><style>body {font-family: Verdana, Arial, sans-serif;background: #fff;margin: 0px auto;padding: 0 0 20px [...]


by way of a response, the browsers were just getting hit directly with

Code: [Select]

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><title>Automated Reptile Environment Control System</title><style>body {font-family: Verdana, Arial, sans-serif;background: #fff;margin: 0px auto;padding: 0 0 20px [...]


The differences in the browsers behavior just reflected how they individually handled the (fairly extreme) situation. IE10 and Chrome were the most liberal, simply recognizing it as html, and assuming some default response header values that seemed to work pretty well. Firefox was a bit more  conservative, and treated it as text/plain rather than text/html content-type, and so it would show a nice screen full of html tags. Safari just turned it's nose up at the whole thing and did a good job of pretending it didn't receive anything.

As I was debugging this today, it took a while for the penny to drop, as it didn't occur that browsers like IE and Chrome could work at all without getting some sort of response headers. It was looking a FF "View Page Info" that first alerted me to the fact that the content-type was being set as "text/plain" rather than "text/html". How could that be? Something subtly wrong with the response headers? Well, yeah... like they were completely missing!  :)

Anyway, I also found a few other less serious bugs, which I will document more fully after I've had a bit more of a play with and test the system, but I thought I'd give you all a quick heads up on the "incompatible browser" issue. I've got it working using RFXduino with IE10, Chrome (WIndows and Android), Firefox (Windows and Android), and Safari (Windows).

Basic changes (should also work if using ethernet shield, although I haven't tested directly):

In sendPage(), change

Code: [Select]

   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_TOP])));
   //wdt_reset();
   //sendUriContentByIndex(client, nUriIndex, requestContent, 0, CONT_TOP);


to

Code: [Select]

   // send response headers
   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_HEADER])));

   // send HTML header
   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_TOP])));
   //wdt_reset();
   //sendUriContentByIndex(client, nUriIndex, requestContent, 0, CONT_TOP);


and in htmldata.h, change

Code: [Select]

PROGMEM prog_char content_main_header[] = "HTTP/1.0 200 OK\nServer: arduino\nCache-Control: no-store, no-cache, must-revalidate\nPragma: no-cache\nConnection: close\nContent-Type: text/html\n";


to

Code: [Select]

PROGMEM prog_char content_main_header[] = "HTTP/1.1 200 OK\r\n"
"Server: arduino\r\n"
"Cache-Control: no-store, no-cache, must-revalidate\r\n"
"Connection: keep-alive\r\n"
"Content-type: text/html\r\n"
"\r\n";


Thanks for your nice system -- I hope you don't mind me adapting it as a demo for my purposes! It has some nice technical and design features that makes it quite a nice showcase program for my system, I think. In my configuration, all the SD card storage can be on the Raspberry Pi gateway device, and so it's like a mini "cloud" server for all the Arduinos connecting through it. Also, things like the RTC function are also virtualised, so the only thing the ARECS Arduino has to do in terms of real devices is manage the relays. I'm thinking I'll be able to get the fully converted program running on a 328 Uno-class board, but we'll have to see how converting the html strings from progmem to SD flash affects performance. But I thought you'd like a heads-up on the browser issue in the meantime.

WiFi shields/Yun too expensive? Embeddedcoolness.com is now selling the RFXduino nRF24L01+ <-> TCP/IP Linux gateway: Simpler, more affordable, and even more powerful wireless Internet connectivity for *all* your Arduino projects! (nRF24L01+ shield and dev board kits available too.)

wallaceb

wow, thanks for the report. i cannot believe i forgot to include that in my header  =(

at least it is a "simple" mistake in that it is not an issue with the fundamental code base used in the project.

i will make the change. i also look forward to hearing about the other bugs you discovered.

i am glad you like the code. i have no issues with you using it, that is why i posted it here  ;)  8)



I've been adapting your ARECS code as a demo to connect to the Internet using my wireless RFXduino system, and you'll be pleased to hear that in the process I found your browser bug!

It's a doozy, actually. Basically, you were forgetting to send response headers before your html  in your sendPage() function! So instead of getting

Code: [Select]

HTTP/1.1 200 OK
Server: arduino
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Content-type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><title>Automated Reptile Environment Control System</title><style>body {font-family: Verdana, Arial, sans-serif;background: #fff;margin: 0px auto;padding: 0 0 20px [...]


by way of a response, the browsers were just getting hit directly with

Code: [Select]

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><title>Automated Reptile Environment Control System</title><style>body {font-family: Verdana, Arial, sans-serif;background: #fff;margin: 0px auto;padding: 0 0 20px [...]


The differences in the browsers behavior just reflected how they individually handled the (fairly extreme) situation. IE10 and Chrome were the most liberal, simply recognizing it as html, and assuming some default response header values that seemed to work pretty well. Firefox was a bit more  conservative, and treated it as text/plain rather than text/html content-type, and so it would show a nice screen full of html tags. Safari just turned it's nose up at the whole thing and did a good job of pretending it didn't receive anything.

As I was debugging this today, it took a while for the penny to drop, as it didn't occur that browsers like IE and Chrome could work at all without getting some sort of response headers. It was looking a FF "View Page Info" that first alerted me to the fact that the content-type was being set as "text/plain" rather than "text/html". How could that be? Something subtly wrong with the response headers? Well, yeah... like they were completely missing!  :)

Anyway, I also found a few other less serious bugs, which I will document more fully after I've had a bit more of a play with and test the system, but I thought I'd give you all a quick heads up on the "incompatible browser" issue. I've got it working using RFXduino with IE10, Chrome (WIndows and Android), Firefox (Windows and Android), and Safari (Windows).

Basic changes (should also work if using ethernet shield, although I haven't tested directly):

In sendPage(), change

Code: [Select]

   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_TOP])));
   //wdt_reset();
   //sendUriContentByIndex(client, nUriIndex, requestContent, 0, CONT_TOP);


to

Code: [Select]

   // send response headers
   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_HEADER])));

   // send HTML header
   sendProgMemAsString(client, (char*)pgm_read_word(&(contents_main[CONT_TOP])));
   //wdt_reset();
   //sendUriContentByIndex(client, nUriIndex, requestContent, 0, CONT_TOP);


and in htmldata.h, change

Code: [Select]

PROGMEM prog_char content_main_header[] = "HTTP/1.0 200 OK\nServer: arduino\nCache-Control: no-store, no-cache, must-revalidate\nPragma: no-cache\nConnection: close\nContent-Type: text/html\n";


to

Code: [Select]

PROGMEM prog_char content_main_header[] = "HTTP/1.1 200 OK\r\n"
"Server: arduino\r\n"
"Cache-Control: no-store, no-cache, must-revalidate\r\n"
"Connection: keep-alive\r\n"
"Content-type: text/html\r\n"
"\r\n";


Thanks for your nice system -- I hope you don't mind me adapting it as a demo for my purposes! It has some nice technical and design features that makes it quite a nice showcase program for my system, I think. In my configuration, all the SD card storage can be on the Raspberry Pi gateway device, and so it's like a mini "cloud" server for all the Arduinos connecting through it. Also, things like the RTC function are also virtualised, so the only thing the ARECS Arduino has to do in terms of real devices is manage the relays. I'm thinking I'll be able to get the fully converted program running on a 328 Uno-class board, but we'll have to see how converting the html strings from progmem to SD flash affects performance. But I thought you'd like a heads-up on the browser issue in the meantime.



mkcinek

Hi Wallaceb,

would you share your sketch if updated ? I messed too much with mine and saved ;)
thanks!
Merry Xmas !

wallaceb


Hi Wallaceb,

would you share your sketch if updated ? I messed too much with mine and saved ;)
thanks!
Merry Xmas !


no problem, i actually changed a few things anyways.

https://www.dropbox.com/s/0twzm2aztgg8o22/webserver-1.3%2012-22-2013%20-%20Copy.zip

i changed the way the system calculates the system up-time on page9 with code i created for a countdown clock so it calculates perfectly.

i also added functionality so the system will send e-mails when any sensor(s) have been detected to be malfunctioning so the user knows to check on things before anything may go too horribly wrong.

i also fixed the issues were certain web-browsers appeared to cause the system to crash. with the help from user pico, and also by changing a buffer size from 128 to 256, all browsers i tested work.

ENJOY!

tmaros

Hi all! I tired to upload (after webserverinit) your sketch webserver.ino, but I get multiple errors, i.e.:
C:\Users\me\Documents\Arduino\libraries\libraries\EthernetDHCP.cpp:545: error: 'ethernet_compat_write_GAR' was not declared in this scope
C:\Users\me\Documents\Arduino\libraries\libraries\EthernetDHCP.cpp:546: error: 'ethernet_compat_write_SUBR' was not declared in this scope
...

I don't know what's wrong. It seems strange that path includes "libraries" two times, but don't know why either..
BTW: it's an excellent job!
Best regards!

wallaceb


Hi all! I tired to upload (after webserverinit) your sketch webserver.ino, but I get multiple errors, i.e.:
C:\Users\me\Documents\Arduino\libraries\libraries\EthernetDHCP.cpp:545: error: 'ethernet_compat_write_GAR' was not declared in this scope
C:\Users\me\Documents\Arduino\libraries\libraries\EthernetDHCP.cpp:546: error: 'ethernet_compat_write_SUBR' was not declared in this scope
...

I don't know what's wrong. It seems strange that path includes "libraries" two times, but don't know why either..
BTW: it's an excellent job!
Best regards!


i am sorry, when i posted my version 1.3 code, i forgot to add the library folder within the zip file.

please download this file:

https://www.dropbox.com/s/cun7nqg9qrlfj9k/1.1%20-%206-6-2013.zip

go into the zip file's libraries folder and copy the following into your Arduino IDE library folder:
EthernetBonjour
EthernetDHCP
EthernetDNS
OneWire

those will give you all the needed external library files the compiler is complaining about.

my bad!  :smiley-roll-blue:

tmaros

No, actually it is my fault - it was other reason. Sorry for offtopic, maybe admin should just delete my last post?

pico



Thanks for your nice system -- I hope you don't mind me adapting it as a demo for my purposes! It has some nice technical and design features that makes it quite a nice showcase program for my system, I think. In my configuration, all the SD card storage can be on the Raspberry Pi gateway device, and so it's like a mini "cloud" server for all the Arduinos connecting through it. Also, things like the RTC function are also virtualised, so the only thing the ARECS Arduino has to do in terms of real devices is manage the relays. I'm thinking I'll be able to get the fully converted program running on a 328 Uno-class board, but we'll have to see how converting the html strings from progmem to SD flash affects performance. But I thought you'd like a heads-up on the browser issue in the meantime.

i am glad you like the code. i have no issues with you using it, that is why i posted it here  ;)  8)


hi wallaceb,

I thought you might like to see my write-up of the RFXduino demo project that ARECS evolved into... ARVCS! ;-)

http://embeddedcoolness.com/rfxduino-demo-projects/

OK, so now it's controlling a roof ventilation system rather than keeping a reptile comfortable, but I think you will instantly spot the family resemblance nevertheless, ;-)

Also, the virtual SD Card and virtual RTC functionality have worked in nicely, and I did get it to fit onto an Uno as result, so it has ticked all the boxes I had in mind originally for a demo project.

Thanks again for your efforts in providing the original program. Nice framework, and I anticipate a few more derivative projects before we're done!
WiFi shields/Yun too expensive? Embeddedcoolness.com is now selling the RFXduino nRF24L01+ <-> TCP/IP Linux gateway: Simpler, more affordable, and even more powerful wireless Internet connectivity for *all* your Arduino projects! (nRF24L01+ shield and dev board kits available too.)

wallaceb




Thanks for your nice system -- I hope you don't mind me adapting it as a demo for my purposes! It has some nice technical and design features that makes it quite a nice showcase program for my system, I think. In my configuration, all the SD card storage can be on the Raspberry Pi gateway device, and so it's like a mini "cloud" server for all the Arduinos connecting through it. Also, things like the RTC function are also virtualised, so the only thing the ARECS Arduino has to do in terms of real devices is manage the relays. I'm thinking I'll be able to get the fully converted program running on a 328 Uno-class board, but we'll have to see how converting the html strings from progmem to SD flash affects performance. But I thought you'd like a heads-up on the browser issue in the meantime.

i am glad you like the code. i have no issues with you using it, that is why i posted it here  ;)  8)


hi wallaceb,

I thought you might like to see my write-up of the RFXduino demo project that ARECS evolved into... ARVCS! ;-)

http://embeddedcoolness.com/rfxduino-demo-projects/

OK, so now it's controlling a roof ventilation system rather than keeping a reptile comfortable, but I think you will instantly spot the family resemblance nevertheless, ;-)

Also, the virtual SD Card and virtual RTC functionality have worked in nicely, and I did get it to fit onto an Uno as result, so it has ticked all the boxes I had in mind originally for a demo project.

Thanks again for your efforts in providing the original program. Nice framework, and I anticipate a few more derivative projects before we're done!



very nice, i am very glad the base code worked so well for you and that your project is working

i really want to ask, how are you doing the web-page graphs of data?  :smiley-eek: i wanted to do something like that  :smiley-mr-green:

pico


very nice, i am very glad the base code worked so well for you and that your project is working

i really want to ask, how are you doing the web-page graphs of data?  :smiley-eek: i wanted to do something like that  :smiley-mr-green:


Well, all the source code is on the site (embeddedcoolness.com) if you want to check into any of the details.

But the basic thing is that data is being logged to the cloud server (Xively in this case), and then the graphs are being displayed as embedded images using the <img src=""> tags in the html pages. Here's a fragment:

Code: [Select]

<img src="https://api.xively.com/v2/feeds/1234567890/datastreams/roof_out_temp.png?width=730&height=250&
colour=F15A24&timezone=+10&duration=^|hours&xlabel=hours&l=temp%20(C)&b=true&g=true"
alt="roof_out_temp.png plot here"/>


Notice that after "duration" I have a ^| placeholder inserted -- that's the equivalent of one of the STX characters in your html template files.

That's so I can dynamically specify how much history to display in the graphs (from 1 hour to ten days.)

(Also, I substituted my actual Xively feed id with "1234567890" for the example above.)

Does that help answer your question at all?

WiFi shields/Yun too expensive? Embeddedcoolness.com is now selling the RFXduino nRF24L01+ <-> TCP/IP Linux gateway: Simpler, more affordable, and even more powerful wireless Internet connectivity for *all* your Arduino projects! (nRF24L01+ shield and dev board kits available too.)

wallaceb



very nice, i am very glad the base code worked so well for you and that your project is working

i really want to ask, how are you doing the web-page graphs of data?  :smiley-eek: i wanted to do something like that  :smiley-mr-green:


Well, all the source code is on the site (embeddedcoolness.com) if you want to check into any of the details.

But the basic thing is that data is being logged to the cloud server (Xively in this case), and then the graphs are being displayed as embedded images using the <img src=""> tags in the html pages. Here's a fragment:

Code: [Select]

<img src="https://api.xively.com/v2/feeds/1234567890/datastreams/roof_out_temp.png?width=730&height=250&
colour=F15A24&timezone=+10&duration=^|hours&xlabel=hours&l=temp%20(C)&b=true&g=true"
alt="roof_out_temp.png plot here"/>


Notice that after "duration" I have a ^| placeholder inserted -- that's the equivalent of one of the STX characters in your html template files.

That's so I can dynamically specify how much history to display in the graphs (from 1 hour to ten days.)

(Also, I substituted my actual Xively feed id with "1234567890" for the example above.)

Does that help answer your question at all?




ok, that was what i was worried about, it requires an outside server, i hoped it was processed by the arduino.

wallaceb

i would like to make a quick update that the latest version of the code i have posted has been running on my system continuously for 2 months and 17 days (as of this posting) and going strong without a single hiccup.

as long as the system stays running and does not fault on its own (for example, if i loose power, then that is not the project's fault) i will update the post with the new run time to document the stability.

pico


i would like to make a quick update that the latest version of the code i have posted has been running on my system continuously for 2 months and 17 days (as of this posting) and going strong without a single hiccup.

as long as the system stays running and does not fault on its own (for example, if i loose power, then that is not the project's fault) i will update the post with the new run time to document the stability.


Good to hear it's working well. BTW, I did promise to let you know about some of the relatively minor bugs I found when porting your code. The most significant problem of those was in the input buffering routines -- some of that just wasn't robust (but I get the impression you may have revamped that code anyway.) The other problems I came across were in in the HTML. When I created external html files from your header file to play around with, I ran them through a HTML checker (the well-known "HTML Tidy"), and found a surprising number of issues with things like opening and closing tags out of order, things like that. Interestingly, it didn't seem to make much difference on the actual performance of the code, as you would be aware, as all the browsers seemed to cope OK. There was one instance when I fixed the order of the tags and something broke! But it wasn't a big deal, and I actually forget what it was now -- sorry.

Anyway, all relatively minor stuff, and just a heads up if you ever want to dive into the HTML again, a check tool like HTML TIdy is useful to have. My philosophy is that if the code is as close to text-book as possible when you write it, it should hold up for the foreseeable future no matter how browsers decide to change in how they support any "gray area" stuff.  

Thanks once again wallaceb for sharing your excellent project! Some extra karma points for you. :-)
WiFi shields/Yun too expensive? Embeddedcoolness.com is now selling the RFXduino nRF24L01+ <-> TCP/IP Linux gateway: Simpler, more affordable, and even more powerful wireless Internet connectivity for *all* your Arduino projects! (nRF24L01+ shield and dev board kits available too.)

Go Up