Today on my Arduino www home automation website I have published my webserver procedure. This procedure receives and processes html requests, takes various actions and causes the next web page to display. You can see this code at:
This code is over 600 lines and supports more that 30 web initiated activities and/or web page displays.
I have published this code for users who want to use their Arduino equipment to publish a full website and are struggling with html request processing. Via http://www.2wg.co.nz/SCRNINFO.DIR/ you can also access the Arduino C language source code that generates the html for all of the application's main web pages. (All the code is not fully current.)
Other major blocks of application functionality that I have published and other Arduino developers might be interested in include:
http://www.2wg.co.nz/PUBLIC/RAMUSAGE.TXT/ - an implementation of realtime memory usage tracking that I use to profile my application and find out which code is likely using up all my memory.
As my application development continues (I am over 6,500 lines of C code now) I will publish other blocks of interesting code including my suite of general code utilities.
The public is of course welcome to browse my Arduino home automation application at http://www.2wg.co.nz and see what can be achieved using Arduino equipment to publish a web site.
I've been studying your code some and I have been 'borrowing' some of it as well. ;D
I have a question about the ReadTimeUDP(). I'm surprised you got that working. I did see a forum append where it was giving you trouble at one point but after you changed some stuff it started to work.
Seems like the only way I keep the UDP port operational is if I start it up once and never stop it.
Any thoughts?
Yes. I don't believe that is true. I use NTP and DNS, and both work fine obtaining a socket, sending and receiving the packets, and releasing the socket.
Did you not see my posts in that discussion? Mine worked fine once I discovered the SD card problem. If you have a SD card in the shield's slot, you must initialize or disable it. It will even cause the dhcp requests to fail.
I do have a SD card, but I have the CS set to HIGH for it... so I think it's out of the picture.
As long as I don't stop/begin the UDP socket, it seems to work just fine; but once I stop it and the socket gets reused for a simple Webserver access, it's toast if I try and restart the UDP socket. If I reset the w5100, it will come back to life.
Have you tried the ShowSockStatus function? This shows the status of all sockets. Try your problem again, then call this function to determine what has happened at the socket level. I would like to see the results of this test if y9ou don't mind.
#include <utility/w5100.h>
byte socketStat[MAX_SOCK_NUM];
void ShowSockStatus()
{
for (int i = 0; i < MAX_SOCK_NUM; i++) {
Serial.print(F("Socket#"));
Serial.print(i);
uint8_t s = W5100.readSnSR(i);
socketStat[i] = s;
Serial.print(F(":0x"));
Serial.print(s,16);
Serial.print(F(" "));
Serial.print(W5100.readSnPORT(i));
Serial.print(F(" D:"));
uint8_t dip[4];
W5100.readSnDIPR(i, dip);
for (int j=0; j<4; j++) {
Serial.print(dip[j],10);
if (j<3) Serial.print(".");
}
Serial.print(F("("));
Serial.print(W5100.readSnDPORT(i));
Serial.println(F(")"));
}
}
Sure, no problem... I've been doing quite of bit of reading over the past 24hrs so I already had that debug code in there. I'm printing out the socket status just before I stop the Udp port.
In the below output, I read the Udp port twice before sending a simple webserver request (actually two web request get sent). The closing client messages are the end of the web requests. Then a upd port starts up again (and in this case, it got assigned back to socket#1) but the udp request fails.
Setup
My IP address: 192.168.1.149.
waiting for good packet 0 <---- Read successful
The EST time is 08:58:41 PM
255.255.255.0
Socket#0:0x14 80 D:255.255.255.255(67)
Socket#1:0x22 8888 D:132.163.4.101(123)
Socket#2:0x0 0 D:0.0.0.0(0)
Socket#3:0x0 0 D:0.0.0.0(0)
All ready.... 2/22/2015 20:58:41
waiting for good packet 0 <---- Read sucsessful
The EST time is 08:59:11 PM
255.255.255.0
Socket#0:0x14 80 D:255.255.255.255(67)
Socket#1:0x22 8888 D:132.163.4.101(123)
Socket#2:0x0 0 D:0.0.0.0(0)
Socket#3:0x0 0 D:0.0.0.0(0)
closing client 2/22/2015 20:59:23 <----- Webpage ack/sent
closing client 2/22/2015 20:59:26 <----- Webpage ack/sent
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0
waiting for good packet 0 <----- upd failed!!!!
255.255.255.0
Socket#0:0x14 80 D:192.168.1.132(58978)
Socket#1:0x22 8888 D:132.163.4.101(123)
Socket#2:0x0 0 D:0.0.0.0(0)
Socket#3:0x0 0 D:0.0.0.0(0)
Does socket#1 show a status of 0x00 after the stop?
Yes
Then when you try to start another UDP socket, that is when the trouble starts?
Yes, but only if I request a few web pages. If I don't 'pollute' the socket with a webpage request, it works just fine.
Hopefully this pseudo code will show exactly what I'm doing...
(Eventually I'll be picking up more of your WebServerST code, but for now I just have stupid code in for the server, it just returns simple web page)
EthernetServer mEServer(80);
void setup() {
...
startEthernet();
...
}
startEthernet() {
....
Ethernet.begin(mMacAddr);
...
mEServer.begin();
...
}
// This is the function that gets called every 30 seconds.
unsigned long readNTPTime() {
EthernetUDP eUdp;
mEUdpOk = eUdp.begin(NTP_LOCAL_PORT);
...
send_the_request/wait for answer;
...
eUdp.stop();
...
}
void loop() {
EthernetClient client = mEServer.available(); // try to get client
if (client and any_request ) { // got client?
....
send_back_response_header
send_back_simple_web_page
...
client.stop(); // close the connection
}
.....
}
I'm using the new web server code I have in the playground, and have merged a NTP client function call, starting and stopping the UDP socket for each NTP call, and it is working fine. I also "polluted" the sockets with web page requests, and both server and NTP still work. ??
The code I released ten months ago is well out of date. The WebServerProcess functionality has been substantially restructured and greatly expanded since then.
100% of my Arduino application is now available on the application's website at http://www.2wg.co.nz - there is a list of files to download on the main dashboard web page.
My application does run successfully and continuously as a web server on port 80 and it uses UDP NTP everyday to update its time. As of now it has been running for 19 straight days.
However please note that I was unable to get UDP NTP working using an IP address assigned by my sketch. The only way i was able to get everything going ok was to let my ADSL modem assign my Arduino board a fixed IP address using DHCP based on a fixed MAC address I have embedded in my sketch. I never tried to find out why this was necessary - I was just happy to be able to keep moving forward.
Thanks for the reply. I just picked up a copy of your code last week from the site you mentioned above, so I do have the latest and greatest. The code looked pretty fault tolerant around the UDP NTP so it was unclear to me if you would even notice if the UDP NTP requests were failing... so I figured it was worth while asking about.
I tried both a fixed IP address and just letting my router assign a IP address, but I get the same results with UDP NTP request failing if I don't keep a 'fixed' socket for them.
I'll debug a bit more while I'm waiting for a few more parts to arrive.. but then to, I'll probably just move on with my workaround.
CatweazleNZ:
My application does run successfully and continuously as a web server on port 80 and it uses UDP NTP everyday to update its time. As of now it has been running for 19 straight days.
Do you use Udp.begin() and Udp.stop() at each NTP request? Or do you keep a socket reserved for that?
CatweazleNZ:
However please note that I was unable to get UDP NTP working using an IP address assigned by my sketch. The only way i was able to get everything going ok was to let my ADSL modem assign my Arduino board a fixed IP address using DHCP based on a fixed MAC address I have embedded in my sketch. I never tried to find out why this was necessary - I was just happy to be able to keep moving forward.
Moving forward is good, but you need to back up and fix that. Most server situations will require a static IP.
SurferTim, do you know what a normal NTP packet looks like? Currently I can only capture packets being directed to my computer... I'm working on fixing that. Anyway, I decided to startup Wireshark and look at incoming packet and surprisingly a NTP packet showed up. (only the 'bad' packets show up, imagine that)
Packet info:
Source Mac Adr: Good - mac of Arduino Dest Mac Adr: BAD - mac of my computer
Source IP Adr: Good - IP Adr of Arduino
Dest IP Adr: Good - IP of the time server
Source Port: Good
Dest Port: Good
As I said, I'm not sure if this get broadcast anywhere else besides my computer or not..... I'm guessing not.
SurferTim, do you know what a normal NTP packet looks like?
Yes.
only the 'bad' packets show up, imagine that
I'm not sure what you mean by "bad". Bear in mind if the NTP server is not on the localnet, then the packet will be sent to your router (gateway), and will return from the gateway device.
As an example,
if you had Internet Connection Sharing enabled on your PC
and
the PC connected to the internet via wireless
and
the Arduino was connected to the PC's ethernet interface
and
the PC was the "router" for the Arduino's network,
then
that would not be a bad packet. All non-localnet packets will show the mac address of the gateway as the remote (destination) mac address.
SurferTim:
Do you use Udp.begin() and Udp.stop() at each NTP request? Or do you keep a socket reserved for that?
Yes - I use udp.begin() and udp.stop(). I run udp ntp on port 8888. And I only run udp ntp operations when there are no active http internet connections running on port 80. So I always have plenty of free ports.
SurferTim:
Moving forward is good, but you need to back up and fix that. Most server situations will require a static IP.
I don't think I need to fix anything. My Arduino system and my internet connection both have static IP addresses. My Arduino static IP address is assigned via DHCP running on my ADSL modem - it assigns a static IP address based on the static MAC address embedded in my application sketch and used in the call to Ethernet.begin().
CatweazleNZ:
I don't think I need to fix anything. My Arduino system and my internet connection both have static IP addresses. My Arduino static IP address is assigned via DHCP running on my ADSL modem - it assigns a static IP address based on the static MAC address embedded in my application sketch and used in the call to Ethernet.begin().
That is a static dhcp assignment, not a static ip, but that is just semantics.
dgbair:
Hi Catweazle,
Thanks for the reply. I just picked up a copy of your code last week from the site you mentioned above, so I do have the latest and greatest. The code looked pretty fault tolerant around the UDP NTP so it was unclear to me if you would even notice if the UDP NTP requests were failing... so I figured it was worth while asking about.
I tried both a fixed IP address and just letting my router assign a IP address, but I get the same results with UDP NTP request failing if I don't keep a 'fixed' socket for them.
I'll debug a bit more while I'm waiting for a few more parts to arrive.. but then to, I'll probably just move on with my workaround.
CatweasleNZ feels nothing needs fixing. I guess you are on your own.
Thanks for the reply. I just picked up a copy of your code last week from the site you mentioned above, so I do have the latest and greatest. The code looked pretty fault tolerant around the UDP NTP so it was unclear to me if you would even notice if the UDP NTP requests were failing... so I figured it was worth while asking about.
I tried both a fixed IP address and just letting my router assign a IP address, but I get the same results with UDP NTP request failing if I don't keep a 'fixed' socket for them.
I'll debug a bit more while I'm waiting for a few more parts to arrive.. but then to, I'll probably just move on with my workaround.
dgbair
SurferTim seems to think this is my problem so I will comment further.
You have access to my current code - although there have been unreleased minor inconsequential updates in the last month.
The discussion over at W5100/Ethernet cannot use socket as UDP after using it as TCP - Networking, Protocols, and Devices - Arduino Forum is too complicated for me. But I take up SurferTim's comment about SD card initialisation. During my application development I have had a lot of problems with the fact that both ethernet and the SD card share my Freetronics Ethermega SPI interface. But provided you initialise each one correctly and enable/disable (switch between) them correctly everything works OK.
In summary about SPI - make sure both ethernet and SD card are initialised and never enable both at the same time, make sure you never send ethernet calls to the SD card, make sure you never send SD card calls to ethernet.
Within my implementation look at the SPIDeviceSelect() procedure - and make sure you understand how it is used within the application to switch between my two SPI devices - including the situation where my application is reading an SD card file and writing the data to an ethernet web page at the same time. (Well almost.)
If you create a test app with a simple web page and an UDP NTP function I will try to debug it on my Freetronics hardware. I should be able to get something working since I have a well proven working solution and I would be happy to do some further testing to see if I can get my system working with its own static IP address rather that the current ADSL modem assigned static IP address.
I agree with the guys on the other discussion. The W5100 is messing up the ARP tables. It's assigning wrong MAC addresses to the NTP packets. Using wireshark on the computer and tcpdump on my router it's very obvious the NTP packets contain the wrong header information.... hence they are getting routed to my computer instead of being routed out bound. ie the router recognizes the MAC address and sends it to the proper place where the MAC address lives.
At some point I may try updating the library and using SEND_MAC vs SEND for UDP/NTP packets (just to see if I can work around this) but that's a install/build process I don't feel like doing right now.
