Pages: [1] 2   Go Down
Author Topic: Where do I post code for help with a large (29K) NTP/UDP and TCP/ip sketch  (Read 1759 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi, I have a problem with a control system/data loging sketch and would like some help. There seams to be 3 problems

a) The NTP code works on start up (in 'setup') to set the UNIX time but updates (in the 'loop')  are more likely to return 0 than the time stamp. If left it can return a value after half a day and then fail again.

b) The Web Server code stops talking to the local network, so you cant pull the data down to a PC.  (this is usually after the NTP has missed 6 or 7 times) This can only be recovered with a hard reset.

c) The interrupt 'one second' drum beat on Timer 1 usually caries on and the control functions and data recording functions continue to work, but the drumbeat has been know to latch up and the whole thing crash.

In short it works for a few hours then progressively gives up.

However as its quite large programme (29k .ino file plus about 12k of HTML) my first request is advice on where to post the code so members can see it. Its far too big for an attachment and presumably posting it here in a dozen or so chunks would be frowned on.

All the bits of code work individually and I have eliminated the possibility of the problem arising from the mains power supply , environmental conditions and shortage of SRAM (650bytes free). Ironically adding Serial.print () statements for debugging appear to make the programme more stable, until the serial monitor/ USB chip over flow.

The code is on a Uno with Ethernet shield a Sandisk micro SD card and I'm now using V1.0.4 of the IDE. I've also tried the suggested EthernetUDP, Socket, W5100 fixes from Issue 1007 on GIT Hub but to no avail. When I started on this 7 months ago I thought it was a SRAM/Stack clash or Malloc problem but those have been resolved. All strings are arrays or static using the 'F' macro.

Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
b) The Web Server code stops talking to the local network, so you cant pull the data down to a PC.  (this is usually after the NTP has missed 6 or 7 times) This can only be recovered with a hard reset.
That sounds like there are no sockets remaining. Normally that is due to leaving characters in the w5100 rx buffer when closing the connection. You can't attach the sketch code as a file attachment?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi SurferTim,
  Using up the sockets was one of my thoughts but Parse.Packet() returns 0 when the NTP request fails. I also put a couple of Serial.print(_sock) and Serial.print(socket) lines into EthernetUDP.ccp and W5100.cpp. Only once did the sketch climb through the sockets going 1,2,3,1 in succession it never got to 4.As far as I can make out there isn't a socket zero as that is the value returned on failure.

  I'm not convinced that is a socket problem because the NTP bit still fails even if you deliberately avoid making a web query on the server part.

How do you know that there is debris in the RX buffer? Looking at the library code Parse.Packet()  looks for W5100.getRXReceivedSize(_sock) to be >0. a Serial.print here shows the received size as 56 which is to be expected and 0 when it isn't.

I've got this feeling that the three problems are interrelated  but I'm at a loss to see or find where the clash is. The Web server file send loop goes round about 10,000 times a second when there are no web enquiries and about 300 then the PC is GETing info. This fills up the serial monitor/hyper terminal long before anything fails

Sorry but the forum webpage wont let me attach more than 4K of code and I don't have an FTP server to do it that way.
Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Do you have a SD card in the shield's slot? If so, remove it for a test.

Does the NTP code on this link work ok for you?
http://arduino.cc/forum/index.php/topic,137789.msg1038071.html#msg1038071

edit: I just checked the file attach under Additional Options. It certainly isn't bigger than 4Mbytes, is it?
Maximum attachment size allowed: 4096 KB, per post: 4
« Last Edit: April 08, 2013, 06:42:56 am by SurferTim » Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi SurferTim,

I'll try your code after lunch but I don't expect a problem from it as it is doing essentially the same thing as mine except that I'm an interrupt routine/counter  to ask for an update every 20 min not mills(), the original system design was for this to be once a day but I've speeded the cycle up so I don't have to wait forever for things to fall over.

I've attached the code file and the HTML files. Your missing the Mystyle.css file but that is only window dressing anyway. The Setup.htm file allows you to set the max, min and default values for the various settings. The server checks they are valid settings before saving them in the EEPROM. (By the way I was reading the 4096Kb and thinking 4096 bytes is not a lot - new glasses required I guess)

You'll see that there is a one second interrupt that sets the bits of 'TickTock' which the main code then uses to update the readings from the sensors every second, update the state of the relays every 15 seconds and then on the 10min interval write data to a file. Then at the moment it should get a NTP update at the 20min - 23 second mark ie just before the next write to the SD card.

Enquiries 'GET' over the Ethernet from the PC to 192.168.1.65 invoke the Index.htm which uses the $Xy replacement flag to put the various temperatures and strings back to the client, you can also change the various thermostat parameters here. Hitting the 'Get saved thermal data' button at the bottom delivers you the current record file on the SD card straight to Excel so you can analyse it.

I'm using Fixed IP addresses as DHCP seems to add 4k to the sketch and I did think of using a RTC but the code for that and using 'Time'h' to set the clock also gave me more than 32k of code for the whole sketch. (Ideally there are a couple of things still to add, The ability to list the files on the card and chose which one to down load and to add more sensors but there is no point in redesigning the input board with MC14051 chips until the fundamental problem is solved. I know now that I would probably be better off with a mega board but there isn't room in the control box for one.)

* TimeStampedTemperatureData40e.ino (31.8 KB - downloaded 12 times.)
* index.htm (7.93 KB - downloaded 11 times.)
* GetData.htm (0.45 KB - downloaded 8 times.)
* response.htm (0.45 KB - downloaded 7 times.)
Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I think you are running out of sockets. I see where you read the first 12 characters of the GET request from the rx buffer, but I do not see where your code reads from the rx buffer until a blank(empty) line. That is "\r\n\r\n". I see an attempt to do that if the request is a POST. Even  when I get that, I insure I have emptied the rx buffer. If you leave characters in that buffer, the socket does not close correctly.

This is the code I use. It has a proven way of reading the request. Feel free to borrow the parts you need.
http://playground.arduino.cc/Code/WebServerST
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi SurferTim,
Well your NTP code ran all night without a hitch. So there is no problem with my broadband NTP service. Now for tweaking my own code to sort out the POST / GET requirements. couple of  things I have noticed:

In my 'actionUpDate()' code iI've got

'if (client.findUntil("thermostat","\n\r")){'     This would leave a '\n' in the buffer. I'll try turning that round to '\r\n' first and that bit should get to the end of the buffer.

I'm also wondering if a 'client.flush();'  after the client.stop(); would do the trick at the end of the code that sorts out POST & GET
Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
'if (client.findUntil("thermostat","\n\r")){'     This would leave a '\n' in the buffer. I'll try turning that round to '\r\n' first and that bit should get to the end of the buffer.
Careful there. I would change that to "\r\n\r\n". If you don't read until the double cr/lf, you risk leaving characters in the rx buffer.

edit:
This is from the reference guide. http://arduino.cc/en/Reference/StreamFindUntil
Quote
findUntil() reads data from the stream until the target string of given length or terminator string is found.
If it finds the target string in the stream, it stops reading and leaves the rest of the characters in the rx buffer.
« Last Edit: April 09, 2013, 04:44:06 am by SurferTim » Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well - if I rem out the part of my code that sorts the HTTP GET and POST then the NTP code keeps working. So clearly the part that sends and receives data to/from web pages is causing the problem.

Changing   'if (client.findUntil("thermostat","\n\r")){'

to  'if (client.findUntil("thermostat","\r\n\r\n")){'

Hasn't made any difference

Nor does including client.flush() just before client.stop()

I'm looking for a way to purge the W5100 RX buffer as it would appear that debris left there is clogging up the works. As far as I can make out client.find(..) and findUntil(..) do not advance the RX pointer but just returns a boolean value. Client.read and readBytesUntil do however consume the contents of the buffer but you have to do something with the values they return.

I'm contemplating putting a blind

while(client.available());{
client.read();
}
client.stop();

on the end of the code rather than the simple client.stop()

Code:
//Ethernet outputs here
 // listen for incoming clients
  byte requestType; 
  client = server.available();
  if (client) {
    while (client.connected()){
      if (client.available()) {
    /////////////////////////////////
    //Wink to show where we've got to 
    digitalWrite(7,LOW);
        memset(fileName, 0 ,sizeof(fileName)); //clear the inputBuffer
        if (client.readBytesUntil('/',fileName,MAX_PAGE_NAME_LEN)){
          if (strcmp(fileName,"GET ") == 0){
          requestType = 1 ; //search for 'GET'
          }
          else if (strcmp(fileName,"POST ") == 0){
          requestType = 2;  //search for 'POST'
          }
        //gather what comes after the '/'
        memset(fileName, 0 ,sizeof(fileName)); //clear the inputBuffer
        //EofN = 0;
          if( client.find("") ){
            fileBufferLen = 0;
            *fileName = 0;
            while(fileBufferLen < MAX_PAGE_NAME_LEN ){
              char c = client.read();
              if( c == 0 ){
                fileBufferLen = 0;   // timeout returns 0 !
              }
              else if((c == 32)||(c == 63)) {  // space character or ?
                fileName[fileBufferLen] = 0; // terminate the string
                break;
              }
              else{
                fileName[fileBufferLen++] = c;
              }
            }
            fileName[fileBufferLen] = 0;
            // Note: inputBuffer full before the closing post_string encountered
          }
          else fileBufferLen = 0;    //failed to find the prestring

        }//end if(client.readBytes
        if (requestType == 2){       //do a POST
          //skip the rest of the header and find the carriage return
         
          actionUpDate();           //check for key words and action
          }//end if (requestType == 2)
        // GET or POST Give the string asked for
        // no file name so give index
        if (fileBufferLen == 0) {
         
          strcpy(fileName,"index.htm");
         
        }//end if
        sendFile(fileName);
        delay(1);
       
       
      }//end while (client.connected()
    }//end(client.avaialble .... for reading the POST/GET file required NB the buffer may not be empty!!
    delay(1);
    while(client.available());{       //make sure w5100 RX buffer is empty
        client.read();
        }
     client.stop();   
      }//end if (client..

digitalWrite(7, HIGH);


} //end 'loop'       

Any body else  tried anything like this?
Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Changing   'if (client.findUntil("thermostat","\n\r")){'

to  'if (client.findUntil("thermostat","\r\n\r\n")){'

Hasn't made any difference

Nor does including client.flush() just before client.stop()
client.flush() won't help. It is a send function that causes the tx buffer to write to the stream.
http://arduino.cc/en/Reference/StreamFlush

The blind read idea is a keeper tho.  smiley-wink
I "blind read" until the \r\n\r\n, then call client.stop() while in the while(client.connected()) loop. That seems to close the socket correctly.
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Changing   'if (client.findUntil("thermostat","\n\r")){'

to  'if (client.findUntil("thermostat","\r\n\r\n")){'

Hasn't made any difference
Then, clearly, the thing to do is to stop trying to actually use the data until you know what the data looks like. Forget about trying to use the data. Simply print it to the serial port, in multiple formats - char and hex. Only AFTER you KNOW what the data looks like can you successfully parse it.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well

Yes I do know what is coming down the wire its everything from "HTTP1.1........... through to...  keep connection live. \r\n\r\n"

If I eliminate all of it from the RXbuffer then the NTP part of the code works BUT the web page does not come up on the browser. it just sits there waiting. If I leave any of the '\r\n\r\n' in the buffer then the NTP code returns zero.

I guess I need to find a way of getting the TCP bit and the UDP bit to use different sockets.

Unfortunately simply moving

client = server.available();
clinet.connected();

to the front of the loop code doesn't stop UDP hijacking the TCP socket.

All ideas welcome
Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

It doesn't hijack anything. It took me about 5 minutes to put a NTP client and a web server together in the same sketch. It is running right now. The NTP time is correct to the second, and the server works fine. I used the server code in a function called doServer() instead of loop(), and call it every iteration through loop() just before the NTP stuff.

I used this NTP code:
http://arduino.cc/forum/index.php/topic,137789.msg1038071.html#msg1038071
I used this server code:
http://playground.arduino.cc/Code/WebServerST
It is still running both just fine.

edit: On the NTP code, I replaced the 10 minute delay with a millis() time check so it wouldn't block there for 10 minutes.
« Last Edit: April 10, 2013, 09:16:23 am by SurferTim » Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 28
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well folks I've built a piece of code that merges the two SurferTim offered up, and that works in its raw state. I've added my interrupt routine which just sets a flag and I've replaced the millis/delay counters with an routine in the loop that is triggered by the interrupt flag.
That did not work at first, untiI put a delay(1) between the trigger and the call to the NTP code.

So I now need to work out how to extract the GET ,POST, fileName, and the feedback of the values to be changed by the post instruction from the buffer (tBuff) into which the browser requests are written. Why was 64 bytes chosen for this buffer? The returned bytes from my browser IE10 run to 240+ from the 'HTTP/1.1'.....'bit through to the .....'keep connection alive\r\n\r\n'

You say the socket is not being hijacked but why when I put some Serial.print debug lines in the Library code di both the TCP and UDP bits of code nearly always come back as _sock = 1? Given that the server is talking to one IP address on port 80 and the UDP is talking to another IP on port 8888 surely they should be using two different sockets.

Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 138
Posts: 5833
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Why was 64 bytes chosen for this buffer?
That was all I needed for my request string. If you need more, feel free to make it bigger.

Quote
You say the socket is not being hijacked but why when I put some Serial.print debug lines in the Library code di both the TCP and UDP bits of code nearly always come back as _sock = 1? Given that the server is talking to one IP address on port 80 and the UDP is talking to another IP on port 8888 surely they should be using two different sockets.
What makes you think that? And you say "nearly always" as if occasionally they are not. If there is no port 80 request pending, the UDP function will use the first socket available.
« Last Edit: April 16, 2013, 10:27:27 am by SurferTim » Logged

Pages: [1] 2   Go Up
Jump to: