Sd Card & Ethernet Problems

Physical hardware: Uno, Ethernet card, SanDisk Class 4 4Gbyt MicroSD card(formatted as FAT32) , IDE 1.0.4

I have a data logging sketch that records info on an SD card at 10min intervals. It also serves up a web page that allows the current info to be viewed over the Ethernet. Some times it will run for weeks and other times the Ethernet shield does not respond to a GET request from the PC after a couple of hours (The Wiz 5100 does not respond to a Ping either. Ping just tells me the device is unreachable).

I have set the SD card CS pin high at the beginning of the programme, So I'm now wondering if there is a subtly in the order of the Ethernet/SD card setups that Ive got wrong. Or is a class 4 micro SD card is too slow? (does any one know if an SD card slows down when it gets warm?)

The ambient temperature where the unit is 25-28DegC The chips do warm up but you can still touch them with out going 'ouch'

The Sketch is big (30K) and FreeRam() is telling me that I've got 450 - 500 bytes of SRAM free depending where I put it in the code. And the fact that it can run some times for months suggests that there is no memory leak.

This is the 'Setup' part of the code

// Chip Select Pins
#define sdChipSelectPin 4
#define ethernetChipSelectPin 10
// Relay Output Pins
#define RelayPinDifferential 8        //limit stat defaults to 60C
#define RelayPinVariable 9            //variable stat with calculated temperature
// Define Clock Variables
volatile boolean TickTock [8];         //Used as a set of flags to control interupt output and timed events
volatile unsigned long systime = 1385798455UL; //946684800UL; //default time at start of year 2000;
long timeError;                       //error in systime when getting ntpEpoch

   
char dateTime [12] = "01 12:59:59";       //to hold dateTime

char fileName [13];                   //to hold file on SD card nb the date (year&month) is overwritten each month
char fileRequest[13];                 //File Client requests
const byte MAX_PAGE_NAME_LEN = 12;    //max characters in page name
byte fileBufferLen = 0;

#include <SPI.h> 
#include <SD.h>
#include <Ethernet.h>
#include <EthernetUdp.h>

#include <EEPROM.h>

// MAC address for controller below.
// 
byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //was ED!

const unsigned int localPort = 8888;           // local port to listen for UDP packets
// list of NTP servers so we don't have to look them up again
//ntp_servers[0] timeServer(213,123,26,170);   // time at BT
//ntp_servers[1] timeServer(94,44,157,90);     // Europe pool
//ntp_servers[2] timeServer(192,43,244,18);    // time.nist.gov NTP server US

const IPAddress ntp_server (213,123,26,170);   // use BT
const IPAddress ip (192,168,1,62);             //62 Fixed IP for server DCHP takes up too much space
const IPAddress localdns (192,168,1,254);
const IPAddress gateway (192,168,1,254);
const IPAddress subnet (255,255,255,0);

// setup for ethernet
// A UDP instance to let us send and receive packets over UDP
EthernetUDP Udp;
// Initialize the Ethernet server library
// HTTP defaults to port 80
EthernetServer server(80);
EthernetClient client;

const byte BUFFER_LEN = 48;           //TC/IP Output buffer
byte Buffer[BUFFER_LEN];
  
//setup for temperature measurement N.B. arrays are zero indexed
unsigned int readings[6];       // accumulates the readings from the analog inputs 
                                // NB these numbers are 16 times A/D conversion figures
boolean temperatureRising[6];   // HIGH indicates thermostat has switched OFF on temperature rise

int TCTemps[24];               // set points for transfer curve 0-5 for used figures,6-11 default, 
                                // 12-17 for max 18- 23 for min
int SetPoints [24];            // Temeratures which will always cause a switch OFF
                                // resistors in measurement bridge 
const long Pullup = 4700;       // 4700 measured value
const long VoltDiv1 = 4690;     // 4700 measured value 
const long VoltDiv2 = 1426;     // 1436 measured value in parallel with internal resistor
int counter;
//DEFINITION for bottom of 'convert2temp' just so it does not have to be calulated every time and we dont have to divide the toals either
const long VoltDivider = (VoltDiv1+VoltDiv2)*1024*16/VoltDiv2; 

//SETUP
void setup()
{
  // flip the relays so we know its all connected
  pinMode(RelayPinDifferential, OUTPUT);   //Set relay driver pins to OUTPUT
  pinMode(RelayPinVariable, OUTPUT);       //ie Digital pins 8 & 9
  
  digitalWrite(RelayPinDifferential, HIGH);//Set them HIGH so inital state of relays 
  digitalWrite(RelayPinVariable, HIGH);    //is power off

  // Set up CS pins for SD card and Ethernet 
  pinMode (ethernetChipSelectPin,OUTPUT);   // make sure its OUTPUT
  pinMode (sdChipSelectPin,OUTPUT);         // make sure its OUTPUT
  
  digitalWrite(sdChipSelectPin, HIGH);      // make sure its off
  digitalWrite(ethernetChipSelectPin,HIGH); // make sure its off
  
/////////////////////////
//  Serial.begin(9600);                         // for debuging
/////////////////////////   
//Get other info from EEPROM 
  for (byte i = 0; i<=23; i++){
    SetPoints[i] = Byte2Int(EEPROM.read(i));    
    TCTemps[i]   = Byte2Int(EEPROM.read(i+24));  
  }  
//Set up reference for Analog input
  analogReference (EXTERNAL);//Use External to avoid any dependancy on Vcc
  analogRead (A0);           //force the procesor to listen  

// use fixed IP address (DHCP seem to add 4k to sketch)
  Ethernet.begin(mac,ip,localdns,gateway,subnet);
  digitalWrite(ethernetChipSelectPin,HIGH); // make sure its off
// line start the Ethernet server
  server.begin();
  Udp.begin(localPort);
//Check presence of SD card
  if (!SD.begin(sdChipSelectPin)) { //card initialized.
    // don't do anything more:
    return;
  }

///////////////////////////////////
  // initialize timer1 
  noInterrupts();           // disable all interrupts
  TCCR1A = 0;
  TCCR1B = 0;
  TCNT1  = 0;
                            // runs slow @ 15624 fast @ 15563 try 15601which is slightly too
                            // fast and causes double dipping for time stamp after 24hours
  OCR1A = 15602; //0x3CF2;  // compare match register for 1Hz output
  TCCR1B |= (1 << WGM12);   // CTC mode
  TCCR1B |= (1 << CS10);
  TCCR1B |= (1 << CS12);    // 1024 prescaler 
  TIMSK1 |= (1 << OCIE1A);  // enable timer compare interrupt
  interrupts();             // enable all interrupts

//Get start time from NTP
  TickTock[4] = true;       // Flag start up for record in data file
  sendNTPpacket();          // send an NTP packet to a time server using a range of addresses it could still come back
                            // with zero but its unlikely It will be flaged and try again on first 10min cycle
  breakTime(systime);       // So make up the file name for the current data This gets updated automatically
                            // but we might not want to wait after a restart
  
  
  //get initial fill of average temperature data from each sensor
  for (byte j = 0; j<128; j++){ //use 128 lots of readings for the first load of the array
    GetReadings();
  }
//  Serial.print (systime);
} //end 'setup'

Interestingly the sketch continues to loop and do all the things It's supposed to even when it stops talking to the pc on the other end of the local network.

This has been bugging me for a year now, I've already tried Surfer-Tim's Timeout trick, in case it was a hardware/Ethernet wiring problem, to no avail. I've also tried to introduce several Serial.print(... statements to debug and find if any of the code stops working but usually the serial monitor overflows and crashes before the communication with the network falls over

This is the check I use. I usually do a Serial character check to get this data when I want it.

byte socketStat[4];

void ShowSockStatus()
{
  for (int i = 0; i < MAX_SOCK_NUM; i++) {
    Serial.print(F("Socket#"));
    Serial.print(i);
    uint8_t s = W5100.readSnSR(i);
    socketStat[i] = s;
    Serial.print(F(":0x"));
    Serial.print(s,16);
    Serial.print(F(" "));
    Serial.print(W5100.readSnPORT(i));
    Serial.print(F(" D:"));
    uint8_t dip[4];
    W5100.readSnDIPR(i, dip);
    for (int j=0; j<4; j++) {
      Serial.print(dip[j],10);
      if (j<3) Serial.print(".");
    }
    Serial.print(F("("));
    Serial.print(W5100.readSnDPORT(i));
    Serial.println(F(")"));
  }
}

Then in loop()

void loop() {
   if(Serial.available()) {
      if(Serial.read() == 's') ShowSockStat();
   }
   // rest of your loop
}

Hi thanks for the cod,e but I cant make it compile with my sketch!!

I get two errors

A stray $) at the end of the uint8_t dip[4]; line which I presume is a typo
and ' 'w5100' was not declared in this scope' despite my putting an #include <w5100.h> at the front of the code

The actual file w5100.h is in the utility directory under the Ethernet directory on the PC which is where it always ha been .

You got a stray smiley face error? Not surprising, I guess, as smiley faces are not understood by most compilers.

Well the smiley is in fact a '$ )' couplet, but that's not the problem I cant get Surffer-Tim's bit of code to compile once its added to my own code.

I've put

#include <w5100.h> in with the other include statements but all I get out is:-

TimeStampedTemperatureData44f.ino: In function 'void ShowSockStatus()':
TimeStampedTemperatureData44f:452: error: 'w5100' was not declared in this scope

when it hits the

uint8_t s = w5100.readSnSR(i);

line in 'ShowSockStatus() function.

Now if I go
#include "w5100.h" then I get the following:-

TimeStampedTemperatureData44f.ino:125:19: error: w5100.h: No such file or directory
TimeStampedTemperatureData44f.ino: In function 'void ShowSockStatus()':
TimeStampedTemperatureData44f:452: error: 'w5100' was not declared in this scope

Which tells me that the use of <w5100.h> is right but the compiler/linker is loosing its ability to see the relivant files during the compile.

As my sketch is falling over on its Ethernet connection and stops behaving as a web server it would be rally nice to know if I'm running out of sockets.

solar_eta:
Well the smiley is in fact a '$ )' couplet, but that's not the problem I cant get Surffer-Tim's bit of code to compile once its added to my own code.

I've put

#include <w5100.h> in with the other include statements but all I get out is:-

TimeStampedTemperatureData44f.ino: In function 'void ShowSockStatus()':
TimeStampedTemperatureData44f:452: error: 'w5100' was not declared in this scope

when it hits the

uint8_t s = w5100.readSnSR(i);

line in 'ShowSockStatus() function.

Now if I go
#include "w5100.h" then I get the following:-

TimeStampedTemperatureData44f.ino:125:19: error: w5100.h: No such file or directory
TimeStampedTemperatureData44f.ino: In function 'void ShowSockStatus()':
TimeStampedTemperatureData44f:452: error: 'w5100' was not declared in this scope

Which tells me that the use of <w5100.h> is right but the compiler/linker is loosing its ability to see the relivant files during the compile.

As my sketch is falling over on its Ethernet connection and stops behaving as a web server it would be rally nice to know if I'm running out of sockets.

Have you resolved this? I run an Ethermega card with SD card, ethernet functionality, 256KB Flash, 100KB program, 8KB RAM, 4KB EEPROM.

My system seems robust now but I have done a lot of work to get the memory use correct. I extensively use F() Strings, have 2KB of strings in EEPROM and have developed comprehensive RAM utilisation functions.

Given that both ethernet and SD card functionality can grab 500 bytes of RAM on the heap at any time for buffering etc I have seen crashes occur whenever my normal free RAM drops below 1KB. Also the length of an HTML Get request does not seem controllable - a web crawler might send a long GET request that might cause a memory crash at any random time. When my system responds to GET requests it quickly determines valid requests and then discards (flush and stop) anything invalid. Is your system quickly discarding browser icon file requests?

My system seems to do some of the things yours does. You can see it at http://219.88.69.69/2WG/. Have a look on the RAM usage page - you will see I am tracking minimum free RAM, maximum heap size, maximum stack size and the free heap list. Those stats are reset everyday at midnight and recorded in an activity log so that I can review daily minimum RAM etc over recent weeks at any time. When I get an unexpectedly low RAM statistic I can review the procedure call hierarchy and activity log to work out what caused the low RAM at that time and to see if I can do anything to minimise the issue.

Cheers

Catweazle NZ

Also the length of an HTML Get request does not seem controllable - a web crawler might send a long GET request that might cause a memory crash at any random time.

It is controllable. The w5100 rx buffer holds it until you read it. Just read the amount you can store in the array. Here is my server code. It has code that prevents an array overflow.
http://playground.arduino.cc/Code/WebServerST

Here is the section of the code that limits it. No matter how large the GET line is, this stores only the first 63 characters in tBuf and ignores the rest.

        if(currentLineIsGet && tCount < 63)
        {
          tBuf[tCount] = c;
          tCount++;
          tBuf[tCount] = 0;          
        }

SurferTim:

Also the length of an HTML Get request does not seem controllable - a web crawler might send a long GET request that might cause a memory crash at any random time.

It is controllable. The w5100 rx buffer holds it until you read it. Just read the amount you can store in the array. Here is my server code. It has code that prevents an array overflow.
Arduino Playground - HomePage

Here is the section of the code that limits it. No matter how large the GET line is, this stores only the first 63 characters in tBuf and ignores the rest.

        if(currentLineIsGet && tCount < 63)

{
          tBuf[tCount] = c;
          tCount++;
          tBuf[tCount] = 0;         
        }

Its not just the first 63 or 100 characters of a GET or POST line that you need to process. For a POST you need to parse the following lines of the html request to extract the client/user input fields. I also parse those lines for the HOST value as a cross check against the client IP address to be certain if the call is coming from my LAN or externally. And I also parse those lines to extract session cookie information that is an integral part of my application security,

Depending in the length of your normal URLs (and mine can be longish because they contain directory paths when extracting/displaying text files from my SD card) and the number of other fields being extracted from the html request you can use a fair bit of memory/string/String space when processing html requests. I am not sure that non trivial contemporaneous ethernet and SD card operations on an Arduino 2K machine is viable. If you have a browser input form that accepts multiple variable string lines (such as an address or email content) then processing it in a 2K Arduino environment might be impossible.

Catweazle NZ

That is why I use a Mega.

It's now looking as there is nothing wrong with the code or with the Uno or the Ethernet card or the SD card!!!

I was reading elsewhere about electrical interference and then my son mentioned that when his microwave started up his main ADSL modem/router had a funny five minutes I started looking at interference rather than for a code bug. It turns our that the W5100 chip is particularly sensitive to magnetic spikes from adjacent curcitry .

I can now make the problem happen at my command (or not as the case might be) but I needed a couple of Capacitor resistor snubbers across some relay contacts and ferrite sleves on the 9V supply lead to the Uno.

solar_eta:
It's now looking as there is nothing wrong with the code or with the Uno or the Ethernet card or the SD card!!!

I was reading elsewhere about electrical interference and then my son mentioned that when his microwave started up his main ADSL modem/router had a funny five minutes I started looking at interference rather than for a code bug. It turns our that the W5100 chip is particularly sensitive to magnetic spikes from adjacent curcitry .

I can now make the problem happen at my command (or not as the case might be) but I needed a couple of Capacitor resistor snubbers across some relay contacts and ferrite sleves on the 9V supply lead to the Uno.

My ADSL modem/router and Ethermega system are in a cupboard and on the otherwise of the wall is the microwave oven in the kitchen. I have not had any interference problems that I am aware of but I must check if any odd experiences coincide with use of the microwave.

Cheers

Catweazle NZ