Uploading file from SD card to remote server?

I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

I started playing with Pachube but it's going to be way too slow, just initializing it at startup hangs my sketch enough that I can't get any response from buttons I'm using for a menu.

I'm going to write the data to an SD card and then I'd like to FTP it to a remote server. I can't find any FTP libraries. What are my options for getting the file off the SD short of writing my own FTP library?

I started playing with Pachube but it’s going to be way too slow

Why do you want to send the data to a web server? Is there any reason to publish the data, rather than just put it on your PC?

I’m going to write the data to an SD card and then I’d like to FTP it to a remote server. I can’t find any FTP libraries. What are my options for getting the file off the SD short of writing my own FTP library?

Wouldn’t it be simpler to remove the SD card from the Arduino, put it in the PC, and then deal with the data there?

Wouldn't it be simpler to remove the SD card from the Arduino, put it in the PC, and then deal with the data there?

Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

I can see where FTP would be really helpful. I have not gotten that far along where I need that much data yet, but I can see it coming. Doesn't look that bad to me.

Don't let the thought that "nobody has done it before" is a reason for you not to do it. I took the time to write email sending code. I will post a link to that. It uses a two-way communication like you will need for the FTP protocol. It sends a command, then listens for, and serial.prints, the response. http://arduino.cc/forum/index.php/topic,69647.15.html

Here is the formats for the commands and responses for FTP. http://www.w3.org/Protocols/rfc959/8_PortNumber.html

I like the idea! :)

Here's what I'm working on. http://www.youtube.com/watch?v=p8amtkAbAWo

A custom Arduino board, LCD screen, buttons and power supply will be in a separate box off to the side. There's no easy way to design the case so that the SD is accessible from the outside. I don't want users to have to open the case up and dig the SD card out, especially if they have boxing gloves on.

I'd like to send the data to our server where we can graph the results for them. How many punches they landed,force, g's, etc.

I've done all the programming to this point (menu, lcd, motors, sensors) but I think writing FTP code is above my head, might have to find a pro to handle that.

I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?

But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?

Nobody said anything about high impact! :astonished: But why not? If the thing survives, ok with me.

I've already got my ethernet shield connected to my FTP server. I'll let you know how it goes.

edit: I just logged in with a username and password. So far, it works pretty clean, but the difficult part is to come. Will I be able to keep the command connection to port 21 open when I open a data connection to port 20?

I'd love to hear how you progress on an FTP connection. If I end up building a library I'll release the code.

If you want to try what I have so far, this connects in passive mode to both the command and data ports. Change network settings and user/password down in the sendFTP function. Press the ‘f’ key and enter.

It just connects and disconnects to insure it can. See if it works on your ftp server.

// w5100 FTP passive client
// IDE v1.0 only

#include <SPI.h>
#include <Ethernet.h>

// Set to your network settings
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };  
IPAddress ip( 192, 168, 0, 2 );    
IPAddress gateway( 192, 168, 0, 1 );
IPAddress subnet( 255, 255, 255, 0 );

// Change this to the ip of the ftp server
IPAddress server( 192, 168, 0, 3 );

EthernetClient client;
EthernetClient dclient;

char outBuf[128];
char outCount;

void setup()
{
  Serial.begin(9600);
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);
  Ethernet.begin(mac, ip, gateway, gateway, subnet); 
  delay(2000);
  Serial.println("Ready. Press f");
}

void loop()
{
  byte inChar;

  inChar = Serial.read();

  if(inChar == 'f')
  {
    if(sendFTP()) Serial.println("FTP OK");
    else Serial.println("FTP failed");
  }
}

byte sendFTP()
{
  if (client.connect(server,21)) {
    Serial.println("Command connected");
  } 
  else {
    Serial.println("Connamd connection failed");
    return 0;
  }

  if(!eRcv()) return 0;

// Change to your user and password  
  client.write("USER myusername\r\n");

  if(!eRcv()) return 0;

  client.write("PASS mypassword\r\n");

  if(!eRcv()) return 0;

  client.write("SYST\r\n");

  if(!eRcv()) return 0;

  client.write("PASV\r\n");

  if(!eRcv()) return 0;

  char rtnVal[32];
  
  sscanf(outBuf,"%*s %*s %*s %*s %s",rtnVal);

  unsigned int hiPort,loPort;

  sscanf(rtnVal,"%*c%*d%*c%*d%*c%*d%*c%*d%*c%d%*c%d",&hiPort,&loPort);
  
  Serial.print("Port ");
  hiPort = hiPort << 8;
  loPort = loPort & 255;
  hiPort = hiPort | loPort;
  Serial.println(hiPort);
  

  if (dclient.connect(server,hiPort)) {
    Serial.println("Data connected");
  } 
  else {
    Serial.println("Data connection failed");
    client.stop();
    return 0;
  }

  client.write("QUIT\r\n");

  if(!eRcv()) return 0;

  dclient.stop();
  client.stop();
  Serial.println("disconnected");
  return 1;
}

byte eRcv()
{
  byte respCode;
  byte thisByte;

  while(!client.available()) delay(1);

  respCode = client.peek();

  outCount = 0;
  
  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);

    if(outCount < 127)
    {
      outBuf[outCount] = thisByte;
      outCount++;      
      outBuf[outCount] = 0;
    }
  }

  if(respCode >= '4')
  {
    efail();
    return 0;  
  }

  return 1;
}


void efail()
{
  byte thisByte = 0;

  client.write("QUIT\r\n");

  while(!client.available()) delay(1);

  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);
  }

  client.stop();
  dclient.stop();
  Serial.println("disconnected");
}

Try using syslog. This could could give you (near) realtime access to the data too.

PaulS:

I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?

I'm a little confused. You're quoting two different people. The boxing robot is not on a rooftop.

jrburden: Try using syslog. This could could give you (near) realtime access to the data too.

I'm assuming you're talking about this? There doesn't seem to be much activity there. http://code.google.com/p/ardusyslog/

My worry is latency where the program hangs as it's sending the data and waiting for confirmation. Is there a way to write it all to SD, close the file, open the file again and initiate a syslog transfer?

Just an update. I have my FTP code reading from and writing to files on my FTP server. I have no SD interface set up yet tho. The test code compiles to 14524 bytes.

Interested?

Update: I have the full package working now. Upload files from the FTP server to the SD card, or download files from the SD card to the FTP server. Right now, I have a define that allows me to compile either upload or download to save program memory.

The code with debugging compiles to 25856 bytes. A little steep for an Uno, but for my Mega2560, not a problem.

If you are interested, let me know. I have a little error checking to do to insure all works ok. If anyone shows interest, I will post the code then.

If you are not interested, at least I'll have it when I need it. :)

I'd love to see your code. I am in the final phases of programming an ftp server on my Mega. I took a different path, figuring it was easier to use an ftp client as needed on the main machine instead of having a server running to recieve the data.

Had some problems with getting the passive data channel to connect since the ftp client (Filezilla in my case) doesn't send any data upstream on the data channel, it just listens (as called out in the ftp standard). The server.available call expects the Rx buffer to have something before it connects. I had to do a terrible thing and modify the ethernet library to be able to get the data channel to connect to the socket without any data in the Rx buffer. I'd like to see how you got that to work....I was going to delve into the ethernet chip library and do it at a low level to get away from modding the library.

I also tweaked the SD library to get the file modified date output rather than use the SDFAT library directly. I use the EPLF format to transfer the directory information, and although the date is not required, I always like know which file is newer....

I have it running fairly well but there seems to be some timing issues where if I comment out some of my debugging serial.print lines it breaks the program....also need to make the code somewhat more presentable, lots of commented dead ends leftover from the development.....

I'll upload my code when I get back to my shop at the end of the week if anyone is interested. If I recall it runs about 38k but has internet time, software RTC, a webserver (based on the http example), and the ftp server all packaged together.

Here is the code I have so far. It is not documented as well as I would like it, but I can do that later.

The “#define FTPWRITE” determines whether it reads or writes to the FTP server. Commented out, it reads from the FTP server and writes to the SD.

Change your network settings, and the user/password in the doFTP() routine.
Place a file named test.txt in your FTP server.

Upload and open serial monitor. Wait for ready.
Press ‘r’ and enter to view SD file contents.
Press ‘f’ and enter to transfer.

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

// reads from server and writes to SD file
// uncomment next line to write SD file to server.
// #define FTPWRITE

byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };  
IPAddress ip( 192, 168, 0, 2 );    
IPAddress gateway( 192, 168, 0, 1 );
IPAddress subnet( 255, 255, 255, 0 );

// FTP server ip
IPAddress server( 192, 168, 0, 3 );

EthernetClient client;
EthernetClient dclient;

char outBuf[128];
char outCount;

void setup()
{
  Serial.begin(9600);

  if(SD.begin(4) == 0)
  {
    Serial.println("SD init fail");          
  }

  Ethernet.begin(mac, ip, gateway, gateway, subnet); 
  digitalWrite(10,HIGH);
  delay(2000);
  Serial.println("Ready. Press f or r");
}

void loop()
{
  byte inChar;

  inChar = Serial.read();

  if(inChar == 'f')
  {
    if(doFTP()) Serial.println("FTP OK");
    else Serial.println("FTP FAIL");
  }

  if(inChar == 'r')
  {
    readSD();    
  }

}

File fh;

byte doFTP()
{
#ifdef FTPWRITE
  fh = SD.open("test.txt",FILE_READ);
#else
  SD.remove("test.txt");
  fh = SD.open("test.txt",FILE_WRITE);
#endif

  if(!fh)
  {
    Serial.println("SD open fail");
    return 0;    
  }

  Serial.println("SD opened");
  
  if (client.connect(server,21)) {
    Serial.println("Command connected");
  } 
  else {
    Serial.println("Command connection failed");
    fh.close();
    return 0;
  }

  if(!eRcv()) return 0;

// change to your user
  client.write("USER myusername\r\n");

  if(!eRcv()) return 0;

// change to your password  
  client.write("PASS mypassword\r\n");

  if(!eRcv()) return 0;

  client.write("SYST\r\n");

  if(!eRcv()) return 0;

  client.write("PASV\r\n");

  if(!eRcv()) return 0;

// new code

  char *tStr = strtok(outBuf,"(,");
  int array_pasv[6];
  for ( int i = 0; i < 6; i++) {
    tStr = strtok(NULL,"(,");
    array_pasv[i] = atoi(tStr);
    if(tStr == NULL)
    {
      Serial.println("Bad PASV Answer");    
      client.stop();
      fh.close(); 
      return 0;  
    }
  }

  unsigned int hiPort,loPort;

  hiPort = array_pasv[4] << 8;
  loPort = array_pasv[5] & 255;

// end of new code

  Serial.print("Data port: ");
  hiPort = hiPort | loPort;
  Serial.println(hiPort);
  
  if (dclient.connect(server,hiPort)) {
    Serial.println("Data connected");
  } 
  else {
    Serial.println("Data connection failed");
    client.stop();
    fh.close();
    return 0;
  }

#ifdef FTPWRITE 
  client.write("STOR /test.txt\r\n");
#else
  client.write("RETR /test.txt\r\n");
#endif

  if(!eRcv())
  {
    dclient.stop();
    return 0;
  }

#ifdef FTPWRITE
  Serial.println("Writing");

// This needs improvement

  while(fh.available())
  {
    char c = fh.read();
    dclient.write(c);
  }

#else
  while(dclient.connected())
  {
    while(dclient.available())
    {
      char c = dclient.read();
      fh.write(c);      
      Serial.write(c); 
    }
  }
#endif

  dclient.stop();
  Serial.println("Data disconnected");
  
  if(!eRcv()) return 0;

  client.write("QUIT\r\n");

  if(!eRcv()) return 0;

  client.stop();
  Serial.println("Command disconnected");

  fh.close();
  Serial.println("SD closed");
  return 1;
}

byte eRcv()
{
  byte respCode;
  byte thisByte;

  while(!client.available()) delay(1);

  respCode = client.peek();

  outCount = 0;
  
  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);

    if(outCount < 127)
    {
      outBuf[outCount] = thisByte;
      outCount++;      
      outBuf[outCount] = 0;
    }
  }

  if(respCode >= '4')
  {
    efail();
    return 0;  
  }

  return 1;
}


void efail()
{
  byte thisByte = 0;

  client.write("QUIT\r\n");

  while(!client.available()) delay(1);

  while(client.available())
  {  
    thisByte = client.read();    
    Serial.write(thisByte);
  }

  client.stop();
  Serial.println("Command disconnected");
  fh.close();
  Serial.println("SD closed");
}

void readSD()
{
  fh = SD.open("test.txt",FILE_READ);

  if(!fh)
  {
    Serial.println("SD open fail");
    return;    
  }

  while(fh.available())
  {
    Serial.write(fh.read());
  }
  
  fh.close();
}

If you have questions, ask. My apology in advance for any typos.

edit: This is FTP client code. I could not find a way to use passive mode with the Arduino as the server.

edit2: Code change on passive mode port retrieval, thanks to cedric2. :slight_smile:
That saves about 2K of memory.

Here is my sloppy code, please excuse the commented out sections, I am still changing the code to minimize the need for modifications to the standard libraries. The program is still sensitive to edits and I believe that there are timing issues to address, any help will be appreciated.

The changes to the standard libraries are bad form and I apologize for them but, the edits were required to get this to work. Adding the functions worked almost instantly after a month of trying to use the unmodified libraries. I will continue to try to get the standard libraries to work, there are several interesting ideas in other threads that I am going to try…

First the Changes to the Standard 1.0 Libraries;

// In the Ethernet libraries

// in EthernetClient.h Add
 uint8_t getsocket();
//after uint8_t status();

//in EthernetClient.cpp Add
uint8_t EthernetClient::getsocket()
{
  return _sock;
}
// to the end of the file

// in Ethernet Server.h add
 uint8_t getsocket();
 uint16_t getport();
// after  EthernetClient available();

// in EthernetServer.cpp add
uint16_t EthernetServer::getport()
{
  return _port;
}

uint8_t EthernetServer::getsocket()
{

	for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
	  if (EthernetClass::_server_port[sock] == _port) {
	    	return sock;
	    }
	  }
	return MAX_SOCK_NUM;
}
// at the end of the file

// in the SD library

//in SD.h add
 void dirEntry(dir_t* sd_dir);
//before  using Print::write;

//in SD.cpp add
void File::dirEntry(dir_t* sd_dir) {
	_file->dirEntry(sd_dir);
}
// at the end of the file

After making those edits, the code in the attached file should compile and run. The listing is too long for posting.

You will need to edit the IP and MAC address to work with your setup.
You also need to comment out and/or adjust the pin settings if you have an LCD screen, otherwise comment out any LCD lines.

Let me know if this works for you.

MegaWebserverPlus.ino (35.5 KB)

@michael635: I took a look at your code. Looks ok, but I have not run it yet.

Why do you need to modify the Ethernet library? I did not look at the code that closely, but is that how you listen on two separate ports? That is the part I couldn't figure out with my ftp server code, besides the router masquerade (can't reach it from the internet without some internal routing).

The problem with the Ethernet library is that to use the server.available function to start he client, there must be data available in the Rx buffer. No data , no connection. To over come this I added a simple function server.getsocket that returns the socket number for the server that can be used to connect the client to the socket to send data. Also added some similar functions to expose the PRIVATE data for debugging. The library should be able to listen on two ports, but it seems to always default to the first port with Rx data. Checking the server socket allows the client to connect to the correct server, not the first available.

See the following link that has some relevance, although I couldn't get the suggested code to function.

http://arduino.cc/forum/index.php?topic=89881.0;wap2

I'm still working on a workaround, but haven't been able to overcome the limitation in the Ethernet Library. My next attempts will be to use the W5100 library directly, but that has a penalty for folks not using that exact board...If anyone has ideas please share....

I'm still sitting behind the router, haven't attempted to go to the otherside, but I expect that I will need to use some DNS service to overcome the dynamic IP issue.

I had not even considered the "no request send" on the data connection. Maybe it is best I went with the client end code instead.

The router routing challenge is the reason I use client passive mode. Client active mode requires the ftp (edit: client) to contact the (edit: server) on a port specified from a port specified by the client (through my firewall/masquerade). edit: You would have to know what port the Arduino is being masqueraded as to send to the server for that. Very complex to determine.

I can route my stuff no problem, but explaining that to someone who is not familiar with routing can be challenging.

If you want to access the Arduino from the internet in server active mode, you only need port 21 routed to the localnet ip of the Arduino. Port 20 will contact the client in client mode on the port specified in the cmd channels PORT send. In server passive mode, you need to route port 21 and the data channel port through the router to the Arduino.

Then there is the user/password thing. If you are going to have an ftp server that allows writing, I highly suggest you use at least some form of hack prevention. If you leave it open on the internet, you will amazed at how quickly those files will either disappear or get filled with junk.

edit: My typing got ahead of my brain in ftp active mode explanation. I'll blame it on "logic reversal".

I am hoping passive modes works without need for routing ports, like you say it is tough to explain. I prefer the KISS principle (Keep It Simple Silly)and passive ftp is a standard function and in my case Filezilla has worked flawlessly through the router in passive mode to other sites without changes...