Go Down

Topic: Uploading file from SD card to remote server? (Read 44819 times) previous topic - next topic

chrishjones

I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

I started playing with Pachube but it's going to be way too slow, just initializing it at startup hangs my sketch enough that I can't get any response from buttons I'm using for a menu.

I'm going to write the data to an SD card and then I'd like to FTP it to a remote server. I can't find any FTP libraries. What are my options for getting the file off the SD short of writing my own FTP library?


PaulS

Quote
I started playing with Pachube but it's going to be way too slow

Why do you want to send the data to a web server? Is there any reason to publish the data, rather than just put it on your PC?

Quote
I'm going to write the data to an SD card and then I'd like to FTP it to a remote server. I can't find any FTP libraries. What are my options for getting the file off the SD short of writing my own FTP library?

Wouldn't it be simpler to remove the SD card from the Arduino, put it in the PC, and then deal with the data there?

SurferTim

Quote
Wouldn't it be simpler to remove the SD card from the Arduino, put it in the PC, and then deal with the data there?

Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

I can see where FTP would be really helpful. I have not gotten that far along where I need that much data yet, but I can see it coming. Doesn't look that bad to me.

Don't let the thought that "nobody has done it before" is a reason for you not to do it. I took the time to write email sending code. I will post a link to that. It uses a two-way communication like you will need for the FTP protocol. It sends a command, then listens for, and serial.prints, the response.
http://arduino.cc/forum/index.php/topic,69647.15.html

Here is the formats for the commands and responses for FTP.
http://www.w3.org/Protocols/rfc959/8_PortNumber.html

I like the idea!  :)

chrishjones

Here's what I'm working on. http://www.youtube.com/watch?v=p8amtkAbAWo

A custom Arduino board, LCD screen, buttons and power supply will be in a separate box off to the side. There's no easy way to design the case so that the SD is accessible from the outside. I don't want users to have to open the case up and dig the SD card out, especially if they have boxing gloves on.

I'd like to send the data to our server where we can graph the results for them. How many punches they landed,force, g's, etc.

I've done all the programming to this point (menu, lcd, motors, sensors) but I think writing FTP code is above my head, might have to find a pro to handle that.

PaulS

Quote
I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

Quote
Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?




SurferTim

#5
Feb 24, 2012, 06:54 pm Last Edit: Feb 24, 2012, 07:03 pm by SurferTim Reason: 1
Quote
But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?

Nobody said anything about high impact! :smiley-eek:
But why not? If the thing survives, ok with me.

I've already got my ethernet shield connected to my FTP server. I'll let you know how it goes.

edit: I just logged in with a username and password. So far, it works pretty clean, but the difficult part is to come. Will I be able to keep the command connection to port 21 open when I open a data connection to port 20?


chrishjones

I'd love to hear how you progress on an FTP connection. If I end up building a library I'll release the code.

SurferTim

If you want to try what I have so far, this connects in passive mode to both the command and data ports. Change network settings and user/password down in the sendFTP function. Press the 'f' key and enter.

It just connects and disconnects to insure it can. See if it works on your ftp server.
Code: [Select]
// w5100 FTP passive client
// IDE v1.0 only

#include <SPI.h>
#include <Ethernet.h>

// Set to your network settings
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };  
IPAddress ip( 192, 168, 0, 2 );    
IPAddress gateway( 192, 168, 0, 1 );
IPAddress subnet( 255, 255, 255, 0 );

// Change this to the ip of the ftp server
IPAddress server( 192, 168, 0, 3 );

EthernetClient client;
EthernetClient dclient;

char outBuf[128];
char outCount;

void setup()
{
 Serial.begin(9600);
 pinMode(4,OUTPUT);
 digitalWrite(4,HIGH);
 Ethernet.begin(mac, ip, gateway, gateway, subnet);
 delay(2000);
 Serial.println("Ready. Press f");
}

void loop()
{
 byte inChar;

 inChar = Serial.read();

 if(inChar == 'f')
 {
   if(sendFTP()) Serial.println("FTP OK");
   else Serial.println("FTP failed");
 }
}

byte sendFTP()
{
 if (client.connect(server,21)) {
   Serial.println("Command connected");
 }
 else {
   Serial.println("Connamd connection failed");
   return 0;
 }

 if(!eRcv()) return 0;

// Change to your user and password  
 client.write("USER myusername\r\n");

 if(!eRcv()) return 0;

 client.write("PASS mypassword\r\n");

 if(!eRcv()) return 0;

 client.write("SYST\r\n");

 if(!eRcv()) return 0;

 client.write("PASV\r\n");

 if(!eRcv()) return 0;

 char rtnVal[32];
 
 sscanf(outBuf,"%*s %*s %*s %*s %s",rtnVal);

 unsigned int hiPort,loPort;

 sscanf(rtnVal,"%*c%*d%*c%*d%*c%*d%*c%*d%*c%d%*c%d",&hiPort,&loPort);
 
 Serial.print("Port ");
 hiPort = hiPort << 8;
 loPort = loPort & 255;
 hiPort = hiPort | loPort;
 Serial.println(hiPort);
 

 if (dclient.connect(server,hiPort)) {
   Serial.println("Data connected");
 }
 else {
   Serial.println("Data connection failed");
   client.stop();
   return 0;
 }

 client.write("QUIT\r\n");

 if(!eRcv()) return 0;

 dclient.stop();
 client.stop();
 Serial.println("disconnected");
 return 1;
}

byte eRcv()
{
 byte respCode;
 byte thisByte;

 while(!client.available()) delay(1);

 respCode = client.peek();

 outCount = 0;
 
 while(client.available())
 {  
   thisByte = client.read();    
   Serial.write(thisByte);

   if(outCount < 127)
   {
     outBuf[outCount] = thisByte;
     outCount++;      
     outBuf[outCount] = 0;
   }
 }

 if(respCode >= '4')
 {
   efail();
   return 0;  
 }

 return 1;
}


void efail()
{
 byte thisByte = 0;

 client.write("QUIT\r\n");

 while(!client.available()) delay(1);

 while(client.available())
 {  
   thisByte = client.read();    
   Serial.write(thisByte);
 }

 client.stop();
 dclient.stop();
 Serial.println("disconnected");
}

jrburden

Try using syslog. This could could give you (near) realtime access to the data too.

chrishjones


Quote
I'm logging data from a 250G accelerometer (boxing punches) and will be sampling data as fast as possible so as not to miss an event.

Quote
Not always. If I want to retrieve data from my rooftop unit, and the area is being pummeled by lightning, going to the roof of a 22 story building would not be a healthy thing to do.

But, you think it is OK to have the Arduino/accelerometer/SD card/boxer out there?


I'm a little confused. You're quoting two different people. The boxing robot is not on a rooftop.

chrishjones


Try using syslog. This could could give you (near) realtime access to the data too.


I'm assuming you're talking about this? There doesn't seem to be much activity there.
http://code.google.com/p/ardusyslog/

My worry is latency where the program hangs as it's sending the data and waiting for confirmation. Is there a way to write it all to SD, close the file, open the file again and initiate a syslog transfer? 

SurferTim

Just an update. I have my FTP code reading from and writing to files on my FTP server. I have no SD interface set up yet tho. The test code compiles to 14524 bytes.

Interested?


SurferTim

Update: I have the full package working now. Upload files from the FTP server to the SD card, or download files from the SD card to the FTP server. Right now, I have a define that allows me to compile either upload or download to save program memory.

The code with debugging compiles to 25856 bytes. A little steep for an Uno, but for my Mega2560, not a problem.

If you are interested, let me know. I have a little error checking to do to insure all works ok. If anyone shows interest, I will post the code then.

If you are not interested, at least I'll have it when I need it.  :)

michael635

I'd love to see your code.  I am in the final phases of programming an ftp server on my Mega.  I took a different path, figuring it was easier to use an ftp client as needed on the main machine instead of having a server running to recieve the data.

Had some problems with getting the passive data channel to connect since the ftp client (Filezilla in my case) doesn't send any data upstream on the data channel, it just listens (as called out in the ftp standard).  The server.available call expects the Rx buffer to have something before it connects.  I had to do a terrible thing and modify the ethernet library to be able to get the data channel to connect to the socket without any data in the Rx buffer.  I'd like to see how you got that to work....I was going to delve into the ethernet chip library and do it at a low level to get away from modding the library.

I also tweaked the SD library to get the file modified date output rather than use the SDFAT library directly.  I use the EPLF format to transfer the directory information, and although the date is not required, I always like know which file is newer....

I have it running fairly well but there seems to be some timing issues where if I comment out some of my debugging serial.print lines it breaks the program....also need to make the code somewhat more presentable, lots of commented dead ends leftover from the development.....

I'll upload my code when I get back to my shop at the end of the week if anyone is interested.  If I recall it runs about 38k but has internet time, software RTC, a webserver (based on the http example), and the ftp server all packaged together.

SurferTim

#14
Feb 28, 2012, 04:39 am Last Edit: Mar 22, 2012, 02:38 am by SurferTim Reason: 1
Here is the code I have so far. It is not documented as well as I would like it, but I can do that later.

The "#define FTPWRITE" determines whether it reads or writes to the FTP server. Commented out, it reads from the FTP server and writes to the SD.

Change your network settings, and the user/password in the doFTP() routine.
Place a file named test.txt in your FTP server.

Upload and open serial monitor. Wait for ready.
Press 'r' and enter to view SD file contents.
Press 'f' and enter to transfer.

Code: [Select]
#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

// reads from server and writes to SD file
// uncomment next line to write SD file to server.
// #define FTPWRITE

byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };  
IPAddress ip( 192, 168, 0, 2 );    
IPAddress gateway( 192, 168, 0, 1 );
IPAddress subnet( 255, 255, 255, 0 );

// FTP server ip
IPAddress server( 192, 168, 0, 3 );

EthernetClient client;
EthernetClient dclient;

char outBuf[128];
char outCount;

void setup()
{
 Serial.begin(9600);

 if(SD.begin(4) == 0)
 {
   Serial.println("SD init fail");          
 }

 Ethernet.begin(mac, ip, gateway, gateway, subnet);
 digitalWrite(10,HIGH);
 delay(2000);
 Serial.println("Ready. Press f or r");
}

void loop()
{
 byte inChar;

 inChar = Serial.read();

 if(inChar == 'f')
 {
   if(doFTP()) Serial.println("FTP OK");
   else Serial.println("FTP FAIL");
 }

 if(inChar == 'r')
 {
   readSD();    
 }

}

File fh;

byte doFTP()
{
#ifdef FTPWRITE
 fh = SD.open("test.txt",FILE_READ);
#else
 SD.remove("test.txt");
 fh = SD.open("test.txt",FILE_WRITE);
#endif

 if(!fh)
 {
   Serial.println("SD open fail");
   return 0;    
 }

 Serial.println("SD opened");
 
 if (client.connect(server,21)) {
   Serial.println("Command connected");
 }
 else {
   Serial.println("Command connection failed");
   fh.close();
   return 0;
 }

 if(!eRcv()) return 0;

// change to your user
 client.write("USER myusername\r\n");

 if(!eRcv()) return 0;

// change to your password  
 client.write("PASS mypassword\r\n");

 if(!eRcv()) return 0;

 client.write("SYST\r\n");

 if(!eRcv()) return 0;

 client.write("PASV\r\n");

 if(!eRcv()) return 0;

// new code

 char *tStr = strtok(outBuf,"(,");
 int array_pasv[6];
 for ( int i = 0; i < 6; i++) {
   tStr = strtok(NULL,"(,");
   array_pasv[i] = atoi(tStr);
   if(tStr == NULL)
   {
     Serial.println("Bad PASV Answer");    
     client.stop();
     fh.close();
     return 0;  
   }
 }

 unsigned int hiPort,loPort;

 hiPort = array_pasv[4] << 8;
 loPort = array_pasv[5] & 255;

// end of new code

 Serial.print("Data port: ");
 hiPort = hiPort | loPort;
 Serial.println(hiPort);
 
 if (dclient.connect(server,hiPort)) {
   Serial.println("Data connected");
 }
 else {
   Serial.println("Data connection failed");
   client.stop();
   fh.close();
   return 0;
 }

#ifdef FTPWRITE
 client.write("STOR /test.txt\r\n");
#else
 client.write("RETR /test.txt\r\n");
#endif

 if(!eRcv())
 {
   dclient.stop();
   return 0;
 }

#ifdef FTPWRITE
 Serial.println("Writing");

// This needs improvement

 while(fh.available())
 {
   char c = fh.read();
   dclient.write(c);
 }

#else
 while(dclient.connected())
 {
   while(dclient.available())
   {
     char c = dclient.read();
     fh.write(c);      
     Serial.write(c);
   }
 }
#endif

 dclient.stop();
 Serial.println("Data disconnected");
 
 if(!eRcv()) return 0;

 client.write("QUIT\r\n");

 if(!eRcv()) return 0;

 client.stop();
 Serial.println("Command disconnected");

 fh.close();
 Serial.println("SD closed");
 return 1;
}

byte eRcv()
{
 byte respCode;
 byte thisByte;

 while(!client.available()) delay(1);

 respCode = client.peek();

 outCount = 0;
 
 while(client.available())
 {  
   thisByte = client.read();    
   Serial.write(thisByte);

   if(outCount < 127)
   {
     outBuf[outCount] = thisByte;
     outCount++;      
     outBuf[outCount] = 0;
   }
 }

 if(respCode >= '4')
 {
   efail();
   return 0;  
 }

 return 1;
}


void efail()
{
 byte thisByte = 0;

 client.write("QUIT\r\n");

 while(!client.available()) delay(1);

 while(client.available())
 {  
   thisByte = client.read();    
   Serial.write(thisByte);
 }

 client.stop();
 Serial.println("Command disconnected");
 fh.close();
 Serial.println("SD closed");
}

void readSD()
{
 fh = SD.open("test.txt",FILE_READ);

 if(!fh)
 {
   Serial.println("SD open fail");
   return;    
 }

 while(fh.available())
 {
   Serial.write(fh.read());
 }
 
 fh.close();
}


If you have questions, ask. My apology in advance for any typos.

edit: This is FTP client code. I could not find a way to use passive mode with the Arduino as the server.


edit2: Code change on passive mode port retrieval, thanks to cedric2.  :)
That saves about 2K of memory.

Go Up