Email attachments from arduino as an SMTP client

Hello All,

I tried searching the best keywords before posting here. I apologize if I am asking something that is obvious. I am aware of the fact that it is possible to send emails to the SMTP server using the arduino ethernet shield. I want to send picture attachments along with the mail. I am aware of the fact that it is possible to send attachments with some encoding inside the mail server. I would like to know whether there is any documentation of this being done using the arduino? I would be happy if some links are shared.

Thanks,
Sai

I am aware of the fact that it is possible to send attachments with some encoding inside the mail server. I would like to know whether there is any documentation of this being done using the arduino?

It is exactly the same as sending the attachment from the PC. Except sssslllloooowwwweeeerrrr. Much sssslllloooowwwweeeerrrr.

Is it going to take for ever to transfer a 50KB picture?

Is it going to take for ever to transfer a 50KB picture?

You can do the math. A 50,000 byte file is 400,000 bits. A 115200 baud rate means that 115200 bits per second get sent. Of those, 8 out of 10 are payload. 400,000 bits/8 = 50,000 packets. 11,500 packets per second will require about 5 seconds to send.

Your baud rate may be different, so the file might take longer to transfer.

The more important question is where in the Arduino's 2KB of SRAM are you planning on storing a 50KB picture?

oh I had plans of storing it in the SD card shield. Am I building forts in the air here? I thought 5 seconds is not bad...

Where is/are the picture(s) coming from? Getting the data from the SD card will take time. The data will not be being sent while it is being read. So, you need to add the time needed to read the data from the card to the time needed to send the data to get the total time required to read and send a picture.

Thanks for enlightening me about the read and write problem. I did not consider it.

I was planning to take pictures using the camera:

From what I read, the minimum file size of a captured frame is 3KB to 9KB. Hence If I store this image in SD card, I thought I could access it using an ethernet shield.

Sai

I don't know of any Arduino code to do it, but the first thing you'll need to find is an encoder to take whatever format the picture is in, ensure it turns up as a JPEG, PNG, or other file format, then encode that file for transmission by email. A common encoding mechanism is Base 64.

The Internet RFC that covers sending SMTP messages with MIME attachments is RFC 3030 - SMTP Service Extensions for Transmission of Large and Binary MIME Messages.

You could trawl through that one and try to interpret it yourself. Alternately you could send an email with an image attachment using your normal mail client and packet-sniff the exchange with the non-SSL mail server. Being able to replay that message would be an ideal starting point. You can "cheat" and simply take the encoded image data, store that on the SD Card, and write your code to assemble the SMTP message using the pre-encoded image. The next step would be writing the code to take the original image and end up with the encoded version.

Encoding files is not as scary as it sounds. Base64 encoding as defined in RFC 4648 is actually quite simple.

I would suggest the following steps:

• Write a function that can send a simple email containing "hello world"
• Write a function that will take a given Base 64 encoded file and wrap it with the appropriate SMTP & MIME leaders/trailers (i.e.: 'your' side of the SMTP conversation) as a script in a third file
• Now simply play the script to the SMTP server, replacing "hello world" with the content of your script
• Finally, write a function to take a given file on the SD card and convert it to Base 64 in a second file, reading a chunk, converting it and writing that chunk to the SD card (you'll want to do this in the smallest sized chunks you possibly can)

This will not handle errors gracefully, but handling SMTP errors gracefully is usually a matter of disconnecting and forgetting the transaction ]

I guesstimate that a brain-dead implementation of the above will take somewhere in the order of a minute to send a 10kB image, most of which will be transferring data back and forth between the Arduino and the SD Card. A smarter implementation which can generate the SMTP conversation and do Base64 encoding on the fly should manage to send the entire message in a few seconds (limited by how fast you can read the data off the SD Card).

But baby steps. Remember Knuth:

• First, make it work
• Then, make it work right
• Only then, make it work fast

In my brief flirtation with Bing and Google just now, I couldn't find any pointers to a Base64 library for Arduino. If anyone can find one, please mention it in this thread, 'cos I want to do Base64 encoding too

If no such beast exists, that will probably be my homework for the weekend.

Thanks.. This suggestion was valuable

I haven't tried sending an attachment, but sending plain text is pretty straightforward.

My problem was that I was trying to use a GMail account, and it needs TLS, something Arduino can't. But hopefully I found that Yahoo Mail doesn't need cipher.

Function would be something similar to:

// Hello World! Example: mail from foo@yahoo.com to bar@fakemail.com
client.connect("smtp.mail.yahoo.com",25);
client.print("HELO\n");
client.print("AUTH PLAIN\n");
client.print("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\n");  // x: base64 encoded login: \0foo@yahoo.com\0password
client.print("MAIL FROM:<foo@yahoo.com>\n");
client.print("RCPT TO:<bar@fakemail.com>\n");
client.print("DATA\n");
client.print("From: \"Foo\" <foo@yahoo.com>\r\n");
client.print("To: bar@fakemail.com>\r\n");
client.print("Subject: Buy Cialis\r\n");  // xD
client.print("\r\n");
client.print("Hello World!\r\n");  // Lines of text
// ......................................
client.print(".\r\n");
client.print("quit\n");
client.stop()

Sending photo attachments is not a good task for the Arduino. The ATmega328 is a microcontroller not a microprocessor. Encoding, attaching, and sending a 50kb file through the poor little ATmega328 probably isn't worth the trouble. There are a lot of limitations you'll have to solve with very creative solutions.

Now, if you are trying to do this under the category of "Let's see if it can be done" that is a different story. If you are trying to accomplish any other goal (like build a system of automated webcams) the Arduino isn't a good choice.

My main objective is to transfer pictures from the JPEG cam that I posted above and I wanted to do it via ethernet shield/ arduino black widow shield. I am looking into best methods to do this.

Now that I posted here, I am able to gauge the possibilities before I hit a dead end. From what I read, the camera that I posted in my previous thread generates images from 3KB to 9KB. I do not want my arduino to be tethered to a computer via USB.

Will something like make sense for a picture transfer?

He has installed Linux inside an Asus router. Will hosting a mail client inside the router be a more sensible solution?

some cam's already have email and ftp build-in like the Trendnet TV-IP110 (cheap as in like $49)

but your issue with that model and google is ssl - there might be other cams that support ssl.

You could also look into plug computers or routers that support open source firmware (I use a reflashed unslung nslu2) then you can use bash, c or php to do what ever you want.

I looked at trying to reflash the cams but way less info and way over my head to try and be the first to do it - when compared to plug/router/nas reflashing there's a ton of info for those devices.

Am I invisible?

[quote author=James C4S link=topic=67701.msg498070#msg498070 date=1311777685]
Sending photo attachments is not a good task for the Arduino. The ATmega328 is a microcontroller not a microprocessor. Encoding, attaching, and sending a 50kb file through the poor little ATmega328 probably isn't worth the trouble. There are a lot of limitations you'll have to solve with very creative solutions.[/quote]
Pardon? What is the trouble? What limitations?

This would be how to base64 encode & send to the SMTP server a file named picture from a SD:

static const char cb64[]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
void encodeblock(unsigned char in[3],unsigned char out[4],int len) {
  out[0]=cb64[in[0]>>2]; out[1]=cb64[((in[0]&0x03)<<4)|((in[1]&0xF0)>>4)];
  out[2]=(unsigned char) (len>1 ? cb64[((in[1]&0x0F)<<2)|((in[2]&0xC0)>>6)] : '=');
  out[3]=(unsigned char) (len>2 ? cb64[in[2]&0x3F] : '=');
}
void encode() {
  unsigned char in[3],out[4]; int i,len,blocksout=0;
  while (picture.available()!=0) {
    len=0;
    for (i=0;i<3;i++) { in[i]=(unsigned char) picture.read(); if (picture.available()!=0) len++; else in[i]=0; }
    if (len) { encodeblock(in,out,len); for(i=0;i<4;i++) client.write(out[i]); blocksout++; }
    if (blocksout>=19||picture.available()==0) { if (blocksout) client.print("\r\n");  blocksout=0; }
  }
}

Interesting problem. Obviously, the hardest part is to build the SMTP client state machine that implements something like section 4.2 in RFC 3030 (see link in reply by ManicDee). Assuming your SMTP server supports chunking, then all you have to do is the following:

1. Get the binary data of the image file as a stream of octets (bytes)
2. Invoke the state machine and follow the protocol (plus whatever cleartext auth needed) up until the BDAT part
3. Issue a BDAT for the size of the binary data (or send them in chunks)
4. Blast the binary data across the wire.
5. Issue a BDAT 0 LAST
6. Wait for confirmation of all data chunks, issue QUIT and close the connection.

You should be able to find C-based SMTP client code in the open-source realm and adapt it for the Arduino. In any case, sounds like quite a bit of work. Also, I would imagine it would be very, very hard to build an SSL stack on the Arduino.

Just my 2 cents' worth from a backseat commentator.

There's no state required, SMTP is a very simple protocol and you can pre-program the entire client side of the conversation, only waiting to see "OK" from the server and dropping the connection if anything looks different to what you expect.

The only problem with trying to use the Arduino for SMTP will be those servers that insist on using SSMTP or SMTP/SSL. Base64 encoding is simple. SMTP conversation is trivial. Ethernet communication is a little harder. I expect SSL requires more data in the form of keys to be held in memory than the Arduino has memory. That might make an interesting experiment for some time in the future.

The obvious workaround is to have a plain-text SMTP server living on the network that the Arduino will connect to, which then forwards the email to wherever it's supposed to go. Of course there are probably "better" ways of getting data off the Arduino, but SMTP is certainly a convenient one (the message goes straight from the Arduino to your mail inbox) and IMHO is the simplest conversation to handle. Other options include FTP or HTTP, but both those protocols are far more complex than SMTP. FTP requires multiple connections and requires the Arduino to listen for incoming connections - no thanks! HTTP will require Base64 encoding, and involves more header generation and a more complex conversation.

I'll be fiddling with my Arduino tonight, let's see if I can't get a simple Base64 encoder working. Of course I'm currently limited to encoding about 400 bytes of data since I don't have an SD card interface yet (it's in the mail). So 400 bytes for input leads to about 530 bytes of output, which ends up leaving 70 bytes of RAM free (on my Uno I'll have more, but you might be using something different). With an SD card (or other external storage), the Base64 encoding operation only needs to use 7.25 bytes of RAM (3 bytes input, 4 bytes output, 2 bits for length of input), "some" memory for an input buffer, "some" memory for an output buffer, and lots of reading and writing over the SD Card interface to fill the input buffer and empty the output buffer.

And then there's the "clever" option of having one buffer for input and output, with the input starting 1/4 of the way through the buffer, overwriting the buffer with the output as it is generated (since we don't need the input data that we've already encoded).

There's also the question of how to get a camera, an Ethernet shield, and an SD Card shield connected to the Arduino at the same time. OH THE WIRING! OH THE HUMANITY!

Razorblade:
Am I invisible?

Seems like yes, I am.

Is this a joke? Are you really not reading me? Why the h**l are you still talking about how difficult it would be, etc... when I've written functional examples with Arduino of both sending e-mail and base64 encoding, and demonstrated that with 25 lines of code it's all done?

Razorblade:
Is this a joke? Are you really not reading me? Why the h**l are you still talking about how difficult it would be, etc... when I've written functional examples with Arduino of both sending e-mail and base64 encoding, and demonstrated that with 25 lines of code it's all done?

No you're not and yes indeed you have. Apologies for missing that post.

Sorry for the anger, but I was getting worried.

This would be the whole program. Reads picture.jpg from microSD card, sends an e-mail with picture.jpg attachment:

#include <SPI.h>
#include <Ethernet.h>
#include <SD.h>

byte MAC[]={0x,0x,0x,0x,0x,0x};  // Your MAC
File picture; Client client;

static const char cb64[]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
void encodeblock(unsigned char in[3],unsigned char out[4],int len) {
  out[0]=cb64[in[0]>>2]; out[1]=cb64[((in[0]&0x03)<<4)|((in[1]&0xF0)>>4)];
  out[2]=(unsigned char) (len>1 ? cb64[((in[1]&0x0F)<<2)|((in[2]&0xC0)>>6)] : '=');
  out[3]=(unsigned char) (len>2 ? cb64[in[2]&0x3F] : '=');
}
void encode() {
  unsigned char in[3],out[4]; int i,len,blocksout=0;
  while (picture.available()!=0) {
    len=0; for (i=0;i<3;i++) { in[i]=(unsigned char) picture.read(); if (picture.available()!=0) len++; else in[i]=0; }
    if (len) { encodeblock(in,out,len); for(i=0;i<4;i++) client.write(out[i]); blocksout++; }
    if (blocksout>=19||picture.available()==0) { if (blocksout) client.print("\r\n");  blocksout=0; }
  }
}

void setup() {
  Ethernet.begin(MAC); delay(2000);
  pinMode(10,OUTPUT); SD.begin(4);
}

void loop() {
  picture=SD.open("picture.jpg",FILE_READ);
  client.connect("smtp.mail.yahoo.com",25);
  client.print("HELO\nAUTH PLAIN\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\n");  // x: base64 encoded login: \0foo@yahoo.com\0password
  client.print("MAIL FROM:<foo@yahoo.com>\nRCPT TO:<bar@fakemail.com>\nDATA\nFrom: \"Foo\" <foo@yahoo.com>\r\nTo: bar@fakemail.com\r\nSubject: Picture attached\r\n");
  client.print("Content-Type: image/jpeg; name=\"picture.jpg\"\r\nContent-Disposition: attachment; filename=\"picture.jpg\"\r\nContent-Transfer-Encoding: base64\r\n\r\n");
  encode();
  client.print("\r\n.\r\nQUIT\n");
  client.stop();
  picture.close();
  while(1);
}

Takes less than 30 seconds to send a 50 KiB picture.

Thanks for sharing RazorBlade! It's quite interesting to see that the Ethernet shield does not have a fixed MAC address which is really weird and makes it susceptible to MAC collissions (though I doubt anybody is doing MAC-based routing much anymore). What will probably happen is that we end up with everyone using all 00-00-00-00-00-00 as their MAC. I wonder if it might be an issue if you have multiple devices with the same MAC address on the same subnet though.

Also, does the Ethernet shield have existing libraries for DNS lookup and all that?

Finally, what's the bottleneck in this system? Ethernet is really fast so why would sending a 50KB picture take 30 seconds?

Sorry for hi-jacking this thread with n00b questions.