Sending emails from arduino

Does anyone have considerable experience with emails and particularly how anti-spam software decides that an email is spam.

I have my arduino emailing me alarms, and quite frequently while I have been testing. I suspect my email account has again decided that the emails are spam and have stopped letting them through.

Originally I was not filling in a ‘from’ field and the same thing happened. I subsequently setup a gmail account and used that email address in the ‘from’ field - I then started receiving the emails from my arduino again.

So obviously I need more than just from, to, subject and body etc in order for my emails to not be flagged as spam.

I looked at the message header of a simple text based email in my inbox:

Return-Path: <gregsind@hiruko.instanthosting.com.au>
Received: from extmail.bigpond.com ([10.10.26.4])
          by viclafep25p-svc.bpe.nexus.telstra.com.au with ESMTP
          id <20161212011226.ICBA16686.viclafep25p-svc.bpe.nexus.telstra.com.au@extmail.bigpond.com>
          for <gregplants@bigpond.com>; Mon, 12 Dec 2016 12:12:26 +1100
X-Junkmail-Premium-Raw: score=19/50,refid=2.7.2:2016.12.12.4521:17:19.782,ip=61.9.189.149,rules=__TO_MALFORMED_2,
 __TO_NO_NAME, __SUBJ_ALPHA_END, __HAS_X_PHP_SCRIPT, WEBMAIL_X_PHP_SCRIPT,
 __HAS_FROM, __FRAUD_WEBMAIL_FROM, __FROM_GMAIL, __CSHC_NS_B_FN_FA,
 __HAS_MSGID, __SANE_MSGID, __FRAUD_ANTIABUSE, __X_MAILSCANNER,
 __FRAUD_CONTACT_NUM, __FRAUD_URGENCY, __C230066_P5, __NO_HTML_TAG_RAW,
 BODYTEXTP_SIZE_400_LESS, BODYTEXTP_SIZE_3000_LESS, BODY_SIZE_300_399,
 __MIME_TEXT_P1, __MIME_TEXT_ONLY, HTML_00_01, HTML_00_10,
 BODY_SIZE_5000_LESS, __FRAUD_WEBMAIL, WEBMAIL_SOURCE, __FRAUD_COMMON,
 NO_URI_FOUND, NO_CTA_URI_FOUND, FRAUD_LITTLE_BODY, __PHISH_SPEAR_STRUCTURE_1,
 BODY_SIZE_1000_LESS, BODY_SIZE_2000_LESS, FORGED_FROM_GMAIL, __MIME_TEXT_P,
 BODY_SIZE_7000_LESS, NO_URI_HTTPS, HAS_X_PHP_SCRIPT
Received: from nschwmtas05p.mx.bigpond.com (61.9.189.149) by extmail.bigpond.com (9.0.019.11-1)
        id 58246FB8065C80C7 for gregplants@bigpond.com; Mon, 12 Dec 2016 12:12:26 +1100
Received: from nschwcmgw04p ([61.9.190.164]) by nschwmtas05p.mx.bigpond.com
          with ESMTP
          id <20161212011144.WKLJ1005.nschwmtas05p.mx.bigpond.com@nschwcmgw04p>
          for <gregplants@bigpond.com>; Mon, 12 Dec 2016 01:11:44 +0000
Received: from hiruko.instanthosting.com.au ([116.0.23.229])
	by nschwcmgw04p with BigPond Inbound
	id JpBk1u00D4wZw5801pBklH; Mon, 12 Dec 2016 01:11:44 +0000
X-Authority-Analysis: v=2.1 cv=UcCsM/mN c=1 sm=1 tr=0
 a=+DGuJkF3DqeAtMcz2AcYfw==:117 a=+DGuJkF3DqeAtMcz2AcYfw==:17
 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10
 a=9+rZDBEiDlHhcck0kWbJtElFXBc=:19 a=v808_MQBxgwA:10 a=4oOGamPeoiMA:10
 a=n5n_aSjo0skA:10 a=OmkCvELxlHwA:10 a=N5mI7RuQtHDImP-ZjN8A:9
 a=-1M1NXTPTxw8Ig3_:21 a=kvenG6QAvXpI2uUc:21 a=L0NOnLaf9m8A:10
 a=fqJV5JGUUjgA:10
Received: from gregsind by hiruko.instanthosting.com.au with local (Exim 4.87)
	(envelope-from <gregsind@hiruko.instanthosting.com.au>)
	id 1cGF9Z-003xu8-Fa
	for gregplants@bigpond.com; Mon, 12 Dec 2016 12:11:41 +1100
To: gregplants@bigpond.com
Subject: Query from Rees Campbell
X-PHP-Script: www.gregsindigenouslandscapes.com.au/Contact.php for 147.69.19.28
From: Rees Campbell<reescampbell1@gmail.com>
Message-Id: <E1cGF9Z-003xu8-Fa@hiruko.instanthosting.com.au>
Date: Mon, 12 Dec 2016 12:11:41 +1100
X-cPanel-MailScanner-Information: Please contact the ISP for more information
X-cPanel-MailScanner-ID: 1cGF9Z-003xu8-Fa
X-cPanel-MailScanner: Found to be clean
X-cPanel-MailScanner-SpamCheck: not spam, SpamAssassin (not cached,
	score=1.203, required 3, DKIM_ADSP_CUSTOM_MED 0.00,
	FREEMAIL_FROM 0.00, FROM_MISSP_FREEMAIL 0.00,
	NML_ADSP_CUSTOM_MED 1.20, NO_RELAYS -0.00,
	TO_NO_BRKTS_FROM_MSSP 0.00)
X-cPanel-MailScanner-SpamScore: s
X-cPanel-MailScanner-From: gregsind@hiruko.instanthosting.com.au
X-Spam-Status: No
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - hiruko.instanthosting.com.au
X-AntiAbuse: Original Domain - bigpond.com
X-AntiAbuse: Originator/Caller UID/GID - [2201 501] / [47 12]
X-AntiAbuse: Sender Address Domain - hiruko.instanthosting.com.au
X-Get-Message-Sender-Via: hiruko.instanthosting.com.au: authenticated_id: gregsind/only user confirmed/virtual account not confirmed
X-Authenticated-Sender: hiruko.instanthosting.com.au: gregsind
X-Source: 
X-Source-Args: 
X-Source-Dir:

So what would be the bare minimum that I would need out of the above to avoid my arduino emails being flagged as SPAM.

So what would be the bare minimum that I would need out of the above to avoid my arduino emails being flagged as SPAM.

A chat with your e-mail provider would be in order. The Arduino isn't sending spam. Your e-mail provider is making the decision that the e-mail, completely independent of the that fact that an Arduino was involved, is spam. Talk to them about why.

PaulS:
A chat with your e-mail provider would be in order. The Arduino isn't sending spam. Your e-mail provider is making the decision that the e-mail, completely independent of the that fact that an Arduino was involved, is spam. Talk to them about why.

I understand that.

The issue is that I am making a device prototype that I intend to sell and I can't have this conversation with every ISP of every client.

I assume that there would be some key red flags (re email fields that I am not filling in) that SPAM filters particularly look for

I assume that there would be some key red flags (re email fields that I am not filling in) that SPAM filters particularly look for

Like many emails from the same source in a very short period of time?

YOU need to talk to your email provider, to determine why they are flagging the emails as spam. If they have a set of rules, you can be reasonably certain that other email providers have very similar rules (except ComCast, which doesn’t filter off a f**king thing).

PaulS:
Like many emails from the same source in a very short period of time?

YOU need to talk to your email provider, to determine why they are flagging the emails as spam. If they have a set of rules, you can be reasonably certain that other email providers have very similar rules (except ComCast, which doesn't filter off a f**king thing).

OK - I am with you.

However we are talking about Australian Telstra here.

I will probably talking to an Indian call center dip shit who knows jack $hit about SPAM filters, or anything outside their 'script'.

And also has no f'ing idea who in the organisation that would be able to answer my technical question.

I reckon I would be better off finding this information from another easier to deal with source.

Any alternative suggestions?

The spam filter is up to the ISP, but there are some steps you can take to prevent some rejections or spam folder results.
Insure the domain you use in the FROM can be resolved.
Insure all the fields have an entry.
Insure the body of the email (message) doesn't read like spam (advertisement).
Don't send too many emails with the same FROM and SUBJECT.

SurferTim:
Insure the domain you use in the FROM can be resolved.

I created a gmail address to stick in the FROM field. But I was wondering if I should just stick the TO email address in the FROM email address. The emails from the arduino device are simple notifications anyway and there is no point in replying to them.

SurferTim:
Insure all the fields have an entry.

Know any good web pages that details all the fields?

SurferTim:
Insure the body of the email (message) doesn’t read like spam (advertisement).

No problem there.

SurferTim:
Don’t send too many emails with the same FROM and SUBJECT.

But this is likely to be my problem in this instance. Both are identical with all emails.
I suppose I could vary the subject a little, but it needs to be easily recognizable as a notification from the irrigation controller.
I guess I could see if there is provision to can change the SPAM settings in my email account to wave through those particular emails.

But this is likely to be my problem in this instance. Both are identical with all emails.

If the subject is "Irrigation system @ 08:30 on 12/15/2016", you would not be sending identical messages.

PaulS:
If the subject is "Irrigation system @ 08:30 on 12/15/2016", you would not be sending identical messages.

Ta - didn't think of doing something like that.

Could only be enough that you might be sending from an IP range that is used for "dial-up" (that is, private customers).
The usual way around this is to relay your mail over a server from a trusted (that is, assigned to a datacenter) IP range.
I run a mail server at home like that with a 4€/month vserver with some hoster. Last weekend I accidentally dropped the relay from my config and immediate, friends reported they hat to fish me from the spam folder because my IP was blacklisted.

The header tags seem strange, I guess for example you did not set the mail to urgent, so why "__FRAUD_URGENCY"?
FORGED_FROM_GMAIL could be a problem though. The mailserver might detect that it does not actually come from gmail.

Best bet is to actually send it over a real SMTP server.
Here is a discussion about using GMail, but that will send your credentials unencrypted: http://forum.arduino.cc/index.php?topic=126829.0

Safer would be to use some computer in your network, which in turn relays it to a trusted server.

ElCaron:
Could only be enough that you might be sending from an IP range that is used for “dial-up” (that is, private customers).
The usual way around this is to relay your mail over a server from a trusted (that is, assigned to a datacenter) IP range.
I run a mail server at home like that with a 4€/month vserver with some hoster. Last weekend I accidentally dropped the relay from my config and immediate, friends reported they hat to fish me from the spam folder because my IP was blacklisted.

The header tags seem strange, I guess for example you did not set the mail to urgent, so why “__FRAUD_URGENCY”?
FORGED_FROM_GMAIL could be a problem though. The mailserver might detect that it does not actually come from gmail.

Best bet is to actually send it over a real SMTP server.
Here is a discussion about using GMail, but that will send your credentials unencrypted: http://forum.arduino.cc/index.php?topic=126829.0

Safer would be to use some computer in your network, which in turn relays it to a trusted server.

Unfortunately I can’t send emails using any SMTP server other than mail.bigpond.com - my ISP seems to be one that blocks sent emails via any other server.

I did try ringing Australian Telstra support, as previously suggested, and it was a total waste of time as I suspected. I spent an hour talking to a succession of bloody Filipinos following scripts but with no ability to answer such a technical question.

I did however find this info on how to construct a message ID after some more googling:

Message-ID: <4129F3CA.2020509@dc.edu>
								
Append "<".

Get the current (wall-clock) time in the highest resolution to which you have access (most systems can give it to you in milliseconds, but seconds will do);

Generate 64 bits of randomness from a good, well-seeded random number generator;

Convert these two numbers to base 36 (0-9 and A-Z) and append the first number, a ".", the second number, and an "@". This makes the left hand side of the message ID be only about 21 characters long.

Append the FQDN of the local host, or the host name in the user's return address.

Append ">".

So my total code for constructing my emails is as follows. So I guess I have no choice but to just suck it an see if this is enough to prevent my emails being blocked.

		write(F("AUTH LOGIN\r\n"));
		if (waitAck())
		{
			memset(m_strTemp, 0, m_nMaxShort);
			base64_decode(m_strTemp, m_strUsername, strlen(m_strUsername));
			strcat_P(m_strTemp, (PGM_P)F("\r\n"));
			
			WiFiEspClient::write(m_strTemp);
			if (waitAck())
			{
				memset(m_strTemp, 0, m_nMaxShort);
				base64_decode(m_strTemp, m_strPassword, strlen(m_strPassword));
				strcat_P(m_strTemp, (PGM_P)F("\r\n"));
				
				WiFiEspClient::write(m_strTemp);
				if (waitAck())
				{
					strcpy_P(m_strTemp, (PGM_P)F("MAIL From: <"));
					if (m_strFromEmail)
						strcat(m_strTemp, m_strFromEmail);
					strcat_P(m_strTemp, (PGM_P)F(">\r\n"));
					WiFiEspClient::write(m_strTemp);
					if (waitAck())
					{
						strcpy_P(m_strTemp, (PGM_P)F("RCPT To: <"));
						strcat(m_strTemp, m_strToEmail);
						strcat_P(m_strTemp, (PGM_P)F(">\r\n"));
						WiFiEspClient::write(m_strTemp);
						if (waitAck())
						{
							write(F("DATA\r\n"));
							if (waitAck())
							{
								strcpy_P(m_strTemp, (PGM_P)F("To: <"));
								strcat(m_strTemp, m_strToEmail);
								strcat_P(m_strTemp, (PGM_P)F(">\r\n"));
								WiFiEspClient::write(m_strTemp);

								strcpy_P(m_strTemp, (PGM_P)F("From: Irrigation controller <"));
								if (m_strFromEmail)
									strcat(m_strTemp, m_strFromEmail);
								strcat_P(m_strTemp, (PGM_P)F(">\r\n"));
								WiFiEspClient::write(m_strTemp);

								strcpy_P(m_strTemp, (PGM_P)F("Sent: "));
								if (m_strDateTime)
									strcat(m_strTemp, m_strDateTime);
								strcat_P(m_strTemp, (PGM_P)F("\r\n"));
								WiFiEspClient::write(m_strTemp);

								char strData[100];

								memset(strData, 0, 100);								
								strcpy_P(strData, (PGM_P)F("X-Accept-Language: en-us, en\r\n"));
								WiFiEspClient::write(strData);
								
								memset(strData, 0, 100);								
								strcpy_P(strData, (PGM_P)F("MIME-Version: 1.0\r\n"));
								WiFiEspClient::write(strData);
								
								memset(strData, 0, 100);								
								strcpy_P(strData, (PGM_P)F("Content-Type: text/plain; charset=us-ascii; format=flowed\r\n"));
								WiFiEspClient::write(strData);
								
								memset(strData, 0, 100);								
								strcpy_P(strData, (PGM_P)F("Content-Transfer-Encoding: 7bit\r\n"));
								WiFiEspClient::write(strData);

								strcpy_P(strData, (PGM_P)F("Message-ID: <"));								
								ultoa(m_nTime, m_strTemp, 36);
								strcat(strData, m_strTemp);
								strcat_P(strData, (PGM_P)F("."));
/*
								Message-ID: <4129F3CA.2020509@dc.edu>
								
								Append "<".

								Get the current (wall-clock) time in the highest resolution to which you have access (most systems can give it to you in milliseconds, but seconds will do);

								Generate 64 bits of randomness from a good, well-seeded random number generator;

								Convert these two numbers to base 36 (0-9 and A-Z) and append the first number, a ".", the second number, and an "@". This makes the left hand side of the message ID be only about 21 characters long.

								Append the FQDN of the local host, or the host name in the user's return address.

								Append ">". 
*/
								randomSeed(analogRead(0));
								uint32_t nRandom = random();
								ultoa(nRandom, m_strTemp, 36);
								strcat(strData, m_strTemp);
								
								strcat_P(strData, (PGM_P)F("@gmail.com>\r\n"));
								write(strData);
								write(F("Subject: "));
								write(m_strSubject);								
								write(F("\r\n\r\n"));
								if (m_strContent)
									WiFiEspClient::write(m_strContent);
								write(F("\r\n.\r\n"));
								if (waitAck())
								{
									write(F("QUIT\r\n"));
									if (waitAck())
									{
										bResult = true;
										strcpy_P(m_strSendStatus, (PGM_P)F("SMTPClient::sendEmail(): email sent!"));
										stop();

Do you HAVE to use emails? There are a lot of options to communicate from host to host. With email I fear you are going to have to fight about stuff like this with everyone, forever.

-jim lee

jimLee:
Do you HAVE to use emails? There are a lot of options to communicate from host to host. With email I fear you are going to have to fight about stuff like this with everyone, forever.

-jim lee

The device is an irrigation controller that a small commercial nursery might use. So if a solenoid fails, for example, and a particular station is no longer being irrigated over the xmas period then the owner needs to get a warning of the failure when it happens and not the next time they load a web page.

Could do a similar thing with a mobile phone shield instead of a wifi shield and send an SMS rather than an email. But that is a whole other project.

Did you follow my advice? It appears you are not sending some parts of the email that the receiving server is expecting. insure you have the names in the To and From after the DATA send. This is from my playground email client code.

// change to your public ip
  client.println(F("helo 1.2.3.4"));

  if(!eRcv()) return 0;
  Serial.println(F("Sending From"));

// change to your email address (sender)
  client.println(F("MAIL From: <me@mydomain.com>"));

  if(!eRcv()) return 0;

// change to recipient address
  Serial.println(F("Sending To"));
  client.println(F("RCPT To: <you@yourdomain.com>"));

  if(!eRcv()) return 0;

  Serial.println(F("Sending DATA"));
  client.println(F("DATA"));

  if(!eRcv()) return 0;

  Serial.println(F("Sending email"));

// change to recipient address
  client.println(F("To: Your Name <you@yourdomain.com>"));

// change to your address
  client.println(F("From: My Name <me@mydomain.com>"));

  client.println(F("Subject: Arduino email test\r\n"));

SurferTim:
Did you follow my advice? It appears you are not sending some parts of the email that the receiving server is expecting. insure you have the names in the To and From after the DATA send. This is from my playground email client code.

// change to your public ip

client.println(F(“helo 1.2.3.4”));

if(!eRcv()) return 0;
  Serial.println(F(“Sending From”));

// change to your email address (sender)
  client.println(F(“MAIL From: me@mydomain.com”));

if(!eRcv()) return 0;

// change to recipient address
  Serial.println(F(“Sending To”));
  client.println(F(“RCPT To: you@yourdomain.com”));

if(!eRcv()) return 0;

Serial.println(F(“Sending DATA”));
  client.println(F(“DATA”));

if(!eRcv()) return 0;

Serial.println(F(“Sending email”));

// change to recipient address
  client.println(F(“To: Your Name you@yourdomain.com”));

// change to your address
  client.println(F(“From: My Name me@mydomain.com”));

client.println(F(“Subject: Arduino email test\r\n”));

The above looks like the code I based my SMTP class on - plus the extra fields that I added and seem to be in all email headers (including the message ID field).

I think your code is not sending a To name.This is the hint:
__TO_NO_NAME

// change to recipient address
  client.println(F("To: Your Name <you@yourdomain.com>"));