Go Down

Topic: Home automation using Twitter, Ethernet Shield (W5100), and Apache2 - PHP (Read 3839 times) previous topic - next topic

Jesterbaze

Just wanted to get an opinion on how my code looks for this litte project of mine. I recently finished the first functional 'alpha' of my HTML reading code.

The code searches for a start and finish marker within the website (using '::').

Some of the code I have looks like it could be MUCH better, & MUCH smaller (memory size wise), however I'm a self-taught coder, and I still have a ton to learn.

Later on I'm looking to add more functionality to this, such as setting up different users, reading more than once command at a time. Reading time-date info (UTC or whatever) to parse which commands were sent first. Just haven't got that far yet. Any input is welcome, I'm here to learn  :)

The end goal is to have a Twitter command sent to my Twitter account, and the Arduino will read and process the command, then control a pin on the board to turn on/off. Run the pin to a transistor operating as a 12V (or whatever volt) switch, to control lights, turning my PC on/off, whatever.

Such as:
Twitter -> PHP  Script that reads twitter posts -> Arduino reading commands (ex: '::PC Off::') -> Arduino pin 'x' -> Transistor -> PC power supply signal wire.

Misc. Specs:
PC - Custom build Ubuntu 13.1 Machine
Board - Arduino Uno
Using Apache2 + PHP mods to host website on 'localhost'
Using PHP to create the website HTML (index.php) -- Need to fix my Apache2 server at the moment, the directories aren't working right. Apache2 isn't easy.

Without further ado, here is the HTML code:

Code: [Select]
<html>
<head>
 <title>PHP is working!</title>
</head>
<body>
<p>This is to test if my Arduino twitter code works!</p><p>::This is the data I want read KTHX:: --NOT THIS DATA</p>
</body>
</html>


And here is my (rather long winded) Arduino 1.0.5 code:

Code: [Select]
#include <TextFinder.h>
#include <Ethernet.h>
#include <SPI.h>

int led = 5;

//Server location you wish to connect to
//USE: xxx,xxx,xxx,xxx OR    (IP Values are separated with a comma, not a period)
//HTTP://www.thisisanexample.com
byte server[] = { 192,168,1,4 }; //ip Address of the server you will connect to

//Board info
//Must define an IP address for the board, and a MAC address, DNS is optional.
IPAddress ip(192,168,1,100);
IPAddress myDns(192,168,1,1);
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

//Where on the site to connect to
String location = "/TwitterPHP/grabtwitter.php HTTP/1.1";  //Where on the site to connect to

//Initialize the Ethernet Client
EthernetClient client;

//String to capture the website
String webData = ("");

//Strings used in readPage()
String detData, tempWebData;  

void setup()
{
 Ethernet.begin(mac, ip);
 Serial.begin(9600);
 pinMode(led, OUTPUT);
}

void loop()
{
 webData = ("");  //Clear out previous webdata.
 String pageValue = connectAndRead(); //connect to the server and read the output

  //Looking to read "blink" from Twitter, via Apache2.
  //You must text "::blink::" to twitter for the program to read "blink".
 if (detData == "blink")
 {
   int numberOfBlinks = 8;
   for (int i=0; i<numberOfBlinks; i++)
   {
   int delayTime = 75;
   digitalWrite(led, HIGH);
   delay(delayTime);
   digitalWrite(led, LOW);
   delay(delayTime);
   }
 }
 //Debugging information
 /*
 Serial.println();
 Serial.println("****************************");
 Serial.println(":::STRING TESTING SECTION:::");
 Serial.println(webData);
 Serial.println(":::END STRING TESTING:::");
 Serial.println("****************************");
 Serial.println();
 Serial.println();
 */
 delay(15000); //wait '15 seconds' before connecting again
}

String connectAndRead()
{
 Serial.println("Connecting to server.");
 if (client.connect(server, 80))
 {
   Serial.println("Connection established.");
   client.print("GET ");
   client.println(location);
   client.println("Host: 192.168.1.4");
   client.println();
   //Connected - Read the page
   return readPage(); //go and read the output
 }
 else
 {
   return "Connection failed.";
 }
}

String readPage()   //This part seems rather weird.
                   //While true -> if avail -> while avail...??? Can't this be shorter?
{
 while(true)
 {
   if (client.available())
   {
     while (client.available())
     {
       char c = client.read();
       webData += c;
     }
     if (!client.available())  //Connection closed, data is retrieved hopefully.
     {
       //Begin string parsing
       //What needs retrieved, first & second marker, storage for truncated string,
       //and the String length of webData.
       int firstMark, secondMark;
       const String splittingChar = ("::"); //This marks the data
       
       //Find the markers, and then capture the data between them.
       firstMark = webData.indexOf(splittingChar);
       secondMark = webData.indexOf(splittingChar, firstMark+1);
       detData = webData.substring((firstMark + 2),secondMark);
       
       //Value of '-1' means nothing was found.
       //Returns all the captured website data
       if ((firstMark == -1) || (secondMark == -1))  //First or second mark not found.
       {
         Serial.println("**********************");
         Serial.println("****Marker missing****");
         Serial.println("**********************");
         return(webData);
       }
       
       //Something was found!
       else
       {
         Serial.println("Here are your '::' markers locations!");
         Serial.print("Location 1: ");
         Serial.println(firstMark);
         Serial.print("Location 2: ");
         Serial.println(secondMark);
         Serial.print("The code you are looking for is: ");
         Serial.println(detData);

       }          
       //Some debug output.
       /*
         Serial.print("The data recieved is ");
         Serial.print(webDataLength);
         Serial.println(" chars long.");
         Serial.println("Disconnecting...");
       */
       
       //Close the connection, flush the data out, return the command from the website.
       client.stop();
       client.flush();
       return (detData);
     }
   }
 }
}


SurferTim

Why do you need Twitter if you have your own Apache server?

You can't host the Apache server on localhost alone. You can only access it from the computer it is installed on. It must be exposed to the internet.


Jesterbaze


Why do you need Twitter if you have your own Apache server?

You can't host the Apache server on localhost alone. You can only access it from the computer it is installed on. It must be exposed to the internet.


I just use Twitter so I can text commands to my computer from anywhere, I couldn't think of a better way to do that (especially because I have a 'dumb phone'). Even running Apache 2 on my local network, the .php file I use will grab data from my Twitter account, and create a webpage on my network (I guess I didn't mean localhost I'm sorry lol) displaying the 5 most recent tweets to my account.

I didn't host the website on Arduino because trying to host a server, and process commands on the same board takes around 28kb of the 32kb alotted if I remember right. It's been a while since I've read in to that however, I could be way off. It would be super awesome to have a completely stand-alone Arduino home automation set-up that could read commands texted to it =)

Jesterbaze

Continued:
The Apache2 end uses a custom .php file that gets my Twitter data from https://github.com/abraham/twitteroauth

I unfortunately forget the author of this PHP file, and I'm having trouble finding it right now... If anyone can cite the author of this code please let me know, or post it here. I don't want to take any credit for that, I don't understand PHP at all.

PHP Code:
Code: [Select]

<?php

require_once('twitteroauth/twitteroauth.php');

//Insert your information into these define fields. I can't go showing mine.
define('CONSUMER_KEY''**');
define('CONSUMER_SECRET''**');
define('OAUTH_TOKEN''**');
define('OAUTH_TOKEN_SECRET''**');

$connection = new TwitterOAuth(CONSUMER_KEYCONSUMER_SECRETOAUTH_TOKENOAUTH_TOKEN_SECRET);


$content "";
if (!isset(
$username)){
    
$username "Baze_HomeAuto";
}
if (!isset(
$number)){
    
$number 3;
}

$feed $connection->get('statuses/user_timeline', array('screen_name' => $username'count' => $number));

//$url = "http://api.twitter.com/1/statuses/user_timeline.xml?screen_name={$username}&count={$number}";
//$tweets = file_get_contents($url);
//$feed = new SimpleXMLElement($tweets);

function time_stamp($date){
    if (empty(
$date)){
        return 
"No date provided";
    }
    
$periods = array("second""minute""hour""day""week""month""year""decade");
    
$lengths = array("60","60","24","7","4.35","12","10");
    
$now time();
    
$unix_date strtotime($date);
    if (empty(
$unix_date)){
        return 
"Bad date";
    }
    if (
$now $unix_date){
        
$difference $now $unix_date;
        
$tense "ago";
    } else {
        
$difference $unix_date $now;
        
$tense "from now";
    }
    for (
$j 0$difference >= $lengths[$j] && $j count($lengths)-1$j++){
        
$difference /= $lengths[$j];
    }
    
$difference round($difference);
    if (
$difference != 1){
        
$periods[$j] .= "s";
    }
    return 
"$difference $periods[$j] $tense";
}

for (
$i 0$i <= $number-1$i++){
    
$status $feed[$i];
    
//$status = $feed->status[$i];

    
if (isset($status->id)){
        
$id $status->id;
    }

    
$created_at $status->created_at;
    
$text $status->text;
    
$text preg_replace("#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t< ]*)#""\\1<a href=\"\\2\" target=\"_blank\">\\2</a>"$text);
    
$text preg_replace("#(^|[\n ])((www|ftp)\.[^ \"\t\n\r< ]*)#""\\1<a href=\"http://\\2\" target=\"_blank\">\\2</a>"$text);

    if (
preg_match("/#(\w+)/"$text$matches)){
        foreach(
$matches as $match){
            
$match str_replace("#"""$match);
            
$text preg_replace("/#(\w+)/","<a href=\"http://twitter.com/search?q={$match}\">$0</a>",$text);
            
$text str_replace("<a href=\"http://twitter.com/search?q={$match}\">#","<span class=\"hash\">#</span><a href=\"http://twitter.com/search?q={$match}\" target=\"_blank\">",$text);
        }
    }

    if (
preg_match("/@(\w+)/"$text$matches)) {
        foreach(
$matches as $match) {
            
$match str_replace("@"""$match);
            
$text preg_replace("/@(\w+)/""<a href=\"http://www.twitter.com/{$match}\">$0</a>"$text);
            
$text str_replace("<a href=\"http://www.twitter.com/{$match}\">@""<span class=\"at\">@</span><a href=\"http://www.twitter.com/{$match}\" target=\"_blank\">"$text);
        }
    }

    
$content .= "<div class=\"status\"><div class=\"text\">{$text}</div>";

    if (isset(
$id)){
        
$content .= "<div class=\"date\"><a href=\"http://twitter.com/Baze_HomeAuto/status/{$id}\" target=\"_blank\" title=\"".date("g:i A M jS"strtotime($created_at))."\">".time_stamp($created_at)."</a></div>";
    } else {
        
$content .= "<div class=\"date\"><a href=\"\" target=\"_blank\" title=\"".date("g:i A M jS"strtotime($created_at))."\">".time_stamp($created_at)."</a></div>";
    }
    
$content .= "</div>";
}

echo 
$content;
?>


Please feel free to ask any questions you have, leave comments, etc. I would be more than happy to answer them!

Limitations of using Twitter for Home Automation
There are a some limitations I have found using this system for Home Automation:

-Twitter doesn't like you re-posting the same information (more below).

-You computer must remain on to allow the hosting of your Apache2 serve (unless you host your own website, which would be so awesome  XD)

-If a buddy of yours posts a command on your Twitter account, in the correct format, it's going to execute that command. This could be cool, or it could be awful depending on what your aim is.

-Arduino 'String's can only contain so much data, I forget what the byte limit is, however it isn't enough to store a full-fledged website. If you are hosting a site with say >800 chars of data, including the header data you recieve on connection, this program doesn't work. I had to change my 'grabtwitter.php' file to only display the 3 most recent posts. I haven't found a work-around for this, but it would be nice to have one, because then I could totally bypass the Apache2 server, and just have my board connect directly to Twitter.

Concerning re-posting
So far I've noticed if you are looking to turn the same item on, then off, then on again, etc. (such as you're trying to test your program for completion  :smiley-roll-sweat: Twitter doesn't allow for the same post to be re-sent within some certain limit. I don't know how many other posts you need before you can re-post the same Tweet.

The dumb way for me to work around this, is to change the string Arduino is looking for.

Such as, changing "on" to "blink" inside of the code. But this is really annoying, and doesn't make much sense in terms of practicality.

The other way I've found, and probably a more sensible way to fix this is lets say you try this:
Code: [Select]

You text "::on::" then you text "::off::"

This works just fine! Good, it's working right?  ...Kindof, not really though :'(

Next time you text "::on::" Twitter will respond and say something to the effect of "You've already posted that!"

Since your program is only reading the data between your two designated markings, next time you text Twitter, just text:
Code: [Select]

"::on::2"
"::off::2"
"::on::3"
"::off::3"  and so on.

or append your command in some other creative way.  Your program will still only read "on", but Twitter is happy because you aren't just reposting "::on::" and "::off::".

Please feel free to modify this code, or offer some creative solutions to some of these issues if you have any ideas!  I would love to see this project grow!

Jesterbaze

[font=Verdana]Some useful links to help in making / modifying this project:
[/font]
[/b][/color]
Twitter OAuth 1.1 Resource
https://dev.twitter.com/docs/auth/oauth

PHP Starter guide -- Including installation
http://www.phpbuddy.com/

'Simple' how to install Apache2 and PHP5 on Ubuntu
http://www.howtoforge.com/installing-apache2-with-php5-and-mysql-support-on-ubuntu-13.04-lamp

Creating a Twitter Application Account
http://iag.me/socialmedia/how-to-create-a-twitter-app-in-8-easy-steps/


zoomkat

Quote
Arduino 'String's can only contain so much data, I forget what the byte limit is, however it isn't enough to store a full-fledged website.


I've got a 4 GB SD card in my Ethernet shield, which can hold a fair amount of html stuff.
Google forum search: Use Google Advanced Search and use Http://forum.arduino.cc/index in the "site or domain:" box.

Jesterbaze

Quote

I've got a 4 GB SD card in my Ethernet shield, which can hold a fair amount of html stuff.


I like where you're going with that idea, I do have an SD card slot in my W5100 Ethernet shield. The problem I'm hitting more specifically is the part in my code where the website data is read

Code: [Select]

//Under the readPage() function

      while (client.available())
      {
        char c = client.read();
        webData += c;
      }


webData only retains so much information, and I'm not really sure of a good way to store more data in that string, or even a good elegant way to use a String[] array...

Ihatehandles


Continued:
Next time you text "::on::" Twitter will respond and say something to the effect of "You've already posted that!"

Since your program is only reading the data between your two designated markings, next time you text Twitter, just text:
Code: [Select]

"::on::2"
"::off::2"
"::on::3"
"::off::3"  and so on.

or append your command in some other creative way.  Your program will still only read "on", but Twitter is happy because you aren't just reposting "::on::" and "::off::".

Please feel free to modify this code, or offer some creative solutions to some of these issues if you have any ideas!  I would love to see this project grow!

I had similar problems with Twitter... Couldn't you just create two strings from the read in tweet, using one and make the other junk... So if the tweet could be manually incremented "ON1, ON2, ON3" while the string from the read function only took the first two characters and junked the rest.... Then the logic for your output command could remain the same and only look for the "ON" message.

I hope this helps

Jesterbaze

Quote

I had similar problems with Twitter... Couldn't you just create two strings from the read in tweet, using one and make the other junk... So if the tweet could be manually incremented "ON1, ON2, ON3" while the string from the read function only took the first two characters and junked the rest.... Then the logic for your output command could remain the same and only look for the "ON" message.

I hope this helps


2 strings from the read in tweet? Like, where the code sorts out the Tweets from the HTML it pulls?

Oh or like, have a starter marker, without a finish marker. So it doesn't pull only whats between the two '::" marks?? I think I see what you mean now... So I don't have to type '::' at the end of my command, and my program will still read what I'm saying. That actually works pretty well. Only issue is if I want to add other commands to my code instead of just ::ON:: and ::OFF:: . But actually for what I'm using it for right now, that would be perfect! Thanks man!

Not sure if this is related, but I really wish this code could take the standard HTML headers and everything and delete them from the memory. Like a 24 char read-in buffer. Pull 24 chars of the HTML, decide if it's the command or not, then scrap them. This way I could pull pages of HTML code from my site, instead of just having a small Apache PHP page. IE: Make a website instead of a lame page with the latest 3 tweets from my account... I know this can be done, but I haven't messed around with using bits/bytes of data yet in my code. Anyone have any ideas how to do that?

In short to the idea above:
-I need a buffer to read 'x' amount of chars from the website. Lets say 24.
-Place the 24 chars in a temp string, and decide if it's a command or not.
-If yes - run the command
-If no - scrap the 24 chars from the memory. (Replace the string as " " or something??? IDK if that would clear the memory or add a space char to the code every 24 'junk chars'.)
-Ultimately, I want this to allow for MUCH more capacity to read entire websites for command strings, instead of tiny little webpages.

Lets get some ideas rolling =) I think this could be a badass little project  XD

farsi

indeed interesting project - I was thinking on observing some news on a website and display this news in a physical form via an Arduino. Your project might be a good start.

I was wondering about the Ethernet setup - what exactly are the options? It might be possible to take an Arduino Yun and run the Twitter client from there?

What about a wifi setup?

Go Up