Arduino+Linksys+internet to set and get Arduino variables over the internet

Hi
I'm trying to setup a solution to read and set Arduino variables over the internet without any shield.
The solution is based on a Arduino and a hacked Linksys WRT54GL. The Linksys has a serial port that can be hacked. This way you can connect the serial port (pin 0 and 1) of the Arduino to the Linksys.
On the Linksys I run openwrt http://wiki.openwrt.org/. Which is a Linux flavor. I installed a serial monitor called minicom.
Because the Linksys is connected to the same pins as the USB; the serial monitor used by the Arduino IDE and minicom are both "connected to Serial".

Until here all is already working fine. Unfortunately :cold_sweat: I want to control this by a website and not a serial monitor. So I need to do some coding to send messages to and receive messages from the Arduino. I however do not know any languages for creating server side dynamic web pages. I'm also not used to developing on Linux. So I'm looking for some advice on how to create the dynamic web pages.

Best regards
Jantje

You probably need to install a web server on the linksys router that supports cgi functions.

Some more background on what is already achieved.
I use C++ classes of which I create global instances to avoid new and delete. Each of these classes has a void Setup(); and a void Loop(); (Notice the first letter is capital). The basic idea is to only do real actions in the Loop(). All getters and setters should only set the variables but not act upon it*.
This way I know when the action will be done and as such I avoid complex sequence problems.

The result is that I call all Setup() of all my classes in setup() of the sketch and the Loop() in the loop() of the sketch.
In the program I used to test the serial communication the setup and Loop looks like:

void setup()
{
	wdt_reset();
	wdt_disable();
	Serial.begin(115200);
	myStringSerial.Setup();
	myDataDummyClass.Setup();
	myMessageHandler.Setup(&myDataDummyClass);
}

void loop()
{
	myStringSerial.Loop();
	if (myStringSerial.MessageReceived())
	{
		myMessageHandler.SetReceivedMessage(myStringSerial.getMessage());
	}
	myMessageHandler.Loop();
        myDataDummyClass.Loop();
}

wdt_reset(); wdt_disable(); are not really needed when you run the bootloader. If not you need them. Then there is Serial.Begin() and then all setups.
myStringSerial is a simple class that reads strings from the serial and returns it line by line.
myDataDummyClass is a class containing data that I use to modify and read.
myMessageHandler links the lines received from the serial ports via myStringSerial and executes the commands on myDataDummyClass. This is why myDataDummyClass is given as parameter to the Setup of myMessageHandler.

The commands I can set to Arduino are

		Serial.println(F("Following commands are supported"));
		Serial.println(F("? to show this info"));
		Serial.println(F("DUMP to show all variables and their values"));
		Serial.println(F("GET [FieldName] to show one variable value"));
		Serial.println(F("SET [FieldName] [value] to set one variable value"));
		Serial.println(F("ISDIRTY to see all the classes with the dirty flag set"));
		Serial.println(F("LOG VALUE               to LOG the values of the fields"));
		Serial.println(F("LOG HEADER              to LOG the names of the fields"));
		Serial.println(F("RESET to reset the arduino"));

Most of them are self explanatory except for LOG VALUE; LOG HEADER; ISDIRTY.
Lets start with ISDIRTY
To save memory I use pointer to the actual variables. That means that when you do a SET the code does not know the variable has been changed. I have foreseen a overloadable method to know but I prefer not to do so.
If I do a SET I also set the ISDIRTY flag of the class. This is possible because each class that you want to change with the serial monitor must be derived from the class DataInterface wich holds the ISDIRTY flag.
The LOG VALUE dumps all values just like DUMP but only the values and with ';' in between the values.
The LOG HEADER dumps all names of the variable just like DUMP but only the names and with ';' in between the values.
So 1 LOG HEADER and many LOG VALUE's gives you a csv file with header 8)

What is the impact on your code?
Functionally you will need to be able to cope with someone changing your variables behind you back (This is done by the Setup Loop approach )
Secondly you will have to check and reset the ISDIRTY flag to process Serial monitor driven changes
Thirdly you will need to use classes
Fourthly you will need to derive the classes you want to control from the web from DataInterface
And last you have to fill in the data DataInterface needs in your setup.
That is all.

As an example I dumped the creator of MyDataDummyClass below

DataInterface * globalChildren[NUMCHILDREN];
MyDataDummyClass::MyDataDummyClass():myLeftDataClass(F("MyDataDummyClass.left")),myRightDataClass(F("MyDataDummyClass.right"))
{
	myFields[0].ClassName=F("MyDataDummyClass");
	myFields[0].FieldName=F("Field_1");
	myFields[0].type=Uint16_t;
	myFields[0].isWritable=false;
	myFields[0].value.pUint16_t=&myField0;

	myFields[1].ClassName=myFields[0].ClassName;
	myFields[1].FieldName=F("Field_2");
	myFields[1].type=Uint8_t;
	myFields[1].isWritable=true;
	myFields[1].value.pUint8_t=&myField1;

	myFields[2].ClassName=myFields[0].ClassName;
	myFields[2].FieldName=F("Field_3");
	myFields[2].type=Uint16_t;
	myFields[2].isWritable=true;
	myFields[2].value.pUint16_t=&myField2;



	DataInterface::myFields=myFields;
	myNumFields=NUMFIELDS;
	myNumChildren=NUMCHILDREN;
	globalChildren[0]=&myLeftDataClass;
	globalChildren[1]=&myRightDataClass;
	myChildren= globalChildren;
	myField0=10;
	myField1=11;
	myField2=12;
}

The class has 2 children which can be changed over the Serial monitor. myLeftDataClass and myRightDataClass
The class has 3 fields: Field_1 Field_2 Field_3

	myFields[0].ClassName=F("MyDataDummyClass");
	myFields[0].FieldName=F("Field_1");
	myFields[0].type=Uint16_t;
	myFields[0].isWritable=false;
	myFields[0].value.pUint16_t=&myField0;

Field_1 is an Uint16_t and can not be changed using the Serial Monitor.
As I said I point directly to the memory. This can be seen at the last line.

For the children classes only the pointers to the classes need to be registered.

	myNumChildren=NUMCHILDREN;
	globalChildren[0]=&myLeftDataClass;
	globalChildren[1]=&myRightDataClass;
	myChildren= globalChildren;

I hope this background info makes my question a bit more clear.

Best regards
Jantje

*Note this is a strong rule as I break it in the example 8)

Zoomkat
Thanks for the info.
OpenWRT uses Lua. Which seems ok. My main concern is that I need read and write to the serial connection. Because I do not use the USB (no DRT line) I do not have the reset problem so I can connect and disconnect easily.

But I want to use the Linksys for dumping my debug info using the LOG HEADER and LOG VALUE. Basically I now run LOG VALUE in each loop and dump it on a SD card on the ethernet shield. As I dumped the ethernet shield I want to change this to dumping it to a SD card on the Linksys.
This means I need a program to listen to the Serial port all the time and I still need the capability to write to the Serial port.

Best regards
Jantje

I'm having trouble visualizing the entire setup. Is this right? ...

  • The Arduino is measuring stuff (temperatures, say).
  • You connect it via the serial port to the Linksys (not USB, serial).
  • The Linksys is plugged into your network
  • The Linksys is not acting as a router but as a small, cheap PC
  • The Linksys is saving incoming data to its internal flash memory
  • You want to run a web server on the Linksys
  • On receiving an incoming brower connection it will, on request, send back some saved data
  • Once the data is sent it can be deleted from the Linksys flash?

Why do you need to write to the serial port?

Where does the USB come into this?

Nick
Thanks for joining in.

  • The Arduino is measuring stuff (temperatures, say). YES
  • You connect it via the serial port to the Linksys (not USB, serial). YES
  • The Linksys is plugged into your network YES (via wireless)
  • The Linksys is not acting as a router but as a small, cheap PC I would say both; Here it is a a cheap PC but I may want to I add a webcam
  • The Linksys is saving incoming data to its internal flash memory That is likely to be (part of) the solution
  • You want to run a web server on the Linksys YES (It is already running one and Lua is running on it)
  • On receiving an incoming browser connection it will, ..... YES and No see below
  • Once the data is sent it can be deleted from the Linksys flash? YES (note I plan to add a SD card to)

Why do you need to write to the serial port?

This is related with "On receiving an incoming browser connection it will, ....."
For most commands reading already received data would be fine. However ....
I have a SET and a RESET command. The RESET command resets the Arduino (using watch dog). The SET command changes the value in the memory of the Arduino. As such these commands need to be send to the Arduino over the serial port.

The minicom (serial monitor) dump below shows what I mean when I talk about the SET command. I put Echo on and added ">" to the response lines (I know: GET and SET are pretty verbose)

GET MyDataDummyClass.right.direction
>GET started
>name            MyDataDummyClass.right.direction
>Writable        1
>Type            Uint16_t
>Value           90
>
>GET done
SET MyDataDummyClass.right.direction 223
>SET started
>name            MyDataDummyClass.right.direction
>Writable        1
>Type            Uint16_t
>Value           223
>
>SET done
GET MyDataDummyClass.right.direction
>GET started
>name            MyDataDummyClass.right.direction
>Writable        1
>Type            Uint16_t
>Value           223
>
>GET done

What this shows is that I changed the value of MyDataDummyClass.right.direction from 90 to 223.
because the MyDataDummyClass does not reset the ISDIRTY flag the ISDIRTY command returns MyDataDummyClass.right

ISDIRTY
>Dirty Classes:
>MyDataDummyClass.right

Where does the USB come into this?

I mentioned USB because I have read about this subject and when you use USB connecting and disconnecting may reset the Arduino. This is at least a problem I don't have.

Best regards
Jantje

You might benefit from some ideas in Bitlash Communicator:

I don't think it's a direct drop-in for your problem, but the combination of Bitlash as the client program (or anything speaking the same prompt-delimited protocol), speaking over USB to a server that provides a web interface / control panel sounds like it's in the neighborhood.

Good luck with your project,

-br

Billroy
Thanks for the reference. It comes close. Really close.
Unfortunately the code is javascript. I googled for running javascript on openwrt and it is not encouraging.
I also don't read javascript fluently (I know; I only know old languages) so it will probably cost me more time to translate than to write a new program.
Best regards
Jantje

I'm not sure this is going to be particularly easy. Whether or not you use Lua.

It seems to me you have two client/slave relationships going here. One is the web server->client relationship. Typically once an incoming brower request arrives you try to satisfy it fairly quickly. The other is the Linksys->Arduino relationship (eg. for setting variables). And then there is the data logging. So maybe that is three things you are trying to do at once.

I think personally I would buy an Ethernet shield and contain all the processing in one place.

I feel I'm nearly there. I found out that when I have mincom (the serial monitor) open and I run the commands

root@OpenWrt:~# echo 'SET MyDataDummyClass.Led_13 1' > /dev/ttyS1
root@OpenWrt:~# echo 'SET MyDataDummyClass.Led_13 0' > /dev/ttyS1

They are send to Arduino and Arduino responds and the output is set in minicom

SET started
name            MyDataDummyClass.Led_13
Writable        1
Type            Uint8_t
Value           1

SET done
SET started
name            MyDataDummyClass.Led_13
Writable        1
Type            Uint8_t
Value           0

SET done

So All I need now is a SEND URL where I can send my commands to. And a GET URL that can get me the Arduino values.
The SEND URL will take a param that is the actual command to send and echo the command to the serial port.
I need to think about the GET as getting value by value may impact performance.

I know for sure. If it is easy I'm not interested 8)

Best regards
Jantje

Would a serial to network proxy get you what you want? Here's one in Python, but there are others around:

This one lets you telnet to the host and type to the arduino connected to it.

If you want GET and PUT you need something to act as a web server, right?

-br

billroy
I use shttp to my router and run minicom from there. This looks like the same functionality as your project.
I want more than that. I want a nice web GUI automatically created based on a standard communication protocol with arduino.

The idea is that the tool runs DUMP against arduino and makes html files based on that.
Then the HTML pages used GET and SET URL's to send commands to Arduino based on the outcome of the DUMP.
This makes for a fully dynamic way to control your arduino in detail. I actually want to make a (tab) page for each child. So you get tab and children tabs .....
(Now I think about it it may be a great way to do Arduino debugging as well; but to difficult for beginners)

I can live with a timer on my HTML page to run the GET.
Living with a timed GET means I can fire the SET to Arduino and do not need to bother with the response and the next GET will get the result anyway.

The problems I'm facing now I know an approach are

  • How to write a program that reads the serial port (several options each with a learning curve)
  • How to run a system call in a dynamic page (I think I remember the Lua examples of Nick contained an example)
  • Finding time to make it all work

I'm not sure I explained this properly.
Best regards
Jantje

I'm proud to say I got it to work :smiley:
I only used a cgi scrip and an ash script
Basically it should work on any linux box.

I run this comand

./test <  /dev/ttyS1

Which reads from the serial port and dumps a html file based on the DUMP command and to the ArduinoValue file based on the LOG VALUE command
This is the test code

#!/bin/ash
InDumpFlag=1
InDump=0
NUM=0
OutputHtml=robot.html
OutputValue=ArduinoValue.txt
outputHeader=ArduinoValuHeader.txt
BeginHtml=input.beginHtml
EndHtml=input.endHtml
while read line
do
    echo "$line" >> all.txt
    if [ "$line" = "Dumping all fields" ]; then
        InDump=1
        NUM=0
        rm $OutputHtml
        cat $BeginHtml > $OutputHtml
    else
      if [ "$line" = "Dump done" ]; then
        cat $EndHtml >> $OutputHtml
        InDump=0
      else
        if [ "$InDump" = "$InDumpFlag" ]; then
          FIRST=`echo $line | cut -d' ' -f1`
          VALUE=`echo $line | cut -d' ' -f2`
          if [ "$FIRST" == "name" ]; then
            let NUM=NUM+1
            NAME=$VALUE
          elif [ "$FIRST" == "Writable" ]; then
            WRITABLE=$VALUE
          elif [ "$FIRST" == "Type" ]; then
            TYPE=$VALUE
          elif [ "$FIRST" == "Value" ]; then
            echo -n "<tr><td><div id=\"n_"$NUM"\">"$NAME"</div></td>"  >> $OutputHtml
            echo -n "<td><div id=\""$NUM"\"></div></td>" >> $OutputHtml
            if [ "$WRITABLE" -eq "1" ]; then
              echo -n "<td><input type=\""$TYPE"\" id=\"t_"$NUM"\"/></td>" >> $OutputHtml
              echo -n "<td><button type=\"button\" onclick=\"UpdateValue('"$NUM"');\">set</button></td>" >> $OutputHtml
            fi
            echo "</tr>" >> $OutputHtml
          fi
        else
          FIRST=`echo $line | cut -d';' -f1`
          if [ "$FIRST" == "LOG VALUE" ]; then
            echo "$line" > $OutputValue
          fi
          if [ "$FIRST" = "LOG HEADER" ]; then
            echo "$line" > $outputHeader
          fi
        fi
      fi
    fi
done

The code to set the variables is a cgi script and is even simpler

#!/bin/ash
echo SET $QUERY_STRING | sed 's/=/ /'   >> /dev/ttyS1
echo LOG VALUE >> /dev/ttyS1

For completeness here is the html starting code (not all of it is needed)

<html><head><script type="text/javascript">

function loadValues()
{
        var xmlhttp;
        if (window.XMLHttpRequest) {xmlhttp=new XMLHttpRequest(); }
        else { xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");}
        xmlhttp.onreadystatechange=function()
        {
                if (xmlhttp.readyState==4 && xmlhttp.status==200)
                {

                        var lineContent = xmlhttp.responseText.split(";");
                        for ( curValue =1; curValue  <(lineContent.length-1) ; curValue ++)
                        {
                          var curelement = document.getElementById( curValue );
                          if (curelement == null)
                          {
                             document.getElementById( "t_" + curValue ).value=lineContent[curValue];

                          }
                          else
                          {
                            curelement.innerHTML=lineContent[curValue];
                          }
                        }
                        setTimeout("loadValues();",1000);
                }
        }
        xmlhttp.open("GET","/robot/value.txt" ,true);
        xmlhttp.send();
}

function UpdateValue(Param)
{
        var xmlhttp;
        if (window.XMLHttpRequest) {xmlhttp=new XMLHttpRequest(); }
        else { xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");}
        xmlhttp.open("POST","/cgi-bin/robot/set.cgi?" + document.getElementById( "n_" + Param  ).innerHTML + "=" + document.getElementById( "t_" + Param  ).value ,true);
        xmlhttp.send();
}
function UseCurLoc(Param)
{
        var xmlhttp;
        if (window.XMLHttpRequest) {xmlhttp=new XMLHttpRequest();}
        else { xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");}
        xmlhttp.open("POST","/cgi-bin/robot/Param?" + document.getElementById( "n_" + Param  ).innerHTML + "=CurrentLocation" ,true);
        xmlhttp.send();
}
function GoToInitializingState()
{
        var xmlhttp;
        if (window.XMLHttpRequest) {xmlhttp=new XMLHttpRequest();}
        else { xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");}
        xmlhttp.open("POST","/robot/Param?GoToInitializingState=true" ,true);
        xmlhttp.send();
}
</script></head>
<body onload="loadValues()";>
<h2>MowRobot control page</h2>
<table>
<tr><td>param</td><td>value</td></tr>

and the html ending code

</table>
</body>
</html>

and attached the code I run on arduino

If you have time to look at it.... tell me what you think

Best regards
Jantje

GenericSerialWeb.zip (10.1 KB)