Reading a decimal numbur using Bridge.Get() on the Arduino side.

Hi all.

This is my first post on the forum. I am new to Arduino and must say I love. So much you can do with this technology it boggles the mind.

OK so here is my question. I am using the the Bridge library to pass values between the 2 processors.
I can pass a float (2 decimal number) to the linux side and read it without any problem. But I am not able to read it with Bridge.Get() on the Arduino side. I am not sure what Char size to use.

I can see the values is correct when I use the url http://10.64.9.72/data/get/

Also I am not sure, but it seems slower if you use the bride library in Python than when you call http://10.64.9.72/data/get/.

thanks all. I hope somebody can help.

Code?

jfourie:
I am not sure what Char size to use.

The Bridge passes everything as strings, including your floats which are converted to string before being stored. The char buffer you use to read from the Bridge should be large enough to hold the largest possible value you are trying to read, plus at least one more character for the NULL string terminator. If it's too small, you will only read the beginning part of the string, and not the whole value.

Also I am not sure, but it seems slower if you use the bride library in Python than when you call http://10.64.9.72/data/get/.

I'm not quite getting what you're saying. Can you rephrase it?

Hi. Tanks for the reply. I understand it is strings. Not shure why the put is string but the get is char?

How big should the char be for 99.99
I get the value but I loose the decimal places.

My code is

Char ttemp[8];

Bridge.get("TTEMP", ttemp, 8);

Then i cast the char to a float.

For my second statement. I wrote a Paython script to get a value from the bridge. When I execute it it takes up to a second to run and to print the value to the command line. But if I open a browser and got to the /data/get/ url. It will show all the values without a delay. Its very fast.

jfourie:
Not shure why the put is string but the get is char?

String is a smart wrapper for character arrays. I think pit() will also accept character arrays in addition to string. Get() probably just uses character arrays because they are more efficient, and because it makes the coding simpler (for the Bridge library developer, not necessarily for you!)

How big should the char be for 99.99

Six should be enough (two digits, one decimal point, two more digits, and a NULL terminator.) But I wouldn't bother going less than the 8 you have.

I get the value but I loose the decimal places.

Have you tried printing the character string once you get it to determine whether you have the whole thing!

Then i cast the char to a float.

How? A simple type cast like (float)ttemp isn't going to do it. I suspect you're reading the string properly, and the issue is the conversion.

For my second statement. I wrote a Paython script to get a value from the bridge. When I execute it it takes up to a second to run and to print the value to the command line. But if I open a browser and got to the /data/get/ url. It will show all the values without a delay. Its very fast.

Curious, I've not seen that happening. Maybe it's something in the way you are fetching the value in the Python code. Please post your code, otherwise we're just guessing.

This is how I cast the char to float.

target_temp = atof(ttemp);

Here is the Python script to show all the values.

#!/usr/bin/python

import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

print("T1 : " + value.get("T1") + " T2 : " + value.get("T2") + " T3 : " + value.get("T3") + " T4 : " + value.get("T4") + " H1 : " + value.get("H1") + " H2 : " + value.get("H2"))

Here is a link to a YouTube video that shows what I mean by slower. I can still use it, just wondering why there is a difference in speed. Is it Python that is the problem ?

OK so the problem seems to be with target_temp = atof(ttemp);

I get the correct char as its prints 99.99 to the lcd when I print the char.
When I convert the char it looses the .99 and makes it .00.

I tried to replicate your web/Python timing. My web page response isn’t quite as fast as yours, but the Python timing is similar. One thing that immediately jumped out at me is that the web request is doing a single access and returning all values, while the Python code is making six separate calls to get the data, so I would expect the Python code to have a little more overhead.

I added some instrumentation to try to figure out what is taking the time in the Python code. I changed the script to this:

#!/usr/bin/python

import datetime

t0 = datetime.datetime.now()

import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')

t1 = datetime.datetime.now()

from bridgeclient import BridgeClient as bridgeclient

t2 = datetime.datetime.now()

value = bridgeclient()

t3 = datetime.datetime.now()

print("T1 : " + value.get("T1") + " T2 : " + value.get("T2") + " T3 : " + value.get("T3") + " T4 : " + value.get("T4") + " H1 : " +  value.get("H1") + " H2 : " + value.get("H2"))

t4 = datetime.datetime.now();

print "t0->t1:", (t1-t0).microseconds
print "t1->t2:", (t2-t1).microseconds
print "t2->t3:", (t3-t2).microseconds
print "t3->t4:", (t4-t3).microseconds
print "t0->t4:", (t4-t0).microseconds

And when I run it, I get this output:

root@Yun3:/mnt/sda1# ./getTest.py
T1 : 123 T2 : 234 T3 : 345 T4 : 456 H1 : 111 H2 : 222
t0->t1: 325
t1->t2: 179537
t2->t3: 261
t3->t4: 752686
t0->t4: 932809

Translation:

  • Importing sys and inserting the bridge directory into the path takes about a third of a millisecond
  • Importing BridgeClient takes about 180 milliseconds
  • Creating the brindgeclient takes about a quarter of a millisecond
  • Getting the data, formatting the output, and printing the output takes about 3/4 of a second.
  • The whole thing takes just about a second.

The time from hitting return to getting the output is several seconds. This instrumentation shows that the script itself takes just under a second, the remaining time is likely the time it requires to launch Python, read and compile the script, and start running the script.

Another observation is that there is a lot more going on in your last line than fetching the text. My next test script:

#!/usr/bin/python

import datetime

import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')

from bridgeclient import BridgeClient as bridgeclient

value = bridgeclient()

t0 = datetime.datetime.now()

T1 = value.get("T1")
t1 = datetime.datetime.now()

T2 = value.get("T2")
t2 = datetime.datetime.now()

T3 = value.get("T3")
t3 = datetime.datetime.now()

T4 = value.get("T4")
t4 = datetime.datetime.now()

H1 = value.get("H1")
t5 = datetime.datetime.now()

H2 = value.get("H2")
t6 = datetime.datetime.now()


print "T1 : ", T1, " T2 : ", T2, " T3 : ", T3, " T4 : ", T4, " H1 : ", H1, " H2 : ", H2

t7 = datetime.datetime.now();

print "t0->t1:", (t1-t0).microseconds
print "t1->t2:", (t2-t1).microseconds
print "t2->t3:", (t3-t2).microseconds
print "t3->t4:", (t4-t3).microseconds
print "t4->t5:", (t5-t4).microseconds
print "t5->t6:", (t6-t5).microseconds
print "t6->t7:", (t7-t6).microseconds
print "t0->t7:", (t7-t0).microseconds

And the output from running it:

root@Yun3:/mnt/sda1# ./getTest2.py
T1 :  123  T2 :  234  T3 :  345  T4 :  456  H1 :  111  H2 :  222
t0->t1: 121992
t1->t2: 121407
t2->t3: 121735
t3->t4: 111502
t4->t5: 131603
t5->t6: 112625
t6->t7: 1325
t0->t7: 722189

The overall time is still about 3/4 of a second to fetch, format, and print all of the values, so restructuring things didn’t drastically change the timing. It looks like it takes about 1/8 of a second to get a value, and a bit over a millisecond to format/print it. I’m a bit surprised by these numbers, I suspected that the printing/formatting would take longer.

So there are some simple benchmarks. It does take some time to read the values from the bridge, but the majority of the overall execution time time appears to be the time required to launch/prepare the script in the first place.

It is because of this, that I write my systems so that the Python code is launched once in the beginning, and keeps looping forever. It is much more efficient to loop the script to handle multiple data points over time, rather than to re-launch the script after each set of data points are collected.

I also tend to not use the Bridge for sending data to the Python script. I use a Process object to launch the script when the sketch starts, and then I use Process.write() to send data to the script, which the script receives by reading stdin, just like it would read from the keyboard if you were running it a the command line. To send data back, the Python code simply prints the data, and that is received in the sketch by calling Process.available() and Process.read().

jfourie:
When I convert the char it looses the .99 and makes it .00.

atof() should work. How are you printing out the value? Is there something in the data path that is converting the value to an int along the way?

This is why it’s very helpful to have the complete code posted right from the beginning. So many times when just snippets are posted, the problem turn out to be in the part that is omitted.

Also, when posting code, please use code tags: it improves readability, makes it easier to select/copy the code, and prevents the code from turning into smiley faces (like the Bridge.get() call in reply 3.) To do that, click the </> button above the post edit widow, and post your code between those tags.

I got it working decimals working.

If you have a small sample of how to send data between the 2 systems that would be great.

PS I find it slower with only one value as well. So it seems it is as you say the startup of Python.

Wonder what is sending the data if you use /data/get/ as that dis not slow. Or is it because the code is running and does not need to start up, it just replies.

curl on the os level is also very fast. So if I type curl http://10.64.9.72/data/get/T1/ or /data/get/

All seem the same speed and a lot faster than Python.

I’m not sure why the web process is so much faster - could be that it doesn’t have to launch Python every time, could be that it’s using something faster then interpreted Python for the internal CGI type call to get the data.

For the sake of communicating with a Python script, once you’ve started it using a Process object, talking to the Python is exactly the same as using a Serial port - both Serial and Process derive from the Stream class, and Stream is the class that provides the read(), write(), print(), and available() functionality.

Here is fairly simple sketch that shows two way communications to a Python process. It’s not quite as trivial as some examples, as I wanted to show asynchronous two-way communications, and also have the sketch free to do other things while waiting for input from the Python process. What’s going on should be fairly well commented:

#include <Bridge.h>
#include <Process.h>
#include <Console.h>

Process proc;

unsigned long timer;
#define INTERVAL 1000

word counter = 0;

#define LED 13

void setup()
{
   pinMode(LED, OUTPUT);
   digitalWrite(LED, HIGH);
   
   Bridge.begin();         // Initialize the Bridge
   Console.begin();        // Initialize the network Console

   // Wait until a console is connected, fast LED flash while waiting.
   while (!Console)
   {
      digitalWrite(LED, HIGH);
      delay(100);
      digitalWrite(LED, LOW);
      delay(100);
   }

   Console.println("Simple Process Test");

   // Start the process, let it run in the background while sketch continues.
   proc.begin("/mnt/sda1/simpleProcessTest.py");
   proc.runAsynchronously();

   // Start the timer.
   timer = millis();
}



void loop()
{
   // If there is anything from the process, echo it to the console.
   if (proc.available())
   {
      // Got something from the process.
      Console.print("From process: ");

      // As long as there is data from the process, copy it to the console.
      // There is a (char) type cast because read() returns an int, and without making
      // it a char, Console.print() will print the ASCII number instead of the character.
      while (proc.available())
         Console.print((char)proc.read());

      // A real application would probably be putting the received characters into a
      // character buffer, waiting for a terminator, and then parsing the buffer.
      // This is EXACTLY the same idea as reading input from a serial port.

      // No more output, print out a line terminator to the console
      Console.println();
   }


   // Just to test, send something to the process once per second
   // We'll just increment a counter and send that as a string
   if ((millis() - timer) > INTERVAL)
   {
      // The timer has expired. Advance the timer to the next interval
      timer += INTERVAL;

      // A real application will try to be doing something more interesting here.
      counter++;
      proc.println(counter);

      Console.print("Sent ");
      Console.print(counter);
      Console.println(" to process.");
   }
}

This is the Pyhon code, which is also not quite trivial as it is also set up similarly to the sketch where it uses a timer to periodically do things while waiting for input:

#!/usr/bin/python -u
# Note the "-u" above - it's very important so that iytoyt us unbuffered.
# Without it, any output from the script will be buffered and won't be
# sent to the sketch until the buffer is full (which can take a long time.)


import time
import sys
import select


print "simpleProcessTest Python script"
print "Reads in input, expecting integer numbers."
print "For each value received, it is doubled and printed back out."
print "It also asynchronously will print out a message every five seconds."

lastUpdateTime = time.time()

# Repeat forever
while True:

   # Check if there is a line of input ready. Wait up to 0.1 seconds
   # This allows the loop to keep runing even if there isn't input,
   # much like calling Serial.available() to check whether data is ready.
   # If you don't want to do anything else while waiting for input, then
   # skip this part and call sys.stdin.readline() directly.
   ready = select.select([sys.stdin], [], [], 0.1)[0]
   if ready:
      # There is a line of input ready, it's an integer counter
      line = sys.stdin.readline()
      if (line.rstrip()):   # Skip empty lines
         try:
            # Convert the character string to an integer
            value = int(line)

            # Double the value and send it to the sketch
            print value*2

         except:
            print "Exception trying to convert input line!"

   else:

      # There is no input waiting. Is it time to print a message?
      now = time.time()
      if ((now - lastUpdateTime) > 5):
         lastUpdateTime = now;
         print "Timer tick!"

And this is the output from running it: (To make it easier to follow, I’ve manually highlighted the data that was echoed from the Python process in red. Anything that’s black was generated by the sketch.)

Simple Process Test
Sent 1 to process.
From process: simpleProcessTest Python script
Reads in input, expecting integer numbers.
For each value received, it is doubled and printed back out.
It also asynchronously will print out a message every five seconds.
2

Sent 2 to process.
From process: 4

Sent 3 to process.
From process: 6

Sent 4 to process.
From process: 8

Sent 5 to process.
From process: 10

Sent 6 to process.
From process: 12

From process: Timer tick!

Sent 7 to process.
From process: 14

Sent 8 to process.
From process: 16

Sent 9 to process.
From process: 18

Sent 10 to process.
From process: 20

Sent 11 to process.
From process: 22

From process: Timer tick!

Sent 12 to process.
From process: 24

Sent 13 to process.
From process: 26

Sent 14 to process.
From process: 28

Thank you for the sample. I have learnt a lot from it. I have also found that if you access the bridge to fast it will send back garbage data. I extended the code above to write all the values to a sqlite database. I did not put a sleep in to see how fast it would go running in a while loop. The bridge will start sending wrong data and then the script fails as it can not convert the data type. Now I run it every 20 seconds and all seems fine.

But here is the part I need to figure out next. I want to build a web front end to the sensors and write it to a local db. There will be a server that requests all the data logged to the local db from the last time it synced and then a Python script will create a json object and send all the data to the server. When the server responds that it got the data then those records will be deleted from the local db.

So what worked well from the bridge is that it is a area that more than one process can update and query. So the db process can read every 20 seconds and write the values and a web page can show real time graphs. But now it seems that the bridge will go unstable if too many proceses start to access it.

This console method you have shown here is a one to one comunication. I guess it could update a table in the db and then the web page could read the table. But that will be even slower.

Mmmmmmm so I have some thinking to do.

I have had good success having the Python script receive data from the sketch, and then write the data to a SQLite3 database. Then, as a separate process, a web application written with the Bottle framework reads from the database and generates the web pages.

If you are not permanently storing the data on the Yun, you might be interested in the RRD - Round Robin Database, which I describe in this topic: http://forum.arduino.cc/index.php?topic=367981.0 It’s been working very well for me. It stores regular sequences of time ordered data, and you can define the interval between data samples, and the number of samples to be stored - once that limit is reached, the oldest samples are automatically overwritten. That way, you don’t have to worry about deleting data as it’s transferred, and you don’t have to worry about the database getting infinitely big. In that thread, I talk about how I set up additional databases that automatically condense the data over longer time periods, but you don’t have to do that if you don’t want that feature.

If your ultimate goal is to get the data to another server, you might also be interested in collectd, which allows automatic forwarding of data through RRD databases to one or more servers. If this suits your needs, it may eliminate the need to develop your own store and forward mechanism. There is a discussion of collectd in this thread: http://forum.arduino.cc/index.php?topic=267847.0

An interesting tidbit of collectd is that it is already supported by LUCi and the Yun’s advanced configuration web pages, which simplifies setting it up. This is discussed here: http://forum.arduino.cc/index.php?topic=238693

Good luck with your project!

Thank you very much for all your help.

Is see there is another way to have the 2 processors talk to each other. You disable the bridge and talk via the com port like you would from a PC to the Arduino,

http://tavendo.com/blog/post/arduino-yun-with-autobahn/