Go Down

Topic: Using the Plotly web api to display data from Yun (Read 10605 times) previous topic - next topic

DrSpeed

Hi All,

If you're gathering data on your Yun and want to store/display it as graphs/charts/etc. you can use the Plotly web site ( https://plot.ly/ ). Plotly seems to be free and they have a python api that you can call from the linux side on the Yun to upload data that in my case is stored on the SD card.

To get the python API to work I needed to get a couple of openssl packages for python:
opkg install pyopenssl
opkg install python-openssl

and get the Plotly python api code from thier web site.

Then the python (I'm not a python person) is pretty straight forward. Parse the data file and pass it to the plotly web api:
Code: [Select]

import plotly
from array import *

data_filename = "../datalog.txt";   #SD Card data
N_POINTS = 300

file = open(data_filename, 'r')
nLines = 0
nShortLines = 0
maxLines = 10
count = 1

ticks = list()
temps  = list()
humids = list()
dewys = list()


for line in file:
    if (len(line) > 1):
        nLines = nLines + 1
        words = line.split(' ')
        if (len(words) == 2):
            fields = words[1].split(',')
            time = fields[0].strip()
            temp = fields[1].strip()
            humid = fields[2].strip()
            dewy = fields[3].strip()
            ticks.append(count)
            temps.append(float(temp))
            humids.append(float(humid))
            dewys.append(float(dewy))
            count = count + 1

            print("Time: {} Temp: {}  Humid: {}  DewPt: {}".format(time, temp, humid, dewy));
    else:
        nShortLines = nShortLines + 1

    if (nLines > N_POINTS):
        break

y = plotly.plotly( USER, APP_KEY)  # this is YOUR account information from plotly

response = py.plot(ticks, temps, ticks, humids, ticks, dewys, filename='Arduino Yun Data')
url = response['url']
filename_resp = response['filename']

sonnyyu


Plotly seems to be free and they have a python api that you can call from the linux side on the Yun to upload data that in my case is stored on the SD card.


Nice.

Anderson69s

Hey i'm very interesting by you project could you share it a little bit more?

Sea2605

#3
Feb 28, 2014, 08:43 pm Last Edit: Mar 01, 2014, 09:21 am by Sea2605 Reason: 1
Yeah, I'm also very interested in this! Please give us some more details. For example:

1. Can you upload your sketch? How did you log the data in the txt?
And to run the Python File: Did you just include a runShellCommand(...) to run the python file?

2. I couldn't figure out how your lines in your log.txt look like can you upload that file also?

EDIT:
Ok, I figured out a little by myself and will give some more detailed info for beginners

1. As was already written: Install OpenSSL etc. via SSH (How to enter SSH: open a "Terminal" window on MacOsX and enter "ssh root@XXX.XXX.XXX.XX" (XXX.... shall be the currrent iP of your Yun in the network)). After you entered your root passwort and are now in the SSH-Command-Line enter this:
opkg update
opkg install pyopenssl
opkg install python-openssl

1. What we all need is to get "distribute", "easy_install" and "pip" via SSH-Commands (so we can install plotly over ssh on the linino):
opkg update
opkg install distribute #it contains the easy_install command line tool (this can take some time)
opkg install python-openssl #adds ssl support to python
easy_install pip #installs pip  (this can take some time)

2. Now install plotly via SSH-Command:
pip install plotly (this can take some time)

3. After that you can create a sketch that makes a datalog.txt on your SD-Card (see e.g. DataLogger-Tutorial  @arduino.cc)

4. Now create the .py-file with the code Dr Speed posted ***  (use e.g. TextWrangler for this), BUT change the path to your datalog.txt for example and all the plotly-Credentials (name, key).... Then: import that .py-file on your Yun (e.g. with an SFTP-Client like Cyberduck or just put it on the SD-Card with your computer), after that enter in SSH:

python /mnt/sd/Filename.py

This should run the transfer (of the datalog.txt) to plotly and you can see your graph on their website.

*** = BE CAREFUL: The code posted by Dr_speed has some little mistakes (copy-paste-related I guess). The variable "y" somewhere at the bottom: needs to be "py" instead of "y". And somehow python/yun wouldn't accept e.g. the line "temps.append(float(temp))"  because it can't convert the String to a float... So I temporarily just put a float inside it: temps.append(1.99)

Anderson69s

#4
Mar 01, 2014, 08:32 pm Last Edit: Mar 01, 2014, 08:51 pm by Anderson69s Reason: 1
Ok, I made something similar with a dht21, bmp085, RTC module, temboo and google drive... (for free)  :smiley-yell:



But I really want to test your way beacause mine take 80% of memory on the Yùn  :~

And I still have to add 8 relays driven by temperature and time and the tracking on gsheet... Not sure if i'll have enough space on Yùn...

I have installed the dependencies and plotly everything is ok so far. Right know, I'm starting to modify the datalogger code for a dht21 and a bmp085 because apparently your way don't need a RTC module...

Then, the py file... Hoping commands for nano in Yùn are the same on Raspberry Pi otherwise it'll be hard...

Here is the code I use for now :

Quote

#include <Wire.h>
#include <Bridge.h>
#include <Console.h>
#include <Temboo.h>
#include "TembooAccount.h" // contains Temboo account information
#include <Adafruit_BMP085.h>
#include "DHT.h"
#include "RTClib.h"

#define DHTPIN 4
#define DHTTYPE DHT22
const String GOOGLE_USERNAME = "xxxxxxxxxxx";
const String GOOGLE_PASSWORD = "xxxxxxxxxx";
const String SPREADSHEET_TITLE = "ArduinoYun";
const unsigned long RUN_INTERVAL_MILLIS = 3600000; // how often to run the Choreo (in milliseconds)
unsigned long lastRun = (unsigned long)-60000;
Adafruit_BMP085 bmp;
DHT dht(DHTPIN, DHTTYPE);
RTC_DS1307 RTC;

void setup() {
 
  Serial.begin(9600);
  Bridge.begin();
  Console.begin();
  bmp.begin();
  dht.begin();
  RTC.begin();
  if (! RTC.isrunning()) {
    Console.println("RTC ERREUR !!!");// la ligne suivante définit l'heure du RTC à celle ou le code a été compilé
    RTC.adjust(DateTime(__DATE__, __TIME__));
  }

}

void loop()
{
  // get the number of milliseconds this sketch has been running
  unsigned long now2 = millis();

  // run again if it's been 60 seconds since we last ran
  if (now2 - lastRun >= RUN_INTERVAL_MILLIS) {

    // remember 'now' as the last time we ran the choreo
    lastRun = now2;
    DateTime now = RTC.now();
    Console.println("Getting sensor value...");

    // get the value we want to append to our spreadsheet
    int date = (now.year());
    int heure = (now.hour());
    unsigned long Pression = bmp.readPressure();
    int TBMP = bmp.readTemperature();
    int TDHT21 = dht.readTemperature();
    int Humidite = dht.readHumidity();

    Serial.println("Appending value to spreadsheet...");

    // we need a Process object to send a Choreo request to Temboo
    TembooChoreo AppendRowChoreo;

    // invoke the Temboo client
    // NOTE that the client must be reinvoked and repopulated with
    // appropriate arguments each time its run() method is called.
    AppendRowChoreo.begin();
   
    // set Temboo account credentials
    AppendRowChoreo.setAccountName(TEMBOO_ACCOUNT);
    AppendRowChoreo.setAppKeyName(TEMBOO_APP_KEY_NAME);
    AppendRowChoreo.setAppKey(TEMBOO_APP_KEY);
   
    // identify the Temboo Library choreo to run (Google > Spreadsheets > AppendRow)
    AppendRowChoreo.setChoreo("/Library/Google/Spreadsheets/AppendRow");
   
    // set the required Choreo inputs
    // see https://www.temboo.com/library/Library/Google/Spreadsheets/AppendRow/
    // for complete details about the inputs for this Choreo
   
    // your Google username (usually your email address)
    AppendRowChoreo.addInput("Username", GOOGLE_USERNAME);

    // your Google account password
    AppendRowChoreo.addInput("Password", GOOGLE_PASSWORD);

    // the title of the spreadsheet you want to append to
    AppendRowChoreo.addInput("SpreadsheetTitle", SPREADSHEET_TITLE);

    String rowData(now2);
    rowData += ",";
    rowData += date;
    rowData += "/";
    rowData += now.month();
    rowData += "/";
    rowData += now.day();
    rowData += ",";
    rowData += heure;
    rowData += ":";
    rowData += now.minute();
    rowData += ":";
    rowData += now.second();
    rowData += ",";
    rowData += Pression;
    rowData += ",";
    rowData += TBMP;
    rowData += ",";
    rowData += TDHT21;
    rowData += ",";
    rowData += Humidite;
    rowData += ",";

AppendRowChoreo.addInput("RowData", rowData);

    // run the Choreo and wait for the results
    // The return code (returnCode) will indicate success or failure
    unsigned int returnCode = AppendRowChoreo.run();

    // return code of zero (0) means success
    if (returnCode == 0) {
      Serial.println("Success! Appended " + rowData);
      Serial.println("");
    } else {
      // return code of anything other than zero means failure 
      // read and display any error messages
      while (AppendRowChoreo.available()) {
        char c = AppendRowChoreo.read();
        Serial.print(c);
      }
    }

    AppendRowChoreo.close();
  }





Anderson69s

#5
Mar 01, 2014, 10:39 pm Last Edit: Mar 03, 2014, 05:28 pm by Anderson69s Reason: 1
Here is the modified code from the datalogger tuto. I can record the dht21, BMP085 and hour without RTC Module save 7% of memory not bad...

Quote
#include <FileIO.h>
#include <Wire.h>
#include <Bridge.h>
#include <Console.h>
#include <Adafruit_BMP085.h>
#include "DHT.h"

#define DHTPIN 4
#define DHTTYPE DHT22

Adafruit_BMP085 bmp;
DHT dht(DHTPIN, DHTTYPE);

void setup() {
 
  Bridge.begin();
  Console.begin();
  Serial.begin(9600);
  FileSystem.begin();
  bmp.begin();
  dht.begin();

  while(!Console);  // wait for Serial port to connect.
  Console.println("Filesystem datalogger\n");
}


void loop () {
 
  unsigned long Pression = bmp.readPressure();
  int TBMP = bmp.readTemperature();
  int TDHT21 = dht.readTemperature();
  int Humidite = dht.readHumidity();
  // make a string that start with a timestamp for assembling the data to log:
  String dataString;
  dataString += getTimeStamp();
  dataString += ",";
  dataString += Pression;
  dataString += ",";
  dataString += TBMP;
  dataString += ",";
  dataString += TDHT21;
  dataString += ",";
  dataString += Humidite;
  dataString += ";";


  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  // The FileSystem card is mounted at the following "/mnt/FileSystema1"
  File dataFile = FileSystem.open("/mnt/sd/datalog.txt", FILE_APPEND);

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.close();
    // print to the serial port too:
    Console.println(dataString);
  } 
  // if the file isn't open, pop up an error:
  else {
    Console.println("error opening datalog.txt");
  }

  delay(5000);

}

// This function return a string with the time stamp
String getTimeStamp() {
  String result;
  Process time;
  // date is a command line utility to get the date and the time
  // in different formats depending on the additional parameter
  time.begin("date");
  time.addParameter("+%D/%T");  // parameters: D for the complete date mm/dd/yy
                                //             T for the time hh:mm:ss   
  time.run();  // run the command

  // read the output of the command
  while(time.available()>0) {
    char c = time.read();
    if(c != '\n')
      result += c;
  }

  return result;
}


And the python code without errors. Don't forget to install dependencies as describe before
Quote

import plotly
py = plotly.plotly( "xxxxxxxx", "xxxxxxx")
from array import *
data_filename = "/mnt/sda1/datalog.txt";

N_POINTS = 500

file = open(data_filename, 'r')
nLines = 0
nShortLines = 0
maxLines = 500
count = 0

date = list()
pressure = list()
tbmp = list()
tdht = list()
humidite = list()
fields = list()

for line in file:
    if (len(line) > 2):
        nLines = nLines + 1
        words = line.split(';')
       
        if (len(words) > 1):
           
            fields = words[0].split(',')
            pprint.pprint(fields)
            date.append(fields[0].strip())
            pressure.append(int(fields[1].strip()))
            tbmp.append(int(fields[2].strip()))
            tdht.append(int(fields[3].strip()))
            humidite.append(int(fields[4].strip()))
            count = count + 1
            print("Date: {} Pressure: {} Tbmp: {} Tdht: {} Humidite: {} ".format(date, pressure, tbmp, tdht, humidite));
    else:
        nShortLines = nShortLines + 1
   
    if (nLines > N_POINTS):     
        break


print("reponse " + str(count))
response = py.plot(date, pressure, date, tbmp, date, tdht, date, humidite, filename='Arduino Yun Data')
url = response['url']
filename_resp = response['filename']


I still need to modify it for the 2 temperature, the pression and the humidity. But for now it connects and send the wrong values...

bream

I'm also posting DHT22 and DS18B20 oneWire sensors  direct to plotly with an Uno.

It's working well but the uno crashes when trying to send multiple streams. the guys at Plotly are investigating at the moment.

Instead I'm just using a single stream but a the moment you're limited to a single chart linked to a streamed data file, which is a shame because I'd like to have multiple charts directly linked to a single file.

My build details and the sketch code are available at http://www.rowan-moore.net

bolatu

Hi guys,

I have a problem about installing plotly to Yun. I followed this tutorial :https://github.com/plotly/arduino-api/tree/master/plotly_yun

However, I couldnt copy the files from PC to Yun with this commands:

Code: [Select]
$ cd ~/Downloads/plotly_yun
$ ls
Arduino         Linino          README.md
$ scp -r Linino/* root@arduino.local:/root/
$ ssh root@arduino.local
root@Arduino:~# opkg update
root@Arduino:~# opkg install python-openssl


Then, I tried your commands and after I use the

Code: [Select]
pip install plotly

I am receiving error which is sofware causing abortion.

I have to install this plotly for my bachelor thesis. Can someone help me because I dont have any knownledge about linux.???

federicofissore


bolatu

1)

Code: [Select]
root@Arduino:~# cd ~/Downloads/plotly_yun
-ash: cd: can't cd to /root/Downloads/plotly_yun


Code: [Select]
root@Arduino:~# scp -r Linino/* root@192.168.1.102:/root/
root@192.168.1.102's password:
Linino/*: No such file or directory


I cannot access the file which is in my pc and copy that plotly_yun file

2)
Code: [Select]
opkg update
opkg install python-openssl #adds ssl support to python
opkg install distribute #it contains the easy_install command line tool (this can take some time)
easy_install pip #installs pip  (this can take some time)
pip install plotly #(this can take some time)


After pip install plotly, I am waiting like 2 minutes and it is giving error:

"network error : software caused connection abort"

P.S. I am using PuTTY to use terminal

federicofissore


Code: [Select]
root@Arduino:~# cd ~/Downloads/plotly_yun
-ash: cd: can't cd to /root/Downloads/plotly_yun



I don't thing this will work. You're running those commands from the yun. I guess you want to run them from your computer


After pip install plotly, I am waiting like 2 minutes and it is giving error:

"network error : software caused connection abort"


Sorry if I insist, but I don't want you to quote the error, I want the whole output. Copy and paste it please. The whole thing.

bolatu


Code: [Select]
pip install plotly

after this command it says:

Code: [Select]
unpacking plotly

But, It terminal freezes and after I waiting some time, I am getting this error:

"network error : software caused connection abort"

This error some kind of windows error. There is no error on the terminal. After I wait 3-5min, I am getting this error and terminal is crashing down and I cannot write any command to the terminal anymore.

federicofissore

Ahh that makes more sense. I guess your yun ran out of disk space. Try power it on again and run command
Code: [Select]

df -h

Can you paste the output?

If the issue is with disk space, I strongly suggest you to use and SD to expand it: follow this tutorial http://arduino.cc/en/Tutorial/ExpandingYunDiskSpace

bolatu

Code: [Select]
root@Arduino:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    7.5M      3.9M      3.6M  52% /
/dev/root                 7.0M      7.0M         0 100% /rom
tmpfs                    29.9M    144.0K     29.7M   0% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock3            7.5M      3.9M      3.6M  52% /overlay
overlayfs:/overlay        7.5M      3.9M      3.6M  52% /
/dev/sda1                 1.8G         0      1.8G   0% /mnt/sda1


Is it memory issue?

sonnyyu


Before pip install plotly:

Code: [Select]
df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                16130332    523296  14797772   3% /


After pip install plotly:

Code: [Select]
df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                16130332    527556  14793512   3% /


~4MB has been used.

show us your output of df.


Very likely, you have 3.6M and 4.0M is needed.

Go Up