problems calling python script on yun

:repo:

On button is pressed in Arduino Yun, I want to run a python script which is placed in root’s Poet dir (A dir I made in root dir to store all my scripts).

The python script is capable of reading a list of links from a (previously generated) txt file in the same dir and download texts from those links using
elinks
(terminal based browser) and store them in another txt file.

I’m running some
os.system
and
subprocess call
from within python to run some terminal shortcuts.

When I ssh and run a script(let’s say “link_data_extractor.py”) every thing works but whenever I run it from a Process call
or
runShellCommand()
call from within Arduino Yun, it doesn’t work … It only spits the beginning of the few python print lines and get’s stuck at from the point where it’s loading a file in the python program.

You can look at the python scripts from the above repo link.

here was my test Arduino script:

#include <Process.h>

void setup() {
  Bridge.begin();
  // Wait until a Serial Monitor is connected.
  while (!Serial);
  Serial.println("Serial has been established");
  dataBroadcast();
}
void loop() {
}

void dataBroadcast() {
  Serial.println("Starting the process of link extraction and poem generation");
  Serial.println("...It will take some time...");

  Process p;
  p.runShellCommand("python /root/Poet/link_data_extractor.py");

  while (p.available() > 0) {
    char c = p.read();
    Serial.print(c);
  }
  // Ensure the last bit of data is sent.
  Serial.flush();
  
  /*I ran PWD from here and saw that the current default directory is 
  /usr/lib/python2.7/bridge 
  */
}

Note: The Yun Boots from a micro SD card(Usual process followed for FS expansion as used in openWRT and it boots softly - using a 16GB card), not sure if I’ve to make PATH changes from python side…

Almost every time when a script runs from the command line and can access files, but does not run properly from a Process object, it's because the script is not using absolute file paths. You are using the full path name to run the script, but the script itself is not executing with /root/Poet as the current directory.

The first thing I would try is go to a different folder on the command line, like /, and try to run your script using the full path name. If it doesn't run, you've found your problem. You will either have to set the current directory at the beginning of your script, or use full explicit path names for all files.

tried them before only .. doesn't work .. Still would give it another shot ..

I don't have time to debug Yun's Bridge pyc files and check them their default run modes. It's so stupid >:( Why can't it just run shell commands like "shell"..

Okay here's the situation:

When I SSH into my Yun from terminal , I create a folder called "Poet", in the root dir (Which is default log-in dir after *ssh*ing). Here all my scripts are stored. Now for example say: in one of the scripts.

From Arduino process calls, when I do 'pwd' command, it showed me it's current dir is "/usr/lib/python2.7/bridge".

Problem is I tried through some other process calls like doing

Process p;
p.runShellCommand("pwd");
while (p.running());
p.runShellCommand("cd \"); // to get out of that bridge dir
while (p.running());
p.runShellCommand("cd /root/Poet");
while (p.running());
p.runShellCommand("pwd"); // to ensure the current path again.

It seems that after the first "pwd", it's not running other commands or is unable to get out of this default log-in dir from Arduino.

Also for in Python scripts for saving or loading files , the path I modified then is : "/root/Poet/filename..."

Also tried moving the whole Poet folder in /usr/lib/python2.7/bridge and then tried to run both:

  • python /usr/lib/python2.7/bridge/Poet/link_data_extractor.py
  • python Poet/link_data_extractor.py

The Python script then doesn't even run, meaning process cannot get into the Poet dir when it is kept inside bridge dir.

Note to test and help me: Can someone first move the fs to a microsd card and expand the fs (to mimic my situation), create a dir in root and run a simple python script that loads and saves data into some text file in that same dir. The python script to be executed by Process-calls of Yun magical Bridge

It would be a great help ..

Thanks

dattasaurabh82:
Problem is I tried through some other process calls like doing

Process p;

p.runShellCommand(“pwd”);
while (p.running());
p.runShellCommand("cd "); // to get out of that bridge dir
while (p.running());
p.runShellCommand(“cd /root/Poet”);
while (p.running());
p.runShellCommand(“pwd”); // to ensure the current path again.

What’s happening is that each time you start a Process object, either by using runShellCommand(), run(), or runAsynchronously(), you are starting a new Linux process. When the run is done, the process exits. No context is saved between runs, each stands alone.

Think of it this way: run your same four commands, but for each one start an SSH session, run one command, then exit the SSH session. Start a new SSH session for the next command. Would you expect your command sequence to work? Of course not.

If you want to run a series of commands in a row like that, and they depend on each other like your example does, you will have to do it in one Process call. Either combine them into one command line, or put them in a shell script and run that script on the Process call.

Also tried moving the whole Poet folder in /usr/lib/python2.7/bridge and then tried to run both:

  • python /usr/lib/python2.7/bridge/Poet/link_data_extractor.py
  • python Poet/link_data_extractor.py

The Python script then doesn’t even run, meaning process cannot get into the Poet dir when it is kept inside bridge dir.

I disagree with your conclusion. I think it means that there is a problem in your code somewhere that is still dependent on assumed file locations. I think it is false assumption that there is something special about the bridge directory that is preventing it from running, that assumption may be blinding you to the real causes.

You are on the right track for figuring this out. Keep your files (including the Python code) in /root/Poet, and make sure you can properly run your script from any folder. Don’t worry about your sketch or the Process class until this works.

You will have to make sure that ALL file references are fully qualified, or set the current directory at the beginning of the process. Note that as discussed above, you cannot use one Process call to set the directory, then another to run the script. It must all be done in the same call. Either set the current directory inside your Python call (using the os.chdir() function) or create a shell script to change the directory then call the Python script and run that shell script in the Process call.

Note to test and help me:
Can someone first move the fs to a microsd card and expand the fs (to mimic my situation), create a dir in root and run a simple python script that loads and saves data into some text file in that same dir. The python script to be executed by Process-calls of Yun magical Bridge

I do almost the same thing all the time and have never had a problem. The only difference is that I create the folder in /mnt/sda1/ and put the files there. For simple file accesses, I just use a full path name. For more complex accesses, I will set the current directory at the beginning of the script.

Think of it this way: run your same four commands, but for each one start an SSH session, run one command, then exit the SSH session. Start a new SSH session for the next command. Would you expect your command sequence to work? Of course not.

This information helps a lot @shapeShifter.

I'll give it a try in your proposed way and if stuck again will seek help.. Thanks for directing me ..

Okay with the help here .. I figured out that all my python os.system and other python os calls happens before the rest of python processes and subProcess calls from python doesn't even happen when I run it from Arduino Process calls .

Unless that now after fixing the path issues a simple python script runs; but that doesn't solve the problem I mentioned above.

When I ssh into Yun and run simply the python calls they run smoothly .. as the flow of the program , but when I use Arduino's process calls they break ..

why any idea?

dattasaurabh82:
Okay with the help here …
I figured out that all my python os.system and other python os calls happens before the rest of python processes and subProcess calls from python doesn’t even happen when I run it from Arduino Process calls .

I understand all of the individual words in that sentence, but I have no idea what the sentence is actually trying to say. Can you rephrase it? Preferably in a few shorter sentences rather one long sentence?

You say you’ve made some changes, can you post your current code?

The latest I see in this thread is that the sketch uses a Process class to run “python /root/Poet/link_data_extractor.py”.

And in your repo link, the very first things that ink_data_extractor.py does is:

# Open the file
link_file = open("links.txt", "r")

As mentioned before, this will not work. So either you haven’t updated the repo to show your latest edits, or you haven’t made the right changes. I’m not going to analyze the rest of the code if the first statement fails.

@ShapeShifter & @dattasaurabh82,

I apologize for not giving a clear example before.

The issue is when you SSH into the server, you have a certain ENVIRONMENT setup; and when you use process it is somewhat different.

Here are the examples you need.

0) Make sure you are on the same network as the Yun.

1) I have just ssh'ed in to my Yun. I give the command env

root@Arduino:~# env
USER=root
HOME=/root
SSH_TTY=/dev/pts/0
PS1=\u@\h:\w\$ 
LOGNAME=root
TERM=xterm
PATH=/bin:/sbin:/usr/bin:/usr/sbin
SHELL=/bin/ash
PWD=/root
SSH_CONNECTION=192.168.43.192 54326 192.168.43.237 22

Notice

PATH=/bin:/sbin:/usr/bin:/usr/sbin
SHELL=/bin/ash
PWD=/root

2) Now I modify the sketch File->Examples->Bridge->Process, as such; and run.

//p.begin("cat"); // Process that launch the "cat" command
  //p.addParameter("/proc/cpuinfo"); // Add the cpuifo file path as parameter to cut
  p.begin("env");

The output looks like this:

USER=root
OLDPWD=/
HOME=/root
PS1=\u@\h:\w\$ 
TERM=vt102
PATH=/bin:/sbin:/usr/bin:/usr/sbin
SHELL=/bin/sh
PWD=/usr/lib/python2.7/bridge

Now side by side:

PATH=/bin:/sbin:/usr/bin:/usr/sbin
PATH=/bin:/sbin:/usr/bin:/usr/sbin

Looks good, but

SHELL=/bin/ash
SHELL=/bin/sh

and

PWD=/root - ssh PWD=/usr/lib/python2.7/bridge - process env

Oops, what's the difference. Not much for SHELL, but it can make a difference sometimes. More on that later. But PWD, wow!!

3. Now let's try it with a simple python program:

root@Arduino:~# pwd
/root
root@Arduino:~# cat env.py 
import os

# https://docs.python.org/2/library/os.html#process-parameters

print os.environ;

Then we add to the sketch

  Process p2;
  p2.runShellCommand("python /root/env.py");
  while (p2.available() > 0) {
    char c = p2.read();
    Serial.print(c);
  }

What's the output (reformated for viewing.)

{
'TERM': 'vt102',
'SHELL': '/bin/sh',
'OLDPWD': '/',
'PWD': '/usr/lib/python2.7/bridge',
'USER': 'root',
'HOME': '/root', 
'PATH': '/bin:/sbin:/usr/bin:/usr/sbin', 
'PS1': '\\u@\\h:\\w\\$ '
}

Now all three:

PATH=/bin:/sbin:/usr/bin:/usr/sbin
PATH=/bin:/sbin:/usr/bin:/usr/sbin
'PATH':'/bin:/sbin:/usr/bin:/usr/sbin'
SHELL=/bin/ash
SHELL=/bin/sh
'SHELL': '/bin/sh'

PWD=/root - ssh PWD=/usr/lib/python2.7/bridge - process env 'PWD':'/usr/lib/python2.7/bridge' - process python

As should be visible now, when process is run - it changes the environment. The two (2) big changes are to SHELL and PWD

On SHELL, changes are not as obvious, but trying to fix this can be frustrating. For the most part, /bin/ash should be more capable than the standard /bin/sh, but the differences are scantily documented(see 12.) BusyBox ('01-current)). This is because /bin/ash was not document well, but was added to the Linux world without much supporting documentation; and because busybox, the actual implementation, is scantily documented as well. You can find more notes on my blog post about the YUN CLI.

NOTE: There is a compile time option "ASH_BASH_COMPAT". SEE 12.) BusyBox ('01-current). But there is no way to know if that was compiled in or not.

The best and easist thing to do is run your own shell script when running process, as is often the advice given.

On PWD, the difference is obvious, but the effects are more sutble. The first most obvious issue is that shell scripts that expect to be in /root or /home don't find things where they are expected.

This also mean other scripts have the same problem. They also expect things in certain locations, but they are not where expect. In the end, it means scripts cannot be developed on another system, such as a laptop or even on the Yun itself, but must developed from the Arduino processor using process.

Now the creators of bridge could have not changed PWD, and used a separate ENVIRONMENT variable for the bridge library and we would not be having this discussion. But they did what they did.

Questions? Jesse

Thank you so much for explaining this . Tis was very helpful.

I've not pushed the latest mods. (My bad).. My git is kind of broken only works on certain networks(Don't know why)..

I'll let you know as soon as I get time to update it ..

I've one more question(Since you explain very well and with basic knowledge I could understand a lot).

I want to ssh into another wrt router from my Yun with a script and do some task for example.. Generally to do so manually we do: ssh root@IP_ADDRESS

to do from openwrt(yun's ver) with a .sh file. I found out the need of empty(equivalent to normal Debain's expect).

or

sshpass

http://forum.arduino.cc/index.php?topic=253943.0

After trying both methods I failed.

Using sshpass in both /usr/bin and /usr/sbin folder gives permission denied to ssh while after installation of empty gives fileIO error.

even tried ssh key authentication method(Like we generally do for long-term ssh usage) https://wiki.openwrt.org/oldwiki/dropbearpublickeyauthenticationhowto http://crossbar.io/iotcookbook/Arduino-Yun-Basic-Setup/ https://yorkspace.wordpress.com/2009/04/08/using-public-keys-with-dropbear-ssh-client/

It still ask for password in this method.

Going :confused: nuts

dattasaurabh82: ::::SNIP::::

It still ask for password in this method.

Going :confused: nuts

I wrote these instructions for myself exactly one year ago. I think they still work.

Best of Luck Jesse

= NOTES = Date: 2014-12-22

1) Add your (computer) public key to:

/etc/dropbear/authorized_keys

then

chmod 600 /etc/dropbear/authorized_keys

2) ssh -q -o StrictHostKeyChecking=no root@arduinoIP_or_LocalName

-q - quiet, don't complain about things -o StrictHostKeyChecking=no - else it will complain not knowing the remote host

Reference: http://wiki.openwrt.org/doc/uci/dropbear http://wiki.openwrt.org/oldwiki/dropbearpublickeyauthenticationhowto

http://stackoverflow.com/questions/3663895/ssh-the-authenticity-of-host-hostname-cant-be-established

Put shebang line (#!/usr/bin/python) at every python file

nano link_extractor.py
#!/usr/bin/python
import os
import subprocess
...

Set every python file executable

chmod 755 link_extractor.py

put every thing (python and txt) into Yun's folder

"/usr/lib/python2.7/bridge/"

Test:

cd /usr/lib/python2.7/bridge/
./link_extractor.py

ATmega32u4 code:

#include <Process.h>
void setup() {
 Bridge.begin();  // Initialize Bridge
}
void loop() {
 Process p;              
 p.begin("link_extractor.py");      
 while (p.running());
 while (p.available()) {
    char c = p.read();
    Serial.print(c);
 }
 delay(1000000); 
}

Thank You every one .. I made it work .. I don't know Git is not working from my home and hence can't push latest modified code .. (Will look into it later.. Currently auto saving parallely into my hard-disk so code is at-least safe)-

One more problem guys ..

can't wrap my head around it .. It says in documentation.

say for a process p

process p;
// process does what-ever it is supposed to do
while(p.running());
// Here I want to blink an LED while the process is running.
// As a showcase to say that something is going on
// I don't want to use the runAsynchronously() as I want some blocking structures for next processes

how do I do it? i couldn't find much documentation about this function..

dattasaurabh82: // I don't want to use the runAsynchronously() as I want some blocking structures for next processes

Then it can't be done. You MUST use runAsynchronously() if you want the sketch to do something while the process runs.

You have only two choices for running a process:

  • run() or runShellCommand() which do not return to the sketch until the process is complete (sketch can't do ANYTHING else)
  • runAsynchronously() or runShellCommandAsynchronously() which returns to the sketch after the process is started and allows the sketch to keep running while the process is running.

The difference between the functions that include "ShellCommand" and those that don't are the shell command version expects a single command string, and combines the begin() and run() (or runAsynchronously()) calls into a single call which simplifies the code, but doesn't allow you to use addParameter().

When you start a process with run(), the run() function will not return until the called process is complete. The sketch will not be able to do anything else, because it will be stuck inside the run() function, although interrupts will still be serviced by the sketch - but if the interrupt sets a flag that is checked by the main code of the sketch, that main code will not run.

Since you want the sketch to do something else while the process is running, you must use runAsynchronously() which returns as soon as the process is started. That will allow you to do what you want:

Process p;
p.begin("foo");
p.addParameter("bar");
p.runAsynchronously();
while (p.running())
   blinkLED();

// At this point, the while loop has dropped out, so you know the process is complete.

That code will start the process, and then spin in a loop calling blinkLED() as long as the process is running. Once the process is complete, the while loop will exit, and execution will continue. Even though the runAsynchronously() call is not blocking, you have essentially created your own blocking mechanism with the while loop: you have blocked the execution of the sketch from reaching the last comment until the process is complete.

That is the answer to what I assume is your concern: you don't want another Process object to start another process until the first one is complete - the while loop does that. You have two choices: use run() and then not have a way to blink your LED, or use runAsynchronously() and add your own blocking mechanism.

Works like magic . :) .

Thanks a lot

Thank you a lot for this information.
Helped a lot with my shell vs procress testing.