[SOLVED] Read output of Asynchronous Process

I'm making a little project where the Arduino has to do several things:

  • Every 5 seconds, check for an update from my website
  • Monitor a motion sensor
  • Monitor a button
  • When motion is detected, take a picture and upload it

and one thing that is annoying is that when I start an asynchronous process, I can see if it's running, but not get the output. How can I do this other that setting curl to output to a file, waiting for it to finish, and then read the file?

What is the asynchronous process? A shell command, an ash script, a Python script, etc?

You can read the asynchronous output with process.available() and process.read().

But some processes, like Python, buffer the output and don't send it back to the Process class until there is a full buffer. The answer, if you have control over the process, is to flush stdout periodically.

For example, if your process is written in Python, add import sys at the top of the file, and then call sys.stdout.flush() after sending important output that you want the sketch to see right away.

For example, here is a trivial asynchronous process implemented as a Python script:

#!/usr/bin/python

import time

forever = 1;

while forever:
  for i in range(10):
    print i,
    time.sleep(0.1)

It forever prints out "0 1 2 3 4 5 6 7 8 9 0 1 2 3 4..." fairly slowly. If spawned as an asynchronous process, process.available() will return zero for a very long time, and then will suddenly return a large amount of data all at once.

With the addition of those two lines mentioned above, it looks like this:

#!/usr/bin/python

import time
import sys

forever = 1;

while forever:
  for i in range(10):
    print i,
    sys.stdout.flush()
    time.sleep(0.1)

When spawned as an asynchronous process, a continuous stream of data is available.

In both cases, this is the sketch that spawns the process and looks for output, and echos anything received to the serial port

#include "Bridge.h"
#include "Process.h"

Process p;

void setup() {
  Serial.begin(19200);
  Bridge.begin();
  
  p.begin("/mnt/sda1/test.py");
  p.runAsynchronously();
}

void loop() {
  while (p.available() )
    Serial.print((char)p.read());
}

I'm using curl. Here is a snip:

Process p;
p.runShellCommandAsynchronously("curl -L http://example.com");

and I'm trying to get the output of it. One solution (that I should have done LONG ago, but I'm too lazy) is just to learn python. Also, one thing: when I don't run curl asynchronously, I get the output perfectly fine.

Anyways, I know enough python to do what I want to do.

lights0123: ::::SNIP:::: and I'm trying to get the output of it. One solution (that I should have done LONG ago, but I'm too lazy) is just to learn python. Also, one thing: when I don't run curl asynchronously, I get the output perfectly fine.

@lights0123, Perhaps it's not clear that running a process Asynchronously means that you will NOT get something back - unless you go out of your way to save that "data" somewhere. In the case of curl you can tell it where to put the data - but you'll still have to fetch it.

In case this is not clear, let me quote the documentation:

runAsynchronously()

Unlike run(), runAsynchronously() is not-blocking. Your sketch will continue to run while the Linux process runs in the background.

To wrap up, save the data somewhere and fetch it. And how long that takes in not determined, you'll have to check for the data and see if it completed as expected.

FWIW, asynchronous calls on the webbrowser usually have a "callback" - you have no such means here.

hope this all make sense to you. Jesse

jessemonroy650: Perhaps it's not clear that running a process Asynchronously means that you will NOT get something back - unless you go out of your way to save that "data" somewhere. In the case of curl you can tell it where to put the data - but you'll still have to fetch it.

I'm sorry, but that's wrong. That may be true in some web programming examples, but not for the Process class. The difference between Process.run() and Process.runAsynchronously() is as the reference you quoted indicates:

In case this is not clear, let me quote the documentation:

runAsynchronously()

Unlike run(), runAsynchronously() is not-blocking. Your sketch will continue to run while the Linux process runs in the background.

In both cases, whatever the Linux process writes to STDOUT is captured and retained by the Process class so that it can be read by Process.read() calls. The only real difference between the two calls is that run() waits for the Linux process to complete before it returns, while run Asynchronosly() doesn't wait. Running a process either way causes data to be returned in exactly that same way. There is no need for any kind of callback notifications.

Take a look at my example above: it works, and it doesn't do anything special to field notifications, or buffer data. The implicit pipe that redirects STDOUT of the process to the Process object takes care of all of it.

Hi, I’m having trouble with that sketch. Here is my code:

At /test.py:

#!/usr/bin/python

import time
import sys

forever = 1;

while forever:
  for i in range(10):
    print i,
    sys.stdout.flush()
    time.sleep(0.1)

and in Arduino:

#include <Bridge.h>
#include <Process.h>

Process p;

void setup() {
  Bridge.begin();
  Console.begin();
  while(!Console.connected()){
    analogWrite(10,255);
    delay(250);
    analogWrite(10,0);
    delay(250);
  }
  Console.println("Init");
  analogWrite(10,255);
  p.begin("/test.py");
  p.runAsynchronously();
}

void loop() {
  while (p.available() ){
    Console.print((char)p.read());
  }
}

What kind of trouble are you having? What are the symptoms?

Do you really have test.py stored at the root level of the Linux file system? Or is it at the root level of the SD card (a big difference!)

If you use SSH to connect to the Yun, and you type "/test.py" at the command line, does the script run and produce output?

I'm 100% sure it's at the root of the filesystem (not SD). I've typed in /test.py and it printed numbers. The problem is that when I connect via Console, it says "Init", but nothing else. NOTE: For some reason, printing something else causes it to suddenly print the output of the python file! NOTE 2: Ah. Printing a string (not a char) causes it to appear.

Where's the button to mark this as solved?

ShapeShifter:
I’m sorry, but that’s wrong. That may be true in some web programming examples, but not for the Process class. The difference between Process.run() and Process.runAsynchronously() is as the reference you quoted indicates:

::::SNIP::::

I stand corrected. From what you say, Process.read() is the call back. I confess, I have not used Process that much. :slight_smile:
Jesse

lights0123:
Where’s the button to mark this as solved?

Go to your first post, click the “More…” drop down on the bottom right of that post, and select Modify. You can then edit the first post, including changing the subject.

jessemonroy650:
I stand corrected. From what you say, Process.read() is the call back. I confess, I have not used Process that much. :)[/quote]
I think of a callback as a function you provide, and the system calls it when necessary (basically an event handler.) Process.read() is not such a function, it is a function provided by the system that your program calls to poll for and read data (exactly the way that Serial.read() behaves.)
Something like JavaScript is event driven, and you need to provide a callback when you do something asynchronously so you know when it’s done (or at least when there is some data ready.) An Arduino sketch is procedural, and it has to periodically poll to see when new data is available. The asynchronicity comes not from being event driven like JavaScript, but from actually running on two processors. (True multitasking. :sunglasses: )
Maybe this is what you are trying to say and I’m just being pedantic and keying on semantics. Or maybe it’s a different type of scheme than what you were thinking. Either way, from the view point of the sketch and the sketch’s programmer, reading from a process started with Process.runasynchronously() is done EXACTLY the say was as if the process were on another computer and you were reading it from the Serial port. (Which in a way is actually what is happening, it’s just the two computers are on the same board and there’s a few more layers of software involved.)

ShapeShifter: ::::SNIP::::

Maybe this is what you are trying to say and I'm just being pedantic and keying on semantics. Or maybe it's a ::::SNIP::::

YEP. :) ;) :D ;D :( :o ??? ::) :fearful: :sunglasses: :disappointed_relieved: :stuck_out_tongue:

Thanks to Jesse's recent bridge documentation post there is a simpler solution to the buffering issue. I noticed that in his discussion he mentions the -u option to run Python in unbuffered mode. Well, the same thing can be used for Python scripts that we launch asynchronously. When launching an asynchronous process, instead of just specifying the name of the Python script, we could use this instead for the command line:

python -u test.py

Note the addition of the -u option to Python.

But wait, there's an even easier way, which is applicable to launching an asynchronous process, and also works when running the script at the command line. And there is no need to remember to use the -u option all the time.

Here's an updated version of the test.py script mentioned earlier, can you spot the difference?

#!/usr/bin/python -u

import time

forever = 1;

while forever:
  for i in range(10):
    print i,
    time.sleep(0.1)

Note the addition of the "-u" to the first line. That first line is a hash-bang or she-bang that tells the Linux system what process to launch to run the script. In simple terms, when you just type the name of the file, the system reads the first line of the file looking for this hash-bang. If it is found, it builds a command string by pre-pending the line to the command you typed, to come up with the process it must launch. So, before, if I just typed "test.py" the system would add the command from the hash-bang and actually launch "/usr/bin/python test.py"

By adding the -u to the hash-bang, the system now comes up with the command string "/usr/bin/python -u test.py" and the output is now magically unbuffered.

Easy. 8)