Go Down

Topic: Understanding the Linux to Arduino communication process (Read 3463 times) previous topic - next topic

Robin2

A dialogue had developed in this EzScrn Thread which seems more appropriate here in the Yun section so I am hoping to continue it here.

For ultimate flexibility, I would think it's valuable to have the major part of the Python code simply read from STDIN and write to STDOUT.
...snip...
What you would need is a low level wrapper: some code that finds and opens the tty port, and redirects it to STDIN/STDOUT, then calls the core code.
I've been thinking some more about this and doing some reading.

As I understand it the normal startup process of the Yun links the serial port ATH0 to a Linux "terminal" (for want of a better description). That allows Arduino code to control the linux side by sending commands similarly to what one can do in an SSH terminal.

The only (?) linkage between Linux and Arduino is via ATH0 on the Linux side and Serial1 on the Arduino side.

When I disable the linkage to ATH0 in the startup process (so that my Python code can use ATH0) my Python code has access to the SSH terminal from which I start the Python program AND to ATH0. It can communicate with the Arduino over ATH0 and it can print debug messages on the terminal - just the same as if the Python code was running on my laptop.

I think what @ShapeShifter has been suggesting is to write some code so that STDIN and STDOUT for my Python program piggy-back on top of the connection to ATH0 established in the startup process.

But if I were to do that I believe I would lose the ability to have two streams of communication.

If I have got this all wrong, perhaps someone with more knowledge of the Linux - Arduino process would be kind enough to explain.


...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ShapeShifter

I think what @ShapeShifter has been suggesting is to write some code so that STDIN and STDOUT for my Python program piggy-back on top of the connection to ATH0 established in the startup process.
No, that's not quite it. But first:

Quote
But if I were to do that I believe I would lose the ability to have two streams of communication.
Correct, you would lose that second communications stream. But in most cases, that second stream wouldn't be there in the first place. You are manually launching the command from the SSH command line, so STDIN/STDOUT/STDERR for the process are connected to the SSH session. But once you get past the development stage, are you still going to be manually launching that process through the command line? Probably not, you're likely to use an automated method like rc.local to launch it so it starts whenever the system is booted. Once you do that, STDIN/STDOUT/STDERR are not going to be connected to anything useful. Or perhaps you will have the sketch launch it (discussed below) in which case STDIN/STDOUT/STDERR will already be automatically connected to ATH0.

Which brings us back to the original point, piggy-back STDIN/STDOUT on top of the ATH0 connection: what I was saying is that if you write your process to use STDIN/STDOUT as the main I/O channel, for example, by sending all your output through Python print statements, it will automatically be connected to the right communications stream in most cases, and can be manually connected in the more unusual cases.

In the standard configuration, what's happening is that Linux is running a command interpreter on ATH0. When the sketch starts the Bridge, it is literally typing a command to the command interpreter to run Python and execute the Python code to handle Bridge communications. In this case, the STDIN/STDOUT of the process is connected to ATH0. Nothing special is needed to open that serial port connection.

Now, if the sketch then launches another process using the Process class, that request is sent to the Python side of the Bridge code, a process is spawned, and the STDIN/STDOUT of that process is connected to a net socket, the other end of which is connected to the Process object in the sketch. Anything that the spawned process sends to STDOUT is available to be read from the Process object in the sketch, and anything written to the process object in the sketch is available to the spawned process by reading STDIN.

Now, say you don't want to use the bridge. If you don't disable the command processor on the Linux side, the sketch can start your Linux side process by sending a command to start the process through the serial port. You would send the exact same sequence of characters that you would type on the SSH command line. By sending those characters out Serial1, you are literally typing that command out on the command interpreter connected to ATH0, and the process will be started in exactly the same way. The difference is that once running, STDIN/STDOUT of the process will automatically be connected to ATH0 and any writes to STDOUT will be available by reading Serial1, and anything written to Serial1 will be available to the process by reading STDIN.

The key to it all is that here are three different scenarios that communicate under three different connections, with absolutely no change to the code of the process. In each case, the process simply reads from STDIN and writes to STDOUT:
  • Process: communicates with the sketch through a Process object when it is launched by the Process object using the Bridge.
  • Serial: communicates with the sketch over ATH0 when it is launched by the sketch writing the name of the process to Serial1.
  • SSH: communicates with you over the SSH connection when it is launched by you typing the name of the process on the SSH connection.

That's just three examples. In reality, the process can communicate over any stream by redirecting STDIN/STDOUT to that stream. But in most cases, you don't have to do that manually, it is automatically done by the system (like in the examples above.)

Just as the Linux process can handle a variety of communications methods without change by using the abstraction of STDIN/STDOUT, the sketch could also use a variety of communications methods by being written to use the underlying Stream object from which the hardware serial, software serial, Process, network client, and other communications classes are derived. We talked about that in the other thread.

Now, you express the desire to have two communications streams: ATH0 and the SSH shell. That throws a wrench into the equation. I assume that you are using the SSH connection for debug messages? If they are output only, that could possibly be handled by writing to STDERR, and having STDERR connected to the SSH session. That way, STDIN/STDOUT would go to the sketch (using whatever communications method) and STDERR would go to the SSH session. You could accomplish this by using redirection operators in the command line that launches the process, so that STDIN and STDOUT go to /dev/ttyATH0, while leaving STDERR alone so it stays with the SSH connection.

What are you doing with the SSH stream connection (which is currently connected to STDIN/STDOUT)? Maybe there is a better (or at least more generic) way to handle it?

Yes, it does add a little more complexity to the way of thinking, and in some cases might involve a little extra code. In that way, it may not be appropriate for something targeted at rank beginners (as we mentioned in that other thread.) But it does provide a very powerful and generic way of handling I/O, one which is a standard programming model (for good reason!) and one which opens up a wide range of possibilities with no additional coding.



And a final unrelated comment:

Quote
The only (?) linkage between Linux and Arduino is via ATH0 on the Linux side and Serial1 on the Arduino side.
That's the normal connection, but not the only one. There is also a handshaking line between D7 on the Arduino side, and GPIO22 on the Linux side. This is normally disabled (high impeadence on both sides) but can be enabled in code.

There is also an SPI connection between the two processors. This is also disabled by default, but could also be used (and is used when loading sketches over the network.)

Robin2

Thank you again @ShapeShifter. More food for thought and much useful clarification.

In my way of thinking so far having the two streams of communications has always seemed important - or at least very valuable when they are available "for free".

Writing a PC program to communicate with an Uno over the USB connection makes for a lot of extra complexity if the Arduino is allowed to send debug messages as well as "proper" data. But it would be utterly pointless for a PC program to send its debug messages to an Arduino :)

I had thought (briefly) of creating an automated startup process for the Python code but I don't plan to do that because of the extra complexity for a novice user if it goes wrong, or if s/he does not want that.

I get the impression from your comments that your mindset favours the Arduino code starting the Python code. But my mindset is very much the other way round. What I have in mind is that a user can write the exact same code for a Yun as for an Uno - only needing to change Serial to Serial1 with the text editor. And, for the avoidance of doubt, I mean the sort of Uno code that is generally used by Arduino beginners, not some "advanced" Uno code to make it compatible with a Yun.

I would also like to think a user who had developed a simple Python program to link his laptop to her Uno could use that code unchanged on the Yun - just changing from (say) ttyACM0 to ttyATH0.

None of that is intended as criticism of your very useful input - just an attempt to explain where I am coming from.

I will certainly give your comments a lot more study.

Thanks

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ShapeShifter

In my way of thinking so far having the two streams of communications has always seemed important - or at least very valuable when they are available "for free".
I certainly understand the benefit of having a debug interface - on every embedded project I do I try to make sure there is a serial port just for debugging output and for entering debugging commands (I always add a small command interpreter that lets me see status, change configuration settings, manually control operations, etc.)

Quote
I get the impression from your comments that your mindset favours the Arduino code starting the Python code. But my mindset is very much the other way round. What I have in mind is that a user can write the exact same code for a Yun as for an Uno - only needing to change Serial to Serial1 with the text editor.
As far as starting the Python process on a Yun, what's worse: making the new user deal with the Linux command line to manually start the Python process every time they want to run? Or simply adding a line of code to the beginning of the sketch and not have to worry about it after that? After all, they still have to make changes (potentially many of them) to go from Serial to Serial1. What's one more?

Quote
I would also like to think a user who had developed a simple Python program to link his laptop to her Uno could use that code unchanged on the Yun - just changing from (say) ttyACM0 to ttyATH0.
And if the Python was written to use STDIN/STDOUT, there would need to be no changes at all.

And that's true even if you manually start the Python from the command line, it just changes the command you would use to launch it: you still wouldn't have to change the code since the command line would be the only thing changes as the communications device changes.


I will certainly concede that it's not an easy design decision. There are several factors to consider. going to STDIN/STDOUT makes some portions easier (no need to edit for ttyACMO/ttyATH0/COMx) and makes other things more complicated (having a separate debug stream.) It's not clear-cut in either direction.

I guess I just don't like the idea of hard-coding a serial port name, especially since you know it will have to be changed based on operating system (Linux vs Windows vs Mac vs Yun) and also because it can change from run to run (the COMx port number under Windows will often change depending on which physical USB socket you use.)

Robin2

As far as starting the Python process on a Yun, what's worse: making the new user deal with the Linux command line to manually start the Python process every time they want to run? Or simply adding a line of code to the beginning of the sketch and not have to worry about it after that?
But this assumes they are more familiar with Arduino than with Linux or with PCs and Python.

In any case I don't, in principle, like hiding the system from the user. It causes too many problems when the hiding-process fails or does not meet the user's need. If you think of the time needed to develop the hiding-process - I would prefer to spend that time teaching the user. Give a man a fish and he can feed his family for a day. Teach him to fish and he can feed them for his life time. See also the law of leaky abstractions. There is no shortage of them in the Arduino sphere.

To be perfectly honest, if it had been my decision there would never have been a Bridge system. I would have  considered it perfectly reasonable for people to use Serial1 given that they are already completely familiar with that sort of thing when using an Uno or Mega.


Quote
I guess I just don't like the idea of hard-coding a serial port name,
There is no question of that. My Python code presents the ports to the user so s/he can choose the correct one. I was just using fixed names for illustration.

As usual, your contributions are much appreciated.


...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ShapeShifter

But this assumes they are more familiar with Arduino than with Linux or with PCs and Python.
Correct. It was my assumption you were trying to accommodate the naive Arduino user, and that is why you wanted to minimize Arduino code changes. It is also my assumption that the naive Arduino user is likely also naive in other computer environments, particularly Linux. If the user is comfortable in Linux, I think they would not be scared by changing a few lines of Arduino code; and as a corollary, the user that is afraid of changing a few lines of Arduino code is likely terrified by a Linux command prompt.

Quote
To be perfectly honest, if it had been my decision there would never have been a Bridge system. I would have  considered it perfectly reasonable for people to use Serial1 given that they are already completely familiar with that sort of thing when using an Uno or Mega.
I don't disagree with you there. But you are looking at it from the point of view of an Uno user talking to a PC over the USB port. The bridge is not necessarily designed for that, it is designed to simplify the transition of someone moving from an Uno with an Ethernet or WiFi shield who wants to make network connections from the sketch. You want to use the Arduino processor to communicate with the Linux side like it is PC. The Bridge is primarily designed to allow the Arduino processor to talk to other computers over the network. I think that's why you have a philosophical conflict with the Bridge philosophy.

Robin2

But you are looking at it from the point of view of an Uno user talking to a PC over the USB port. The bridge is not necessarily designed for that, it is designed to simplify the transition of someone moving from an Uno with an Ethernet or WiFi shield who wants to make network connections from the sketch. You want to use the Arduino processor to communicate with the Linux side like it is PC. The Bridge is primarily designed to allow the Arduino processor to talk to other computers over the network. I think that's why you have a philosophical conflict with the Bridge philosophy.
I wish I had been able to make my point so clearly. :)

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ShapeShifter

See also the law of leaky abstractions.
I finally got around to reading this today, it's an interesting article. I've felt that way for a long time, I just didn't have a fancy name for it.

Almost 25 years ago, my boss wanted me to come up with a GUI to display RADAR pulse trains, display pulse processing results, and allow visual analysis of the data. He wanted me to develop it on his Mac because it had a 21" high resolution display, while all of the PCs around still had CGA or basic VGA graphics, which were primitive in comparison. I had never done any Mac development, but he had a copy of ZBasic that abstracted a lot of details. It made the easy stuff very easy, but the hard stuff was either very hard or downright impossible. There were certain UI elements that he wanted me to implement, but it just couldn't be done within the ZBasic framework.

He was dismayed when I said we needed to get the official CodeWarrior object oriented C development environment, as well as a tall stack of "Inside Macintosh" manuals from Apple. It was all rather expensive, compared to ZBasic, and it took time to rewrite the code using the new language and programming paradigm. But the results were that I could finally do all of the things that couldn't be done before. ZBasic made it very easy as long as what you wanted to do was what the language provided, but as soon as you wanted something to look different or do something more advanced, you were stuck because the low-level APIs were simply not available.

I see the Arduino world as being very similar: it is several layers of abstraction, starting with hardware abstractions (pin numbers and digitalRead/digitalWrite), continuing with software abstractions (setup() and loop() instead of main()) and compiler abstractions (pre-processing the .ino file to build include paths and generate function prototypes.) And then there are all of the libraries that make high level operations easy.

I think all of this is wonderful, as it lowers the entry barrier and allows beginners easy access to embedded programming. These abstractions allow useful programming without the need to study processor data sheets, learn low-level programming constructs, program device control registers, etc. I'm sure that there are lots of people who are doing wonderful things with embedded programming who would've never gotten started without these friendly abstractions to lead the way. Some have moved on to more advanced development environments, some have stayed with Arduino and are pushing the envelope with new techniques and libraries, and some are content to stay in Arduinoland and keep using the abstractions.

I think this is the most interesting passage in that article:
Quote
The law of leaky abstractions means that whenever somebody comes up with a wizzy new code-generation tool that is supposed to make us all ever-so-efficient, you hear a lot of people saying "learn how to do it manually first, then use the wizzy tool to save time."
It's a very valid point. A lot of the questions we see on this forum are because of the lack of understanding of "how to do it manually." Even with the abstractions, you need to understand what you're really doing, even if you don't plan on becoming more advanced. In this way, the Arduino abstractions can be a hindrance. But I still think that overall, it's good that the abstractions are there, as it encourages a lot of people who wouldn't be otherwise involved in the hobby, and it's also a practical solution for those quick projects that live well within the constraints and don't need something more advanced.

I've been doing this for many years, and I've done many embedded projects ranging from small and simple to very large and complex, and with commercial, industrial, medical, and military applications. I've just about done it all from mainframes to microcontrollers, from GUIs to faceless embedded devices. I'm very comfortable working on a bare metal project where I hook into the power up reset vector and handle everything myself with no support libraries.

I think I can do pretty much anything with an embedded processor. And yet, I've had several personal projects that I've wanted to work on for my own purposes, and never did because it was just too much work to build a board and implement all of the low-level interfaces, even though I've done it hundreds of times before. Then I discovered Arduino. I don't need all of the abstractions, and yet they let me whip together a project in an afternoon that would otherwise take me days or weeks, and therefore never get done.

Are there limitations and side effects to the abstractions? Sure there are! But on the whole, I think it's good that they are there, since they open up so many doors. (It's just unfortunate that some folks are going to get a black eye as they walk into the door frame while they try to pass through.) To paraphrase the old saying about love: It's better to have programmed with Arduino and lost, then to have never programmed at all.  :smiley-mr-green:

This has turned into more of a philosophical discussion than a technical one. But, while your original question was about the communications process, I think you were really trying to understand the philosophy behind it. So in that regard, I think this discussion has been worthwhile. At least, I'm enjoying it, as it's making me think, and I hope it's doing the same for you.

Robin2

I'm only an amateur programmer but it is something I find easy to grasp from the first time I came across it -  a long time ago (1970 ?). Then when the Apple II and IBM PC and clones came out it was a long time before I could afford one.

Of course programming would be very difficult without abstractions - including such things and C++ and Python.

But it seems to me there are people (programmers) whose principal interest is creating abstractions - perhaps as a demonstration of how clever they are. I got very frustrated by some of the common ones in the Ruby world. You can find pages and pages on the web about "how do I ..." or "I did ... and nothing happened". And when I figured out for myself how to do the task without the abstraction it was actually quite straightforward. And an explanation would have taken a fraction of the effort of writing the abstraction - as well as sharing (rather than hiding) knowledge.

In the OpenSource world things are made much more complex by the almost universal unwillingness to write comprehensive (or any) documentation. If there is "documentation" it is one or two examples rather than an explanation of what can and cannot be done. The "cannot" is often the most useful information.

I guess I have no argument against a system that allows someone who has only a basic grasp of Arduino programming to access the power of the Yun Linux system using familar concepts.

But I do object to the fact that the Yun developers made no easy way to not use the Bridge, and don't market the Linux side as a small alternative to an RPi.

Also, from what I can see, many people are using Uno's with Ethernet shields who are already very competent with Web programming. The Yun seems ideal for them - it can run lots of the PC software they are already familiar with.

If you are not familiar with Joel Spolsky's blog it is well worth a read. Another guy worth reading is Jeff Attwood (Coding Horror). Spolsky and Attwood created StackExchange.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ShapeShifter

I am in 100% agreement with you. Especially this part:

In the OpenSource world things are made much more complex by the almost universal unwillingness to write comprehensive (or any) documentation. If there is "documentation" it is one or two examples rather than an explanation of what can and cannot be done. The "cannot" is often the most useful information.
A big part of my job is writing documentation. I hate doing it, but it's a very necessary evil. Even if the customer doesn't require (and isn't paying for) documentation, I still have to write some level of documentation, if for no other reason than to remind of what the hell I was thinking when the customer comes back months or years later and asks questions.

The worst case I recall was a military contract years ago. The documentation requirements were so large and rigid that I spent something like 3 months writing the documentation, and 3 days writing the code. (And people wonder why military projects cost so much...)

While that was excessive, documentation in general is very important, and unfortunately all too lacking.

Go Up