Ash: Slowing it down to prevent overflow on 32u4's serial. [solved]

Working on an open source haptic interface
More info about the project for the curious, here->Neotype: Haptic Computing | Hackaday.io

The project uses the premise of using key combinations or "chords" to get every
character with ten keys.
With the same principle, combinations of vibratory motors are used
to display letters haptically in a way that is "felt" as opposed to heard or seen.

Goal: Get a haptic output from Yun's command line.
Would be nice to cat files and get used to feeling the chords.

Problem: The serial buffer overflows before the 32u4 has time to haptically display.
Each character takes between 100-1000 ms to display in a human readable way.
The serial buffer can only take 64 bytes before it overflows.
In this way the 32u4 hangs on the output result of runShellCommand("ls");
on the 64th char. By the time one letter is printed the buffer has run over.

Hypothetical Solution (maybe there is a better): Slow down terminal(ash) output.

Question: How can ash output to the 32u4 be slowed down on the linux side?

An alternative solution: software handshaking

If you send a control-S (XOFF) to the Linux side, it will stop sending output. Send a control-Q (XON) to restart it.

The general idea is that you monitor how many characters are already in the serial buffer (Serial.available())

The standard way of doing this is to have two threshold values: high-water and low-water. When the number of characters already in the buffer reaches or exceeds the high-water mark, the buffer is getting too full so send the XOFF character to stop the incoming data flow. As the buffer is drained by your output process, when the number of characters in the buffer reaches or falls below the low-water mark, then the buffer has enough room, so send the XON character to resume receiving data.

The proper values for the high-water and low-water thresholds will depend on the size of the data buffer, how quickly the sending process (Linux in this case) responds to the XON/XOFF, the communications baud rate, and how often you are checking the available space.

Normally, when I implement something like this, I'm handling it in a serial data receive interrupt routine. That means I can check the available space on each incoming character (and not have to poll the free space) and I can respond very quickly if getting full. In this case, I can usually set the high-water mark very close to the end of the buffer.

With the Arduino Serial library, where you don't have direct access to the serial interrupt, you will have to frequently poll Serial.Available() for the amount of space used. In this case, you will probably have to set your high-water mark lower. You want to make sure that the number of characters that could possibly be received between polls is less than the amount of room in the buffer.

For example, at 250000 baud, it takes about 0.04 milliseconds (40 microseconds) to transmit a character. That means a 64 character buffer can fill up in 2.5 ms. That's not much time! In practical terms, you would probably want to poll the available space about every millisecond, and set your high-water mark so that you leave a good 50 (or even 60?) characters of space.

To do this, you will probably need to poll Serial.Available() on every pass through loop(), and keep the other processing in loop() as short as possible, using the "BlinkWithoutDelay" concepts to maintain any required timing. Calling delay() inside of loop() will cause serious problems (unless you only do it when you sure that output has been stopped by XOFF.)

In fact, now that I've said that last sentence, maybe a workable scheme would be: (psuedocode, not compilable)

void setup()
{
   setup the serial port
   send an XOFF character to Linux
}

void loop()
{
   if (serial.Available())
   {
      read a character and process it.
      doesn't really matter how long it takes.
   }
   else
   {
      // No data is available
      send XON character to Linux
      delayMicroseconds(500)
      send XOFF chacracter to Linux
   }
}

The idea is that normally the character flow is stopped. If there is no data available, start the flow, wait a short time (less time than it could possibly take to fill the buffer) and then stop the flow. Process characters as long as something is there, and once empty start the flow for a short time to get another chunk of data to work on.

It's kind of a messy way to do it, and it's not the most efficient way to get high throughput. But without access to the serial data interrupt, it's a rather safe way to make sure you don't get overruns, and from the sound of your mechanical output device, it doesn't sound like throughput is a major concern.

This is a really great answer ShapeShifter, I have been thinking about this most of the week.
This is a good approach for what I'm trying to do.

One important question remains though. What are the values of CTRL_S & CTRL_Q?
In the bridge library I found that a def CTRL_C was used, but I have yet to find what it inherits from to see its value.

ShapeShifter:
For example, at 250000 baud, it takes about 0.04 milliseconds (40 microseconds) to transmit a character. That means a 64 character buffer can fill up in 2.5 ms. That's not much time! In practical terms, you would probably want to poll the available space about every millisecond, and set your high-water mark so that you leave a good 50 (or even 60?) characters of space.

Got the feeling this is going to be really tight in Tenkey. That is if anything else is going on.
Might be safe to say any water will be high water that might breach the sides of the tub before the next poll.
Will be interesting to test this to see what it would take for that to be a problem.

In the bridge library I found that a def CTRL_C was used, but I have yet to find what it inherits from to see its value.

found: CTRL_C = 3

This is the actual mapping for CTRL_C ( tested it in my terminal example )

It makes sense if certain commands are reserved in the 0-31 or 128-255 value number space

I'll have to test other values and report back unless anyone knows a reference

[quote author=Paul Beaudet date=1414881714 link=msg=1944774]Might be safe to say any water will be high water that might breach the sides of the tub before the next poll.
Will be interesting to test this to see what it would take for that to be a problem.
[/quote]

LOL! I love that analogy, really paints a picture. I think you're right on.

ASCII Table

Look in the third or fourth column and find a letter, like C. Then look in the same row in the first column for the corresponding control character. The whole first column is considered control characters.

CtrlS = XON = DC3
CtrlQ = XOFF = DC1

People typically call them XON/XOFF (which helps remember the meaning) but the characters are really called Device Control 3 and 1 (not so easy to remember.)

An alternate method might be to run a C program on the linux side that takes a file name and number of bytes to send as arguments. You then would not to be concerned about overruns.

It would read the file and print number of bytes then wait for a newline character until it sends the next block.
I have written in on my yun and I can send you the source and binary in a tar file if you email me.

steve_at_heggood.com

That is also a good solution. My first thought was that it wouldn't work well, because I was under the impression that he wanted to see the boot up messages on his interface. But as I re-read the OP I see he's mostly interested in reading files. So that would work.

If that's all that's wanted, it might be possible to use the Bridge library Process calls to run the cat command, and then read the response characters at whatever speed is needed. I haven't studied the Bridge code, but it seems to do its own buffering and handshaking.

BTW Paul: I keep forgetting to say that this sounds like an interesting project!

heggood:
An alternate method might be to run a C program on the linux side that takes a file name and number of bytes to send as arguments. You then would not to be concerned about overruns.

This is a good idea, same thing could be said about a python program, might make things a lot easier.
I did mention reading in particular to Illustrate an example of use. Inevitably however I'm interested in
building a wearable computer where full access to a shell is preferable. I imagine the problem will
end up being that, text based and haptic based shells are different things and I'll have to write my own.
Which leads back to this idea. Alas it will be easier to start with Shapshifter's idea for now. So I can
get something done by the end of the day.

ShapeShifter:
Look in the third or fourth column and find a letter, like C. Then look in the same row in the first column for the corresponding control character. The whole first column is considered control characters.

This was my next hunch! Thank you for confirming!

Got a bit distracted this weekend, but today I'll should get some form of a prototype put together.

Working and integrated into tenkey!

Pulled out the functions made based on Shapeshifter's advice

boolean serialBowl()
{ // keep the alphabits from overflowing
  static boolean printing = false;   
   //signal activity to outside loop
    if(hapticMessage(MONITOR_MODE))   //letter played or boot has occurred
    {
      byte incoming = Serial1.read(); 
      //read off a byte regardless
      if (incoming == 255){printing = false;} 
      //255 = -1 in byte land
      else if (incoming && terminalToggle(1))
      {
        printing = true;               
        //prevents stop case
        hapticMessage(incoming);      
        //set incoming byte to be played
        Serial.write(incoming);        
      }
    }
    if(Serial1.available() > 3){Serial1.write(XOFF);}
    //pause ash output to keep up
    else{Serial1.write(XON);} 
    //resume output of ash
  return printing;
}

boolean terminalToggle(boolean returnVar)
{
  static boolean terminalMode = false;
  if(returnVar){return terminalMode;} 
  //preclude toggle
  #ifdef YUN
    terminalMode = !terminalMode;
    //terminal mode possible on yun
  #endif
}

To see how these integrate with the rest of the code, the commit can be found
here-> yun version is now a haptic computer · PaulBeaudet/tenkey@fc3b02d · GitHub

Unfortunately its a bit of a messy commit because, I also cleaned up my botched attempt to achieve similar functionality
with bridge. Ended up saving about +10% of the space over that attempt.

Thanks for the feedback, great to hear that it's working!

I love your comments and function names, especially this pair! 8)