Large delays in code when using Serial.print, etc.

Hi All,
I seem to be receiving huge delays with writing to the serial port in a teensy.
I'm currently running the following code:

void loop() {
 // put your main code here, to run repeatedly:
   for(int i = 0; i < 30; i++) { 
   bool sysReady = initiate();
   if(sysReady){
     readInput();      //read input from Pi
   }
 }
   draw();       //send entire array from Pi to LED's
    if(receivedData && shiftRegCount == 0) {
     readSensors();  //Send entire array to Pi
   }
}

This code continually grabs 30 bytes to be read from a device on the other end of the serial port, then draws one column on my matrix, then sends information back to the device on the other end of the serial.

When I don't send serial communication from the teensy to the other device (in this case my PC running pySerial) I get awesome framerates, maybe 60fps. However when I start sending data back, my program slows down to a crawl. See attached video:

Any suggestions?

The functions referenced in the above code are as follows:

//========================READ SENSORS==========================
void readSensors(){
  for(int i=0; i<yMax; i++){
    //
    //digitalWrite(S0,bitRead(i,S0));
    //digitalWrite(S1,bitRead(i,S1));
    //digitalWrite(S2,bitRead(i,S2));
    //digitalWrite(S3,bitRead(i,S3));
    //digitalWrite(4,0);        we can just ground this
    
    for (int j=0; j<xMax; j++){
        //Serial.write(char(255));
        //Serial.write(char((i)*10 + (j) + 1));
        //Serial.write(char(i+j+1));
        //Serial.write(char(0));
    }
  }
  Serial.println("This is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, ");
} 
void draw(){
  for(int j = 0; j<48; j++) {
      setGSData(j,ledArray[shiftRegCount][j%16][j/16]);
  }
    //Serial.println(j);
    //delay(50);  
  //Send out values and shift the shift register
    sendGSData(shiftSelect[shiftRegCount]);
  //Increment count
   ++shiftRegCount;
   if(shiftRegCount > 7) {
      shiftRegCount = 0;
   }
}

//=========================READ INPUT FROM PI==============================
void readInput() {
   inByte = Serial.read();  //reads first character in buffer from incoming serial data
   payloadByte = int(inByte);  //typecasts inByte from a char to an int
   if (inPacket) {  //If we're in a packet (have read start bit), then continue
     if(payloadByte == 0) {  //Checks to see if we're reading the stop bit
         inPacket = false;      //Sets inPacket to false, meaning we're not in a packet and will look for another start bit
         //Once we hit the stop bit, package has been read and we update the mask
         //ledArray[currentValue[0]][currentValue[1]][6]=255; 
         if(++updateCount > 159) {
           updateCount = 0;
           if(writeToLeftArray == 0) {
             writeToLeftArray = 1; 
             shifter = 3;
           }
           else {
              writeToLeftArray = 0; 
              shifter = 0;
           }
         }
     }
     else {
         if(payloadByte == 255) {  
           inPacket = true;
           packetPlacement = 0;
         }
         if(packetPlacement == 0) {                  //First piece of data in packet denote position on table
             int x = 0;
             int y = 0;
             if(payloadByte > 160) {
               payloadByte = 160; 
             }
             xyconvert(payloadByte-1, &x, &y);       //Converts position to x-y coordinates
             currentValue[0] = x;
             currentValue[1] = y;
         }                                           //Next three bytes will denotes r,g,b brightness,
         else if(payloadByte == 1) {     //                      
             payloadByte = 0;
         }
         else if(payloadByte == 254) {
             payloadByte = 255;
         }
         
         if(packetPlacement > 0 && packetPlacement < 4) {
           if(writeToLeftArray == 1){                                                                  //Determines which array to write to
             ledArray[currentValue[0]][currentValue[1]][packetPlacement-1] = map(payloadByte,0,255,0,4095);            //Writing to first array
           }else{
             ledArray[currentValue[0]][currentValue[1]][packetPlacement+2] = map(payloadByte,0,255,0,4095);            //Writing to second array
           }
         }
         if(packetPlacement < 0 || packetPlacement > 3) {
             Serial.println("Error: Incorrect data format");
         }
         if(++packetPlacement > 6) {
           inPacket = false;
         }    
     }
   }
   else {
     if(payloadByte == 255) {  //ASCII start bit
         inPacket = true;
         receivedData = true;
         packetPlacement = 0;
     }
   }
 //Logic to flip from reading to left to reading from right and vice versa for data input from serial
}

I can't make sense of your code. If it is not very long can you post all of it. If it is long can you make a short version that illustrates the problem?

I noticed a commented out Serial.println(j); followed by a commented out delay(50);. If that delay is used its hardly surprising things are slow.

What baud rate are you using?

...R

  Serial.println("This is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, this is a test, ");

Maybe you need several dozen exclamation points to go along with this nonsense.

@Paul.
The number of bytes i'm sending per cycle via serial will be large. (approx 640bytes), that string is to emulate exactly that. The alternative is to repeatedly send char values through the function, but it also causes a similar delay.

@ Robin
115200 is what i'm using. There are no delays in my code, and i'm using the "This is a test, " repeatedly in my sending.

So, why are you surprised that sending 640 bytes takes time?

Why does readSensors() write to the serial port? Nothing in its name suggests that it is proper for it to do so.

readSensors is more complex than just writing to the serial port. In reality it will have a function that will be called "send to Pi" or similar that will do this, but please lets not focus on the semantics of my code.

Even when I write small amounts of data (say 8 bytes) per cycle, it will still cause flicker (as seen in the video).

@mHo2, you are not providing the sort of comprehensive picture of your project requirement that would enable me to help.

At 115200 baud the Arduino should send or receive about 11000 bytes per second so 640 bytes should take about 60 millisecs. I know from trials I carried out a few months ago that this is achievable.

...R

Robin2:
@mHo2, you are not providing the sort of comprehensive picture of your project requirement that would enable me to help.

At 115200 baud the Arduino should send or receive about 11000 bytes per second so 640 bytes should take about 60 millisecs. I know from trials I carried out a few months ago that this is achievable.

...R

Hey Robin,
Sorry about that. I'll try to fill you in without wasting too much of your time.
The end goal here is to have a multiplexed RGB LED array (16x10). This incorporates the following:
160 common anode RGB LED's
3x TLC5940
2x 595 shift registers with Transistors for current supply.

When I loop through and 'draw' to the screen I am shifting the register and setting which anodes to turn on with the TLC5940.

Inbetween my draw function I am both sending and receiving data on the microcontroller.
This data is being sent over serial to pySerial to be dealt with. (in this came a custom python game).
The data being SENT from the uC is proximity sense data of 160 IR photodiodes
The data being RECEIVED by the uC is which LED's to turn on and at what intensity.

Hope that helps?

EDIT:
Further more, it seems as though as long as i'm not writing my serial data to python, this code works flawlessly (i.e. just writing to the serial monitor). What i'm thinking is one of two things:
1: Serial is blocking and python cannot read the data as quickly as the serial monitor
2: Something is messed up in the python code

2 seems very unlikely as the reading and writing is done in separate threads. Furthermore the code runs fine if i'm not writing to python.

1: Serial is blocking

The outgoing buffer is 64 characters long. When it gets full, Serial.print(), etc. block until there is room.

and python cannot read the data as quickly as the serial monitor

Read? Probably. Read AND interpret? Possibly not.

@PaulS
Alright, so that seems like the likely culprit. Is there any way to have a pseudo multithreaded type of code within the microcontroller? It seems like that's the only way to get around this blocking issue.

I don't believe that Python is incapable of reading data quickly enough - the processor in a PC is 1000 (?) times faster than the Arduino. Of course your particular Python program may not be written to get the best throughput.

If you need to break up the Arduino transmissions into smaller chunks I suggest you send (say) 60 bytes and then wait for an acknowledgment from the PC before sending the next part. In reality, however, this will just slow things down compared to what could be achieved with a suitable Python program.

I know very little about Python - I use JRuby - and I wonder if it has the concept of background Threads. I use a Thread in JRuby to read and transmit data via the serial port independently of the running of the main user-facing program.

...R