SD library interferes with Mega Serial?

Im working on a device for a dance project that uses body-mounted IMUs to feed information to a processing sketch over bluetooth serial.

-There are three 3.3v Pro Minis (https://www.sparkfun.com/products/11114), each communicating with two IMUs over IC2.
-These feed data over Serial into a 3.3v Mega (Mega Pro 3.3V - DEV-10744 - SparkFun Electronics). Each one goes into its own serial port on the Mega (Serial1, Serial2, Serial3).
-The Mega gets data from each Mini every time through loop(), and then when Processing asks for it, sends data in nine-byte chunks to a Processing sketch through Serial via a bluetooth module (SparkFun Bluetooth Mate Gold - WRL-12580 - SparkFun Electronics).
-The Processing sketch reads the incoming data and moves stuff around on screen corresponding to input coming from the sensors.
-The Mega also monitors battery voltage levels over I2C using a Lipo Fuel Gauge (SparkFun LiPo Fuel Gauge - TOL-20680 - SparkFun Electronics)

Everything above works perfectly. No problems anywhere. Problems begin below:

I want to have an SD card logging the sensor data on-board in case the Bluetooth connection craps-out during a recording (and as a backup), and I want to use an RTC to give the data context in time. I want to prompt the data-logging to start and stop from Processing.

I have tested the RTC (https://www.sparkfun.com/products/10160) and the SD card (MicroSD card breakout board+ : ID 254 : $7.50 : Adafruit Industries, Unique & fun DIY electronics and kits). Both use SPI. Both work great with the Mega by themselves, and I did a test of logging the RTC data to the SD card each time through loop(), and that worked great too.

The problem is when I bring it all together. Just introducing the RTC works fine. I can send the time and date to Processing along with the IMU data. BUT: when I try to use the SD card, the info coming in to Processing FREAKS OUT! That is, I can move my IMUs around and see smooth changes on screen, and then when I enable logging to SD, everything starts moving randomly.

My code is below. Is there anything I'm missing? I have tried both the included SD.h library as well as the SdFat.h library. Below is the code for logging the actual data, but even if I change it to just write a constant variable to the SD card it still messes with the info going to processing.

Help?

#include <SD.h>
#include <SPI.h>
#include <Wire.h>
#include <RTClib.h>
#include <RTC_DS3234.h>
#include "MAX17043.h"

MAX17043 fuelGauge(20,10);

File myFile;

// Avoid spurious warnings                                     //This is from RTC Example code. Might not need it?
#undef PROGMEM
#define PROGMEM __attribute__(( section(".progmem.data") ))
#undef PSTR
#define PSTR(s) (__extension__({static prog_char __c[] PROGMEM = (s); &__c[0];}))

// Create an RTC instance, using the chip select pin it's connected to
RTC_DS3234 RTC(53);


char armR[8] = {0,0,0,0,0,0,0,0};
char armL[8] = {0,0,0,0,0,0,0,0};
char legs[8] = {0,0,0,0,0,0,0,0};

int led = 13;
int battStatus = 1;

boolean logging = false;

void setup() {
  pinMode(led, OUTPUT);   //LED setting
  pinMode(A7, OUTPUT);    //Mini 1 RST
  pinMode(A6, OUTPUT);    //Mini 2 RST
  pinMode(A5, OUTPUT);    //Mini 3 RST
  pinMode(10, INPUT);     //Fuel Gauge
  pinMode(A8, OUTPUT);    //SD Card
  
  digitalWrite(A7, HIGH);
  digitalWrite(A6, HIGH);
  digitalWrite(A5, HIGH);
  
  SPI.begin();
  RTC.begin();
  
  SPI.setDataMode(SPI_MODE0);
  
  
  // initialize all serial ports:
  Serial.begin(38400);
  Serial1.begin(38400);
  Serial2.begin(38400);
  Serial3.begin(38400);
  
  fuelGauge.begin();
  fuelGauge.setAlertThreshold(3);
  
  if (!SD.begin(A8)) {
    //Serial.println("initialization failed!");            //Indicate SD init error
    return;
  }
  
  
  
}

void loop() {
  
  const int len = 32;
  static char buf[len];

  DateTime now = RTC.now();
  SPI.setDataMode(SPI_MODE0);
  
  battStatus = digitalRead(10);
  //Serial.print(battStatus);
  if (battStatus==0) {
                                           //Battery Alert here 
  }
  
  
  
  Serial1.write(1);                        //Get data from ProMinis
  if (Serial1.available()) {
    
    int starter = (int)Serial1.read();
    
    if (starter==201) {
      Serial1.readBytes(armR, 8);
    }
    
  }
  
  Serial2.write(1);
  if (Serial2.available()) {
    
    int starter = (int)Serial2.read();
    
    if (starter==202) {
      Serial2.readBytes(armL, 8);
    }
    
  }
  
  Serial3.write(1);
  if (Serial3.available()) {
    
    int starter = (int)Serial3.read();
    
    if (starter==203) {
      Serial3.readBytes(legs, 8);
    }
    
  }
  
  
  if (logging) {                                         //logging the data to the SD card
    myFile = SD.open("dance1.txt", FILE_WRITE);
    now.toString(buf,len);
    //Serial.println(buf[0]);
    if (myFile) {
      //myFile.println(now.toString(buf,len));
      for (int i=12; i < 20; i++){
        myFile.print(buf[i]);
      }
      myFile.print("-");
      myFile.print("A:");
      for (int i = 0; i < 8; i++) {
        myFile.print((byte)armR[i]);
        myFile.print(",");
      }
      myFile.print("B:");
      for (int i = 0; i < 8; i++) {
        myFile.print((byte)armL[i]);
        myFile.print(",");
      }
      myFile.print("C:");
      for (int i = 0; i < 8; i++) {
        myFile.print((byte)legs[i]);
        myFile.print(",");
      }
      myFile.println("#");
      
      myFile.close();
    } else {
      //Serial.println("error opening file");       //Was there an SD error?
      logging = false;
      Serial.write(223);
      for (int i = 0; i < 8; i++) {
        Serial.write(1);
      }
    } 
  }
  
  
                                                          //Inout from Processing
  if (Serial.available()) {
    int recd = Serial.read(); 
    if (recd==49) {                                      //received a "1" ---> send position data!
      Serial.write(201);
      for (int i = 0; i < 8; i++) {
        Serial.write((byte)armR[i]);
      }
      
      Serial.write(202);
      for (int i = 0; i < 8; i++) {
        Serial.write((byte)armL[i]);
      }
      
      Serial.write(203);
      for (int i = 0; i < 8; i++) {
        Serial.write((byte)legs[i]);
      }
      
    } else if(recd==91) {                                 //received a "[" ---> reset first mini
        digitalWrite(A7, LOW);
        delay(1000);
        digitalWrite(A7, HIGH);
    } else if(recd==113) {                                //received a "q" ---> check battery level and report
        float batt = fuelGauge.getBatteryPercentage();
        //gaugeOutput();
        /*
        Serial.print((int) floor(batt));
        Serial.print(".");
        int frac = (batt - int(batt)) * 100;
        Serial.println(frac);
        //*/
        ///*
        Serial.write(220);
        Serial.write((byte) floor(batt));
        Serial.write(46);
        int frac = (batt - int(batt)) * 100;
        Serial.write((byte) frac);
        for (int i = 0; i < 5; i++) {
        Serial.write(1);
        }
        //*/
        
        if (batt<=3.00) {
         //Do emergency stuff here 
        }
    } else if(recd==32) {                                 //received a SPACEBAR ---> start/stop logging!
        logging = !logging;
        if (logging) {
          Serial.write(221);
          for (int i = 0; i < 8; i++) {
            Serial.write(1);
          }
        } else if (!logging) {
          Serial.write(222);
          for (int i = 0; i < 8; i++) {
            Serial.write(1);
          }
        }
    }
  }
  
  
  
  //delay(1000); 
}


void gaugeOutput() {
  Serial.print("Version: ");
  Serial.println(fuelGauge.getVersion());
  Serial.print("Alert Threshold: ");
  Serial.println(fuelGauge.getAlertThreshold());
  Serial.print("Alert Threshold Register Version: ");
  Serial.println(fuelGauge.getAlertThresholdRegister());
  Serial.print("Battery Voltage: ");
  Serial.println(fuelGauge.getBatteryVoltage());
  Serial.print("Battery Percentage: ");
  Serial.println(fuelGauge.getBatteryPercentage());
  Serial.print("Is Alerting? ");
  Serial.println(fuelGauge.isAlerting());
  Serial.print("Is Sleeping? ");
  Serial.println(fuelGauge.isSleeping());
  Serial.print("Is Sleeping Register Version? ");
  Serial.println(fuelGauge.isSleepingRegister()); 
  Serial.println("");
}

Try using fat16lib's SDfat.h library instead.
Not sure how to assign the chip select pin, will have to do a little reading.

Maybe here:

I found opening and closing the file can cause delays. Have you tried opening the file when you enable logging, and close it when you disable logging? Then just write to the SD in between. You might want to include a SD.flush() call after your write to save the data to the SD. Just a thought...

I just tried that, and got the same results.

The strange thing is that once I send it the "start logging" command and it starts misbehaving, then it doesn't acknowledge my subsequent "stop logging" command. That's why I think it's interfering with the Serial communications somehow.

SurferTim:
I found opening and closing the file can cause delays. Have you tried opening the file when you enable logging, and close it when you disable logging? Then just write to the SD in between. You might want to include a SD.flush() call after your write to save the data to the SD. Just a thought...

Maybe too much SD write and not enough serial read? Try reading the serial port until it is empty.

// change this
  if (Serial.available()) {
    int recd = Serial.read(); 

// to this
  while (Serial.available()) {
    int recd = Serial.read();

SurferTim:
Maybe too much SD write and not enough serial read?

Now it will acknowledge my "stop logging" command and go back to normal, non-logging behavior, but it still is erratic when logging.

I changed the processing sketch so that it would just dump the values it was getting into the terminal. It seems like when I'm not logging this code works as expected:

...
      Serial.write(201);
      for (int i = 0; i < 8; i++) {
        Serial.write((byte)armR[i]);
      }
...

And Processing receives: "201 -x-x-x-x-x-x-x-x"

But when I'm logging, Processing is getting repeats of the ID byte: "201-201-201-201-x-x-x-x" or "201-201-201-201-201-x-x-x"

Any idea why "Serial.write(201);" would send multiple values? or is the problem on the Processing side?

EDIT: I double checked in the Arduino terminal and it's definitely sending the ID byte again and again ("201-201-201-201-x-x-x-x"). So it's not a problem with Processing.

SurferTim:
Maybe too much SD write and not enough serial read?

I looked at your comment again and I decided to change all of the Serial.available() to while loops (on Serial, Serial1, Serial2, etc.) and that totally took care of it. I guess the serial buffer was getting flooded with input from the IMUs and when I enabled logging it just couldn't keep up and was sending those values at output? I'm not exactly sure, but this fixed the problem:

...
 Serial1.write(1);
  while (Serial1.available()) {
    
    int starter = (int)Serial1.read();
    
    if (starter==201) {
      Serial1.readBytes(armR, 8);
    }
    
  }
  
  
  Serial2.write(1);
  while (Serial2.available()) {
...

Thanks SurferTim!

  Serial1.write(1);                        //Get data from ProMinis
  if (Serial1.available()) {
    
    int starter = (int)Serial1.read();
    
    if (starter==201) {
      Serial1.readBytes(armR, 8);
    }
    
  }

Send a byte. 62.5 nanoseconds later, begin reading the response. It doesn't work that way. You need to send the request for data and then wait for data to arrive. Since delaying until the data begins to arrive isn't a good idea, you should send the request for data in setup(), and then again AFTER getting a response.

The readBytes() method returns the number that were actually read. This is a blocking function, which you really should not be using.

Yeah, this was definitely part of the problem, thanks.

PaulS:
The readBytes() method returns the number that were actually read. This is a blocking function, which you really should not be using.

What do you mean that it's a "blocking function"? And why shouldn't I be using it? It seemed to me to be a more elegant way than doing this:

for (int i = 0; i < 8; i++) {
  armR[i] = Serial1.read();
}

It seemed to me to be a more elegant way than doing this:

You can only do that if you KNOW that there are 8 bytes to read. If you KNOW that there are 8 bytes to read, then you can use readBytes() (which does that).

If you do not KNOW that there are 8 bytes to read, the readBytes() function can take up to 8 seconds to return, since it waits a full second for each byte to arrive. If each byte arrives 998 milliseconds apart, you can wait almost 8 seconds for the method to return.

Blocking functions and speed do not go together.