Sketch Only Runs When Connected To USB

So for my project I am basically using a wireless mouse to control a wheeled bot with modified servos for motors. I am using a BotBoarduino with a USB Host shield, and this is the USB Host library I am using.

Everything runs fine while connected to my computer. But if I disconnect, nothing seems to run. Everything is being powered by battery so it shouldn't be drawing power from the USB, and I checked with a multimeter and I am seeing 5V from the hosted USB port while disconnected. Though the weird part is that if I write in code to directly command the servos, that will run just fine without being connected, its just the USB Mouse code that is stalling. I also experimented by plugging in the USB cord to my iPhone charger and it will run, so it's not actually communicating with the computer. Somehow is it checking for a USB connection?

Here is the sketch, note that I did comment-out everything trying to print to serial.

#include <hidboot.h>
#include <usbhub.h>
// Satisfy IDE, which only needs to see the include statment in the ino.
#ifdef dobogusinclude
#include <spi4teensy3.h>
#endif


#include <Servo.h> 
Servo servoL, servoR;
int posL, posR;

int y;

bool button;


class MouseRptParser : public MouseReportParser
{
protected:
	virtual void OnMouseMove	(MOUSEINFO *mi);
	virtual void OnLeftButtonUp	(MOUSEINFO *mi);
	virtual void OnLeftButtonDown	(MOUSEINFO *mi);
	virtual void OnRightButtonUp	(MOUSEINFO *mi);
	virtual void OnRightButtonDown	(MOUSEINFO *mi);
	virtual void OnMiddleButtonUp	(MOUSEINFO *mi);
	virtual void OnMiddleButtonDown	(MOUSEINFO *mi);
};
void MouseRptParser::OnMouseMove(MOUSEINFO *mi)
{
    //Serial.print("dx=");
    //Serial.print(mi->dX, DEC);
    //Serial.print(" dy=");
    //Serial.println(mi->dY, DEC);
    
    y = -1*(mi->dY);
};
void MouseRptParser::OnLeftButtonUp	(MOUSEINFO *mi)
{
    //Serial.println("L Butt Up");
    button = false;
};
void MouseRptParser::OnLeftButtonDown	(MOUSEINFO *mi)
{
    //Serial.println("L Butt Dn");
    button = true;
};
void MouseRptParser::OnRightButtonUp	(MOUSEINFO *mi)
{
    //Serial.println("R Butt Up");
};
void MouseRptParser::OnRightButtonDown	(MOUSEINFO *mi)
{
    //Serial.println("R Butt Dn");
};
void MouseRptParser::OnMiddleButtonUp	(MOUSEINFO *mi)
{
    //Serial.println("M Butt Up");
};
void MouseRptParser::OnMiddleButtonDown	(MOUSEINFO *mi)
{
    //Serial.println("M Butt Dn");
};

USB     Usb;
USBHub     Hub(&Usb);
HIDBoot<HID_PROTOCOL_MOUSE>    HidMouse(&Usb);

uint32_t next_time;

MouseRptParser                               Prs;

void setup()
{
    //Serial.begin( 115200 );
    //while (!Serial); // Wait for serial port to connect - used on Leonardo, Teensy and other boards with built-in USB CDC serial connection
    //Serial.println("Start");

    if (Usb.Init() == -1)
        //Serial.println("OSC did not start.");

    delay( 200 );

    next_time = millis() + 5000;

    HidMouse.SetReportParser(0,(HIDReportParser*)&Prs);
    
    
    servoL.attach(8);
    servoR.attach(7);
}

void loop()
{
  Usb.Task();
  
  /*
  servoR.write(91);
  delay(1000);
  servoR.write(90);
  delay(1000);
  */
  
  if( button == true )
  {
    posL = 90 - y;
    posR = 90 + y;
  }
  else
  {
    posL = 90;
    posR = 90;
  }
  
  servoL.write(posL);
  servoR.write(posR);
}

The batteries should at least provide 7 Volts according to specification.
How much do you provide?
Electric schema please?
What devices are connected?

I'm using a 9V to power the board and a 6V NiMH for the servos. There's not much to show on a schematic, besides a toggle switch I just have the 9V attached to the VL input and the 6V attached to the VS input. Attached to the BotBoarduino are two servos and the USB Host shield board. Attached to the shield board is the USB dongle for the wireless mouse.

connected the GND's of the batteries?

Yes, they do share a common ground on the BotBoarduino

Ok, so I took a jump wire and attached the 5V output on the BotBoarduino to the 5V input from the USB port and it works now. So basically I'm providing power into the programming USB to make it think its connected. Still have no idea why it should need power input though.

Looks like something is broken on the board to me?