Increasing communication speed (Not Baud Rate)

Hello, I am currently working on a project with the Fast Led library and Serial. The idea is the Arduino receives information about what colors the led strip should be from a connected computer. Right now I've run into the problem that the "frame rate" of the leds is around 7 (I'd like at least 30). What I've done to prevent the Serial buffer from overflowing on the Arduino is implemented a system where the computer waits to send data until the Arduino has echoed that it is ready. I started by using the built in Serial method Serial.readBytesUntil('>', inputBuffer, 15);. This produced some very strange behavior. When Programming I accidentally messed up and ended up sending too many characters to the Arduino in the form of zero's tacked onto the end of the command. To my utter surprise the Arduino was suddenly updating a good 40 times per second. My running theory at the time was that the method wasn't recognizing the end character and was falling back on the timeout of 3ms. By sending extra characters I was able to hit the limit on the method, stopping it, and speeding up the whole program. However, I'm now pretty sure this isn't the case.
After this discovery I figured all I needed to do was get it to recognize my end character and it would fix it. I was wrong. After some trial and error I got my own version of the readBytesUntil method working.

//takes in one character at a time then marks the input data as new once it finds an endmarker
void processSerial() {
	static byte ndx = 0;
	char endMarker = '>';
	char rc;

	while (Serial.available() > 0 && newData == false) {
		rc = Serial.read();
		interSerial();

		if (rc != endMarker) {
			inputBuffer[ndx] = rc;
			ndx++;
			if (ndx >= bufferSize) {
				ndx = bufferSize - 1;
			}
		}
		else {
			inputBuffer[ndx] = '\0'; // terminate the string
			ndx = 0;
			newData = true;
		}
	}

}

It's Basically just copied and pasted from this thread: Serial Input Basics - updated - Introductory Tutorials - Arduino Forum

Anyways I finally got this to work only to find that it was back to running at almost the exact same frame rate I had started at. Furthermore, upon reverting back to one of my backups (I'm around 80% sure it was previously working ), I found that the two zero's method no longer worked and resulted in a mess of corrupted led's followed by the whole thing locking up. At this point I'm pretty much perplexed. I've worked for hours unable to find any leads to possible solutions which is why I've turned here.

I'll post the whole program below as I'm out of space here.

#include <FastLED.h>

#define LED_PIN     8
#define NUM_LEDS    60 
#define BRIGHTNESS  200
#define LED_TYPE    WS2811
#define COLOR_ORDER GRB
CRGB leds[NUM_LEDS];

#define UPDATES_PER_SECOND 300


CRGBPalette16 currentPalette;
TBlendType    currentBlending;

extern CRGBPalette16 myRedWhiteBluePalette;
extern const TProgmemPalette16 myRedWhiteBluePalette_p PROGMEM;

//=====communication====
bool connectionEstablished = false;
//used to iterate over inputBuffer
bool newData = false;

//=====SyncVariables=====
bool LocalAnim = false;
bool Sync = true;
CRGBPalette16 serialPalette;
float brigntness = 0.1;
int bpm = 128;

void setup() {
	delay(3000); // power-up safety delay
	//dissable led dithering to remove flickering when serial is being proccesed.
	FastLED.setDither(0);
	Serial.begin(500000);
	Serial.setTimeout(3);
	FastLED.addLeds<LED_TYPE, LED_PIN, COLOR_ORDER>(leds, NUM_LEDS).setCorrection(TypicalLEDStrip);
	FastLED.setBrightness(BRIGHTNESS);

	currentPalette = RainbowColors_p;
	currentBlending = LINEARBLEND;
}


const int bufferSize = 18;
char inputBuffer[bufferSize];
unsigned long deltaTime = 0;
void loop()
{
	unsigned long oldTime = micros();

	//if data has been sent parse it
	//if (Serial.available() > 0) {
		//Serial.readBytesUntil('>', inputBuffer, 15);
		processSerial();
	//}
	if (newData) {
		parseinput(inputBuffer);
		newData = false;
	}

	//will update the pallet mapping every loop if there are animations running localy on the arduino. 
	//delta time should be used for frame-independent animations
	if (LocalAnim) {
		static uint8_t startIndex = 0;
		startIndex = startIndex + 0; /* motion speed */
		FillLEDsFromPaletteColors(startIndex);
	}
		//FastLED.show();
		//FastLED.delay(positive((1000 / UPDATES_PER_SECOND) - (millis() - deltaTime)));
	

	//calculates the amount of time all of the code took to run and takes that into account with the delay
	//The max resolution is multiples of 4 microseconds
	deltaTime = micros() - oldTime;


}


//takes in one character at a time then marks the input data as new once it finds an endmarker
void processSerial() {
	static byte ndx = 0;
	char endMarker = '>';
	char rc;

	while (Serial.available() > 0 && newData == false) {
		rc = Serial.read();
		interSerial();

		if (rc != endMarker) {
			inputBuffer[ndx] = rc;
			ndx++;
			if (ndx >= bufferSize) {
				ndx = bufferSize - 1;
			}
		}
		else {
			inputBuffer[ndx] = '\0'; // terminate the string
			//debug
			//Serial.print("process");
			//Serial.println(inputBuffer);
			ndx = 0;
			newData = true;
		}
	}

}

//this meathod is run on each character read from serial. This can take up to 20 microseconds without causing significant slowdowns.
void interSerial() {

}


int positive(const int &input){
	if (input < 0) {
		return 0;
	}
}

void parseinput(const char *input) {
	if (memcmp(input, "hex", 3) == 0) {
		char index[3];
		//this meathod directly modifys the input c string 
		getStringBetweenDelimiters(input, ":", ";", index);

		//stores a pointer so that it can iterate over the hex in the string
		char *p;
		p = strstr(input, ";") + 1;
		currentPalette[atoi(index)] = CRGB(
			//p is just a pointer and * gets the value of that memory address. So we add to the pointer to iterate over the array
			HextoByte(*p, *(p+1)), HextoByte(*(p+2), *(p+3)), HextoByte(*(p+4), *(p+5))
		);
	}
	else if (memcmp(input, "ref", 3) == 0) {
		FillLEDsFromPaletteColors(0);
		FastLED.show();
	}
	//prints the last frames deltaTime
	Serial.println(deltaTime);
}


int getStringBetweenDelimiters(const char* string, const char* leftDelimiter, const char* rightDelimiter, char* out)
{
	// find the left delimiter and use it as the beginning of the substring
	const char* beginning = strstr(string, leftDelimiter);
	if (beginning == NULL)
		return 1; // left delimiter not found

	// find the right delimiter
	const char* end = strstr(string, rightDelimiter);
	if (end == NULL)
		return 2; // right delimiter not found

	// offset the beginning by the length of the left delimiter, so beginning points _after_ the left delimiter
	beginning += strlen(leftDelimiter);

	// get the length of the substring
	ptrdiff_t segmentLength = end - beginning;
	// allocate memory and copy the substring there
	//*out = malloc(segmentLength + 1);
	strncpy(out, beginning, segmentLength);
	(out)[segmentLength] = 0;
	return 0; // success!
}


/*
//will parse out all characters found from a starting index to a ending character to an int
int parseIntTillEndChar(const char *input, int startingindex, const char endchar) {
	//this creates a char pointer which is just a number that references memory
	const char *p;
	//p is now pointing to inputs memory address
	p = input;
	//we add to the memory address to get to the first index of the string
	p += startingindex;
	//creates another string to hold the number
	char intbuffer[4];

	//stores the index for iterating over string
	uint8_t index = 0;
	//checks for null terminator, the specified end character, or if the index got to high
	while (*p || *p != endchar || index > 2) {
		intbuffer[index] = *p;
		index++;
	}
	//parses the result to a int and returns it;
	return atoi(intbuffer);
}
*/

uint8_t HextoByte(char& x16, char& x1) {
	uint8_t decimal = HexLetterToNumber(x1) + 16 * HexLetterToNumber(x16);
	return decimal;
}

uint8_t HexLetterToNumber(char& x) {
	switch (x)
	{
	case('a'):
		return 10;
		break;
	case('b'):
		return 11;
		break;
	case('c'):
		return 12;
		break;
	case('d'):
		return 13;
		break;
	case('e'):
		return 14;
		break;
	case('f'):
		return 15;
	default:
		return int(x) - 48;
		break;
	}
}


void FillLEDsFromPaletteColors(uint8_t colorIndex)
{
	uint8_t brightness = 30;
	for (int i = 0; i < NUM_LEDS; i++) {
		leds[i] = ColorFromPalette(currentPalette, colorIndex, brightness, currentBlending);
		colorIndex += 4;
	}
}

Also sorry for all of the commented out code. A lot of it is from me experimenting around.

Please post an example of a complete message that is sent to the Arduino.

One way to make a big improvement to serial speed is to reduce the number of characters that need to be transmitted.

Another is to send the data in a way that it can be used directly by the Arduino.

I have never used the FastLed library but I think I read somewhere that it disables interrupts for significant periods and interrupts are an essential part of serial reception. However if you are not losing characters this may not be a problem.

...R

Here is an example of what might be sent. "hex:4;ffa500>" or "ref>" the first command takes in an index and color, while the second command refreshes the leds. However, I have seen this run at a high frame rate with these before so I know it's possible. I'm just trying to figure out how it worked back then and how I can recreate it without the weird workarounds. Like I said in the title baud rate is not the issue here. if I did my math right at 500,000 baud there should be around 20 microseconds between each character. That times a max command length of 15 characters times the 17 commands to update the leds gives me around 5.1 milliseconds. That kind of delay is a worst case scenario and it still gives a theoretical 192fps. what needs to be faster is the serial processing. I know the code that parses it isn't the problem as I've seen that run fast before and that has had no modifications. The part that is slowing it down is my serial receive methods. Interestingly enough the custom version of Serial.readBytesUntil() method I posted in my initial post and the built in readBytesUntil() methods yield almost the exact fps of 7.18 except in the conditions I described above.

Droid_22:
I'm just trying to figure out how it worked back then and how I can recreate it without the weird workarounds.

Without seeing the program from back then who can tell what was happening?

Here is an example of what might be sent. "hex:4;ffa500>" or "ref>" the first command takes in an index and color, while the second command refreshes the leds.

Why bother to send "hex" or "ref" when 'h' and 'r' would convey the same information?

Is there any value in sending the refresh signal rather than just repeating the colour signal? If not then I reckon you don't even need the 'h' in the colour signal.

What does the number 4 signify?

Do you have to send the colour values in HEX?

Have a look at the parse example in Serial Input Basics - I think it is simpler than your system.

If this was my problem I would write a test program without FastLed and measure the serial performance I could get.

...R

I think it would be faster to convert the HEX characters with something like this

if (hexChar <= '9') {
  return hexChar - '0';
}
else {
  return hexChar - 87;
}

...R

Robin2:
Why bother to send "hex" or "ref" when 'h' and 'r' would convey the same information?

Is there any value in sending the refresh signal rather than just repeating the colour signal? If not then I reckon you don't even need the 'h' in the colour signal.

What does the number 4 signify?

Do you have to send the colour values in HEX?

Have a look at the parse example in Serial Input Basics - I think it is simpler than your system.

If this was my problem I would write a test program without FastLed and measure the serial performance I could get.

...R

the 4 is one of the indexes out of 16. Ref is needed so that I can update the colors without seeing them update individually. I send hex and ref because originally I was planning on scaling this up a bit and I figured 3 characters would allow for more commands. However, in hindsight your probably right when it comes to shortening them. Either way as stated the serial speed is not the immediate issue. I really just want to know if there are any ways I could make this code run faster.

//takes in one character at a time then marks the input data as new once it finds an end marker
void processSerial() {
 static byte ndx = 0;
 char endMarker = '>';
 char rc;

 while (Serial.available() > 0 && newData == false) {
 rc = Serial.read();
 interSerial();

 if (rc != endMarker) {
 inputBuffer[ndx] = rc;
 ndx++;
 if (ndx >= bufferSize) {
 ndx = bufferSize - 1;
 }
 }
 else {
 inputBuffer[ndx] = '\0'; // terminate the string
 //debug
 //Serial.print("process");
 //Serial.println(inputBuffer);
 ndx = 0;
 newData = true;
 }
 }

}

worst case scenario I fallback on my weird solution. somehow by hitting the character limit on the Serial.readBytesUntil once, it speeds up all subsequent calls of the method no matter how many characters are sent to it. So my current solution is just to send a bunch of commands that hit the character limit at the beginning. It isn't pretty and I don't know how it works... but it works so I might just have to settle for it.

edit - Just to clarify the character limit trick works on my old code of which the only difference is this

	//if data has been sent parse it
	if (Serial.available() > 0) {
		Serial.readBytesUntil('>', inputBuffer, 15);
		parseinput(inputBuffer);
	}

The parse input method is exactly the same

Droid_22:
I really just want to know if there are any ways I could make this code run faster.

from what you have said I can't figure out what is running too slowly.

I have never found that the code in Serial Input Basics is an obstacle to performance. In my experience it is the baud rate that is the obstacle, and you are using 500,000 baud - which I also use.

If you are only updating 16 colours then, IMHO, it would be worth experimenting with sending all 16 at the same time - even if they don't change. It would only require 48 bytes. The the receiving program can just iterate over the received data and update all the colours with a simple FOR loop

One thought, about 500,000 baud. Every time a character arrives the Arduino must execute an interrupt to move it to the Serial Input Buffer. At a lower baud rate those interrupts will be less frequent and may actually leave more time for other activities.

...R

The fastled library disables interrupts. Any attempt to send more than one byte at a time to the Arduino will more than likely fail.

A possible solution is to send one character at a time and let the Arduino echo it back to the sender. That way the sender will know when it can send the next character.

If you don't have a need to keep the current pattern running, you can first send a single character to tell the Arduino to switch to 'command mode' and in that 'command mode' handle further serial communication in more convenient way.

I was having issues with the interrupts at first. However, the way I've programmed it now the Arduino sends a message when it's ready for more data. That way the led's are never updating when the Arduino is receiving data.