Max Tokenization Length

mlorenbaker · January 27, 2025, 5:58pm

Hello!
I am a novice Arduino user figuring it out for my job, and need help with the Max Tokenization Length. I have an Arduino Mega 2560 and have this error:

Tokenization is skipped for long lines for performance reasons. This can be configured via editor.maxTokenizationLineLength.

I found another exchange on stack overflow where it was suggested to Go to File--> Preferences--> Setting Search for Max Tokenization Length Change the Max Tokenization Length manually

But there is no spot for me to search for Max Tokenization Length. My code is for an LED lighting system. I inherited the code and have been working to convert it into code that mimics seasonal light transitions. I am happy to post the code if you think it's needed to inform about this error. Sorry if this is a dumb question, and thank you!!

UKHeliBob · January 27, 2025, 6:02pm

Please post your code, using code tags when you do

Which version of the IDE are you using ?

I would be interested in seeing the post that gave you this advice. Please post a link to it

UKHeliBob · January 27, 2025, 6:04pm

This is probably the post. Note that it applies to the VSCode editor

sterretje · January 27, 2025, 6:25pm

I'm not sure if it's really an issue but I have seen the message in the past; can't remember if it was in the compiler output or in the IDE logs.

See if this solves it: 2.0 slows down if very long lines (but OK with CRs / Line feeds inserted) - #8 by ptillisch.

mlorenbaker · January 27, 2025, 6:34pm

Hi,
I am using IDE 2.3.3
You found the correct link on stack overflow, and see that it's for VSC now. I pasted my code below (hopefully correctly). This is supposed to be 14 days of LED lighting (sped up in the delay for testing). It will run correctly for the first 4 days (the first 96 steps) before starting over.

// Board2

//           -------
//      ----/       \----
//-----/                 \----
//
//0-255 is the range for changing light intensities

// LED brightness pattern arrays
int LED_9[336] = { 50, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                                                                                                                // LED 12 230
int LED_12[336] = { 50, 0, 0, 0, 0, 0, 23, 46, 104, 127, 168, 214, 230, 221, 184, 152, 92, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23, 45, 102, 124, 165, 210, 226, 217, 181, 149, 90, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22, 44, 100, 122, 162, 207, 222, 214, 178, 147, 89, 22, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22, 44, 98, 120, 160, 203, 219, 210, 175, 144, 87, 22, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21, 43, 97, 118, 157, 200, 215, 206, 172, 142, 86, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21, 42, 95, 116, 154, 196, 211, 203, 169, 139, 84, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21, 41, 93, 114, 151, 193, 207, 199, 166, 137, 83, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20, 41, 92, 112, 149, 189, 204, 195, 163, 134, 81, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20, 40, 90, 110, 146, 186, 200, 192, 160, 132, 80, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20, 39, 88, 108, 143, 182, 196, 188, 157, 129, 78, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 38, 87, 106, 140, 179, 192, 185, 154, 127, 77, 19, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 38, 85, 104, 138, 175, 189, 181, 151, 124, 75, 19, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 18, 37, 83, 102, 135, 172, 185, 177, 148, 122, 74, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 18, 36, 81, 100, 132, 168, 181, 174, 145, 119, 72, 18, 0, 0, 0, 0, 0, 0 };  // LED 12
int LED_11[336] = { 50, 0, 0, 0, 0, 0, 1, 2, 5, 6, 7, 9, 10, 10, 8, 7, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 9, 10, 9, 8, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 9, 10, 9, 8, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 9, 10, 9, 8, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 9, 9, 9, 7, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 9, 9, 9, 7, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 7, 8, 9, 9, 7, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 6, 8, 9, 8, 7, 6, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 6, 8, 9, 8, 7, 6, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 6, 8, 9, 8, 7, 6, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 6, 8, 8, 8, 7, 6, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 5, 6, 8, 8, 8, 7, 5, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 4, 6, 7, 8, 8, 6, 5, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 4, 4, 6, 7, 8, 8, 6, 5, 3, 1, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                                                                                                          //LED 11: 10
int LED_10[336] = { 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                                                                                                               // LED 10: 0
int LED_5[336] = { 50, 0, 0, 0, 0, 0, 8, 15, 34, 41, 55, 70, 75, 72, 60, 50, 30, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 15, 33, 41, 54, 69, 74, 71, 59, 49, 30, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 15, 33, 40, 53, 67, 73, 70, 58, 48, 29, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 14, 32, 39, 52, 66, 71, 68, 57, 47, 29, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 14, 32, 39, 51, 65, 70, 67, 56, 46, 28, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 14, 31, 38, 50, 64, 69, 66, 55, 45, 28, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 14, 30, 37, 49, 63, 68, 65, 54, 45, 27, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 13, 30, 37, 48, 62, 66, 64, 53, 44, 27, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 13, 29, 36, 48, 61, 65, 63, 52, 43, 26, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 13, 29, 35, 47, 59, 64, 61, 51, 42, 26, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 13, 28, 34, 46, 58, 63, 60, 50, 41, 25, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 12, 28, 34, 45, 57, 61, 59, 49, 41, 25, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 12, 27, 33, 44, 56, 60, 58, 48, 40, 24, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 12, 27, 32, 43, 55, 59, 57, 47, 39, 24, 6, 0, 0, 0, 0, 0, 0 };                                                                                                                                    // LED 5: 75
int LED_2[336] = { 50, 0, 0, 0, 0, 0, 5, 10, 23, 28, 37, 47, 50, 48, 40, 33, 20, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 22, 27, 36, 46, 49, 47, 39, 32, 20, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 22, 27, 35, 45, 48, 46, 39, 32, 19, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 21, 26, 35, 44, 48, 46, 38, 31, 19, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 9, 21, 26, 34, 43, 47, 45, 37, 31, 19, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 9, 21, 25, 34, 43, 46, 44, 37, 30, 18, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 9, 20, 25, 33, 42, 45, 43, 36, 30, 18, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 9, 20, 24, 32, 41, 44, 42, 35, 29, 18, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 9, 20, 24, 32, 40, 43, 42, 35, 29, 17, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 9, 19, 23, 31, 40, 43, 41, 34, 28, 17, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 19, 23, 31, 39, 42, 40, 33, 28, 17, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 18, 23, 30, 38, 41, 39, 33, 27, 16, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 18, 22, 29, 37, 40, 39, 32, 27, 16, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 18, 22, 29, 37, 39, 38, 31, 26, 16, 4, 0, 0, 0, 0, 0, 0 };                                                                                                                                              // LED 2: 50
int LED_13[336] = { 50, 0, 0, 0, 0, 0, 2, 4, 8, 10, 13, 17, 18, 17, 14, 12, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 4, 8, 10, 13, 16, 18, 17, 14, 12, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 10, 13, 16, 17, 17, 14, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 9, 12, 16, 17, 16, 14, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 9, 12, 16, 17, 16, 13, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 17, 16, 13, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 16, 16, 13, 11, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 16, 15, 13, 11, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 11, 15, 16, 15, 13, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 8, 11, 14, 15, 15, 12, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 8, 11, 14, 15, 14, 12, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 7, 8, 11, 14, 15, 14, 12, 10, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 7, 8, 11, 13, 14, 14, 12, 10, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 6, 8, 10, 13, 14, 14, 11, 9, 6, 1, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                         // LED 13: 18
int LED_6[336] = { 50, 0, 0, 0, 0, 0, 2, 4, 8, 10, 13, 17, 18, 17, 14, 12, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 4, 8, 10, 13, 16, 18, 17, 14, 12, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 10, 13, 16, 17, 17, 14, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 9, 12, 16, 17, 16, 14, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 8, 9, 12, 16, 17, 16, 13, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 17, 16, 13, 11, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 16, 16, 13, 11, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 12, 15, 16, 15, 13, 11, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 9, 11, 15, 16, 15, 13, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 8, 11, 14, 15, 15, 12, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 7, 8, 11, 14, 15, 14, 12, 10, 6, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 7, 8, 11, 14, 15, 14, 12, 10, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 7, 8, 11, 13, 14, 14, 12, 10, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 6, 8, 10, 13, 14, 14, 11, 9, 6, 1, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                          // LED 6: 18
int LED_4[336] = { 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                                                                                                                // LED 4: 0
int LED_3[336] = { 50, 0, 0, 0, 0, 0, 3, 6, 14, 17, 22, 28, 30, 29, 24, 20, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 6, 13, 16, 22, 27, 30, 28, 24, 19, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 6, 13, 16, 21, 27, 29, 28, 23, 19, 12, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 6, 13, 16, 21, 27, 29, 27, 23, 19, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 6, 13, 15, 20, 26, 28, 27, 22, 19, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 6, 12, 15, 20, 26, 28, 26, 22, 18, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 5, 12, 15, 20, 25, 27, 26, 22, 18, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 5, 12, 15, 19, 25, 27, 25, 21, 18, 11, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 5, 12, 14, 19, 24, 26, 25, 21, 17, 10, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 5, 12, 14, 19, 24, 26, 25, 20, 17, 10, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 5, 11, 14, 18, 23, 25, 24, 20, 17, 10, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 5, 11, 14, 18, 23, 25, 24, 20, 16, 10, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 5, 11, 13, 18, 22, 24, 23, 19, 16, 10, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 5, 11, 13, 17, 22, 24, 23, 19, 16, 9, 2, 0, 0, 0, 0, 0, 0 };                                                                                                                                                   // LED 3: 30
int LED_8[336] = { 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };                                                                                                                                                                                                                                                                                // LED 8: 60
int LED_7[336] = { 50, 0, 0, 0, 0, 0, 6, 12, 27, 33, 44, 56, 60, 58, 48, 40, 24, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 12, 27, 32, 43, 55, 59, 57, 47, 39, 24, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 12, 26, 32, 42, 54, 58, 56, 46, 38, 23, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 11, 26, 31, 42, 53, 57, 55, 46, 38, 23, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 11, 25, 31, 41, 52, 56, 54, 45, 37, 22, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 11, 25, 30, 40, 51, 55, 53, 44, 36, 22, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 11, 24, 30, 39, 50, 54, 52, 43, 36, 22, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 11, 24, 29, 39, 49, 53, 51, 42, 35, 21, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 23, 29, 38, 48, 52, 50, 42, 34, 21, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 23, 28, 37, 48, 51, 49, 41, 34, 20, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 23, 28, 37, 47, 50, 48, 40, 33, 20, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 22, 27, 36, 46, 49, 47, 39, 32, 20, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 10, 22, 27, 35, 45, 48, 46, 39, 32, 19, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 9, 21, 26, 34, 44, 47, 45, 38, 31, 19, 5, 0, 0, 0, 0, 0, 0 };

//411
//413
//425


//434
//442
//446


//451
//468
//496
//500
//514
//546

int step;
int fadeAmount = 255;
void setup()

{
  step = 0;
  Serial.begin(9600);
  pinMode(2, INPUT);
  pinMode(3, INPUT);
  pinMode(4, INPUT);
  pinMode(5, INPUT);
  pinMode(6, INPUT);
  pinMode(7, INPUT);
  pinMode(8, INPUT);
  pinMode(9, INPUT);
  pinMode(10, INPUT);
  pinMode(11, INPUT);
  pinMode(12, INPUT);
  pinMode(13, INPUT);
}

void loop() {

  // linear scaling that ramps to zero over 255 subdivisions
  // the next point
  // linear scaling that ramps to one.
  byte brightness_2 =
    LED_7[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_7[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_3 =
    LED_8[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_8[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_4 =
    LED_11[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                 // linear scaling that ramps to zero over 256 subdivisions
    LED_11[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);           // linear scaling that ramps to one.

  byte brightness_5 =
    LED_9[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_9[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_6 =
    LED_12[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                 // linear scaling that ramps to zero over 256 subdivisions
    LED_12[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);           // linear scaling that ramps to one.

  byte brightness_7 =
    LED_10[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                 // linear scaling that ramps to zero over 256 subdivisions
    LED_10[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);           // linear scaling that ramps to one.

  byte brightness_8 =
    LED_6[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_6[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_9 =
    LED_4[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_4[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_10 =
    LED_3[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_3[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.


  byte brightness_11 =
    LED_5[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_5[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.

  byte brightness_12 =
    LED_2[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                // linear scaling that ramps to zero over 256 subdivisions
    LED_2[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);          // linear scaling that ramps to one.


  byte brightness_13 =
    LED_13[(step / 256) % 336] *  // the current point for the next 256 subdivisions
      (1.0 - (step % 256 / 256.0))
    +                                 // linear scaling that ramps to zero over 256 subdivisions
    LED_13[(step / 256 + 1) % 336] *  // the next point
      (step % 256 / 256.0);           // linear scaling that ramps to one.

  analogWrite(2, brightness_2);
  analogWrite(3, brightness_3);
  analogWrite(4, brightness_4);
  analogWrite(5, brightness_5);
  analogWrite(6, brightness_6);
  analogWrite(7, brightness_7);
  analogWrite(8, brightness_8);
  analogWrite(9, brightness_9);
  analogWrite(10, brightness_10);
  analogWrite(11, brightness_11);
  analogWrite(12, brightness_12);
  analogWrite(13, brightness_13);

  delay((900000 * 0.33) / 256);  // 900000 is for 6 hours, * 4 is to make it 24 hrs
  step++;

  if (step > 366 * 256) {
    step = 0;
  }
}

UKHeliBob · January 27, 2025, 6:48pm

I can't help with your problem but I have an observation on your code

Why are the LED arrays declared as data type int when none of the values that I can see are greater than 255 and would fit in a byte thus saving memory

mlorenbaker · January 27, 2025, 8:07pm

Thanks for looking over it!
And thanks for point out the int thing- I will look into it. I hadn't considered the data type.

kenb4 · January 27, 2025, 8:40pm

Long lines should be avoided. They make code harder to read, and can hide errors. Instead of trying to "fix" the tokenization, the code can be improved by using a reasonable line length. First, you should note that the compiler does not necessarily enforce the size of the array you have declared

int three[4] = { 1, 2, 3 };       // not an error
int five[4] = { 1, 2, 3, 4, 5 };  // error!

Each of your giant arrays is supposed to have 336 elements. Are you sure? By using a reasonable line length, you can see the pattern in the numbers and notice where you might be off. Then you can have the compiler count for you and then have it tell you if the count is incorrect. For example

int LED_12[] = {
  50, 0, 0, 0, 0, 0, 23, 46, 104, 127, 168, 214, 230, 221, 184, 152, 92, 23, 0, 0, 0, 0, 0, 0,  //
  0, 0, 0, 0, 0, 0, 23, 45, 102, 124, 165, 210, 226, 217, 181, 149, 90, 23, 0, 0, 0, 0, 0, 0,   //
  0, 0, 0, 0, 0, 0, 22, 44, 100, 122, 162, 207, 222, 214, 178, 147, 89, 22, 0, 0, 0, 0, 0, 0,   //
  0, 0, 0, 0, 0, 0, 22, 44, 98, 120, 160, 203, 219, 210, 175, 144, 87, 22, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 21, 43, 97, 118, 157, 200, 215, 206, 172, 142, 86, 21, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 21, 42, 95, 116, 154, 196, 211, 203, 169, 139, 84, 21, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 21, 41, 93, 114, 151, 193, 207, 199, 166, 137, 83, 21, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 20, 41, 92, 112, 149, 189, 204, 195, 163, 134, 81, 20, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 20, 40, 90, 110, 146, 186, 200, 192, 160, 132, 80, 20, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 20, 39, 88, 108, 143, 182, 196, 188, 157, 129, 78, 20, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 19, 38, 87, 106, 140, 179, 192, 185, 154, 127, 77, 19, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 19, 38, 85, 104, 138, 175, 189, 181, 151, 124, 75, 19, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 18, 37, 83, 102, 135, 172, 185, 177, 148, 122, 74, 18, 0, 0, 0, 0, 0, 0,    //
  0, 0, 0, 0, 0, 0, 18, 36, 81, 100, 132, 168, 181, 174, 145, 119, 72, 18, 0, 0, 0, 0, 0, 0,    //
};
static_assert(sizeof(LED_12) / sizeof(int) == 336);

six zeroes on either end of each line (except for the very first element)
the // comment slashes at the end of each line force the auto-formatter to keep the line as-is, instead of breaking it up (one line per element -- don't want that)
the static_assert is evaluated at compile-time, and does basic math to ensure you have the right number of elements. If you hover over the variable name LED_12, you should also see an info box pop up that mentions the number of elements

variable LED_12

Type: int[336]
Value = {50, 0, 0, 0, 0, 0, 23, 46, 104, 127, ...}

One setting you should change is the warning level. Under File | Preferences (on a Mac, it's Settings under the app menu): Compiler warnings. Instead of None, set it to at least Default. Then you will see when compiling for Mega, for the very end of your sketch

warning: integer overflow in expression [-Woverflow]
   if (step > 366 * 256) {
              ~~~~^~~~~

The max value of an int on Mega is 32767, which is of course less than 93696. So the number will wrap and Bad Things will happen. You need to consider this while also perhaps using byte for your arrays.

xfpd · January 27, 2025, 8:49pm

That works out to delay(1160); ... about one second.
For 24 hours: delay(24 * 60 * 60 * 1000UL);?

ptillisch · January 27, 2025, 10:38pm

Hi @mlorenbaker. "Tokenization" is the parsing of the code by the Arduino IDE editor so that it can do things like coloring the code (syntax highlighting).

Please note that this is not an error. This is only an informational message telling you that the editor intentionally skipped tokenization of exceptionally long lines in your code in order to avoid excessive use of computing resources, which could impact usability. The reason the IDE shows a message under these conditions is to make sure you won't be confused by the lack of syntax highlighting.

The message doesn't necessarily indicate anything is wrong, though as has already been explained, it might indicate that your code can be reformatted to be more maintainable and readable.

It is possible to adjust the maximum line length for tokenization via the
Arduino IDE advanced settings, but you should be aware that if you set this too high you might then find that Arduino IDE becomes either unusable or unpleasant to use when you open a sketch with long lines, especially if using a computer with modest specs.

mlorenbaker · January 27, 2025, 10:59pm

Yeah it's sped up on purpose but perhaps a little too fast- thanks for the fix!

mlorenbaker · January 27, 2025, 11:02pm

Thank you so much! This is super helpful, I'll definitely try these suggestions out!

JoeHuber · March 28, 2025, 1:38am

Thanks for your explanation.

I see this message when I mouse over long lines of compiler messages in the IDE's lower Output pane. Is it just a spurious response intended for the upper code editing pane?

I see this often and never understood what tokenization had to do with compiler output messages.....

ptillisch · March 31, 2025, 1:12am

Arduino IDE's "Output" view is provided by the Eclipse Theia Platform IDE framework, which in turn is based the feature of VS Code. This view is a general purpose container for any read-only text. In some cases, tokenization of the content of the Output view is definitely useful. For example, here you can see it used to provide some helpful coloration of logs in the VS Code "Main" output channel:

However, I'm not sure whether the tokenization is used by the "Arduino" output channel. This channels content is defined as being in the "plaintext" language. It seems a bit odd to bother with tokenizing plain text content, but apparently there is a use case for this:

github.com/vsce-toolroom/vscode-textmate-languageservice

Add pseudo-tokenization of `plaintext` language documents

opened 04:17PM - 08 Sep 23 UTC

closed 02:53PM - 10 Nov 23 UTC

zm-cttae

context-services priority-medium status-released scope-functionality type-proposal

## ⚡️️ Feature Request - [x] I'm using the latest version of `vscode-textmate…-languageservice` available. - [x] I searched [existing issues][vsctmls-issues], open & closed. Yes, my feature request is new. ### Is your feature request related to a problem? - **As a:** developer tokenising arbitrary documents that aren't registered - **I want to:** receive a semi-valid facsimile of plaintext tokenization - **So that:** I can universally tokenize and parse all VS Code documents without errors ### Describe the solution you'd like Add semi-valid line tokenization and API results for `plaintext` documents and language. ### Describe alternatives you've considered Document the error as it is technically valid to `vscode-textmate` result in core app. [vsctmls-issues]: https://github.com/vsce-toolroom/vscode-textmate-languageservice/issues?q=is%3Aopen+sort%3Aupdated-desc /label ~feature-request

I don't know whether Arduino IDE uses the tokenization data or not. It does have an awareness of the meaning of some of the output, but it is possible that is done independently from the tokenization data provided by the Theia framework.

system · September 27, 2025, 1:12am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Id.exe fails to compile when my sketch exceeds about 165 lines. IDE 1.x	17	5315	May 5, 2021
warning: internal error: out of range error Programming	17	9076	May 5, 2021
Arduino - Bluetooth - Android - LEDs Deutsch	70	29119	May 6, 2021
Arduino Uno stops working for no reason? Programming	23	8249	May 5, 2021
Problem with a char array Programming	13	3973	May 5, 2021

Max Tokenization Length

Related topics