weird compiler optimization behavior / bug

this is from the french forum discussion - brought by @bidouilleelec and worth additional eyeballs looking at it for what is either a super aggressive supposedly “dead loop” / “dead code” removal optimization or a compiler bug.

(tested with 1.8.5 / avr-gcc/4.9.2-atmel3.5.4)

the conversation iterated over a few codes, and @bricoleau proposed a simplified version of the code showing the weird behavior. here it is

int n = 1;

void setup()
{
  Serial.begin(115200);
  Serial.println(F("STARTING"));
}

void loop()
{
  if ( n <= 3 ) {
    boolean r = false;
    while ( !r ) {
      if (Serial.available()) r = (Serial.read() == 'c' );
    }
    Serial.println(n);
    n++;
  } else while (1);
}

you could argue that the active wait on Serial or dead loop to stop the program in the loop at the end are “ugly” but that’s not the point, this is a legit programming construct.

The code as it is should basically greet you with “STARTING” and every time you enter ‘c’ in the Serial console display the value of n and increase it. Once n reaches 4 the code should go into the infinite else while (1); loop and be stuck there.

And guess what, when you run it and type ‘c’ a number of times, the code never goes into the infinite loop.

console.png

Now if you modify the code to make the boolean r volatile

int n = 1;

void setup()
{
  Serial.begin(115200);
  Serial.println(F("STARTING"));
}

void loop()
{
  if ( n <= 3 ) {
    volatile boolean r = false; // ONLY CHANGE IS r BEING VOLATILE
    while ( !r ) {
      if (Serial.available()) r = (Serial.read() == 'c' );
    }
    Serial.println(n);
    n++;
  } else while (1);
}

then the optimizer cannot make the same assumptions and compiler behaves differently and this time the code does go and get stuck into the infinite loop as expected.

console2.png

In the french discussion some members looked at the generated binary and it turns out that in the non working version the compiler seems to have removed entirely the   if ( n <= 3 ) { and } else while (1); parts, leading to the unwanted behavior.

Keeping the code the same and changing the optimizer parameter in the compilation also lead to expected behavior.

So - what do you think is going on there? why would it be legit in that case for the optimizer to get rid of part of the code?

Behavior confirmed on 1.8.5 and Arduino UNO.

Any idea why? it just does not seem to make sense to me how the optimizer could assume the test against n <=3 is useless… (or does it assume printing and user input would be useless?)

J-M-L:
Any idea why? it just does not seem to make sense to me how the optimizer could assume the test against n <=3 is useless… (or does it assume printing and user input would be useless?)

The optimizer has no clue whether code is printing or fetching user input - it see the code at a MUCH lower level, and judges only based on the side-effects of running the code, or not.

Regards,
Ray L.

What happens if you make r static? Global?

RayLivingston:
The optimizer has no clue whether code is printing or fetching user input - it see the code at a MUCH lower level, and judges only based on the side-effects of running the code, or not.

Regards,
Ray L.

Yes I’m actually familiar with how this works and many of the optimization techniques - but this one makes no sense so wondering if there is a “Voodoo level” that I don’t know about which reads code intents :slight_smile:

PaulS:
What happens if you make r static? Global?

Subtle changes to r or elsewhere would lead to code working as expected or not. for example making it static or global

int n = 1;
boolean r;

void setup()
{
  Serial.begin(115200);
  Serial.println(F("STARTING"));
}

void loop()
{
  if ( n <= 3 ) {
    r = false;
    while ( !r ) {
      if (Serial.available()) r = (Serial.read() == 'c' );
    }
    Serial.println(n);
    n++;
  } else while (1);
}

then code will behave as expected

but keeping it on the stack and adding another statement within the test will also make the code work as expected, for example this works

int n = 1;

void setup()
{
  Serial.begin(115200);
  Serial.println(F("STARTING"));
}

void loop()
{
  if ( n <= 3 ) {
    boolean r = false;
    Serial.println("Can a print change the world?");
    while ( !r ) {
      if (Serial.available()) r = (Serial.read() == 'c' );
    }
    Serial.println(n);
    n++;
  } else while (1);
}

Given critical pieces of codes do disappear at assembly level, it’s feels like a very aggressive optimization decision

arduino-1.8.4 ( arduino.avr-gcc=4.9.2-atmel3.5.4-arduino2 ) on linux.
Does NOT exhibit the problem. Works as expected on UNO.

STARTING
1
2
3

J-M-L:
(tested with 1.8.5 / avr-gcc/4.9.2-atmel3.5.4)

tf68:
arduino-1.8.4 ( arduino.avr-gcc=4.9.2-atmel3.5.4-arduino2 ) on linux.

(emphasis added)

That makes the difference a compile / link switch. Both of you: please turn on verbose build then post the results so we can diff.

(Or the Arduino built toolset is flawed.)

Speaking of optimizer trouble, apparently the compiler is no longer optimizing bit shifts...

Friend, either you're closing your eyes
To a situation you do not wish to acknowledge
Or you are not aware of the caliber of disaster indicated
By the presence of a compile bug in your community . Well, ya got trouble, my friend, right here,
I say, trouble right here in Arduino City.

(apologies to Meredith Willson)

Indeed I had missed the arduino2 in my copy and paste.

avr-gcc/4.9.2-atmel3.5.4-arduino2 on a Mac

</sub> <sub>Compiling sketch... "/Users/jeanmarc/Library/Arduino15/packages/arduino/tools/[color=red]avr-gcc/4.9.2-atmel3.5.4-arduino2[/color]/bin/avr-g++" -c -g [color=red]-Os[/color] -Wall -Wextra -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10805 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR   "-I/Users/jeanmarc/Library/Arduino15/packages/arduino/hardware/avr/1.6.21/cores/arduino" "-I/Users/jeanmarc/Library/Arduino15/packages/arduino/hardware/avr/1.6.21/variants/standard" "/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986/sketch/sketch_jul05a.ino.cpp" -o "/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986/sketch/sketch_jul05a.ino.cpp.o" Compiling libraries... Compiling core... Using precompiled core Linking everything together... "/Users/jeanmarc/Library/Arduino15/packages/arduino/tools/[color=red]avr-gcc/4.9.2-atmel3.5.4-arduino2[/color]/bin/avr-gcc" -Wall -Wextra [color=red]-Os[/color] -g -flto -fuse-linker-plugin -Wl,--gc-sections -mmcu=atmega328p  -o "/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986/sketch_jul05a.ino.elf" "/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986/sketch/sketch_jul05a.ino.cpp.o" "/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986/../arduino_cache_261545/core/core_arduino_avr_uno_326b060ef0db883205dbc87760fa0a68.a" "-L/var/folders/xk/mwl4yt3j5rsgdmgmgjlqsk7w0000gn/T/arduino_build_74986" -lm</sub> <sub>

All the more reason to... please turn on verbose build then post the results so we can difference.

interesting to see the bug is not there in 1.8.4 and may be not same flags?

tf68:
arduino-1.8.4 ( arduino.avr-gcc=4.9.2-atmel3.5.4-arduino2 ) on linux.
Does NOT exhibit the problem. Works as expected on UNO.

STARTING

1
2
3

just to be sure, you tried typing more 'c' in the Serial monitors right?

Still - it's weird that on the same platform/machine/context with same compile flags, adding a Serial.print() in the if () {...} section totally changes the behavior....

Here is an edited compiler output. And yes I used lots of cccccccccccccccc.

Compiling sketch...
"/home/terry/bin/arduino-1.8.4/hardware/tools/avr/bin/avr-g++" -c -g -Os -Wall -Wextra -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10804 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR   "-I/home/terry/bin/arduino-1.8.4/hardware/arduino/avr/cores/arduino" "-I/home/terry/bin/arduino-1.8.4/hardware/arduino/avr/variants/standard" "/tmp/arduino_build_46171/sketch/compilerBugTest.ino.cpp" -o "/tmp/arduino_build_46171/sketch/compilerBugTest.ino.cpp.o"
Compiling libraries...
Compiling core...
Using precompiled core
Linking everything together...
"/home/terry/bin/arduino-1.8.4/hardware/tools/avr/bin/avr-gcc" -Wall -Wextra -Os -g -flto -fuse-linker-plugin -Wl,--gc-sections -mmcu=atmega328p  -o "/tmp/arduino_build_46171/compilerBugTest.ino.elf" "/tmp/arduino_build_46171/sketch/compilerBugTest.ino.cpp.o" "/tmp/arduino_build_46171/../arduino_cache_987557/core/core_arduino_avr_uno_3e8997973f924952b582441c1fc61698.a" "-L/tmp/arduino_build_46171" -lm
"/home/terry/bin/arduino-1.8.4/hardware/tools/avr/bin/avr-objcopy" -O ihex -j .eeprom --set-section-flags=.eeprom=alloc,load --no-change-warnings --change-section-lma .eeprom=0  "/tmp/arduino_build_46171/compilerBugTest.ino.elf" "/tmp/arduino_build_46171/compilerBugTest.ino.eep"
"/home/terry/bin/arduino-1.8.4/hardware/tools/avr/bin/avr-objcopy" -O ihex -R .eeprom  "/tmp/arduino_build_46171/compilerBugTest.ino.elf" "/tmp/arduino_build_46171/compilerBugTest.ino.hex"
Sketch uses 1794 bytes (5%) of program storage space. Maximum is 32256 bytes.
Global variables use 190 bytes (9%) of dynamic memory, leaving 1858 bytes for local variables. Maximum is 2048 bytes.

Do I think right if I say the toolchain on macOS and Linux are the same anyway (very similar UNIX under the hood)

So if you are with arduino.avr-gcc=4.9.2-atmel3.5.4-arduino2 (on linux) this seems the same version for OS X although not the same version of Arduino IDE and I see no meaningful difference for the compilation flags

```
avr-gcc/4.9.2-atmel3.5.4-arduino2/bin/avr-g++"  -c -g -Os -Wall -Wextra -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10805 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR

avr/bin/avr-g++" -c -g -Os -Wall -Wextra -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10804 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR
```

nor at link time

```
avr-gcc/4.9.2-atmel3.5.4-arduino2/bin/avr-gcc" -Wall -Wextra -Os -g -flto -fuse-linker-plugin -Wl,–gc-sections -mmcu=atmega328p  -o

avr/bin/avr-gcc" -Wall -Wextra -Os -g -flto -fuse-linker-plugin -Wl,–gc-sections -mmcu=atmega328p  -o
```

I downloaded IDE 1.8.5

arduino-1.8.5 ( arduino.avr-gcc=4.9.2-atmel3.5.4-arduino2 ) on linux.

Does exhibit the same problem that JML reported.

STARTING
1
2
3
4
5
6
7
8
9
10
...

Just for grins I swapped tools folders between IDE-1.8.4 and IDE-1.8.5.

Using IDE-1.8.5 with tools from IDE-1.8.4 your test program fails.

Using IDE-1.8.4 with tools from IDE-1.8.5 your test program works as expected.

Just to stir the pot, remove loop

int n = 1;

void setup()
{
    Serial.begin(115200);
    Serial.println(F("STARTING"));
    while(1)
    {
        if ( n <= 3 ) {
            boolean r = false;
            while ( !r ) {
                if (Serial.available()) r = (Serial.read() == 'c' );
            }
            Serial.println(n);
            n++;
        } else while (1);
    }
}

works fine 1.8.5/4.9.2

tf68:
Just for grins I swapped tools folders between IDE-1.8.4 and IDE-1.8.5.

Using IDE-1.8.5 with tools from IDE-1.8.4 your test program fails.

Using IDE-1.8.4 with tools from IDE-1.8.5 your test program works as expected.

LOL the plot thickens!!

Yeah, if you turn off lto ( link time optimization ) it works in 1.8.5.

Also DKWatson's sketch will only compile with lto on.

As far as I can tell the build scripts are essentially identical between 1.8.5 and 1.8.3 (presumably also between 1.8.5 and 1.8.4). Is that what everyone else is seeing?

The only notable difference was -w versus -Wall.