using #if WITH_XXXX .... #endif

I have a sketch that runs on a UNO but is too big to port to a LEONARDO so I can use the extra couple of analog inputs. As I'm using fixed IP addresses I'm not using DHCP or DNS and I've managed to reduce the size of the compiled code by wrapping DNS.h and DNS.CPP plus the calling functions in the Ethernet Library with #if WITH_XXXX .... #endif conditional compiling directives (see Reducing code size on Arduino Ethernet boards – Michael Noland's Blog). I've also done the same with DCHP but I still need to take some more bytes out of the compiled code.

I was wondering if I can do the same sort of thing with individual functions within the remaing parts of the library that I do not use by wrapping say 'udp.remoteIP()' in the same #if WITH_? ....#endif.

If I can do this what is the syntax for not compiling a function as opposed to a whole *.h or *.cpp file? I think it would need to look something like #if WITH_UDP_REMOTEIP() but I cant get it to work.

The preprocessing directives are handled first before the rest of the compiler gets
to see the code and are purely textual in operation, so yes you can conditionally
comment out any lines you like.

Since the compiler/linker is smart about not linking in unused functions you probably
only need to conditionally compile the calls to the functions in questions.

MarkT:
Since the compiler/linker is smart about not linking in unused functions you probably
only need to conditionally compile the calls to the functions in questions.

Is this true?
I am taking a C++ class right now and just today I was reading about functions/const/inline (even though I pretty much already know them) and it says that even unused functions take up space while inline is a substitution and will be ignored if the function is not called (which makes sense).

So, which is true D:

Where did you read that claim ?

solar_eta:
I was wondering if I can do the same sort of thing with individual functions within the remaing parts of the library that I do not use by wrapping say 'udp.remoteIP()' in the same #if WITH_? ....#endif.

#if defined (__AVR_ATmega32U4__)
  // Code for Arduino Leonardo here
#else
  // Code for other Arduino boards here
#endif

Ps991:
it says that even unused functions take up space while inline is a substitution and will be ignored if the function is not called (which makes sense).

So, which is true D:

Well, this is really a generalization. Most people here would say that, in general, unused functions do not take up space. The default compile switches cause most unused code to be optimized out. This only works if the compiler can figure out at “compile-time” whether a section of code can ever be called:

   if (false)
      call_expensive_routine();

This code will be completely optimized away. Also, it will not contribute toward “needing” the routine. If the routine isn’t called from anywhere else, it will not be linked in to the program.

If, however, the compiler cannot figure out at compile-time if a section of code can get executed, it is obligated to leave the code in… even if there is no way for the code to get executed at “run-time”. For example:

   if (var)
      call_expensive_routine();

Unless var is set to a value someplace “close”, the compiler will not be able to determine that var can only have a false value, and that the call could never take place. The code will have to stay, and the routine will have to be linked in, taking up space.

switch statements tend to cause lots of unnecessary code to be included, because the switch variable is, umm… a variable that cannot be determined at compile-time. So all the cases are included, and all the routines called from those cases are also linked in (e.g., printf).

I have often wished for a compiler that would do what I mean, not what I say. :wink:

One other thing… you said,

I have a sketch that runs on a UNO but is too big to port to a LEONARDO

You do know about the F macro for string literals, as well as other PROGMEM tricks for const data, right? You’re running out of program space, not RAM?

Cheers,
/dev

You're running out of program space, not RAM?

Just in case you don't recognize it, that's a hint to post your code.

MarkT:
Since the compiler/linker is smart about not linking in unused functions you probably
only need to conditionally compile the calls to the functions in questions.

I would be very surprised if that were true, except for functions included in a single compile. For example, your program may include a class, though it uses only a single function within that class. The entire rest of the class will be included in the generated object code. If you have a function in your program that is never called, that will be optimized out.

Regards,
Ray L.

For example, your program may include a class, though it uses only a single function within that class. The entire rest of the class will be included in the generated object code.

Do you have some proof of that? A class is really nothing more than a collection of functions. Unused functions are unused functions, whether part of a class or not.

First to clear up a couple of points:
A) I do not have enough Programme space on the Leonardo. There is enough Programme space on the Uno but it does not have enough analog inputs to do everything I need.

B) I have limited space in the box this is all going in so a Mega won't fit nor is there space to add a 4051 analog switch.

C) Yes I'm using the F macro

Now reading the above replies I think the message is that the pre-compiler does not know if a function is called so it cant exclude code/functions in #include(d) libraries that will not be used.
If that is the case why does #if WITH_DNS ...#endif work! and reduce the size of my code From the previous replies it should not work because it has not yet looked at the other code in the Ethernet Library and thus not seen any calls to the DNS functions.

I don't think that the compiler/linker is as smart as people think. If it leaves out unused functions then why does telling it to leave out DNS save 2K on the size of the compiled code

I can see that the compiler probably leaves out unused functions that you have written within your sketch but once the linker calls for library code you get all of it

So back to my original question what is the correct syntax for excluding the unused functions within the library code, how do I tell the pre-compiler that they are not wanted but the rest is.

I have just found this in the Atmel AVRLibcReferenceManual at Smart | Connected | Secure | Microchip Technology

"How the Linker Works

The compiler compiles a single high-level language file (C language, for example) into a single object module file. The linker (ld) can only work with object modules to link them together. Object modules are the smallest unit that the linker works with.

Typically, on the linker command line, you will specify a set of object modules (that has been previously compiled) and then a list of libraries, including the Standard C Library. The linker takes the set of object modules that you specify on the command line and links them together. Afterwards there will probably be a set of "undefined references". A reference is essentially a function call. An undefined reference is a function call, with no defined function to match the call.

The linker will then go through the libraries, in order, to match the undefined references with function definitions that are found in the libraries. If it finds the function that matches the call, the linker will then link in the object module in which the function is located. This part is important: the linker links in THE ENTIRE OBJECT MODULE in which the function is located. Remember, the linker knows nothing about the functions internal to an object module, other than symbol names (such as function names). The smallest unit the linker works with is object modules.

When there are no more undefined references, the linker has linked everything and is done and outputs the final application."

Getting the entire THE ENTIRE OBJECT MODULE from Library code is significant

So some how I have to tell the compiler/linker to make a partial version of the object code from a library leaving out the unused functions.

Anybody got any suggestions

Just to avoid further confusion.... The term "library" is rather ambiguous, as there are TWO kinds of "libraries". In Arduino-world, "library" typically refers to a class definition, such as Ethernet, or HardwareSerial. As indicated in the above snippet from the Reference Manual, this kind of library is compiled into an object module, and it needed by an application, the ENTIRE object code from that library is linked into the applications object module. There is no mechanism at link-time to determine "Oh, he doesn't need this function, so it won't be linked in".

But from the linkers perspective, there is another kind of library, a linker library, which is a special object file and contains a few, or many, function and constant definitions that are not necessarily part of any class. Most of the standard C library functions get included in this way. For linker libraries, only those functions that are actually USED get included in the application object file.

To exclude unused code from a class, I would be surprised if there was any way exclude unused functions from the application object file other than by manually using #if/#ifdef/etc. to exclude them from the source code for that library.

If that is the case why does #if WITH_DNS ...#endif work! and reduce the size of my code From the previous replies it should not work because it has not yet looked at the other code in the Ethernet Library and thus not seen any calls to the DNS functions.

Using pre-preprocessor conditionals like #if, #ifdef, etc., if the expression evaluates to false, has the exact same effect as DELETING the affected lines from the source code. This is done by the C pre-processor BEFORE the compiler ever sees the code. The library will be compiled (and it is compiled separately from the application code) as though those lines never existed. This happens WHEN the library is compiled, and NOT when the application it compiled. The linker will then link in this new, smaller "library" object code with the application object code.

Regards,
Ray L.

solar_eta:
First to clear up a couple of points:
A) I do not have enough Programme space on the Leonardo. There is enough Programme space on the Uno but it does not have enough analog inputs to do everything I need.

B) I have limited space in the box this is all going in so a Mega won't fit nor is there space to add a 4051 analog switch.

https://www.pjrc.com/store/teensypp.html

Problem solved.

I think much of how compiler tools work particularly in embedded systems is not
well understood by many developers and with Arduino, the IDE hides much of
how it works with has both good and bad sides.
The bad side is that the typical Arduino user is very ignorant as to how the
modules are compiled and the final linked images are created.

I see much misunderstanding and misinformation being flung around in this thread.
(solar_eta descriptions is correct, however, there have been some recent
additions to the compiler toolsets to aid the less skilled developer in creating smaller linked images -
so some of this is a repeat of solar's description)

A first point of clarification is that what Arduino calls "libraries" are not libraries at all
in terms of the compiler tools.
A true library is a collection of compiled objects that are archived into a .a file.
The Arduino IDE does build a real library for the core code. All the other "libraries" regardless
of whether it is a 3rd party "library" or the "libraries" included with the IDE are NOT libraries
as far as the compiler tools are concerned.
They are just a bunch of source modules that are compiled into .o objects and then linked in.
In fact the goofy way the IDE does its build process the .o files from an arduino "library"
will be compiled to a .o file and linked in if the sketch simply includes a header from that "library's"
directory.

In terms of whether unused code is linked in by the linker or removed by the linker
both can be correct.

The compiler creates compiled .o object modules.
The linker links compiled object .o modules to create a final image.
Those compiled .o files can be directly specified or can be pulled from a library .a file.
The Arduino IDE creates and uses only a single library .a file - it is the core files.

When linking, if the .o object is directly specified the full contents of that .o module
is pulled in. It doesn't matter how many functions or data objects are in that module
or which ones are being used. They are ALL pulled in.

When linking, after all the .o modules are pulled in, if there are any unresolved symbols
and libraries were specified, then the liker will start to look in those .a files for a .o file
that contains the unresolved symbol.
If a .o file is found that contains that symbol, again, ALL of the .o module is pulled in,
regardless of how much of it is really being used.

One thing that made real libraries a bit difficult to deal with in the old days was
that the linker did only a single pass through a .a file. So if a .o file referenced a symbol
in another .o file in .a file, the order of the .o files in the .a file made a big difference.
There was a tool for ordering the .o files and it was possible that a .a could not be created
if there were multiple .o references that depended on each other.
Multi pass through .a files was eventually added to resolve this.

The kicker in all this is that the ENTIRE .o contents are pulled in regardless of what is
actually needed/used within that .o file.
This required developers to deal with this reality by either breaking up their source code across
many files to create many .o modules, properly ordering their .o files in the .a files and properly creating their makefiles to pull in what is actually used
OR to use conditionals in the code to only compile what is needed.
In a REAL library environment you really can't use conditionals since the code must be compiled
prior to knowing what the final target is or what modules many be needed.

As toolsets have moved away from developers developing in unix environments and to using
GUI based IDEs, there is less and less control and less developer knowledge on how set up proper
build environments that only pull in the actual code being used.
These new generation developers were not getting good build images since the images
contained "dead wood" and wanted smarter tools to give them better images.
There was a desire to have smarter tools that could look
into a gigantic shit pile of object files and do garbage collection to
selectively pull out only what is needed
vs what that they told the linker they were using.

Essentially what was wanted was a way to optimize the linking process similar to
what has been done in the compiler over the past few decades:
i.e the compiler can generate good object code from mediocre high level C code.
So why shouldn't the linker be able to do something similar?

The solution was to dream up a way to make an additional linker pass through the linked image and
yank out those pieces that are not needed and then go back and fix up all the addresses/references
to account for all the removed code/data.

Note that this is fairly recent capability in compiler tools and the Arduino IDE
starting using it a few releases before the 1.0 release.

To make this happen with the gcc tools, the compiler has to be involved.
So with the gcc tools, the way this happens is that the compiler is told
to place every single function and data object into its very own linker section
so there is now block of code/data associated with each linker symbol.

The gcc options -fdata_sections and -ffunction_sections tells the compiler to do this.

Then the linker is told it can remove a section of code/text or data if there is no reference
to the symbol associated with that section.
The linker option --gc-sections tells the linker to enable garbage collection of unused sections.

The net result is that while all the dead wood code and data is still pulled in, the linker makes a final
garbage collection pass to remove all the dead wood before creating the final linked image.

The IDE uses these gcc options so the final linked image won't contain unused code or data.
i.e. if you have functions in an arduino "library" that are not called, the code for those functions
will not be linked in. (actually it is linked in but then the garbage collection will rip it back out)
There are some exceptions to this particularly in the AVR environment.

There are some special cases that "break" the linkers ability to detect that a section
is not being used.

ISR routings are one such case. If you link in code that sets up ISRs, that code will
be pulled in even if never used since there is no way to actually determine if the ISR is ever
called. This is because the ISR "magic" for a C level ISR function
installs a the h/w vector and the h/w vector points to the C ISR function. This creates
a reference that the linker "sees" so it will pull in the C ISR function. Then in some cases
the C ISR function calls other functions, and then you can end up with a cascading effect
where lots of functions and data storage is called in.

This is something to keep in mind when using the IDE given its goofy IDE build process.
The IDE will link against a "library's" .o files if its header is used.
For example, if you include the Wire.h header, you get lots of the Wire code even if you
don't use it.

Another area is with PROGMEM. If you use PROGMEM pointers inside data structures,
the linker wont be able to properly resolve that the data is never used and it will pull in the data.

There are also some cases of using data pointers to const data inside data structures that have the same issues.
I'm assuming that data reference issues are because the linker doesn't look beyond a simple reference
the way the compiler does when doing optimizations.

The linker garbage collection stuff is fairly good a removing stuff but it doesn't always remove
everything.

The best way to see what is eating up the space is simply to create a link map and symbol table.
With the 1.5x build rules you can alter the linker rule automatically build them - which is what I do.
That way I can look at them if I ever need to.

--- bill

Without actually looking at your code ( -_- ), I can only offer incremental suggestions. First, there's a few misconceptions:

solar_eta:
the pre-compiler pre-processor does not know if a function is called so it cant exclude code/functions in #include(d) libraries that will not be used.

Yes, but that's not its job. Really, the pre-processor is fairly stupid. It just deals with # statements like #include, #define and #if. Anything that matches a #define gets replaced, just like an editor Search&Replace operation. (It's a little smarter than that.) If there are no # statements in your code, it just passes through to the compiler. Mostly, it globs your file and all the included files (and their includes) into one big file for the next step: compiling.

Although we also say "compile" for the whole process, it really consists of several phases, starting with pre-processor. Then it compiles the resulting code (go read The Dragon Book). The compiler optimizes the generated code, within a fairly narrow scope (mostly within the file). This is where some code and functions may be determined to be unnecessary. You'll end up with an object code file, which may be grouped with other object files into a library. The linker will take multiple object files and libraries to create an executable, and it tries to eliminate any code that is not called by anything.

Without getting into graph theory and cut sets, the linker sees the tangled mess of routine calls as a big graph. It "tugs" on each "string" to see what it's attached to (a reference). If a collection can be removed because it never "tugs" on the loop node, it can be removed. (Yes, I'm simplifying. While composing this, I see bperrybap has posted a very detailed description, along with some history.)

The problem is that most programmers are not careful about how the references are established. You can't always structure your code to allow the compiler and linker to eliminate unused functions, though. For example, handling an ethernet packet may involve a switch statement on the protocol field. Since the compiler cannot tell at compile-time whether that protocol field is possible at run-time, the code must be included. ...And all the functions that code calls must also be included. ...And so on, until all DNS and DHCP code has been included. As far as the linker is concerned, those modules must be included in the executable.

The final solution is, as you found, #ifdef sections. There is probably a best place to insert the #ifdef blocks (i.e., cases of the switch statement), but the sledge-hammer approach will work. I would suggest making sure that the declarations (i.e., in the header files, not just the .CPP files) are also conditional so that the compiler can tell you when a piece of code still tries to use it.

If that is the case why does #if WITH_DNS ...#endif work!

Nobody is surprised that works. Preventing the compiler from seeing the declaration, definition and/or bodies will work.

I don't think that the compiler/linker is as smart as people think.

These are two huge areas of research. You have no idea of the amount of effort spent on creating and maintaining these tools. Don't blame the tools...

Personally, I fault the language for some of these problems. But that's a whole 'nother ball o' wax. :slight_smile:

Cheers,
/dev

RayLivingston:
I would be very surprised if that were true, except for functions included in a single compile. For example, your program may include a class, though it uses only a single function within that class. The entire rest of the class will be included in the generated object code.

No, it won't.

void setup ()
  {
  }  // end of setup

void loop () { }

Code size: 472 bytes.


void setup ()
  {
  Serial.begin (115200);
  }  // end of setup

void loop () { }

Code size: 1842 bytes.


void setup ()
  {
  Serial.begin (115200);
  Serial.println ();
  }  // end of setup

void loop () { }

Code size: 1922 bytes.


void setup ()
  {
  Serial.begin (115200);
  Serial.println ();
  Serial.println ("Hello, world");
  }  // end of setup

void loop () { }

Code size: 2040 bytes.


And so on. As you pull in (call) more functions (from the HardwareSerial library in this case) the code gets larger. This is under IDE 1.0.6, compiling for a Uno.

If I may suggest, test this stuff yourself, rather than speculating.

Nick,
Not quite the best example.
Serial.println() is not part of HardwareSerial but part of the Print class.

In fact, these are great examples of what I was talking about when the garbage collection
of the linker can't remove all the dead code not being used due to coding
structural issues.

(have a look at a link map to see everything that got linked in)

The reason the code jumped up so quickly when Serial.begin() was added is that the IDE linked
in the HardwareSerial code which dragged in quite a bit of dead code including some of the
Stream/Print class.
There are ways this could be coded differently to ensure that less dead code is dragged in.

But with the current coding of HardwareSerial, Stream, and Print classes in 1.06,

you end up with dead code like:
the print class write() function
This is because the linker garbage collection can't optimize out unused virtual functions.

then lots of unused HardwareSerial functions;

read(), available(), peek(), serialEvent(), store_char() [inline], flush(), write(), along with the tx and rx ring_buffer structures.
All this code and data is being dragged in but yet the sketch is not ever doing anything
that would ever need/use it.

There are some functions in HardwareSerial that are not being linked in:
end() but for the most part all of HardwareSerial code was dragged in, including lots
of code that will never be used.

While it is true that in general when code or data is not used/referenced the linker garbage
collection feature can remove it; however,
depending on how the the source code is actually coded and implemented,
the linker may be unable to remove the unused "dead" code and data.

There are several such cases in the Arduino core code and libraries that could be re-coded
to help allow the linker garbage collection to remove unused code and data.

--- bill

Thanks for all the feed back I'm mot going to fight the IDE/compiler/linker on this one. #if WITH_XXXX .... #endif took about 2K out of the sketch which was not quite enough to port the code to the reduced programming space on the Leonardo over the Uno.

Instead I've copied the whole ethernet library *.cpp and *.h files onto individual tabs on the IDE Then I set about renaming them as LocalEthernet.h etc, doctored all the #include statement, then found that I needed to also copy in 'socket','W5100', 'UDP.h' and Client.h' as well to get the code to compile. Finally I progressively removed all the functions that I was not using from the code until I got to a compile size about 200bytes less than the Leonardo will handle.

So now its testing testing testing time. If nothing falls over in my sketch I'll turn the modified Ethernet code into an additional and new reduced Library.