Closed source *.a lib with Arduino, linking issue

The Arduino compiles everything (also the libraries), because the target can be so many microcontrollers and processors. What you want to do is against the way the Arduino environment works.
If you find a way to make it work, a newer Arduino version might not accept the *.a file.

You are compiling for just one microcontroller ?
Perhaps you can create a *.cpp or *.h file with the binary code as a big array of data and call an entry function at the beginning of the data.

// The start of the code should be a function.
// For example: int commonFunction( int command, ....
const byte myClosedLibrary[] PROGMEM = { 0x10, 0x85, 0x9A ...

Now tell us, how can we check your code to see if you have used pieces of code from open source or similar licenses.