What does that tell you about a large crypto library that the Arduino side only needs to compile and link a few functions to do the 'signing'' side of the transaction? You may as well look at the size of the math.h library on disk and claim that won't fit on an Arduino.
Simsam: the internal storage of the core cryptography is probably many times larger than the key. It will teach you a lot by going through the code to find out what functions it actually uses and what storage they use. It is entirely possible that it assigns static buffers for the decryption routines that you aren't even going to use on the Arduino.