Just a random thought, on the box of the remote, is there an agreement? There may be clause in the agreement saying you should not break apart the remote?
I don't want to do anything that might fall even within a grey area
IMHO nobody would sue you unless you start making money. Patent trolling is a business so they will not troll until there is sufficient expected profit. Most probably you will be to small to be of any relevance. However if you should really make money then you should pay an attorney to take care of this.