specifying digital or analog pin?

should pinMode then be set for all the digital I/O that can go each way?

For all the ones that matter, yes. If the pin isn't used, there's no reason to set the mode.