What would be more accurate to measure in this case, the open circuit or short circuit voltage?
The short circuit voltage is zero.
The open circuit voltage (Voc) and the short circuit current (Isc) are of interest, because those allow you to estimate the internal resistance (Voc/Isc) of the module. Your multimeter is fine for those measurements.
For maximum power transfer, the load resistance will match that value (at a given temperature difference, of course).
Do not expose standard Peltier modules to temperatures higher than about 80 C, or you will destroy them. You can buy high temperature (250 C) Peltier modules for power generation. Marlow is a good source of reliable TEGs