- As there is no magic smoke, it is most likely 5V
The amazing part at this point is that you got this far without the grounds connected together.
So keeping the rx as TTL 5v makes sense as the receeving devices can Listen and the tx line makes sense being rs232 signal because having a buffer is important.
Frankly, I don't think you have the RX, TX, and GND identified properly from the mystery source device. I'd bet even money that you have GND confused with either TX or RX.
1) "Just plugging in wires" is a dangerous method. You are lucky you haven't destroyed the serial interface of the mystery box.2) What is powering the mystery box? Are there any other electrical connections to it? Specifically, is there anything that might be GND? Use your meter and try to see which wire seems to be connected to GND.3) Can you look inside the mystery box? Can you trace the wires back to whatever chip they are using to drive the serial port?
Any way to detect this besides just plugging wires and testing like I did?
But did you? There have been three wires to be identified.(1) Ground. This device must be powered somehow. Ground should show a stable 0V WRT the power supply ground.(2) Transmit (from the point of that device) That can be toggling, it depends on the activity of the device. As you said you have already received somthing useful, this is easy!(3) Receive (from the point of that device) That is a (can be) a high impedance input. It should NOT read 0V but some100 milivolts. You can further check be pulling it up and down with a 10k resistor, and you should have 0V and 5V there.Note: Those hints base on the assumption that there is no RS232 interface involved.