Do you happen to know why this happens?
Without the grounds connected you haven't a common reference where the potentials are measured from. The 5V the Arduino is running on is the voltage compared to ground. If you connect the ground from another device (also running on 5V) to the 5V pin of the Arduino, then you have a voltage of 10V from the Arduinos ground to the 5V of the other device (the voltages adding up).
I'm not using any pull-up resistors also. Should I ?
The Arduino has internal pull-ups for the I2C but they are weak (20k-50k), so with more than 1 device connected or with a bus length of more than 10cm you should add external pull-ups but only one pair per bus (one resistor for the SDA, the other for SCL). The value should be somewhere between 2k2 to 4k7, depending a bit on the buses total capacity and the voltage you're pulling to.