What I am saying is that when you disconnected the 5V line, it stayed working because there are diodes built into the chip. This is a very bad way of operating the chip.
When you connected 3.3v, you need to level shift signals from 5v down to 3.3V otherwise there will be excessive currents flowing through the same built protection diodes which will damage the chip.
Level shifters can range from dedicated IC's to Transistors, to basic resistor potential dividers.