And I see two ports are Gnd, what's the difference between them?
Nothing they are the same thing.
You are not giving a full description of what you have. What port expander are you using? Have you got 4k7 pull up resistors on the I2C lines? Have you any supply decoupling on the chips?