Why don't try taking the math out of the equation and just focus on what analogRead is returning first. At the lowest sensitivity setting for that device with it laying "perfectly flat" you should get returned values of ~337 for X and Y and ~501 for Z. This is assuming you are powering the device with 3.3volts and are using the DEFAULT analogReference. If you're values are not close to those numbers and semi stable, you may have a wiring problem.