So, for what it's worth, there seem to have been two issues.
One is that the sensor appears to have been damaged. I tried the same code on two different units (i had a breakout board and a built-in module) and the BoB had the issue of no tilt compensation at all. I think the mag readings on it were just noise, I'll check into it more.
Two was what jremington suggested - signs on axes. Everything is relative, of course, but signs are what's important. Here's code that takes signs into account to reconcile the FreeScale tilt-compensation code with the quaternions produced by DMP:
float magComp[3];
float heading;
float phi = -ypr[2];
float theta = ypr[1];
float mx = mag[1];
float my = mag[0];
float mz = mag[2];
heading = atan2(-(mz * sin(phi) - my * cos(phi)), mx * cos(theta) + my * sin(theta) * sin(phi) + mz * sin(theta) * cos(phi));
notice that i'm explicitly not flipping the z axis (though it is in the axis definitions) and am flipping the sign on phi. this seems to produce a correct, tilt-compensated output.