Sounds like your particular implementation of the Madgwick filter is not working at all.
One of the most common (and fatal) mistakes by beginners is to use code intended for one type of sensor to process data from another type of sensor, with incorrect axis assignment, data scaling or calibration.
All sensors need to be individually calibrated in order for the filters to work.