My personal Scale project has a huge drift in its readings for weight

You remain on the scale for 30 minutes?

Have you measured the resistance of the load cell(s)? Perhaps the current draw is too much for the supply?
Do you have one of these to check the current drawn from the USB?
https://www.aliexpress.com/item/1005006550950357.html?src=google