I'm looking to undertake an ambitious project, which would be to design and build a hexapod that will be capable of self-righting, similar to the Spot robot from Boston Dynamics.
My main question is about the difficulty of programming this functionally. I hear of terms like inverse kinematics and inverse dynamics, and I'd like to know what I'd need to learn in order to be able to create the program. Similarly, I have no clue about whether cheap hobby servos would be sufficient to build such a robot, or whether I would need something more specialized.
Extremely difficult, so start at the beginning. Just getting a hexapod to walk smoothly and with minimum energy expenditure is an interesting and popular challenge, so that is one place to start on the learning curve.
Hint: the search terms "hexapod gait analysis" will turn up lots of research and discussion.
whether cheap hobby servos would be sufficient
Probably not, for inverse kinematics calculations. You can easily determine that by experimenting with a single leg mechanism.
I see. So in that case, what hardware would I need once I get to the point where I'm able to program a self-balancing robot?
Look on the web to see what others have done. There are countless hexapod project posts.
I suggest to get some practical experience by building a leg mechanism and determining for yourself whether its performance will be satisfactory for the project.