To me (and you) they are quite affordable but it still is 10% of the budget of OP.
You can get them direct from Hong Kong for just under $4 (including free shipping). Just search eBay for "USBasp". Note that they usually come with only a 10-pin cable. To use them with an Arduino you will need to adapt them to the 6-pin connector. You can find 10-pin to 6-pin adapters on eBay but they will cost another $4 so someone on a tight budget might want to fabricate their own cable.
4$ is 8% of 50$
With a tight budget like this you need to use what you have in house. So I advice start with arduino isp and a good breadboard.
That said; 50$ is really tight (in the western world) to learn a new technology so it probably will not be sufficient if there are no free resources around (like a school lab).
I don't know much from the hardware side.
From the software side : If you like/know eclipse the paths Arduino IDE -> arduino eclipse plugin -> avr eclipse could be a good path and it is 100% free.
If you're not into eclipse you're probably best looking into the free AVRstudio from Atmel.