I think I'm missing something because I have this problem:
https://www.nkon.nl/rechargeable/18650-size/sanyo-ur18650nsx.html
If I were to buy this battery (2500mAh 3,6 nominal output) so if I wanted to know how much kWh this battery contains I would multiply 2500 by 3,6v and divide it by 1000 right? So now I've got a battery with a capacity of 9 kWh which would mean I could output 9000watt (if there wasn't a current limit of 20A) for 1 hour? Which in my mind just seems unrealistic
2500mAh = 2.5Ah.
2.5Ah * 3.6volt = 9Wh = 0.009kWh.
That would make a lot more sense but on a lot of websites they say you have to use mAh instead of Amp-hour? Like this website:
so why do they say mAh?
Thanks anyway! 
In your OP, you calculated Wh (watt-hours) but called it kWh (kilowatt-hours) so you forgot to divide it by another 1000 to go from watt to kilowatt. The website you linked is only talking watt-hours.
Confusion is often minimized by using straight SI units in your calculation.
That means amp, not milliamp. 3600s, not hour.
Thus 2.5x3600 coulombs x 3.6V = 32.4kJ = 9Wh
(Though here we can directly use hour as input and output use it:)
2.5Ah x 3.6V = 9Wh
So the cell will run 2.5A for an hour generating 9W
In general if you see very big or very small numbers, the prefix is wrong.
10000mAh is much better as 10Ah. Lots of marketing people think they can sell 10000 better
than 10, which is of course true where the victim is innumerate. Its harder to make a mistake
with numbers closer to 1.
This happens with big capacitors too, a 0.1F capacitor is often described as 100,000uF which is
as stupid as calling 10% "100000 parts per million"
The main advantage of SI units everywhere is there are no arbitrary constants to worry about,
1 watt is 1 volt amp, is 1 kg m/s^2, is 1 joule/second - 1 is good, you don't make mistakes with 1
One little wrinkle about SI is that the unit of mass is the kg, not the gramme. Its an historical
accident and its annoying, but its not going to be fixed in for foreseeable future.
Schotsl:
so why do they say mAh?
Because your battery will not have the same voltage for all charging stages. Giving mWh is actually not that easy and often not even desired.
so why do they say mAh?
The marketing dept., 5000 mAh is much more eye catching than 5 Ah.
MarkT:
1 watt is 1 volt amp, is 1 kg m/s^2, is 1 joule/second - 1 is good, you don't make mistakes with 1
Except you just did. 
1 watt = 1 kg m2/s3
outsider:
The marketing dept., 5000 mAh is much more eye catching than 5 Ah.
NEVER EVER EVER USE YELLOW!!!
That's the point. Most people who aren't engineers don't or can't read the units.