In this case the compiler does the work because you only use constants. The compiler knows, that prog[0][4] is 99, inserts that directly and compiles it to x=99;
If you use variables as index this is not possible and the array must be accessed at runtime. Because the AVR processors need special commands to read data from PROGMEM you must advise the compiler to do so. This is done by the pgm_read... functions.