What limits the size of arrays?

I have a program that stores a matrix calculated from inputs and outputs from previous loop{} iterations, then does some linear algebra to calculate the best outputs to send at the present loop{} iteration in order to achieve a desired set of inputs.
That means for N inputs and M outputs and a “Memory” of some integer number of clock cycles, I have an NMemory x MMemory matrix.

It also stores some intermediate matrices that I may or may not figure out how to eliminate in the future.

I am looking to make that matrix as large as possible to make my program as accurate as possible. I found that for
N=4;
M=4; //I plan on using more of both once I get this working.

Memory must be <16 or I get the following error:
/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/bin/…/lib/gcc/avr/4.3.2/…/…/…/…/avr/bin/ld: arduino_code_c.cpp.elf section .bss will not fit in region data
/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/bin/…/lib/gcc/avr/4.3.2/…/…/…/…/avr/bin/ld: region data overflowed by 1360 bytes

I thought it was a little odd that I would get an overflow if the program with Memory=15 is compiled saying:
Binary sketch size: 11212 bytes (of a 32256 byte maximum)

So I tried the following out on a new sketch:
float M[100][100];
void setup() {}

void loop(){}

Compiling that gave me an error:
sketch_nov23a:1: error: size of array ‘M’ is too large

But compiling
float M1[100][50];
float M2[100][50];
void setup() {}

void loop(){}

says:
Binary sketch size: 450 bytes (of a 32256 byte maximum)

Just to make my test a bit closer to my original problem, I tried:
//float M1[100][50];
float M1[100][50];
float M2[100][50];
float M3[100][50];
float M4[100][50];
float M5[100][50];
float M6[100][50];
void setup() {}

void loop(){}

And that again compiled with:
Binary sketch size: 450 bytes (of a 32256 byte maximum)

So what is going on here? I still could not even reproduce the original error of overflowing. I am guessing the compiler realized I never used the variables declared in the test, so it just discarded them.

So then that still leaves me wondering what limits array sizes.

On the Atmega328 you have 2 Kb of SRAM (2048 bytes). Does that answer your question? Bear in mind some is used for purposes other than your arrays.

Binary sketch size: 450 bytes (of a 32256 byte maximum)

The sketch goes into PROGMEM (program memory). That is not where the arrays go. They go into SRAM (RAM). They are different things. You have 32 Kb of PROGMEM, and 2 Kb of SRAM.

Time for an ATMega1284 - 16 KB of SRAM.