Is there a way to know how much RAM is used by an Arduino program?
I'd take that sketch with a grain of salt for two reasons. It's using the C malloc() function to try to allocate memory, and is never deeply invested in the function call stack.
First, the malloc() function requires the C runtimes implement some form of heap management, and each allocation can waste additional bytes to point to the remainder of the heap. Further, trying to allocate and free bytes like this can fragment the heap, wasting more memory each time. There may be more memory available than what malloc() can grab at any given time.
Second, if your function a() calls b() which calls c(), space is allocated on the stack for each call, plus any arguments passed into each function. This is also counted as part of the RAM that is not available to the program, but when each function returns, the stack memory is freed up again. Thus, it depends on WHEN you calculate free memory, to know how much you're really using. In most cases, you want to know the worst case, or "high water mark" for stack (and heap) allocation.
There are some optimizations that I'm sure the libraries are doing to minimize all of this, but 1024 bytes of RAM is not much room for features OR for improvement.
when each function returns, the stack memory is freed up again. Thus, it depends on WHEN you calculate free memory, to know how much you're really using.
Sort of. The "unused" stack memory is still part of the stack, which is statically allocated on cpus like the AVR. While free stack space can be used by other functions, it's NOT available for use by malloc(), or for static variables.
(This also means that there are two kinds of ram memory that you can use up. Stack memory and "other ram." In theory, for some applications you can save memory by putting variables that were declared at top level on the stack instead. Or vis versa...)
westfw, I'm not fully educated on the model that the ATmegas use, but many tiny platforms have a non-enforced sliding barrier between heap and stack-- that is, if the stack pointer isn't intruding into heap, it's all good. Stack grows down, heap grows up, in the same address space. This may not be the case here, but it's the least memory-constrained implementation. The hardest to debug without a logic emulator, though. So you're saying the ATmega has a hard barrier between the two, but I haven't seen docs that say how much is in each partition. I saw no mention of stack pointer limiting in the ATmega168 datasheet, but such a heap limit might be enforced on malloc() in the runtime library.
"I'd take that sketch with a grain of salt for two reasons."
A third reason is that I don't think it works when you compile for ubuntu (or was it mac?)
There's another version that compares the stack pointer with the address of a local variable, more reliable and much less costly.
Also the tools come with avr-objdump which can tell you the static ram allocations from the elf file (search here for avr-objdump)
Are any of the other options documented on the playground? I guess this is the only method I know of and while it seems it is not perfect it has helped me on a couple of occasions. Maybe some of the drawbacks should be mentioned on the playground?
many tiny platforms have a non-enforced sliding barrier between heap and stack
Now that you mention that, I'm not sure.
many tiny platforms have a non-enforced sliding barrier between heap and stack
Now that you mention that, I'm not sure.
So since this is coming up more and more frequently, I went and took a look at the initialization code in a compiled sketch. It does in fact simply initialize the stack pointer to the top of the RAM area, and it is permitted to grow downward (toward the space occupied by malloc'ed data, uninitialized data, and initialized data) with no checking. Also, malloc() does compare things against the current value of the stack pointer. So it DOES have exactly the sort of sliding window between heap and stack that Halley describes....
Scary, isn't it? 
