In terms of syntax you could return a pointer to the array, but then you would need to ensure that the pointer is still valid when you exit the function. In other words, if you create an array inside the function its memory will be freed when the function returns. There are ways to circumvent this, e.g. you can call malloc(), return the pointer, and then free the memory when you are done with it.
A simpler way is to pass the array as an argument to the function:
void MatMatDot(double dest, double a, double b)
// use dest as the output of your function, don't need to return anything