Setting an array size at compile time with an object parameter

Projects Programming
1

Hi, apologies if this is a dumb question but I’m just starting to learn c++, I’m writing my first library for Arduino, and I’m trying to work something out … all my google searches have failed to get me answers.

I have a class with two members that are arrays of variables, but the size of each array needs to be set by the person using the library.

The array sizes do NOT need to be set at runtime, they can be set at compile time and never changed, so I want to know if there is any clean way to set the array sizes when declaring the object in the sketch without using dynamic memory allocation.

I know I could just use some #define ARRAY_SIZE macros and instruct the user to insert them before declaring the object, but that seems a bit clunky and not very user friendly:

I want something like this:

MyClass myObject(16, 8);
//create myObject containing one 16 element array and one 8 element array

Instead of:

#define ARRAY_1_SIZE 16
#define ARRAY_2_SIZE 8
MyClass myObject;

Is this possible?

2

without using dynamic memory allocation.

Why? Dynamic memory allocation does not use more memory than static allocations.

3

Paul, Thanks for answering my question by asking why I am asking the question - That really helps answer my question!

The Arduino is an 8 bit microcontroler with limited resources, I was taught that it is bad practice to use run time memory allocation if it isn't strictly necessary. As I explained, dynamic arrays aren't required for my application so I want to know if there is a clean way to avoid using them. If there isn't any point in avoiding dynamic memory allocation then why use static allocation at all?

4

bryan_withypool:
Paul, Thanks for answering my question by asking why I am asking the question - That really helps answer my question!

The Arduino is an 8 bit microcontroler with limited resources, I was taught that it is bad practice to use run time memory allocation if it isn't strictly necessary.

Who taught you that? This is over-simplified. Micro-controllers have both limited CPU and limited memory, so you always have to think about both (technically, everything has limited memory and CPU, so you should always be thinking about this but in small projects you my find you have more than enough of both).

So, by deciding it is "bad-practice" to use runtime memory allocation, you are sacrificing memory re-use for better speed.

I also think you are confusing dynamic memory allocation, built once, or not very often at runtime startup, with continuous memory allocation and deallocation.

As I explained, dynamic arrays aren't required for my application so I want to know if there is a clean way to avoid using them. If there isn't any point in avoiding dynamic memory allocation then why use static allocation at all?

Just use dynamic arrays. It's the right way to go if the clients need to be able to define the size of objects passed into your library, even on microcontrollers (mind you, that smells like a design flaw to me. I'm skeptical dynamic memory is actually needed here). There is nothing wrong with dynamic memory allocation when used appropriately.

My suggestion would be to look deeper into when to use dynamic allocation vs static allocation generally, and the trade-offs, as the rule given to you is not an appropriate one, and it does require a better understanding to make good choices.

But, in general, the suggestions would be:

  • if you can't know the object size at compile time, use a dynamic array. No problem.
  • If you are creating and destroying a large object in a tight loop, put the object creation outside the loop and re-use it.
  • If you are running out of memory but you have lots of CPU time, use dynamic memory to enable re-use of memory areas.
  • If you are running out of CPU, and you have already done decent optimisation, get a bigger CPU if you can. Squeezing that last extra 10% out of your CPU takes a huge amount of effort.
5

Thanks, thats a much more useful reply.

The advice I had was just a general rule about not using malloc for everything on a microcontroler, and was backed up by all the internet searches I did aroundthe topic where people warned against using malloc on Arduino... and the fact that the memory requirement is known at compile time ...

I take you point though about it not being much of an issue if it is only used at startup.

My library stores data in some structures, but the user can specify how many structures they want when they write their code - they specify the size of an array of structs within the object - I can't see any way of achieving this without either telling the user to use a macro for the array size before declaring the object variable, or by dynamically allocating memory for the number of elements the user needs.

I agree that it seems like there should be a way of achieving what I want without runtime memory allocation, given that the array size is known at compile time - hence my origonal question!

6

bryan_withypool:
Thanks, thats a much more useful reply.

The advice I had was just a general rule about not using malloc for everything on a microcontroler, and was backed up by all the internet searches I did aroundthe topic where people warned against using malloc on Arduino... and the fact that the memory requirement is known at compile time ...

malloc is C. new is C++. Arduino is C++. So yeah, generally speaking, don't use malloc on Arduino.

7

Very simple task with a template. None of that run-time nonsense (just stirring, but seriously...)

template< int Size0, int Size1 >
  class Foo{
    public:
      int array0[ Size0 ];
      int array1[ Size1 ];     
};

//...  In sketch ...

Foo< 16, 8 > foo;

And that is it, pretty basic huh?

Be sure to use appropriate names, not like I've done here.

8

bryan_withypool:
My library stores data in some structures, but the user can specify how many structures they want when they write their code - they specify the size of an array of structs within the object - I can't see any way of achieving this without either telling the user to use a macro for the array size before declaring the object variable, or by dynamically allocating memory for the number of elements the user needs.

The usual way to do this is to either have the user create the objects then pass a reference in to your library, or for your library to have addObject methods.

I agree that it seems like there should be a way of achieving what I want without runtime memory allocation, given that the array size is known at compile time - hence my origonal question!

Well, as I said, you seem to have a design flaw. The dynamic or otherwise allocation of memory for client objects should be done outside your library by the client. The client creates the objects either statically or dynamically (it's up to the client, not you), then calls your library with the object reference.

9

PaulS:
Why? Dynamic memory allocation does not use more memory than static allocations.

It sure does. Every allocation will use: the requested number of bytes + 2 for the allocation header.
Then you also have to store the returned pointer somewhere, so that is another two bytes. If this variable is a global, then you've effectively used both static and dynamic memory.

arduinodlb:
Well, as I said, you seem to have a design flaw. The dynamic or otherwise allocation of memory for client objects should be done outside your library by the client. The client creates the objects either statically or dynamically (it's up to the client, not you), then calls your library with the object reference.

Encapsulation is a good thing, you should embrace it. Objects should be responsible for their data, which includes allocation and destruction. Just because the client needs to specify certain limitations, does not mean they should be responsible for handling an arbitrary data structure used inside a class.

10

pYro_65:
Encapsulation is a good thing, you should embrace it. Objects should be responsible for their data, which includes allocation and destruction. Just because the client needs to specify certain limitations, does not mean they should be responsible for handling an arbitrary data structure used inside a class.

You are misinterpreting my post. Encapsulation was exactly my point. This is not about limitations. It's about objects being fully responsible for the memory they know about. I never said an arbitrary data structure so I don't know where you got that from. The whole point is any arbitrary data structure is maintained privately in it's own object, and the interface between objects is clearly defined.

11

Hmmm, I do not think I misinterpreted anything.

It's about objects being fully responsible for the memory they know about.

That is what encapsulation involves, however I was responding to this:

The dynamic or otherwise allocation of memory for client objects should be done outside your library by the client.

Which, to me, appears to be in contradiction with your statement above. Do not worry, I get what you are putting forward: When using a linked list, stack, or queue style object, it is simply wrapping or organizing "user/client data".

These types of objects/systems are known to have reference semantics.

However the OP is discussing an object with some feature that, to be efficient, requires the user to provide a setting/limitation. The OP has not shown that it is infact working with user data, and can be dismissed due to not needing dynamic memory (not expecting a user to create a 'stack' style class at some random point).

There is no requirement on the user to know what these settings do inside the class. And if its specifying a memory limit, there is also no requirement for the user to know how the allocated memory is organised, hence arbitrary.

Take a running average class for example, it could contain a buffer of samples, however a user only needs access to the average result. The data may be stored in an array, but it could be a completely illogical storage design, it doesn't matter, good encapsulation separates the external interfaces from the internal implementation.

This style of object is described as having value semantics. And basically the comment which included (it's up to the client, not you) should have the roles reversed.

Cheers.

13

arduinodlb - I am aware of the differences between C and C++ but having spent quite a while trawling the web looking for the best solution to the problem, I was left with the impression that Arduino did NOTimpliment the new operator - google new operator and arduino and you see tons of posts claiming that is is not implimented (and others warning against using non static memory alloocation).

Having had a bit of a dig I can now see that this was added to the 1.0.5 IDE - its a bit of a pain digging through pages and pages of arduino tutorials saying that 'new' will not work, only to find that it now will (but there are no references to it in the Arduino documentation or the AVR lib docs that I can see ...

pYro_65: Thanks - I've never tried templates so I'll go and learn about them! :wink: I also agree with your comments about encapsulation:

My application is a serial port handler for a master/slave multidrop protocol and the Arduino lib is for the slaves. The library handles all communications in a way that is (hopefully) transparent to the user and easy for any novice to use.

Ideally, all the user needs to do is declare the comm object and register with the library any variables in their code that they want the serial comm master to be able to read and write. So long as they call a refresh function periodically the comm library will let the master read and write to the users chosen variables transparently.

In order for it to work the user has to tell the library object how many variables they want to share with the network - Then they have to register their variables (Basically call a function and pass the object pointers to the variable, and type information) The user will know how many variables they want to share at compile time and the maximum allowed is 128 - but the intended use would typically be for much less than this.

At all times the user is responsible for their data and what they do with it - I'm just writing a comm system that allows another connected device to be granted access to that data in a simple and user friendly way.

Delta_G - You are wrong. If the user only needs my library to work on 2 variables then it will consume 8 bytes and the remaining memory can be used for their own purposes. If I did what you suggest then my lib would always use 512 bytes!

16

Delta_G:

That is exactly why I want the user to be able to select the number of entries at compile time - that was the whole point of my question!

17

That is exactly why I want the user to be able to select the number of entries at compile time - that was the whole point of my question!

From my perspective, this statement seems misconceived.

Specifying and fixing the number of entries in an array "at compile time", is the complete antithesis of enabling the "user" to select it.

18

michinyon:

"Specifying and fixing the number of entries in an array "at compile time", is the complete antithesis of enabling the "user" to select it."

The 'user' = the person 'using' my library with their code -- they still need to compile their code (that uses my library) into a binary that runs on the arduino - that means that the user is able to specify parameters relating to my library at compile time.

The key motivation behind my origonal question is the fact that the array in question is hidden within an object, and not directly available to or manipulated by the user - all the user should have to do is specify its size when they invoke the object in their code.

19

bryan_withypool:
The key motivation behind my origonal question is the fact that the array in question is hidden within an object, and not directly available to or manipulated by the user - all the user should have to do is specify its size when they invoke the object in their code.

If it's completely internal to your library, why does the user need to specify the size? Generally speaking, that information should be contained within your library. You could take a suggested size from the user and then resize as necessary similar to what is done with hash buckets, or you could use a different container instead of an array.

I feel that you may be jumping through hoops to avoid dynamic memory allocation, but it doesn't look to me like there is a clear necessity to do so.

20

Delta_G:
The...

Exactly, this is in my opinion the best way to deal with something that does not need to be dynamic.

And is where my template example shines through:

It allows the maximums of the library to be set by the client, so it completely avoids the overhead associated with a client using less elements than the system allows. The maximum limit can also be enforced with a compile time error, meaning a client does not need any code to handle it. Statically sized memory allocation is a win, win, win situation.

Compile time is an important part of the programming environment. Run-time code is inherently just an overhead if it has a compile time equivalent.

If it's completely internal to your library, why does the user need to specify the size? Generally speaking, that information should be contained within your library. You could take a suggested size from the user and then resize as necessary.

Exactly, there is no reason why this cannot be a compile time interaction with the object.

I feel that you may be jumping through hoops to avoid dynamic memory allocation, but it doesn't look to me like there is a clear necessity to do so.

I do not think anyone has mentioned optimizations yet. Using dynamic memory prevents optimizations... lots of them. As you are using pointers you loose all guarantees about the const-ness of expressions using the pointers.

You can see the effect of compile-time vs run-time here:

const long test = 4;

void setup() {
  Serial.begin( 9600 );
  
  Serial.print( test );
  //Serial.print( (int) &test );
}

void loop(){}

If you remove the comment, it prints the pointer to 'test', and as a result the RAM usage jumps as the compiler can no longer guarantee what happens to 'test' and forces it into memory (the linker does not remove it as there are symbols being referenced).

Statically linked code will always optimize better.

21

Several posts on this thread have been removed.

Keep it civil, please.

22

pYro_65,

Thanks, your suggestion and comments on optimisation make a lot of sense. I've got this bit of my library working at the moment by using the new operator so I'm going to stick with that for version 1 - then when I'm more confident I'll have a got at implementing a template version as you suggested and see what that does to the memory use - could be a good learning experience!

23

The key motivation behind my origonal question is the fact that the array in question is hidden within an object, and not directly available to or manipulated by the user - all the user should have to do is specify its size when they invoke the object in their code.

If you want the array size fixed in your object class at compile time, there is still nothing stopping you from including in your class an accessor method that returns the size of the array inside the class object, so that other pieces of code which use that object can determine how many entries it will be able to put into, or obtain from, the instance of the object.