Allocating Memory buffer for class

I’m Struggling for the correct way to allocate memory without using malloc at the time of or prior to the constructor being triggered.
I do not want to pass a pointer to the class of a pre allocated byte array or use a global byte array.
I want to define the amount of memory at the time i create the instance of the class.
I do not need to change the size of the buffer after it is created.
The following examples show what I have and what and my basic research into what I want to do.

My Current Code that works but uses malloc() and free():

ComProtocolClass::ComProtocolClass(uint16_t BufferSize){
	SetBufer(BufferSize);
}
ComProtocolClass::~ComProtocolClass(){
	free( *Buffer );
}
ComProtocolClass & ComProtocolClass::SetBufer(uint16_t  BufferSize){
	free( *Buffer );
	Buffer = (uint8_t*) malloc( BufferSize * sizeof(uint8_t) );
	Length = BufferSize;
	Serial.print("Length ");
	Serial.println(Length);
	return *this; // return Pointer to this class
}// end SetBufer

Creating the class Instance:

ComProtocolClass ComP(100); // allocate 100 bytes of memory for my buffer

I would like to allocate it more like simpleFIFO does

template<typename T, int rawSize>
SimpleFIFO<T,rawSize>::SimpleFIFO() : size(rawSize) {
	flush();
}

Creating the class Instance:

SimpleFIFO<byte, 100> sFIFO;

in my class everything is of type byte.
my goal is to allocate memory using something like:

ComProtocolClass<100> ComP(); // allocate 100 bytes of memory for my buffer

My best guess at this moment is:

template<byte, int rawSize>
ComProtocolClass<byte,rawSize>::ComProtocolClass(){

I just don’t understand how this is processing to achieve the results

I am interested in documentation on how to use the < > is used in classes to allocate memory. Provideing simple descriptions and simple examples, even what this is called so I can google it would be excellent.

Thanks in advance
Z

Stop listening to all the drivel on this forum about how evil malloc is. That is a giant load of BS propagated by people who simply don't know what they're talking about. ANY time you allocate memory from the heap, malloc WILL be used, whether directly by you, or by some other function you call. There is no magical way of allocation heap memory that doesn't use malloc. So, doing it yourself is the most direct and efficient way of doing it, and many times it is the only way that makes any sense. There is NOTHING wrong with malloc, and NOTHING to be gained by trying to avoid using it.

Regards,
Ray L.

RayLivingston:
Stop listening to all the drivel on this forum about how evil malloc is. That is a giant load of BS propagated by people who simply don’t know what they’re talking about. ANY time you allocate memory from the heap, malloc WILL be used, whether directly by you, or by some other function you call. There is no magical way of allocation heap memory that doesn’t use malloc. So, doing it yourself is the most direct and efficient way of doing it, and many times it is the only way that makes any sense. There is NOTHING wrong with malloc, and NOTHING to be gained by trying to avoid using it.

Regards,
Ray L.

Thank you @RayLivingston for your confidence in the direction I started with. With my malloc() version, I’m only calling malloc() at the beginning prior to setup() and although I have the ability to change the size later, I do not plan on changing the buffer size once set.

have I accounted for everything preventing memory management errors?
I’m beginning to agree that the only way to accomplish my goal is to use malloc. the alternate seems to be complex and messy simpleFIFO made it look easy but its requires much more than I expected to get functional code.
Z

zhomeslice:
have I accounted for everything preventing memory management errors?

No, you certainly have not. Whenever you call malloc(), LOOK at the return value. If malloc returns NULL, it failed, and you did not get the requested memory. Failure to do this WILL crash the system when you try to write to the memory you didn’t get. You need to handle this case. You also call free(), without first ensuring it is pointing to a valid block allocated by malloc(). Passing a NULL pointer to free WILL crash the system.

void *p = NULL;
...
p = malloc(SOME_SIZE);
if (!p)
{
    Serial.print("Out of memory\n");
    while (1)
        ;
}
...
if (p)
    free(p);

Regards,
Ray L.

ANY time you allocate memory from the heap, malloc WILL be used, whether directly by you, or by some other function you call. There is no magical way of allocation heap memory that doesn't use malloc.

?

If I may add some drivel, the OP is trying to use templates to bind an array size at compile-time. That is one alternative to using the heap. There are other run-time allocation techniques besides the heap (i.e., malloc and free).

There is NOTHING wrong with malloc

Actually, it is the worst dynamic memory technique, and its problems are well documented. It is possible to use it safely, but that usually requires a good deal of expertise and care.

The OP has not fully described the objects' lifetimes, but if the allocations are made only during setup, then it is quite safe to use. And you must resist the tempatation to realloc, "just this once." :slight_smile: [Edit: I see that this is the case.]

However, if the allocations happen at various times after that, with overlapping lifetimes and different sizes, then it is not safe to use. Allocating random sizes from the heap, at random times, is a sure-fire way to court failure.

NOTHING to be gained by trying to avoid using it.

Actually, you would gain 730 bytes of program space and 4 bytes of RAM per allocation.

the alternate seems to be complex and messy simpleFIFO made it look easy but its requires much more than I expected to get functional code.

Yes, C++ templates usually require the implementations (code for each method) to be in the header. For something like a FIFO, it can be a good solution. For larger, more complex classes, it is not as readable or manageable, IMO.

FWIW, it is important to analyze the buffer requirements, because an embedded system needs to be implemented in a predictable way: determinism is your best friend. Understanding the maximum buffer sizes and maximum number of buffers allows you to dictate (i.e. predict) the response when those limits are reached.

From your last post it appears that you have done some of this analysis. Can I ask why you don't want to pass the storage into the constructor? That isn't much different from passing the size, especially if it only happens at init time (at file scope, before setup):

class ComProtocol
{
  uint16_t Size;
  uint8_t  Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

  ComProtocol( uint16_t size, uint8_t buffer )
    : Size(size), Buffer( buffer )
      {};

   // ... methods...
};

static uint8_t fooBuffer[100];
static uint8_t barBuffer[200];

ComProtocol foo( sizeof(fooBuffer), fooBuffer );
ComProtocol bar( sizeof(barBuffer), barBuffer );

This is a common technique, and can be made even easier to use with a #define:

#define COM_PROTOCOL(name,size) \
    static uint8_t name ## Buffer [ size ];  \
    ComProtocol name( sizeof( name ## Buffer ), name ## Buffer );

And in use:

COM_PROTOCOL( foo, 100 );
COM_PROTOCOL( bar, 200 );

All together, the sketch would be:

class ComProtocol
{
  size_t   Size;
  uint8_t *Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

  ComProtocol( size_t size, uint8_t *buffer )
    : Size(size), Buffer( buffer )
      {};

   // ... methods...
};

#define COM_PROTOCOL(name,size) \
    static uint8_t name ## Buffer [ size ];  \
    ComProtocol name( sizeof( name ## Buffer ), name ## Buffer )

COM_PROTOCOL( foo, 100 );
COM_PROTOCOL( bar, 200 );

void setup() {}

void loop() {}

On an UNO, this uses 536 bytes of program space and 317 bytes of RAM.

The corresponding heap sketch would be this:

class ComProtocol
{
  size_t   Size;
  uint8_t *Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

  ComProtocol( size_t size )
    {
      Buffer = (uint8_t *) malloc( size );
      if (Buffer)
        Size = size;
      else
        Size = 0;
    };

  ~ComProtocol()
  {
    if (Size && Buffer)
      free( Buffer );
  }

   // ... methods...
};

ComProtocol foo( 100 );
ComProtocol bar( 200 );    

void setup() {}

void loop() {}

This uses 1270 bytes of program space and 335 bytes of RAM (27 + 300 + 2*4).

Cheers,
/dev

Perhaps you should actually read the OPs actual posts:

I want to define the amount of memory at the time i create the instance of the class.
I do not need to change the size of the buffer after it is created.

With my malloc() version, I'm only calling malloc() at the beginning prior to setup() and although I have the ability to change the size later, I do not plan on changing the buffer size once set.

Regards,
Ray L.

/dev:
If I may add some drivel...

Yes! Thank you :slight_smile:

the OP is trying to use templates to bind an array size at compile-time.

Yes, compile-time allocation is preferred as it eliminates any of the potential problems over time

The OP has not fully described the objects' lifetimes, but if the allocations are made only during setup, then it is quite safe to use. And you must resist the tempatation to realloc, "just this once." :slight_smile: [Edit: I see that this is the case.]

I have no need to re-allocate the buffer once it is allocated

Can I ask why you don't want to pass the storage into the constructor?

I believed that it would be clumsy by placing my code into a class requiring the buffer to be created prior to calling the class seemed to prevent me from controlling the buffer preventing outside influence... but it isn't required as long as it is handled. I am in favor of your #define trick as it handles everything within the file and simplifies the sketch code.

That isn't much different from passing the size, especially if it only happens at init time (at file scope, before setup):

class ComProtocol

{
  uint16_t Size;
  uint8_t  Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

ComProtocol( uint16_t size, uint8_t buffer )
    : Size(size), Buffer( buffer )
      {};

// ... methods...
};

static uint8_t fooBuffer[100];
static uint8_t barBuffer[200];

ComProtocol foo( sizeof(fooBuffer), fooBuffer );
ComProtocol bar( sizeof(barBuffer), barBuffer );



This is a common technique, and can be made even easier to use with a `#define`:



#define COM_PROTOCOL(name,size)
    static uint8_t name ## Buffer [ size ];  
    ComProtocol name( sizeof( name ## Buffer ), name ## Buffer );



And in use:



COM_PROTOCOL( foo, 100 );
COM_PROTOCOL( bar, 200 );



All together, the sketch would be:



class ComProtocol
{
  size_t   Size;
  uint8_t *Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

ComProtocol( size_t size, uint8_t *buffer )
    : Size(size), Buffer( buffer )
      {};

// ... methods...
};

#define COM_PROTOCOL(name,size)
    static uint8_t name ## Buffer [ size ];  
    ComProtocol name( sizeof( name ## Buffer ), name ## Buffer )

COM_PROTOCOL( foo, 100 );
COM_PROTOCOL( bar, 200 );

void setup() {}

void loop() {}



On an UNO, this uses 536 bytes of program space and 317 bytes of RAM.

The corresponding heap sketch would be this:



class ComProtocol
{
  size_t   Size;
  uint8_t *Buffer;

public:
  ComProtocol()
    : Size(0), Buffer( (uint8_t *)NULL )
      {};

ComProtocol( size_t size )
    {
      Buffer = (uint8_t *) malloc( size );
      if (Buffer)
        Size = size;
      else
        Size = 0;
    };

~ComProtocol()
  {
    if (Size && Buffer)
      free( Buffer );
  }

// ... methods...
};

ComProtocol foo( 100 );
ComProtocol bar( 200 );

void setup() {}

void loop() {}



This uses 1270 bytes of program space and 335 bytes of RAM (27 + 300 + 2*4).

Cheers,
/dev

Thank you @RayLivingston your information on malloc and free is invaluable and I am keeping these as I may one day need this type of memory control. I am now comfortable with using malloc because of your input. But :slight_smile: I need to avoid using the heap and prefer binding this array at compile-time.

@/dev This is excellent and in the direction I am wanting to go.

Thank you

Additional input is always welcome :slight_smile:

Z

zhomeslice:
But :slight_smile: I need to avoid using the heap and prefer binding this array at compile-time.

And what do you imagine is the advantage of that?? If you only allocate the memory once, there is absolutely no difference, other than the specific location of the block., which is irrelevent. The only risk of using malloc is if you are doing a LOT of malloc and frees in a chaotic pattern, you can end up with numerous small blocks of memory and be unable to allocate a single large block. But you are doing only a single allocation, and that's it. There is ZERO risk there. And, the critical I keep pointing out, that keeps getting ignored by the pedants here is that wise use of malloc and free can very often allow you to INCREASE the functionality of your program. In a limited memory system, the very LAST thing you want to do is waste memory. But using static allocation you are ALWAYS using the maximum amount of memory, even though much of it is probably very rarely actually used. With dynamic allocation, you can allocate memory when you need it, free it when you don't, and have, in effect, more total memory, provided you don't actually NEED all of your data at any one time.

And the hazards of fragmenting memory are vastly over-blown here as well. In most applications, memory will be allocated and freed relatively locally, or in a somewhat predictable pattern. This will generally allow free to re-combine small blocks into larger blocks. It is allocating and holding large blocks of memory that is much more likely to become problematic. I have applications that do hundreds of mallocs per second, and run just fine for days and days and days. Those applications would require vastly more RAM to operate with static allocation.

Once again, I'm betting many, if not most, of the people here who talk about the "dangers" of malloc, have never actually used it, and are simply repeating what they've read or heard. malloc comes from the very first implementation of c by Kernighan and Ritchie, and is used in every c compiler, every c program and every version of Unix/Linux/etc. since the mid-'70s. Back in those days, 4K of RAM was a lot on many systems, yet malloc was still used successfully. Go figure...

Regards,
Ray L.

RayLivingston:
And what do you imagine is the advantage of that?? If you only allocate the memory once, there is absolutely no difference, other than the specific location of the block., which is irrelevent. The only risk of using malloc is if you are doing a LOT of malloc and frees in a chaotic pattern, you can end up with numerous small blocks of memory and be unable to allocate a single large block. But you are doing only a single allocation, and that's it. There is ZERO risk there. And, the critical I keep pointing out, that keeps getting ignored by the pedants here is that wise use of malloc and free can very often allow you to INCREASE the functionality of your program. In a limited memory system, the very LAST thing you want to do is waste memory. But using static allocation you are ALWAYS using the maximum amount of memory, even though much of it is probably very rarely actually used. With dynamic allocation, you can allocate memory when you need it, free it when you don't, and have, in effect, more total memory, provided you don't actually NEED all of your data at any one time.

And the hazards of fragmenting memory are vastly over-blown here as well. In most applications, memory will be allocated and freed relatively locally, or in a somewhat predictable pattern. This will generally allow free to re-combine small blocks into larger blocks. It is allocating and holding large blocks of memory that is much more likely to become problematic. I have applications that do hundreds of mallocs per second, and run just fine for days and days and days. Those applications would require vastly more RAM to operate with static allocation.

Once again, I'm betting many, if not most, of the people here who talk about the "dangers" of malloc, have never actually used it, and are simply repeating what they've read or heard. malloc comes from the very first implementation of c by Kernighan and Ritchie, and is used in every c compiler, every c program and every version of Unix/Linux/etc. since the mid-'70s. Back in those days, 4K of RAM was a lot on many systems, yet malloc was still used successfully. Go figure...

Regards,
Ray L.

:slight_smile: This is quite informative. As of now I haven't completely set myself to either method of allocating memory. My first attempt used malloc() worked. Your additional information gave me confidence to pursue this. I'm attempting to create code with the #define trick but while it compiles it immediately crashes resetting the UNO... so this is giving me grief lol. So I my thoughts are returning to using malloc because of your confidence in using this path. I first learned C back in 1992 and used malloc then. so I'm familiar with it but using it on ATMega328 chips that is uniquely different from a DOS PC.

Z

zhomeslice:
I'm familiar with it but using it on ATMega328 chips that is uniquely different from a DOS PC.

Nope, it is absolutely 100% the same in every way.

Regards,
Ray L.

Why not set the buffPtr to NULL in the constructor as a flag. Then use a begin(newBuffPtr,numBytes); to pass in your buffer during setup()? Or just call setBuff(numBytes) during setup()?

-jim lee

The goal is to have a passthrough class that handles connecting to any output Serial,i2c, rs485, SPI. The reason for this class is that I am connecting multiple UNO’s using RS485 with a single MAX485 chip per UNO. this creates issues with integrity so my code will handle checking that everything is good before using the received data. One UNO will be the master and the rest will be slaves (10~15). So the need to handle a buffer to receive data is necessary also the data arriving could be easily corrupted by many sources and so I must check for valid results before using the data. and this is why I have the following ComProtocol code. I designed this so that it could be used with i2c, SPI, Serial, or any other communication connection. I have used the simpleFIFO library to simulate a serial connection so only one UNO is necessary.
The goal of this code is to be non blocking and allow other processes to occur if this caught waiting for the remaining data to arrive.

Working Test Code:

#include <ComProtocol.h>
#include <SimpleFIFO.h>
ComProtocolClass ComP(100);
SimpleFIFO<byte, 100> sFIFO;

uint8_t BS[256];
void setup() {
  Serial.begin(115200);
  Serial.println("Test ");
  ComP.OnFail([](uint8_t Code) {
    Serial.print("Fail Code ");
    Serial.println(Code);
  });
  ComP.CallAvailable([]() {
    return (xavailable());
  });
  ComP.CallRead([]() {
    return (xread());
  });
  ComP.CallWrite([](uint8_t Byte) {// We have to use a lambda function to tell Serial.write what data type we are using
    xwrite(Byte);
  });
  char * test = "This is a test    ";
  delay(1000);
  Serial.println(test);
  ComP.sendMsg (test , strlen(test) + 1);
  Serial.println("1");
  delay(10);
}

void loop() {
  //  ComP.SetBufer(StringHolder);
  if (ComP.recvMsg ()) {
    Serial.println((char *)ComP.Buffer);
    ComP.Buffer[15]++;
    ComP.sendMsg (ComP.Buffer, strlen(ComP.Buffer) + 1);
    Serial.println();
  }
  delay(10);

}

int xavailable() {
  return (sFIFO.count());

}

int xread() {
  return (sFIFO.dequeue());
}
void xwrite(int x) {
  sFIFO.enqueue((byte)x);
  Serial.print((byte)x, HEX);
}
ComProtocolClass::ComProtocolClass(uint16_t BufferSize){
	SetBufer(BufferSize);
}
ComProtocolClass::~ComProtocolClass(){
	if (Buffer) free( Buffer ); 
}
ComProtocolClass & ComProtocolClass::SetBufer(uint16_t  size){
	if(Buffer) free( Buffer );  
	Buffer = (uint8_t *) malloc( size );
	if (Buffer)
	Length = size;
	else{
		Length = 0; // if this occurs I  need to fail this unit thus I'm keeping the following line
		Serial.print("Out of memory\n");
		while (1){
			uint8_t N = 0;
			if(ErrorLED) digitalWrite(ErrorLED, bitRead(B10101000,N%8));
			N++;
			delay(500);
			
		}
	}
	Serial.print("Length ");
	Serial.println(Length);
	return *this; // return Pointer to this class
}// end SetBufer

I didn’t have success in allocating memory at compile time with the #define trick… I got it to compile but it would crash the second it attempted to send load the buffer and it would reset the UNO.

I did have success with malloc I am not as nervous using malloc as I was when starting this post. Please let me know if you spot any other errors.

Z

PS. My code uses anonymous functions (lambda) to handle the function callbacks allowing the ComProtocol class to handle all the direct input from the serial connection.

Interrupts.cpp (6.57 KB)

Interrupts.h (5.44 KB)

RayLivingston:
Passing a NULL pointer to free WILL crash the system.

No, it won't.
free(NULL) does nothing.