Problems with too big vectors

Projects Programming
1

Hi,
I use an Arduino Mega2560 and I'm trying to save a vector in another vector with a higher dimension. For the first vector I use a struct for the elements.

#include <ArduinoSTL.h>

struct exstruct{
  uint16_t exint;
  float exfloat;
};

std::vector<exstruct> exvec = {{5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}};

std::vector<std::vector<exstruct>> exvecvec = {exvec};

void setup(){
  Serial.begin(9600);
  delay(1000);
  Serial.println("Reached setup");
}

void loop(){

}

If the first vector is too big, the program won't reach the setup function and outputs some squares and a mirrored question mark. In this code I left one element too much.
If I don't put it in the second vector, it works fine. The size of both vectors has to be dynamic, so using an array doesn't work.

My guess is that the memory is full at this size, even though it just reports 34% of dynamic memory used.

I tried using a pointer, so that the second vector just saves the position of the first vector but sadly I didn't get that to work. I've never really worked with pointers, so if somebody is more experienced with them and can help me solve the problem with pointers I'd be grateful.

I also tried saving it to PROGMEM, but I that didn't work because of the dynamic size of the vectors.

Is there any other way to solve this problem?
Or is the memory not even the problem?

2

I'm not much of a C++ programmer. Reported dynamic memory usage does not include dynamically allocated memory; that might be the case with your vectors.

Do a search for Arduino Free Memory and implement that; add a call to the function in setup() and check how much is actually free. E.g. Measuring Memory Usage | Memories of an Arduino | Adafruit Learning System. Reduce the size of your exvec by a few elements (e.g. 50) and take it from there.

3

What problem are you trying to solve? What is your ultimate goal? There's likely a better way to do it considering limitations of the processor you're using.

That being said, I'm wondering if an extra copy of the first vector is being made when you include it in the second vector. You might try:

std::vector<exstruct> exvec = {{5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}, {5,0.5}};
std::vector<std::vector<exstruct>> exvecvec = {std::move(exvec)};

But, don't try to access 'exvec' because it's data will have been taken over by 'exvecvec'.

Also, as @sterretje implied, the compiler can't possibly know how much "dynamic" memory (i.e. dynamically allocated) is being used. If it could, then it wouldn't be "dynamic" would it?

4

Your vector has 393 elements of 6 bytes each for a total of 2358 bytes. That shouldn't cause memory problems on a MEGA 2560 (8K SRAM). 34% of 8192 bytes is about 2785 bytes so that seems reasonable. Did you get any compiler warnings?

When I try to compile it I get this error:

Arduino: 1.8.19 (Mac OS X), Board: "Arduino Mega or Mega 2560, ATmega2560 (Mega 2560)"

new.cpp.o (symbol from plugin):
   In function `operator new(unsigned int)':
(.text+0x0): multiple definition of `std::nothrow'

libraries/ArduinoSTL/new_handler.cpp.o (symbol from plugin):(.text+0x0): first defined here

collect2: error: ld returned 1 exit status
exit status 1

There seems to be a conflict between the ArduinoSTL library and the built-in 'new' operator. Are you using an older version of the Arduino IDE or AVR core?

5

Shouldn't you have

#include <vector>

instead of

#include <arduinoSTL .h>?

Edit: I have now put quotes around the code because it did not show up in my post...

6

That’s seems full of issues

Going to one of the 32 bit arduino compatible hardware (eg ESP32) could be useful

7

just tried in wokwi with an ESP32 and it works fine

Serial Monitor (@ 115200 bauds) will show

Reached setup
393 struct members in vector
1 vector in vector of vectors
5 0.500000
5 0.500000
5 0.500000
...

8

Thanks for your answer.
When it still works, it reports 465 bits left. Regarding that one element adds 18 bits, there should be some space remaining.

9

I'm trying to play some music with a passive buzzer. The first element represents the note and the second element the length it is supposed to be played.
The exvecvec vector is supposed to hold multiple tracks which can be played with different buzzers.
I'm sure there are some libraries to achieve something like that, but I wanted to try it myself.

Using std::move doesn't change anything for me. Nontheless thanks for your help.

10

everything is static then, you don't need vectors or complicated constructs...

an array of struct (note, duration) for a song
an array of struct (song pointers, song size) for the song list

everything could be stored in flash

I would advise to store the note/frequencies in a integral format rather than double, and possibly through a look up table to save memory. duration could also be done in the same way ➜ to some extent you could have 1 byte for the note and 1 byte for the duration which would be indexes in look up tables.

11

So your idea is to make a vector of vectors.
That would run out of memory pretty soon if every vector (song) has almost 400 bytes.

3 songs should be possible...

As JML suggested, you can save a factor 3 by using one byte for tone (which piano has 255 keys?) and one byte for duration. If you can do with 8 durations and 32 tones you could even pack it in one byte...

That would yield 18 songs (almost a jukebox)...

vecvec is not the best name...
songLibrary would be better...
And rename vec to song then...
That would make things less abstract and it would be easier for us to understand your intent.

12

That seems to work, thank you.

13

Have fun !

14

vectors are for when the size and contents of the array are going to change at runtime. It doesn't seem like that's happening for your songs. Instead, just store each song in it's own PROGMEM array, then use another array to store references to those arrays. I've fit several songs into this project here that I'll share with you.

Song3.zip (7.7 KB)

It's an old project, so you might have to change some stuff up to get it to work. I know for a fact that the prog_int16_t stuff is deprecated. In my project, I use a special terminator value (0 pitch and 0 duration) so the program knows when it has reached the end of the array and should repeat from the beginning.

Why yes, I am a Pokemon fan, what makes you think that?

15

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.