Storing huge amount of chars in some kind of giant string

I have a massive amount of chars, caused by a buffer. Every X minutes will these chars be saved on a SD. So i want know to store all of those chars, one after the other, in some kind of giant string and after x minutes this string will be saved on a SD. I don’t know how big this string will be after X minutes, cause this depends on the buffer. Maybe 10.000 chars, maybe 1.000.000 chars.

After saving this string to my sd card, the content should be resetet.

for (int i = 0; i < sizeof(buffer); i++) {
                    if (i % 8 == 0) {
                        sprintf(buf, "%06X ", offset);
                        // STORE 'buf' IN A GIANT STRING
                        offset += 8;
                    }

                    sprintf(buf + strlen(buf), "%02hhX ", buffer[i]);
                    if (i % 8 == 7) {
                        sprintf(buf + strlen(buf), "\n");
                        // STORE 'buf' IN A GIANT STRING

                    }
                    else if ((i + 1) == sizeof(buffer)) {
                        sprintf(buf + strlen(buf), "\n");
                        // STORE 'buf' IN A GIANT STRING
                    }
                }
                offset = 0;

→ after 60 seconds, store giant string on sd, delete content of giant string and begin again storing chars in it

Your going to have difficulty finding space for 10,000 chars in an Arduino Mega with 8k of SRAM or an Uno with 2k

I think your aspirations need a make-over. Why not write to the SD Card in 512 byte chunks?

...R

You should make constant size of the buffer and save the data not when some time expires but when you have enough data to save. You also need some "extra space" in the buffer so you can save values that come while you are writing the old data to the SD card. I would do something like this:

buffer=char[1024];
int position=0;
while (true) {
  if (charReady()) {
    buffer[position]=getChar();
    position++;
    if (position==512) {
      saveToSD(0);  //saves first half of buffer to SD card
    }
    if (position==1024) {
      saveToSD(512);  //saves second half to SD card
      position=0;
    }
  }
}

I also thought about saving after a specific amount of bytes to the sd card. But i think, the amount of datas in the buffer and especially the speed how this buffers arrives, is to huge to increment an integer.

What i mean:

int position = 0;
for (int i = 0; i < sizeof(buffer); i++) {
    if (i % 8 == 0) {
        sprintf(buf, "%06X ", offset);
        // STORE 'buf' IN A GIANT STRING
        offset += 8;
    }

    sprintf(buf + strlen(buf), "%02hhX ", buffer[i]);
    if (i % 8 == 7) {
        sprintf(buf + strlen(buf), "\n");
        // STORE 'buf' IN A GIANT STRING
    }
    else if ((i + 1) == sizeof(buffer)) {
        sprintf(buf + strlen(buf), "\n");
        // STORE 'buf' IN A GIANT STRING
    }
}
position++;
printf("%i", position);
if(position == 512{
    printf("save");
    position = 0;
}
offset = 0;

Instead of

1, 2, 3, …, 510, 511, 512, save

i’ll get

1

540490017

(and no, 540490017 is definitly not the size of my buffer)

basically, the buffer is traffic. So it will come huge and fast.

Edit: Ok, but

if ((sizeof)buf % 512 == 0){
printf("512 bytes")
}

seems to work…

Edit2: Yep, worked like charm. Thanks. Quick question:

                if (sizeof(buf) % 512 == 0){
                    printf("save 1");
                    savestuff();                
                }
                if ((sizeof(buf) % 512 == 0) && sizeof(buf) % 1024 == 0) {
                    printf("save 2");
                    savestuff();            
                    memset(&buf[0], 0, sizeof(buf));    
                }
                offset = 0;

is now my code. After 512 bytes → save, after 1024 bytes → save + memset to set my variable buf back to avoid overflow.
Whats a better way instead of

                if ((sizeof(buf) % 512 == 0) && sizeof(buf) % 1024 == 0)

?
When i only ask for ‘% 1024’ it will also save for 512. Whatever, in this ase it’s the same “problem”. Not a real problem - but its save twice in this step. I’ll getb ‘save1’ and ‘save2’ as output.

Can you please show a complete program which demonstrates the issue? These snippets are not showing the cause of your problem.

sizeof() is not going to tell you how many characters are in the buffer. It will tell you how much memory has been allocated to hold the buffer. The characters in the buffer should not occupy all the space allocated for it.

Depending on how you allocated memory for the buffer (the code you didn't show) the sizeof() probably won't ever change for the life of your program. Measuring sizeof() in if() statements is redundant.

The problem is not "caused by the buffer". The problem is caused by how you are using it. SD cards are The Arduino SD library is happy storing individual characters one-at-a-time although they are more efficient (faster) if you give them a 512-byte buffer to store in one command. If you don't need extreme speed then don't use a buffer at all: just put the data onto the SD card as it arrives.

*edit - make it clear that it's the Arduino library doing some of the work, not the card alone.

MorganS:
SD cards are happy storing individual characters one-at-a-time

Oh no. AFAIK SD card is VERY unhappy if you store single bytes. It has large (512 bytes but maybe even larger, I don't know) pages and a whole page needs to be refreshed for each single byte write. It is said there are powerful self-repairing and wear leveling mechanisms implemented in a SD card. Also Arduino is very slow writer from SD card's point of view but still I think it is not good idea idea to write single bytes. Sure, the SD card is so large it may take long time before wear will be apparent but...
I don't have much experience in using SD cards (I did not make wear tests for example) so I cannot say how long it takes to destroy a SD card by single byte writes but I believe it is ugly abuse and if you store lot of data it will ruin the card quickly.
(On second thought: maybe if the card is clever and the used library is clever it will erase the whole page first and then write to it and in that case maybe erasing won't be performed before every write and maybe the writes will not wear the other bytes in the page - but do you know if all maybes are true?)

AFAIK..

You don't know. Read the code or do a search before posting an uneducated guess that can mislead the OP.

The SD library uses a 512-byte buffer for writing. It doesn't do anything with the card until the buffer is full (from single-byte writes), or the file is flushed or closed.

For the OP, MorganS' suggestion is correct:

just put the data onto the SD card as it arrives.

In this embedded environment, process the data as soon as you can. Avoid saving and then processing. The Arduino just doesn't have enough RAM. And where are you going to put new data that starts arriving while you're processing the old data?

You can always close and delete the file if it turns out that you shouldn't have saved the data.

-dev:
You don't know. Read the code or do a search before posting an uneducated guess that can mislead the OP.

Sorry I don't want to start (another) flame.

  1. OP did not say he is using a library to write to the SD card. It is your assumption he uses this one.
  2. MorganS does not mention any library either. He wrote:

MorganS:
SD cards are happy storing individual characters one-at-a-time although they are more efficient (faster) if you give them a 512-byte buffer to store in one command.

As I understood it he suggests it is OK to save single bytes to SD card (physically). Only that saving the data in block from buffer is faster (not specifying if the buffer is from a library or hand-made).
If it was meant as you say I apologize. But note there is space for confusion: I don't consider byte being "saved to SD card" when it is still pending in Arduino's buffer. Then loss of power (or reset) means loss of data. When MorganS said "storing individual characters" I understood bytes send directly to SD card without waiting for full page.

Sorry, I wasn't clear. I've edited my post. Yes it's the SD library which is making it "happy" to save characters.