different kind of memories for storing strings

Hardware Arduino Yún
1

Some questions on manipulating where MCU strings are allocated.

If I understand correctly:

char message[] = "I support the Cape Wind project.";

Q1:
Will put the string in PROGMEM, and at boot copy it to SRAM. With the F() construct this copy can be avoided.
Why is this not the default? I.e. what is the reasoning behind the copy of constant data to SRAM? Is it faster?

Q2:
Strings can also be stored in EEPROM memory. But if you do this to save on constants taking space in PROGMEM, I assume something else than the eventual sketch must put the data there?

2

The programming model for a C/C++ program is simple: there is 'text' (code) and 'data'; the second part is divided in initialized and uninitialized data. The first part needs to be kept somewhere, where the second part (the uninitialized data) can simply be a piece of ram initialized to all bits zero before the program starts.

The initialized part is either writable or not writable. The gnu compiler assumes the data to be writable, so the initialized data has to be put in RAM before the program starts. C++ complicates this model somewhat, because constants are not supposed to be writable, so it can stay either in ROM or the compiler has to do its best and forbid writing to constant memory.

kind regards,

Jos

3

NewLine:
Q1:
Will put the string in PROGMEM, and at boot copy it to SRAM. With the F() construct this copy can be avoided.
Why is this not the default? I.e. what is the reasoning behind the copy of constant data to SRAM? Is it faster?

The ATmega processors use a Harvard architecture. Program memory and data memory are two distinct memory banks. The program memory is optimized to provide efficient instruction fetches, while the data memory is optimized for efficient data storage and manipulation. At system startup, the strings (and other initialized data) is copied to RAM for efficient access. The F() macro prevents this from happening, but it limits you to using only certain functions to access it, those that are specifically coded to use the special instructions needed to access program memory.

So, it copies strings to RAM to make access more efficient. In addition, you didn't define your string as const, so it is a variable string that can be changed, so it MUST be copied to RAM to allow potential changes.

Q2:
Strings can also be stored in EEPROM memory. But if you do this to save on constants taking space in PROGMEM, I assume something else than the eventual sketch must put the data there?

First, keep in mind the EEPROM access is MUCH less efficient than program memory access,which is less efficient than RAM. Any access to EEPROM data requires that it first be copied to RAM.

To get data into EEPROM in the first place, it must be written there by a sketch, I'm not aware of any other way. That means that those strings need to be in the sketch's program memory (and RAM!) before they can go in EEPROM. It would be a terrible waste for the same sketch to write the EEPROM, and then later read it. So that means loading a sketch to put the strings in EEPROM, then another one to access and use them.

Having said all of that, since this is posted on the Yun forum, there is another option for storing a large number of strings: the Linux processor. If only occasional access to a large number of strings are required, storing them on the Linux side and requesting them as needed us a viable solution. There are many ways tondo this. There was a recent thread about a Yun based robot that needed to randomly send a large variety of tweet strings. The developer was struggling with memory constraints. Fetching them from the Linux side was one of the proposed solutions.

4

Thanks for the answers, these are very useful. At the moment I have no issues, just trying to understand.

I realize now that my example was badly chosen.

In my code I have many cases where I do for example:

Bridge.put("this is my key","this is my message");
strncpy(msg,"here I am");

So the strings are not really assigned to variables, rather inputs arguments for functions.

What does the compiler do with these strings? In what memory do they arrive?

--

The strings are mostly "human readable messages" towards the CPU. E.g. if the MCU sees that a sensor is broken it reports this to the CPU by doing something like:

Bridge.put("errors","the temperature sensor is broken!");

Obviously this is wasteful and I could just send a shorter string, or an int, or .... but whilst developing my code, a directly readable message is easier. Therefor getting the message from the CPU to send it back to the CPU isn't really what I want to do :slight_smile:

To get data into EEPROM in the first place, it must be written there by a sketch, I'm not aware of any other way. That means that those strings need to be in the sketch's program memory (and RAM!) before they can go in EEPROM. It would be a terrible waste for the same sketch to write the EEPROM, and then later read it. So that means loading a sketch to put the strings in EEPROM, then another one to access and use them.

Exactly, but I am reading conflicting messages whether writing a new sketch also erases (by default) the EEPROM.

5

NewLine:
What does the compiler do with these strings? In what memory do they arrive?

As I'm sure you suspect from your previous posts, they are stored in PROGMEM, so that they are available when power is applied. Then, as part if the start-up process, those strings are copied to RAM for efficient access. This two-step is needed because the contents of RAM are lost when power is off. PROGMEM is in flash memory, so it is non-volatile and stays until overwritten by a new sketch.

Obviously this is wasteful and I could just send a shorter string, or an int, or .... but whilst developing my code, a directly readable message is easier. Therefor getting the message from the CPU to send it back to the CPU isn't really what I want to do :slight_smile:

Absolutely! There is nothing wrong with "wasting" memory to make operation better or easier. It only becomes an issue when you are running low.

There are those who say you need to be hyper-efficient in code and memory usage all of the time, and always use all of the tricks to minimize memory usage and instruction cycles. I don't agree with this. There is some merit to that when you know your code will be part of a bigger system that is likely to run into memory issues before you're done. But for a small special purpose sketch which will only ever use a fraction of the resources, I hardly feel it's necessary.

There have been times where I've wanted to use short strings for efficiency (either run time processing, or because the code will eventually be part of a larger project) but I wanted to use more informative strings for testing. So, to use your broken sensor example, I would put definitions like this in a header file:

#ifdef VERBOSE_MSGS
  #define ERR_KEY "errors"
  #define TEMP_FAIL "The temperature sensor is broken!"
#else
  #define ERR_KEY "E"
  #define TEMP_FAIL "T"
#endif

This include file (and the fact whether VERBOSE_MSGS is defined) would be included by both the sending and receiving program. Then, when it's time to send the error:

Bridge.put(ERR_KEY, TEMP_FAIL);

The best of both worlds, and easy to switch between modes.

Exactly, but I am reading conflicting messages whether writing a new sketch also erases (by default) the EEPROM.

I've not used the EEPROM in the Arduino environment, so I can't say. It would be an interesting test. If EEPROM is reset by loading a sketch, then perhaps there is a way to preset it as well?

6

BTW, this copying strings to RAM is not limited to Arduino. I'm working several projects using Freescale Kinetis processors that have an ARM core, which is also a Harvard architecture. By default, the CodeWarrior compiler copies all strings to RAM, because fetching data from that bank doesn't need special instructions or extra memory cycles, and also because RAM is faster than flash memory. Reading from internal RAM has zero wait states, while reading from flash adds wait states to give the memory time to access the data.

Fortunately, the CodeWarrior compiler provides a pragma that turns on an option to keep const strings in flash memory and not copy them to RAM.

7

I use additional eeprom to hold strings that can be used by several sketches. They are usually arrays so the sketch only needs to access what it wants, saving memory usage.

eeprom storage survives power off and is only replaced when rewritten.

8

If your intention is to learn and explore, writing to the EEPROM is great.

If you have a Yun, don't want to wear out the limited cycles of writing to the EEPROM and want to permanently store data:

  • have the arduino side store it's data into the bridge (just like you described)
  • have a python (for example) program run on the Yun side that
  • reads
  • whether the whole bridge data as one data structure and writes it to a SD card
  • or individual bridge messages and writes them to a database
  • does this saving of bridge data content whether every second, minute or hour (depending on your project)
  • gets initialized during startup of the Yun
  • pulls all the stored data and writes it to the bridge upon startup

This way your arduino side can permanently store gigabytes of data.

9

NewLine:
Some questions on manipulating where MCU strings are allocated.

If I understand correctly:

char message[] = "I support the Cape Wind project.";

Q1:
Will put the string in PROGMEM, and at boot copy it to SRAM. With the F() construct this copy can be avoided.
Why is this not the default? I.e. what is the reasoning behind the copy of constant data to SRAM? Is it faster?

Most importantly, it is not writeable in flash RAM.

Q2:
Strings can also be stored in EEPROM memory. But if you do this to save on constants taking space in PROGMEM, I assume something else than the eventual sketch must put the data there?

EEPROM is not a good place to put program output strings in order to save RAM. Accessing EEPROM access comes with large penalties in terms of access time and program delay, including disabling of interrupts during EEPROM access. EEPROM should (only) be used for saving program default settings that are once read at sketch startup (if previously saved) and only written when such program/sketch settings change.

Ralf

10

PCWorxLA:
Most importantly, it is not writeable in flash RAM.EEPROM is not a good place to put program output strings in order to save RAM. Accessing EEPROM access comes with large penalties in terms of access time and program delay, including disabling of interrupts during EEPROM access. EEPROM should (only) be used for saving program default settings that are once read at sketch startup (if previously saved) and only written when such program/sketch settings change.

Interesting. But then with the Yun, it is probably better to put the settings on the CPU side.

11

NewLine:
Interesting. But then with the Yun, it is probably better to put the settings on the CPU side.

Well, no, not really. And not sure what you think makes it "better" going through the bridge to access anything on the Linux side. RAM storage "over there" is definitely NOT an alternative for example in cases of a power outage. And you actually dealing with a lot more code to access anything, including an SD card, on the MCU/sketch side...

Ralf

12

The way I see it, the "best" location for storing information depends on many factors. Each method has its pros and cons, and only by looking at the big picture can a decision be made. There is no one-size-fits-all solution.

This whole discussion looks at things from the point of view of the Yun's '32U4 processor. The AR3391 has completely different considerations.

For fixed data (including strings) that never change, the best location is usually in PROGMEM, especially if the data is accessed frequently.

However, if there is a large amount of data that does not fit in PROGMEM, and it is only accessed occasionally, then storing it on SD card, or requesting it from the Linux side may have appeal, even though there is some additional overhead to access it. But I wouldn't consider it unless it doesn't fit in PROGMEM. Storing such data in EEPROM would not be an option since there is little of it (and we're talking large amounts of data here.)

For smaller amounts of configuration data that changes rarely (mostly, but not completely constant) and which must survive power cycles, then EEPROM is the best choice. This data would be read on power up, and only written when it actually changes. Such data could be stored on the Linux side, but the extra overhead of fetching it is probably not worth it, and there is a significant delay before the Linux side boots and is ready -- EEPROM is ready right away.

If there is a larger amount of rarely changing configuration data, which doesn't fit in EEPROM, then storing it on SD card makes sense. Just keep in mind that reading it at startup would have to wait until the the Linux side has booted. A mixed scenario may make sense if there is a smaller subset of data that must be accessed shortly after boot (put that in EEPROM) while the rest that doesn't fit can be accessed later (put that on SD, or request it over the bridge.)

There is no single answer to the best type of memory to use for any particular system. You really need to do the analysis and make the determination based on the specifics of each application.

RAM: fastest access, limited quantity, loses values on reset or power failure

PROGMEM: slower access, higher quantity, changed only when loading sketches, retains values on reset or power failure.

EEPROM: very slow access, very limited quantity, can be updated but limited number of write cycles before failure, retains values on reset or power failure.

SD Card: very slow access, very high capacity, can be updated easily, retains values on reset or power failure.

Bridge: slow access, medium capacity, can be updated very easily, loses values on reset or power failure

13

just to note you can buy 32K/64K bytes eeprom for £1 on ebay