Large strings and concatination with strcat and F()

Projects Programming
21

westfw:
There is no way to concatenate 400bytes worth of separate PROGMEM strings without using 400 bytes of RAM.

Well, I suppose you could implement a python-esque "generator" scheme where you pass a bunch of printf-like descriptors and pointers to a "create" function, and then use an access function to read it back a character at a time. I guess fprintf_P() itself is sort-of like that, and you might be able to use its underlying support functions (vfprintf()?)

I know.
The goal wasn't to avoid this buffer, it was to avoid having to re-allocate it every time and thus avoiding memory fragmentation. This is why the preallocation of it was mentioned earlier in the thread. Along with how to avoid having to re-allocate other strings before the concatenation.

The same with constant string data, which can be stored in PROGMEM and accessed a byte at a time with F(). Also as mentioned by others strcat_P can be used to copy string.

22

UKHeliBob:
Even if you can't change the way the project has been set up in the sense that business logic occurs on the 328 and communication with the Web uses an 8266 is there no prospect of changing the 328 for a processor with more memory ?

It is very likely that no peripherals (the DHT11 for one) or the interface to them need be changed and the majority, if not all of the existing code would run on the new processor, but it is difficult to know without more details

Having said that I would still favour integrating everything on to say an ESP32

I agree completely.
We've known very early on that a different MCU setup would have been better, but the changes to do that now would involve re-engineering quite a few custom shields and other aspects of the server side of things. Something we have decided is better to save for version 2 of the project rather than uproot version 1

23

david_2018:
I would use a char array instead of String, much less problem that way:

This looks promising, thanks.

24

If memory fragmentation is the concern, then use char arrays instead of String. Local char arrays in functions will not fragment memory.

25

Come to think of it, printf does almost what you want, assuming you import the a version with float support (or use other mechanisms to support the float values), and provide the underlying "special" file implementation for the ESP. And it would wind up being more like a program using unix sockets...

const char PROGMEM ReportFormat[] =
    // these will get concatenated into one long PROGMEM string
    // some of these %f's (floating point output) are wrong, of course.
    "GET /grafana.php?DHT11_Temperature=%f"
    "&DHT11_Humidity=%f"
    "&BMP085_Temperature=%f"
    "&BMP085_Pressure=%f"
    "&TSL2591_IR=%f"
    "&TSL2591_Full=%f"
    "&TSL2591_Visible=%f"
    "&TSL2591_Lux=%f"
    "&TSL2561_Visible=%f"
    "&TSL2561_IR=%f"
    "&TSL2561_Lux=%f"
    "&MQ7=%f"
    "&MQ135=%f"
    "&MQ5=%f"
    "&MQ135_Acetone=%f"
    "&MQ135_CO=%f"
    "&MQ135_Alcohol=%f"
    "&MQ135_CO2=%f"
    "&MQ135_Tolueno=%f"
    "&MQ135_NH4=%f"
    );

fprintf_P(esp8266f, ReportFormat,
    DHT.humidity,
    BMP085.readTemperature(),
    BMP085.readPressure(),
    (TSL2591_IR),
    (TSL2591_Full),
    (TSL2591_Visible),
    (TSL2591_Lux),
    (TSL2561_Visible),
    (TSL2561_IR),
    (TSL2561_Lux),
    analogRead(3), //Carbon Monoxide
    analogRead(2), //Air Quality (CO, Ammonia, Benzene, Alcohol, smoke)
    analogRead(1), //Natural gas, LPG
    MQ135_Acetone,
    MQ135_CO,
    MQ135_Alcohol,
    MQ135_CO2,
    MQ135_Tolueno,
    MQ135_NH4);
fflush(esp8266);
26

Ziplock9000:
The goal wasn't to avoid this buffer, it was to avoid having to re-allocate it every time and thus avoiding memory fragmentation.

you can always have a global set of Strings and use reserve() to ensure they have the right buffer size. If you don’t do crazy concats then behavior is close to char array...

But the char array approach gives you full view on what’s going on.

27

westfw:
Come to think of it, printf does almost what you want, assuming you import the a version with float support (or use other mechanisms to support the float values), and provide the underlying "special" file implementation for the ESP. And it would wind up being more like a program using unix sockets...

Very interesting, thank you. I'll take a look

All I'd have to do now is get the length of ReportFormat in PROGMEM once it's populated.

Then replace:
esp8266.println(commandString)
with:
esp8266.println((PGM_P)pgm_read_word(&(ReportFormat)))

Then I should be golden if the syntax of that last line is correct. Again thanks.