Using PROGMEM for large URLS

I am writing an IoT sketch on the Yun that uses HttpClient to makes calls to various web API URLs.

I am running into memory issues, but from what I have read about using PROGMEM, it seams like it is best for working with data in a byte-by-byte (or word) manner.

Since client.get(string) requires a single complete string, is there any way to use flash memory to save ram space?

I might have an incorrect assumptions about how this works, but I thought that building a new string out of characters fetched from flash memory and then using that string for the http call would not help any

thanks for any advice

Using program memory is the right solution to save RAM in this situation. I'm not familiar with the HttpClient, but the Serial.print(...) are overloaded to work with strings in flash memory. This means you can write:

Serial.println(F("Hello world!"));

The string "Hello world!" will be stored in flash memory. Retrieval is handled by println itself.

If that's the case for HttpClient as well, you're good to go. The easiest way to find out is to try wrapping one of your Urls in the F("...") macro.

All is not lost if that doesn't work. Another option is to use the PString library which will work with flash strings.

In that scenario you'd create a temporary buffer inside your function and use PString's print function to copy the string from flash memory. You still end up saving RAM because only the strings you are using are in RAM at any one time.

Good luck. Amazing how quickly that RAM can disappear.

Since client.get(string) requires a single complete string, is there any way to use flash memory to save ram space?

What, exactly, is a string? A string is a NULL terminated array of chars. How big is a char? One byte. How is an array stored? In adjacent memory locations.

So, yes, you can store strings in PROGMEM to save SRAM.

I'm not familiar with the HttpClient... Another option is to use the PString library

HttpClient is derived from Stream, so you don't need PString to stream. Anything you can do with Serial, you can do with an HttpClient:

    client.print( F("flash string") );

Cheers, /dev

oraz: I am writing an IoT sketch on the Yun that uses HttpClient to makes calls to various web API URLs.

I am running into memory issues, but from what I have read about using PROGMEM, it seams like it is best for working with data in a byte-by-byte (or word) manner.

Since client.get(string) requires a single complete string, is there any way to use flash memory to save ram space?

I might have an incorrect assumptions about how this works, but I thought that building a new string out of characters fetched from flash memory and then using that string for the http call would not help any

thanks for any advice

RAM issues, start with using the F() macro to get your constant prints into flash.

You don't have to assemble a complete string to send data through serial. As long as they go in order, you can send in pieces, even single chars.

Print from flash without a buffer:

byte  printFlash( PGM_P FM )
{
  byte c;
  byte  fb; // flash byte
  c = 0;
  do
  {
    c++;
    fb = pgm_read_byte( FM++ );
    if ( fb )  Serial.print(( char ) fb );
  }
  while ( fb );
  return c;
}

You can match serial input chars to text in flash, no-buffer character by character though that's more involved. I have an example sketch with C++ class to show matching one word, the same class can be used to match many different words. It's not beginner code but then beginners use Serial which is way more complicated! I have an example sketch to try and teach how it's done,

You don't have to assemble a complete string to send data through serial. As long as they go in order, you can send in pieces, even single chars.

True, but OP was asking specifically about an instance called client, which suggests the Ethernet class. Each client.print() results in a packet being sent. A one byte payload in a 512 byte packet is a waste of bandwidth and slow.

So, one typically will sacrifice a bit of memory for speed.

PaulS: True, but OP was asking specifically about an instance called client, which suggests the Ethernet class. Each client.print() results in a packet being sent. A one byte payload in a 512 byte packet is a waste of bandwidth and slow.

So, one typically will sacrifice a bit of memory for speed.

Only if you can't get all the data into the output buffer before it empties.

Thanks for the replies. I will try some things out and post some test code.

I can't use the F() macro, because I am constricting URLs with query strings attached depending on cases in the sketch, so it will no longer be a string literal.

Because I am not printing but making an Http call, I assumed retrieving the string characters in a loop would just cause the HttpClient to do a series of get()'s on single characters. But I will try and get to posting a test.

#include <Bridge.h>
#include <HttpClient.h>

HttpClient client;
const char baseUrl[] PROGMEM = {"http://www.brainjar.com/java/host/test.html"};

void setup() 
{
pinMode(13, OUTPUT);
Serial.begin(9600);
while(!Serial);

//client.get(F(baseUrl));
//error: no matching function for call to 'HttpClient::get(const __FlashStringHelper*)'



 //makeCall(F(baseUrl), client);
//error: cannot convert 'const __FlashStringHelper*' to 'char*' for argument '1' to 'void makeCall(char*, HttpClient)'
//error: nvalid initialization of non-const reference of type 'char*&' from an rvalue of type 'const __FlashStringHelper*'
}

void loop() {

}

//F() can never work with a string passed into a function if I'm correct?
void makeCall(String &URL, HttpClient inClient){
   Serial.println(F("calling:")); 
   Serial.println(URL); 
 //  Serial.println(F(URL));         
    
     inClient.get(URL);
     
   digitalWrite(13, HIGH);     
     while (inClient.available())
       {
        char c = inClient.read();  
        Serial.print(c);
        }       
   digitalWrite(13, LOW);  
}

Here is a first unsuccessful attempt:

I could not actually get to the question of using flash memory to make an HTTP call with a non-literal string.

Just using F() at all doesn’t seem to work for me, so I thought I would just post this as it is.

EDIT: Never mind. I thought you were trying to do EthernetClient kinds of things, not the Yun HttpClient. My mistake! The HttpClient class can only take a String (shudder) or a const char *, not an F macro thang.

Cheers, /dev

The ethernet adapters I see are SPI bus which is fast but not so fast that a sequence of output commands would not be taken as one whole.

But more, the library uses a buffer (I saw code, 512 bytes there). so perhaps write your html to that?

//client.get(F(baseUrl)); //error: no matching function for call to 'HttpClient::get(const __FlashStringHelper*)'

The F macro tells the compiler to not generate code to copy the string literal from PROGMEM to SRAM at run time. It is NOT used on a variable that is already using the PROGMEM directive to keep the data out of SRAM.

//F() can never work with a string passed into a function if I'm correct?

Wrong. But, in order for it to work, the F() macro needs to be applied to a string literal AND the function needs to be defined to take an array stored in PROGMEM.

That’s why I posted a print from flash function that takes a pointer to flash that can be passed.

I assumed retrieving the string characters in a loop would just cause the HttpClient to do a series of get()'s on single characters.

If the write-to command makes it so then sure. I notice in /dev’s IoT code that it does not.

@ PaulS

Ah I see I misunderstood the F() macro completely. In the past I had used F(), but it must have slipped my mind that it is not meant to "fetch" data from program memory, that is what pgmspace.h is for.

If the write-to command makes it so then sure. I notice in /dev's IoT code that it does not.

the library uses a buffer (I saw code, 512 bytes there). so perhaps write your html to that?

where is this IoT code you are referring to? Also is YunClient's buffer accessible? I'm not sure how I would do that, but it sounds like it would make things easier.