Include an external binary file as a PROGMEM variable?

I currently am able to include an arbitrary binary blob of data in my code using the PROGMEM keyword, and have code working that can access that data. However, manipulating the data is cumbersome at best. I’m relatively new to C and have little experience with the C preprocessor itself, but I do understand the theory behind what it does. I’m just not sure if it’s possible to accomplish what I’m hoping for.

I’m using HxD to generate/edit raw binary data, and then every time I want to change the data I have to use HxD’s handy “copy as C” function (which forms the data as a long string of “0x” characters, adds the commas, line-breaks it, etc), paste it into the Arduino editor, then remove the variable declaration and closing “};” that HxD includes when you use “copy as C”.

What I’d like to be able to do if possible is simply have the preprocessor include the actual raw binary file directly.

I’m not sure if it’s possible, but ideally I’d like to be able to do something like: static const byte blob PROGMEN =
#include <myBinary.bin>

The key of course is that “myBinary.bin” is actually the raw binary data, nut the string-based representation C requires. So the preprocessor would need to also convert the binary file into the appropriate C++ syntax for a static byte array.

The ideal workflow would be to just edit the .bin file directly with HxD or whatever, then rebuild/reload the sketch and have the new data present. The step of copying, pasting, removing the extra lines, etc. gets annoying after many iterations.

Is this possible?

whatever is in the include file needs to be compileable

why don't you convert your raw binary to hex values in an array using some script?

I could easily write a Python script or something to do the conversion, but I'm not sure how to hook into the build process. In fact, if I could hook into the preprocessing step I could forego worrying about the C++ preprocessor and do my own preprocessing, at which point I could easily devise my own syntax for including the binary file.

Do you have some guidance as to how to hook into the compile/deploy process in Arduino's IDE?

not unless you want to write your own makefile (shouldn't be that hard), no need for the IDE

but why can't you manually rebuild your .h file whenever you recompile that code. not sure how you're creating that code

I could manually rebuild the .h file, but it's just one extra manual step between editing the binary data and hitting "Upload sketch".

A custom makefile is OK, only thing is I'd still like the deploy to board function. I suppose the worst case is that I have to edit the COM port in the makefile every time my system decides to change the port number?

You can simply put an include file in the directory where your sketch resides

C:\Users\sterretje\Documents\Arduino
+-- yourSketch 
    +-- yourSketch.ino
    +-- rawData.h

And in the sketch include rawData.h; note the use of double quotes instead of < and > in below.

#include "rawData.h"

void setup()
{
}

void loop()
{
}

rawData.h could look like

#ifndef _RAWDATA_H_
#define _RAWDATA_H_

const byte blob[] PROGMEN = {
	0x30, 0x31, 0x32
};
#endif

@fdmillion, it is not possible to put a binairy file in a Arduino project. What sterretje showed is how to do it.

There is an other option: The ESP32 can simulate EEPROM and a filesystem SPIFFS in Flash memory. A file from the computer can be uploaded to the SPIFFS and be read in the sketch. It can be read as a file of course, it is not a pointer to memory.

You could also try to find someone who will make a dedicated bootloader that uses a reserved memory location for the binary data and the bootloader could accept the sketch and binary data. This will probably create a whole bunch of new problems. That’s why we use what sterretje showed.

I know that I can't just include a binary file as-is.

What I was hoping for after learning this was a way to hook into Arduino's build process (i.e. what happens when you click the Upload button, or the Compile button) to include a script that will convert my binary file into a properly formatted .h file with the data from that binary file. I can easily write this script in any language I want, but I still have to remember to manually run the script after changing the binary file prior to deploying (I've already scratched my head for many minutes wondering why something wasn't working, only to discover it's because I forgot to run the script to update the binary blob.)

From what I see, there doesn't seem to be an easy way to hook into the build process. The build process is well documented, so my alternative may simply be to write a "deploy.sh" script and forego the Arduino IDE altogether for this particular project, unless there is actually a way to insert a custom build step...

Do you mean these hooks: Redirecting. Your subject title seems to be about something else.
Here are some notes from user pert: https://forum.arduino.cc/index.php?topic=586019.msg3986855#msg3986855.

Do you have a way to tell which version of binary data you have during runtime ?