Adding a collection of .ZIP libraries to Arduino IDE ("esp32_bluepad32" by Ricardo Quesada)

Hello all. I'm in a bit of a pickle.

For a project with students in a school I'm trying to set up Arduino IDE 2.3.8 using a package of libraries (esp32_bluepad32) hosted on a github repository.

Installing using Arduino IDE fails.
#1 the school network blocks githubusercontent so the automated install isn't possible
#2 Installing the package of libraries from ZIP using Add Zip Library fails with error #13.

The repo does successfully install using Arduino IDE's library manager on my home network onto a personal laptop so I know its structure is "good".

The link to add to Arduino IDE additional board manager URLs preferences is at https://raw.githubusercontent.com/ricardoquesada/esp32-arduino-lib-builder/master/bluepad32_files/package_esp32_bluepad32_index.json

The most recent ZIP file with a library is at https://github.com/ricardoquesada/esp32-arduino-lib-builder/releases/download/4.1.0/esp32-bluepad32-4.1.0.zip

For the life of me I cannot get the full suite of libraries installed from the ZIP. The archive as downloaded fails. I've tried extracting the archive and then rezipping the sub folder and even the sub-sub-folder before performing import from zip. None succeed.

Only re-zipping one of the individual libraries from the libraries directory in the "larger" esp32-bluepad32-4.1.0\libraries directory has worked.

Presumably Arduino IDE is processing the esp32-bluepad32-4.1.0.zip file when it does an "automated" online install. How can I mimic that from a ZIP?

Or, will re-zipping every single directory in that "libraries" directory and then importing it work to fully install the package of libraries?

PS I'm now simply going to try a wholesale copy of all the Arduino libraries from a clean install on another computer into %userprofile%/Local and /Roaming to see if I can bypass the whole installation headache (10 GB later). I'd rather have a more flexible solution so I don't have to set up each user with an onerous 10 GB copy-and-paste routine (esp32 is BIIIIIIG).

What is that, show the FULL error output.

This additional boards is no longer needed, I was able to istall the repo from Library Manager. ESP32 boards are included with the IDE.

If you wanted to go the zip route (not needed) this is where you get it.

I am on a Mac, so this may be a bit different, but under each user folder is a hidden folder called .arduinoIDE (NOTE the period), in that folder is arduino-cli.yaml (open with a text editor). It is very self-documenting (look at your copy on your home computer). You could try changing the paths in the arduino-cli.yaml to network shares. You can populate the network shares from your home computer if that is allowed or get the sysadmin to do it for you.

https://github.com/ricardoquesada/esp32-arduino-lib-builder/releases/download/4.1.0/esp32-bluepad32-4.1.0.zip is not a library so trying to install with Add Zip Library will not work.

The workaround You already seem to know this) is to install the board package on another computer (you did that already on your computer) and next copy the directory where the IDE installed it to the computers where you need it.

:warning: The computers must use the same operating system (e.g. Windows and 64 bit).

The board package will be installed in the packages subdirectory of the Arduino15 directory (see https://support.arduino.cc/hc/en-us/articles/4415103213714-Find-sketches-libraries-board-cores-and-other-files-on-your-computer#boards where to find it for your OS).

The following might work

  1. Add the URL as you did.
  2. Copy the files for the platform from the Staging directory in the arduino15 directory. It should be the files in the below screenshot that are dated today.

Install the platform.

Hi sonofcy, thanks for responding. The error was "Error: 13 INTERNAL: Library install failed". Nothing exciting. Nothing informative :).

I do actually need the particular version of bluepad32 that is causing me headaches and the only way to get it installed is by using the JSON hosted at githubusercontent. githubusercontent is blocked, thus I cannot install using the Library Manager. The ZIP fails to install because it's not a single library (as far as I can tell).

Dank je wel sterretje. Good to know about the OS versions--I was wondering about that. Doing the brute force "move everything" method of copying all Arduino folders seems to have worked but it's far from ideal since (a) it can't be updated; and (b) it's an awful lot of data.

I'm wondering if it would be sufficient to copy only the packages subdirectory of the Arduino15 directory rather than everything. That would save an awful lot of space since I now need to move upwards of 10 GB worth of data.

Thank you for the staging directory suggestion. I'll try it out.

I am not hopeful that this will work because adding the URL is a pointless action since it can't be loaded. I have wondered if I could self-host the JSON since the ZIP file itself that's referenced in the JSON file is not hosted on githubusercontent. Now I need to figure out how to host a JSON somewhere... it's been years since I had my own webserver...

I am also tempted to do things the long way and re-ZIP each library in the massive ZIP and load each one separately--maybe that'll work (although, it's not a solution since it's far too many steps for someone else). If only there were a way to smoothly gain access to 'staging' from inside Arduino IDE (I'm guessing that's the solution to my problem).

It's a 13 because it's not a library.
I would NOT touch that so-called library, it's much more and will be a maintenance nightmare. Try to find a normal library for whatever device you have. It sounds like some Bluetooth thing, lots of libraries for those, just make sure you know what type of Bluetooth you are dealing with, I think there are 3 or 4 now? I only know of Classic and LE, but I am sure there is more. AFAIK, none of them works with their cousins.
Good Luck.

Too late :). Bluepad is needed for a bluetooth connection with bluetooth controllers. It is fully embedded in the instructions and sample code for a particular platform that we're using and, aside from having its JSON file hosted on githubusercontent, it works quite well (https://lordofrobots.com/). LoR is now a three year old project affiliated with an FRC robotics team.

With Arduino IDE and CLI, you're downloading two different sets of stuff: Boards and Libraries. For whatever reason, esp32_bluepad32 is a custom Board. Boards do contain their own Libraries. But given the overhead of being a Board, the desired functionality is probably not just in one (or more) of those libraries. Maybe it patches the board core. Note that Bluepad32 v4.1.0 uses arduino-esp32 v2.0.17, the last v2 release (which is almost two years old; the current one is v3.3.7)

Therefore, as you have seen, the features to install a Library do not work. It never claimed to be a Library. Stop using the word Library :slight_smile:

On this Windows box, I have WSL, with Arduino15 (on a Dev Drive) symlinked as ~/ard15. At the top of the following tree, you can see the standard esp32 board, version 2.0.17, which has both a hardware and tools directory as immediate children...

~/ard15/packages$ tree -d -L 4 esp32*
esp32
├── hardware
│   └── esp32
│       └── 2.0.17
│           ├── cores
│           ├── libraries
│           ├── tools
│           └── variants
└── tools
    ├── esptool_py
    │   └── 4.5.1
    ├── mklittlefs
    │   └── 3.0.0-gnu12-dc7f933
    ├── mkspiffs
    │   └── 0.2.3
    ├── openocd-esp32
    │   ├── v0.12.0-esp32-20230419
    │   │   ├── bin
    │   │   └── share
    │   └── v0.12.0-esp32-20230921
    │       ├── bin
    │       └── share
    ├── riscv32-esp-elf-gcc
    │   └── esp-2021r2-patch5-8.4.0
    │       ├── bin
    │       ├── include
    │       ├── lib
    │       ├── libexec
    │       ├── riscv32-esp-elf
    │       └── share
    ├── riscv32-esp-elf-gdb
    │   └── 11.2_20220823
    │       ├── bin
    │       ├── include
    │       └── share
    ├── xtensa-esp-elf-gdb
    │   └── 11.2_20220823
    │       ├── bin
    │       ├── include
    │       ├── lib
    │       └── share
    ├── xtensa-esp32-elf-gcc
    │   └── esp-2021r2-patch5-8.4.0
    │       ├── bin
    │       ├── include
    │       ├── lib
    │       ├── libexec
    │       ├── share
    │       └── xtensa-esp32-elf
    ├── xtensa-esp32s2-elf-gcc
    │   └── esp-2021r2-patch5-8.4.0
    │       ├── bin
    │       ├── include
    │       ├── lib
    │       ├── libexec
    │       ├── share
    │       └── xtensa-esp32s2-elf
    └── xtensa-esp32s3-elf-gcc
        └── esp-2021r2-patch5-8.4.0
            ├── bin
            ├── include
            ├── lib
            ├── libexec
            ├── share
            └── xtensa-esp32s3-elf
esp32-bluepad32
└── hardware
    └── esp32
        └── 4.1.0
            ├── cores
            ├── libraries
            ├── package
            ├── tools
            └── variants

...and at the bottom, the esp32-bluepad32 board, version 4.1.0, with just the hardware directory, but no sibling tools. (There is another tools three levels beneath, along with libraries)

So try first installing the standard esp32 board, version 2.0.17, which if not blocked will install the needed tools. Then when installing the custom board, each required tool will be "already installed". Here's the disk usage breakdown

~/ard15/packages$ du -h -d 1 esp32*
758M    esp32/hardware
2.0G    esp32/tools
2.8G    esp32
583M    esp32-bluepad32/hardware
583M    esp32-bluepad32

The tools are 2GB; each hardware about a quarter to a third of that.

Then you're back to the main question, installing the custom board itself. The hardware is "only" 758MB. If you're thinking about hosting the package-JSON, you might also/instead host the whole esp32-bluepad32 as a ZIP.

BTW, if you're curious about the differences

$ diff --brief --recursive --ignore-space-change -x '*.elf' -x '*.a' esp32/hardware/esp32/2.0.17/ esp32-bluepad32/hardware/esp32/4.1.0/
Files esp32/hardware/esp32/2.0.17/boards.txt and esp32-bluepad32/hardware/esp32/4.1.0/boards.txt differ
Files esp32/hardware/esp32/2.0.17/installed.json and esp32-bluepad32/hardware/esp32/4.1.0/installed.json differ
Only in esp32-bluepad32/hardware/esp32/4.1.0/libraries: Bluepad32_ESP32
Only in esp32-bluepad32/hardware/esp32/4.1.0/: package
Files esp32/hardware/esp32/2.0.17/package.json and esp32-bluepad32/hardware/esp32/4.1.0/package.json differ
Files esp32/hardware/esp32/2.0.17/platform.txt and esp32-bluepad32/hardware/esp32/4.1.0/platform.txt differ
Files esp32/hardware/esp32/2.0.17/tools/platformio-build-esp32.py and esp32-bluepad32/hardware/esp32/4.1.0/tools/platformio-build-esp32.py differ
Files esp32/hardware/esp32/2.0.17/tools/platformio-build-esp32c3.py and esp32-bluepad32/hardware/esp32/4.1.0/tools/platformio-build-esp32c3.py differ
Only in esp32/hardware/esp32/2.0.17/tools: platformio-build-esp32s2.py
Files esp32/hardware/esp32/2.0.17/tools/platformio-build-esp32s3.py and esp32-bluepad32/hardware/esp32/4.1.0/tools/platformio-build-esp32s3.py differ
Files esp32/hardware/esp32/2.0.17/tools/sdk/esp32/dio_qspi/include/sdkconfig.h and esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/dio_qspi/include/sdkconfig.h differ
Files esp32/hardware/esp32/2.0.17/tools/sdk/esp32/dout_qspi/include/sdkconfig.h and esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/dout_qspi/include/sdkconfig.h differ
Only in esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/include: bluepad32
Only in esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/include: bluepad32_arduino
Only in esp32/hardware/esp32/2.0.17/tools/sdk/esp32/include/bt: esp_ble_mesh
Only in esp32/hardware/esp32/2.0.17/tools/sdk/esp32/include/bt: host
Only in esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/include: btstack
Only in esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/include: cmd_nvs_4.4
Only in esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/include: cmd_system_4.4

and a bunch more. For one of those headers

~/ard15/packages$ diff -u esp32/hardware/esp32/2.0.17/tools/sdk/esp32/dio_qspi/include/sdkconfig.h esp32-bluepad32/hardware/esp32/4.1.0/tools/sdk/esp32/dio_qspi/include/sdkconfig.h

an excerpt:

-#define CONFIG_BLE_MESH_TIME_SCENE_SERVER 1
-#define CONFIG_BLE_MESH_LIGHTING_SERVER 1
-#define CONFIG_BLE_MESH_DISCARD_OLD_SEQ_AUTH 1
+#define CONFIG_BT_CONTROLLER_ONLY 1
 #define CONFIG_COAP_MBEDTLS_PSK 1
 #define CONFIG_COAP_LOG_DEFAULT_LEVEL 0
 #define CONFIG_ADC_DISABLE_DAC 1
@@ -670,6 +629,14 @@
 #define CONFIG_RTC_STORE_CRITICAL_DATA_SIZE 2048
 #define CONFIG_RTC_STORE_REPORTING_WATERMARK_PERCENT 80
 #define CONFIG_IO_GLITCH_FILTER_TIME_MS 50
+#define CONFIG_BLUEPAD32_PLATFORM_CUSTOM 1
+#define CONFIG_BLUEPAD32_MAX_DEVICES 4
+#define CONFIG_BLUEPAD32_GAP_SECURITY 1

removes a bunch of BLE stuff and adds BLUEPAD32 stuff

Thank you kindly for the informative post. You've made me wonder if there could be a simpler approach to the project (I lack the experience or time to redo it).

The Lord of Robots project uses the Espressif esp32 board version 3.3.7 in concert with this esp32_bluepad32 'package'/board. I haven't looked closely but I believe esp32-bluepad32 is used to connect with bluetooth PS4 compatible controllers.

I've now set up a localhost of the JSON and the ZIP files instead of setting up an actual webserver (using the portable version of https://simplewebserver.org/). It seems to work, sort of.

Since I'm on my home network now I am no longer subject to the firewall. Just in case something gets pulled in automatically from a blocked site I've turned off internet completely, but I have unmet dependencies that prevent installing the ZIP without an active internet connection. An internet connection is required to pull in the required tools, even after installing esp32 3.3.7 by espressif (an 8 GB install). I'll try again in a few days while on the corporate network to see if my workaround works.

In case anyone else runs into the same problem, it does seem like running a local host is a possible workaround to problems accessing the JSON or resources linked within the JSON from a blocked site.

PS I'll try your suggestion of installing the 2.0.17 package and see if that resolves tool dependency problems.

Hi @erdunbar.

Great work finding an innovative solution to the problem!

You can see the specified tool dependencies that are installed along with the "esp32_bluepad32" platform here:

https://github.com/ricardoquesada/esp32-arduino-lib-builder/blob/master/bluepad32_files/package_esp32_bluepad32_index.json#L38-L94

"toolsDependencies": [
  {
    "packager": "esp32",
    "name": "xtensa-esp32-elf-gcc",
    "version": "esp-2021r2-patch5-8.4.0"
  },
  {
    "packager": "esp32",
    "name": "xtensa-esp32s2-elf-gcc",
    "version": "esp-2021r2-patch5-8.4.0"
  },
  {
    "packager": "esp32",
    "name": "xtensa-esp32s3-elf-gcc",
    "version": "esp-2021r2-patch5-8.4.0"
  },
  {
    "packager": "esp32",
    "name": "xtensa-esp-elf-gdb",
    "version": "11.2_20220823"
  },
  {
    "packager": "esp32",
    "name": "riscv32-esp-elf-gcc",
    "version": "esp-2021r2-patch5-8.4.0"
  },
  {
    "packager": "esp32",
    "name": "riscv32-esp-elf-gdb",
    "version": "11.2_20220823"
  },
  {
    "packager": "esp32",
    "name": "openocd-esp32",
    "version": "v0.12.0-esp32-20230419"
  },
  {
    "packager": "esp32",
    "name": "esptool_py",
    "version": "4.5.1"
  },
  {
    "packager": "esp32",
    "name": "mkspiffs",
    "version": "0.2.3"
  },
  {
    "packager": "esp32",
    "name": "mklittlefs",
    "version": "3.0.0-gnu12-dc7f933"
  },
  {
    "packager": "arduino",
    "name": "dfu-util",
    "version": "0.11.0-arduino5"
  }
]

These are referencing tools that are actually provided by other packages. Those packages are defined in the primary package index:

https://downloads.arduino.cc/packages/package_index.json

For example, this:

{
  "packager": "esp32",
  "name": "xtensa-esp32-elf-gcc",
  "version": "esp-2021r2-patch5-8.4.0"
}

is referencing the tool defined in the primary package index like so:

{
  "name": "xtensa-esp32s3-elf-gcc",
  "version": "esp-2021r2-patch5-8.4.0",
  "systems": [
    {
      "host": "aarch64-linux-gnu",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-arm64.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-arm64.tar.gz",
      "checksum": "SHA-256:b218c11122e5565b6442376ebd21a652abdfcbf90981afa3e177ce978710225d",
      "size": "64233211"
    },
    {
      "host": "arm-linux-gnueabihf",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-armel.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-armel.tar.gz",
      "checksum": "SHA-256:967477434ad5483718915936a77ce915a10c5972a6b3fd02688a5c4e14182bfb",
      "size": "63530586"
    },
    {
      "host": "arm64-apple-darwin",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-macos-arm64.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-macos-arm64.tar.gz",
      "checksum": "SHA-256:c64b05be25d26916c65dcfe11de9e60b96d58980b2df706d3074cb70b1ef6cb9",
      "size": "66791095"
    },
    {
      "host": "i686-mingw32",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win32.zip",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win32.zip",
      "checksum": "SHA-256:658d3036ffdf11ddad6f0a784c8829f6ffd4dbd7c252d7f61722256d0ad43975",
      "size": "82665716"
    },
    {
      "host": "i686-pc-linux-gnu",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-i686.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-i686.tar.gz",
      "checksum": "SHA-256:07671d01a63ebd389912787efb2b263677c7b351c07fe430ded733cdae95e81d",
      "size": "70025439"
    },
    {
      "host": "x86_64-apple-darwin",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-macos.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-macos.tar.gz",
      "checksum": "SHA-256:99b6d44cea5aebbedc8b6965e7bf551aa4a40ed83ddbe1c0e9b7cb255564ded5",
      "size": "75719772"
    },
    {
      "host": "x86_64-mingw32",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win64.zip",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win64.zip",
      "checksum": "SHA-256:9000be38d44bf79c39b93a2aeb99b42e956c593ccbc02fe31cb9c71ae1bbcb22",
      "size": "86022563"
    },
    {
      "host": "x86_64-pc-linux-gnu",
      "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
      "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
      "checksum": "SHA-256:8aa17a6adf01efa5b1628c8ac578063a44d26ae9581d39486b92223a41ef262f",
      "size": "68099473"
    }
  ]
}

When the host is a 64-bit Windows machine (host ID x86_64-mingw32), this specific build of the tool is used:

{
  "host": "x86_64-mingw32",
  "url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win64.zip",
  "archiveFileName": "xtensa-esp32s3-elf-gcc8_4_0-esp-2021r2-patch5-win64.zip",
  "checksum": "SHA-256:9000be38d44bf79c39b93a2aeb99b42e956c593ccbc02fe31cb9c71ae1bbcb22",
  "size": "86022563"
}

Note that this is also served from GitHub.

So you would also need to set up your webserver to serve these tool dependencies.

The situation may be different for the arduino:dfu-util@0.11.0-arduino5 dependency:

  {
    "packager": "arduino",
    "name": "dfu-util",
    "version": "0.11.0-arduino5"
  }

The reason is that this tool is served from downloads.arduino.cc, not GitHub:

{
  "host": "i686-mingw32",
  "url": "https://downloads.arduino.cc/tools/dfu-util-0.11-arduino5-windows_386.tar.gz",
  "archiveFileName": "dfu-util-0.11-arduino5-windows_386.tar.gz",
  "checksum": "SHA-256:6451e16bf77600fe2436c8708ab4b75077c49997cf8bedf03221d9d6726bb641",
  "size": "571340"
}

(i686-mingw32 is used as a fallback on 64-bit Windows machines when there isn't a build of the tool for the x86_64-mingw32 host ID)

So if arduino.cc isn't blocked, then you might be able to get away without having to host that specific tool on your webserver.


If you would like to learn about the structure of the Arduino package index file, it is documented here:

https://arduino.github.io/arduino-cli/latest/package_index_json-specification/

Well perhaps we have come full circle. Despite the README being titled
LoR_v3 — LoR Core V3 Board Support, it's a Library. Further reading makes that clearer.

I'm skeptical. Try this. In the Preferences dialog (under the File menu), enable Show verbose output during compile. Then with an esp32_bluepad32 board selected, build a new empty sketch. The output should start like

FQBN: esp32-bluepad32:esp32:esp32
Using board 'esp32' from platform in folder: E:\Arduino15\packages\esp32-bluepad32\hardware\esp32\4.1.0
Using core 'esp32' from platform in folder: E:\Arduino15\packages\esp32-bluepad32\hardware\esp32\4.1.0

You can see the first segment of the Fully Qualified Board Name is esp32-bluepad32; and from the end of the folder paths, it is using version 4.1.0 of the custom board. It is not using 3.3.7 of anything. Right-click and Copy All, then pass it through grep or paste in an editor and do a Find: lots of 4.1.0 for lots (and lots) of include paths. No 2.0.17 either. The tools related to a given board have their own specific versions, and they're the executables at the beginning of many of the commands, especially when "Compiling" something; for example

"E:\\Arduino15\\packages\\esp32\\tools\\xtensa-esp32-elf-gcc\\esp-2021r2-patch5-8.4.0/bin/xtensa-esp32-elf-g++"
"E:\\Arduino15\\packages\\esp32\\tools\\xtensa-esp32-elf-gcc\\esp-2021r2-patch5-8.4.0/bin/xtensa-esp32-elf-gcc"

uses version esp-2021r2-patch5-8.4.0 of both g++ and gcc

That's probably installing newer versions of lots of the tools: they're not used by the custom board at all, which is based on

~/ard15/packages$ cat esp32-bluepad32/hardware/esp32/4.1.0/cores/esp32/core_version.h
#define ARDUINO_ESP32_GIT_VER 0x5e19e086
#define ARDUINO_ESP32_GIT_DESC 2.0.17
#define ARDUINO_ESP32_RELEASE_2_0_17
#define ARDUINO_ESP32_RELEASE "2_0_17"

Thank you to all who have posted here. Such helpful posts... that will hopefully help someone else in the future too.

I pre-empted that by starting fresh and not installing esp32 by Espressif. No failures during verification so no dependencies, code-wise or tool-wise on that library. Perhaps it's still in the project's v3 set up instructions from an earlier version of the project (Lord of Robots | Linktree). And, yes, the verbose log did show that output. Not installing esp32 by Espressif saves an awful lot of space and about half an hour of installation time each time Arduino is set up so I'm hoping to avoid that step.

tbh I don't know how esp32_bluepad32 is used other than that it is used to provide for bluetooth connection to PS4 controllers. Presumably it also acts as the foundation for programming the esp32 since esp32 by Espressif (or Arduino) doesn't seem to be needed.

I'm fairly sure that most of the other servers aren't blocked. I'll have to experiment after the weekend.

Hi, I thought I'd respond directly since I got a subtly different approach working for me.

I downloaded the JSON, the relevant zip archive and a portable webserver (Rebex Tiny Web Server (free) - Rebex.NET).

Then I modified the JSON to point to the zip on a localhost URL rather than to an external server. As long as the tiny-web-server was running and the JSON and the zip were stored in wwwroot I was able to perform the install. In theory, with the help of some more recent posts immediately above to deal with tool and library dependencies, you should be able to do a full offline "clean" install, if necessary.

From the JSON:
"url": "http://localhost:1180/esp32-bluepad32-4.1.0.zip",

1 Like

As seen in that diff between the custom and standard boards, only one additional board-level Library:

Only in esp32-bluepad32/hardware/esp32/4.1.0/libraries: Bluepad32_ESP32

That's library's src has just one file: dummy.h

// This file is here only to silence warnings from Arduino IDE
// Currently IDE doesn't support no-code libraries, like this collection of example sketches.

So that library is only to host examples sketches: two of them

~/ard15/packages$ ls -1p esp32-bluepad32/hardware/esp32/4.1.0/libraries/Bluepad32_ESP32/examples/
Controller/
WebAppBLEService/

At the top of that first example's .ino is an #include

~/ard15/packages$ head esp32-bluepad32/hardware/esp32/4.1.0/libraries/Bluepad32_ESP32/examples/Controller/Controller.ino
#include <Bluepad32.h>

ControllerPtr myControllers[BP32_MAX_GAMEPADS];

// This callback gets called any time a new gamepad is connected.
// Up to 4 gamepads can be connected at the same time.
void onConnectedController(ControllerPtr ctl) {
    bool foundEmptySlot = false;
    for (int i = 0; i < BP32_MAX_GAMEPADS; i++) {
        if (myControllers[i] == nullptr) {

Where is Bluepad32.h?

~/ard15/packages$ cd esp32-bluepad32/hardware/esp32/4.1.0/
~/ard15/packages/esp32-bluepad32/hardware/esp32/4.1.0$ find . -name Bluepad32.h
./tools/sdk/esp32/include/bluepad32_arduino/include/Bluepad32.h
./tools/sdk/esp32c3/include/bluepad32_arduino/include/Bluepad32.h
./tools/sdk/esp32s3/include/bluepad32_arduino/include/Bluepad32.h

It's a board-level include, one for each chip variant; identical:

~/ard15/packages/esp32-bluepad32/hardware/esp32/4.1.0$ diff --report-identical-files $(find . -name Bluepad32.h | head -n 2)
Files ./tools/sdk/esp32/include/bluepad32_arduino/include/Bluepad32.h and ./tools/sdk/esp32c3/include/bluepad32_arduino/include/Bluepad32.h are identical

So that's in addition to the changes seen earlier in existing board-level headers like sdkconfig.h. That's how/why it is distributed as a custom Board, and not just a Library.

Maybe at one time -- with IDE 1.x? -- it was required to install the standard esp32 board first, but that is not the case now. The custom board forks the hardware side of the board, with changes and additions; while reusing the tools from the standard board, with just one difference:

~/ard15$ diff -u \
<(jq '.packages[] | select(.name == "esp32") | .platforms[] | select(.version == "2.0.17") | .toolsDependencies | sort_by(.name)[]' package_index.json) \
<(jq '.packages[] | select(.name == "esp32-bluepad32") | .platforms[] | select(.version == "4.1.0") | .toolsDependencies | sort_by(.name)[]' package_esp32_bluepad32_index.json)
@@ -21,7 +21,7 @@
 {
   "packager": "esp32",
   "name": "openocd-esp32",
-  "version": "v0.12.0-esp32-20230921"
+  "version": "v0.12.0-esp32-20230419"
 }

a slightly older version of openocd (no idea what it does)

If only githubusercontent.com is being blocked, then maybe you don't need to make that change. And if that's the case, maybe you can just copy package_esp32_bluepad32_index.json into the Arduino15 directory and restart the IDE?

Or maybe the IDE doesn't like having a package-JSON that has no matching source URL. So add the URL, it fails to download, and then you copy the file anyway (and restart)? Then you don't need to setup a web server.

This was the case for Arduino IDE 1.x (with the exception of a special mechanism), but that behavior was changed for Arduino IDE 2.x. Arduino IDE 2.x will still use the data from the local copy of the package index file, even if the source URL of that index is not present in Arduino IDE's "Additional Boards Manager URLs" preference.

Also possibly of interest is that Arduino IDE 2.x supports the use of the file scheme in the package index URL specified via the preference. You can use this to specify an index file from an arbitrary path on the local drive (e.g., file:///C:/package_foo_index.json). However, note that it is not necessary to do this in the case where you only want Arduino IDE to use the package index file that is already located in the standard location under the Arduino15 folder

Thank you. Using file:///c:... worked. I don't know if there are additional firewall problems that'll appear, but, on my home network this allowed me to install esp32_bluepad32. Putting the JSON in Arduino15, or in the packages directory without adding a file:/// entry to the Preferences 'Additional boards manager URLs' entry didn't do anything.

This is what I added to my Arduino Preferences to get the JSON to be recognised (& that's also where I put the JSON file).

file:///C:\Users\myusername\AppData\Local\Arduino15\package_esp32_bluepad32_index.json

How does one get Arduino IDE to use the package index file in the 'standard' location?

I experimented but couldn't get it to work and went with the file:///... solution.

Something important to note is that Arduino IDE caches the files that are downloaded by Boards Manager. When you install a platform, it first checks the local cache for the files that are indicated for download by the entry in the package index. If the file is already present in the cache, and the file passes verification against the data in the index, then the cached file is used instead of downloading it.

This might cause spurious results if you are testing various ways for Arduino IDE to download a file specified by an index.

The problem can be avoided by deleting the file from the cache before installing the platform via Boards Manager. The files are cached in the folder at this path on your hard drive:

C:\Users\<username>\AppData\Local\Arduino15\staging\packages

If Arduino IDE is running at the time you add the file to C:\Users\<username>\AppData\Local\Arduino15, you must restart Arduino IDE to cause it to reload the package index files in that folder. You can do that by selecting File > Quit from the Arduino IDE menus, then starting Arduino IDE once more. After doing so, you should find that Boards Manager is populated with the data from the package index file you added.