I want to download all the libraries

I was recently off the internet for some time, which highlighted for me the need for a downloadable archive of every Arduino library. If there is some disaster whereby internet is out, having these libraries could make the difference between life and death, or at least between boredom and productivity.

I found that there is a github repository for Arduino libraries. What I need to know is how to download all of them. I think there should be a torrent or zip file so they can be quickly downloaded. Software and tech security is not achieved by relying on the internet. We should all have these libraries just in case.

Is there a way to download all these libraries at once? If not, there should be.

You can get a machine readable list of all the repositories under the arduino-libraries GitHub organization here:

It will be a simple task to write a script to parse that data and either clone or download the .zip archive of each. If you like to use shell scripts, I'll recommend the excellent jq tool for working with JSON.

but Arduino Libraries · GitHub doesn't have all Arduino libraries

mechatroniac:
I was recently off the internet for some time, which highlighted for me the need for a downloadable archive of every Arduino library. If there is some disaster whereby internet is out, having these libraries could make the difference between life and death, or at least between boredom and productivity.

I think this is a fantasy problem rather than a real problem. Reminds me of the chap who wanted to take a copy of the internet.

I suspect that you will never need 95% of the libraries and there will always be new libraries added after your "download" completes.

Most of the libraries have little or no documentation so you will probably need access to the internet to seek advice about out how to use them.

And a great antidote for boredom is to get your project to work without using a library. :slight_smile:

...R

pert:
You can get a machine readable list of all the repositories under the arduino-libraries GitHub organization here:
https://api.github.com/orgs/arduino-libraries/repos?per_page=100

It will be a simple task to write a script to parse that data and either clone or download the .zip archive of each. If you like to use shell scripts, I'll recommend the excellent jq tool for working with JSON.

I wouldn't know how to get started with that. I'm just a dumb guy who makes robots out of junk and arduinos. I can barely write a bash script.
Is there a template you can recommend?

21TB of data, was archived to 186 reels of film

mechatroniac:
Is there a template you can recommend?

Software and tech security is not achieved by relying on templates.

-I think this is a fantasy problem rather than a real problem. Reminds me of the chap who wanted to take a copy of the internet.

Nope, it's a real problem. Internet may not always be there. Libraries are finite, and don't take that much space.

-I suspect that you will never need 95% of the libraries and there will always be new libraries added after your "download" completes.

So?

-Most of the libraries have little or no documentation so you will probably need access to the internet to seek advice about out how to use them.

Still better than not having the libraries.

-And a great antidote for boredom is to get your project to work without using a library

Thanks for your "advice" and ridicule.

pert:
Software and tech security is not achieved by relying on templates.

??? Why are you guys being so hostile to me because I want to have a copy of the libraries on my harddrive? Is it so unreasonable?

The ones you've used before are already on your hard drive aren't they?
How many others are you after?

I say download whatever you want, it's no skin off anybody else's nose.

CrossRoads:
The ones you've used before are already on your hard drive aren't they?
How many others are you after?

I say download whatever you want, it's no skin off anybody else's nose.

I would like to archive every library within reason because I might not know what I will need until I need it. I have a lot of sensors that I haven't even used yet. A lot of sensors and things are in consumer electronics that can be recovered after a disaster scenario so that some semblance of technology may be recovered.

The Arduino Library List states that there are 3586 libraries registered. Even if they were 1Mb each it would still be only 3 GB, but I think most are in the tens of kilobytes. So if each were average around 100 Kb, that is only a 359 Mb zip file.

I'm quite astonished that it seems I am the first to think about this.

I want access to the libraries after an asteroid hit or something like that. Just being offline for a month I got to experience the folly of relying on the internet. It can be taken away, by tyrants, by terrorism, by natural disaster. You name it. I don't trust github, they are under control of one of the most tyrannical corporations on earth.

Isn't it a good idea to have a distributed copy of the libraries, just in case?

It's pretty elitist to expect me to pick up some arcane programming discipline just to download something that should be available, by any reasonable measure. Not everyone into Arduino should be expected to be versed in web programming. I didn't appreciate the ridicule either. What if the site that hosts the libraries goes down? Then you might wish you had listened to me and at least had a torrent up.

Please calm down a little @mechatroniac. I don’t know whether you’re trolling or just aggro by nature, but it’s not going to get you anywhere here.

My previous comment was just a joke. Maybe you didn’t notice that I paraphrased your “Software and tech security is not achieved by relying on the internet.”

I will quibble with your “arcane programming discipline” statement though. This is really fundamental stuff that would serve someone like yourself well if you could get past your attitude.

Even though I would never write it for you, I just so happen to have the exact things you are asking for (even though I wrote them for another purpose). But I’m not interested in providing support for using them, so if you have problems you’ll either have to figure it out for yourself or else hope that someone else can help you out (I’m sure they will if you take a bit more of a positive tone to your requests).

#!/bin/bash

# Install the latest release of every library in the Library Manager index

readonly LIBRARY_LIST="$(arduino-cli lib search --names --format json)"

# Install libraries
IFS=

Even though I think we’ve determined that your original request for a download of the arduino-libraries GitHub organization’s repositories was not really what you wanted, I’ll throw in that one as a bonus:

#!/bin/bash

readonly GH_TOKEN="$1"

readonly EXCLUDE_SKETCH_REPOS="false"
readonly EXCLUDE_LIBRARY_REPOS="false"
readonly EXCLUDE_PLATFORM_REPOS="false"

readonly EXCLUDE_PRIVATE="false"
readonly EXCLUDE_FORKS="true"
readonly EXCLUDE_ARCHIVED="true"

readonly CLONE_REPOS="true" # Set to false to download .zip of repo, which is faster

readonly SKETCH_REPO_LIST='
"arduino/arduino-examples"
"arduino/ArduinoTensorFlowLiteTutorials"
"arduino/cloud-examples"
"arduino/Summer-camp-robot-examples"
'

# There are some additional libraries outside the arduino-libraries organization.
# NOTE: there are many more in the bcmi-labs organization which are not included in this list.
readonly ADDITIONAL_LIBRARY_REPO_LIST='
"arduino/Adafruit-GFX-Library"
"arduino/Adafruit-ST7735-Library"
"arduino/ArduinoCloudProviderExamples"
"arduino/EduIntro"
"arduino/BasicEducationShield"
"arduino/CTC-Arduino"
"arduino/HMC5983"
"arduino/pubsubclient"
"arduino/Library-Arduino-Cbor"
"arduino/TFT"
"vidor-libraries/VidorBoot"
"vidor-libraries/USBBlaster"
"vidor-libraries/VidorPeripherals"
"vidor-libraries/VidorGraphics"
'
function get_github_api_response() {
  local -r APIURL="$1"
  curl --silent --header "Authorization: token $GH_TOKEN" "$APIURL"
}

function parse_api_response() {
  local -r apiResponse="$1"
  local -r key="$2"
  # jq uses the system EOL, so when the script is run on Windows you get CRLF. It's easiest just to get rid of that right away, thus the `tr` in this command
  echo "$apiResponse" | jq "$key" | tr --delete '\r'
}

# Get all the repositories in the arduino-libraries organization.
# Inexplicably, there are some non-library repos in this organization, but only a few.
if [[ "$EXCLUDE_LIBRARY_REPOS" == "false" ]]; then
  readonly ORGANIZATION_REPO_LIST_API_RESPONSE="$(get_github_api_response "https://api.github.com/orgs/arduino-libraries/repos?per_page=100")"
  readonly LIBRARY_REPO_LIST="$(parse_api_response "$ORGANIZATION_REPO_LIST_API_RESPONSE" ".[].full_name")$ADDITIONAL_LIBRARY_REPO_LIST"
fi

# NOTE: there are more in the bcmi-labs organization which are not included in this list.
readonly PLATFORM_REPO_LIST='
"arduino/ArduinoCore-avr"
"arduino/ArduinoCore-samd"
"arduino/ArduinoCore-sam"
"arduino/ArduinoCore-arc32"
"arduino/ArduinoCore-mbed"
"arduino/ArduinoCore-megaavr"
"arduino/ArduinoCore-API"
"arduino/ArduinoCore-Z80-MBC"
'

# Generate the list of non-excluded project types repo
if [[ "$EXCLUDE_SKETCH_REPOS" == "false" ]]; then
  REPO_LIST="${REPO_LIST}${SKETCH_REPO_LIST}"
fi

if [[ "$EXCLUDE_LIBRARY_REPOS" == "false" ]]; then
  REPO_LIST="${REPO_LIST}${LIBRARY_REPO_LIST}"
fi

if [[ "$EXCLUDE_PLATFORM_REPOS" == "false" ]]; then
  REPO_LIST="${REPO_LIST}${PLATFORM_REPO_LIST}"
fi

function exclude_repo() {
  local -r exclusion_enabled="$1"
  local -r apiResponse="$2"
  local -r exclusionKey="$3"
  local -r exclusionValue="$4"
  if [[ ("$exclusion_enabled" == "true") && ("$(parse_api_response "$apiResponse" "$exclusionKey")" == "$exclusionValue") ]]; then
    echo "true"
  else
    echo "false"
  fi
}

# Filter the list of repos
while IFS='' read -r repo_slug; do
  if [[ -n "$repo_slug" ]]; then
    REPO_API_URL="https://api.github.com/repos/${repo_slug//\"/}"
    API_RESPONSE="$(get_github_api_response "$REPO_API_URL")"
    if [[ "$(exclude_repo "$EXCLUDE_PRIVATE" "$API_RESPONSE" ".private" "true")" == "false" ]]; then
      if [[ "$(exclude_repo "$EXCLUDE_FORKS" "$API_RESPONSE" ".fork" "true")" == "false" ]]; then
        if [[ "$(exclude_repo "$EXCLUDE_ARCHIVED" "$API_RESPONSE" ".archived" "true")" == "false" ]]; then
          FILTERED_REPO_LIST="$FILTERED_REPO_LIST"

\n’
export ARDUINO_DIRECTORIES_USER="${PWD}"
for library in $(echo “$LIBRARY_LIST” | jq -r “.libraries | .name” | tr --delete ‘\r’); do
  if ! [[ -d “${ARDUINO_DIRECTORIES_USER}/libraries/${library// /_}” ]]; then
    arduino-cli lib install “$library” --no-deps
  fi
done


Even though I think we've determined that your original request for a download of the arduino-libraries GitHub organization's repositories was not really what you wanted, I'll throw in that one as a bonus:

~~~
§_DISCOURSE_HOISTED_CODE_1_§
~~~

\n'"$repo_slug"
        fi
      fi
    fi
  fi
done <<<"$REPO_LIST"

# Download the repos
while IFS='' read -r repo_slug; do
  if [[ -n "$repo_slug" ]]; then
    REPO_SLUG_WITHOUT_QUOTES="${repo_slug//\"/}"
    if [[ -d "$REPO_SLUG_WITHOUT_QUOTES" ]]; then
      echo "WARNING: directory $REPO_SLUG_WITHOUT_QUOTES already exists, skipping"
      continue
    fi

    if [[ "$CLONE_REPOS" == "true" ]]; then
      git clone --depth 1 "https://github.com/${REPO_SLUG_WITHOUT_QUOTES}" "${REPO_SLUG_WITHOUT_QUOTES}"
    else
      REPO_API_URL="https://api.github.com/repos/${REPO_SLUG_WITHOUT_QUOTES}"
      API_RESPONSE="$(get_github_api_response "$REPO_API_URL")"
      DEFAULT_BRANCH_NAME="$(parse_api_response "$API_RESPONSE" ".default_branch")"
      DEFAULT_BRANCH_NAME_WITHOUT_QUOTES="${DEFAULT_BRANCH_NAME//\"/}"
      # Since we don't need the repository data, it's more efficient to use the .zip downloads provided by GitHub instead of `git clone`
      wget --header="Authorization: token $GH_TOKEN" "https://github.com/${REPO_SLUG_WITHOUT_QUOTES}/archive/${DEFAULT_BRANCH_NAME_WITHOUT_QUOTES}.zip"
      ARCHIVE_FILENAME="${DEFAULT_BRANCH_NAME_WITHOUT_QUOTES}.zip"
      mkdir --parents "$REPO_SLUG_WITHOUT_QUOTES"
      unzip -q "$ARCHIVE_FILENAME" -d "$REPO_SLUG_WITHOUT_QUOTES"
      rm "$ARCHIVE_FILENAME"
    fi
  fi
done <<<"$FILTERED_REPO_LIST"
~~~

\n'
export ARDUINO_DIRECTORIES_USER="${PWD}"
for library in $(echo "$LIBRARY_LIST" | jq -r ".libraries[] | .name" | tr --delete '\r'); do
  if ! [[ -d "${ARDUINO_DIRECTORIES_USER}/libraries/${library// /_}" ]]; then
    arduino-cli lib install "$library" --no-deps
  fi
done

Even though I think we’ve determined that your original request for a download of the arduino-libraries GitHub organization’s repositories was not really what you wanted, I’ll throw in that one as a bonus:

§_DISCOURSE_HOISTED_CODE_1_§

mechatroniac:
It's pretty elitist to expect me to pick up some arcane programming discipline just to download something that should be available, by any reasonable measure.

I agree with you about the "arcane programming" but I disagree strongly with your view "should be available, by any reasonable measure"

It is interaction over the internet that has enabled all those libraries to come into being. It's not reasonable to expect programmers who have been comfortable using the internet for at least the last 10 years to think of building a system that works without the internet.

This is not an issue that is peculiar to Arduino programming - and in the grander scheme of programming things Arduino hobby programming is just trivial.

An archive of Arduino libraries might or might not be a good thing - but I don't think you can fault anyone for not producing one. The essence of Open Source programming is the opportunity for YOU to fill a gap that you perceive. But that's back to "arcane programming".

...R

Thank you pert, I really appreciate it. I'll give it a shot, and study that code so I understand how you did it.

You're welcome. I'm glad if I was able to be of assistance. Enjoy!
Per

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.