Creating Custom Boards for Board Manager

Development Tools IDE 1.x
1

Our project uses a custom board based on the ATmega 2560 processor. The firmware can be compiled just fine using the Arduino Mega board in the IDE but we actually use a custom bootloader that takes up less space than the stock bootloader, so computed memory statistics are a bit off with this method (although it doesn't affect operation).

Up to now we have just been adding an entry to Arduino's boards.txt file but with recent versions of the IDE this is no longer the way to go.

Finally we have found issues when compiling with avr-gcc 4.9.2 and so would like to keep avr-gcc 4.8.1. I understand the new boards manager will allow us to do that.

Creating a new entry for board manager appears to be a pretty involved process. I feel confident I'm going to regret ever having to learn about it, but such is the way of infinite progress.

For others embarking on this same journey, documentation is not great but here are some links I've found:

Arduino IDE 1.6.x package_index.json format specification - We are up to 1.8.1 but I presume this information must still be accurate. This page describes how to create a json file that describes your custom board package, but by itself it is only essentially metadata, other elements are required.

Arduino IDE 1.5 3rd party Hardware specification - once again we are way beyond IDE 1.5 but I presume this information is also still relevant. This page is where the pain really starts to set in, as well as perhaps some brain tumors and/or ulcers, but perhaps not all of it is needed...

Boards Manager Support File Locations on GitHub - I also found this page that discusses various places you can host your json file on GitHub, but I think this is a topic that represents one of the last steps rather than the first.

Unofficial list of 3rd party boards support urls - We may glean information from how others have done things.

2

So first of all I started creating a json file using the example on this page. The format of the file is clearly explained, but I have questions about the content. Let's say I want our board to compile with avr-gcc 4.8.1. Do I need to maintain and update my own version/download of the compiler or can I simply refer to this somewhere at Arduino.cc? The example at the top of the page links to Arduino but what is the difference between 4.8.1-arduino2 and 4.8.1-arduino5? Do I need to specify compilers for every OS as implied in the example at the top of the page, or can I omit the tools entirely as implied by the example at the bottom of the page?

I probably don't even know enough to ask the right questions so I will gladly accept guidance on the most efficient way of proceeding from this point.

3

Ok, more fooling about. I've created a json file and an archive package and hosted them on GitHub here. They are not really functional yet but I imagine it is easier to help someone correct a thing they have already created, than to help someone create something that doesn't yet exist.

I am sort of basing my json around Adafruit's and the example listed previously in the documentation, and I organized my GitHub repository in the same way I see SparkFun did theirs. Of course I don't really know what I'm doing, just copying others without knowing the nuances.

So I add the link to my json in the Arduino IDE Preferences -> Additional Board Manager URLs box. Then I open Boards Manager and get a java.lang.NullPointerException error, the full text of which is below. If I close the IDE and try to re-open, the IDE itself it will fail to load, until I delete the local copy of package_openpanzer_index.json from the Arduino15 folder.

(FYI: I am still using IDE 1.6.9 if that makes a difference...)

Here is the link to my json file if you want to crash your own copy:
~~https://raw.githubusercontent.com/openpanzerproject/Arduino_Boards/master/IDE_Board_Manager/package_openpanzer_index.json~~

EDIT: Changed later to this:

https://openpanzerproject.github.io/OpenPanzerBoards/package_openpanzer_index.json

There are no doubt too many errors to count in my setup... I'm just trying to get a framework established so hopefully whatever kind soul attempts to assist doesn't have to walk me through getting GitHub setup, etc...

java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
 at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$19(ContributionManagerUI.java:151)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
 at cc.arduino.contributions.packages.ContributionsIndex.findPackage(ContributionsIndex.java:49)
 at cc.arduino.contributions.packages.ContributionsIndexer.syncBuiltInHardwareFolder(ContributionsIndexer.java:197)
 at cc.arduino.contributions.packages.ContributionsIndexer.syncWithFilesystem(ContributionsIndexer.java:187)
 at processing.app.BaseNoGui.initPackages(BaseNoGui.java:626)
 at processing.app.Base$9.onIndexesUpdated(Base.java:1307)
 at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$19(ContributionManagerUI.java:149)
 ... 1 more
java.lang.NullPointerException
 at cc.arduino.contributions.packages.ContributionsIndex.findPackage(ContributionsIndex.java:49)
 at cc.arduino.contributions.packages.ContributionsIndexer.syncBuiltInHardwareFolder(ContributionsIndexer.java:197)
 at cc.arduino.contributions.packages.ContributionsIndexer.syncWithFilesystem(ContributionsIndexer.java:187)
 at processing.app.BaseNoGui.initPackages(BaseNoGui.java:626)
 at processing.app.Base.openBoardsManager(Base.java:1325)
 at processing.app.Base$10.actionPerformed(Base.java:1348)
 at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
 at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
 at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
 at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
 at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
 at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833)
 at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:877)
 at java.awt.Component.processMouseEvent(Component.java:6535)
 at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
 at java.awt.Component.processEvent(Component.java:6300)
 at java.awt.Container.processEvent(Container.java:2236)
 at java.awt.Component.dispatchEventImpl(Component.java:4891)
 at java.awt.Container.dispatchEventImpl(Container.java:2294)
 at java.awt.Component.dispatchEvent(Component.java:4713)
 at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
 at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
 at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
 at java.awt.Container.dispatchEventImpl(Container.java:2280)
 at java.awt.Window.dispatchEventImpl(Window.java:2750)
 at java.awt.Component.dispatchEvent(Component.java:4713)
 at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
 at java.awt.EventQueue.access$500(EventQueue.java:97)
 at java.awt.EventQueue$3.run(EventQueue.java:709)
 at java.awt.EventQueue$3.run(EventQueue.java:703)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
 at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
 at java.awt.EventQueue$4.run(EventQueue.java:731)
 at java.awt.EventQueue$4.run(EventQueue.java:729)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
 at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
 at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
 at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
4

LukeZ:
documentation is not great

If there's anything specific let me know and I'll see what I can do. Sometimes someone reading documentation for the first time can spot things that seemed obvious to the author only because they have become so familiar with the subject matter.

LukeZ:
Boards Manager Support File Locations on GitHub - I also found this page that discusses various places you can host your json file on GitHub, but I think this is a topic that represents one of the last steps rather than the first.

I wrote that one (actually I'm one of the authors of the official JSON specification too). I found I was discussing those options every time I added Boards Manager support for a hardware package so decided it was easiest to just publish it in a central location. I have been meaning to update the last part regarding use of the archive file that GitHub automatically generates. We had an issue with the first Boards Manager project I did where the checksum of the archive file changed, causing Boards Manager installation to fail. I don't know why this happened but I haven't seen it happen since then so it may not be so inadvisable. It's possible it was caused by a force push. The downside to that is the repository must have the correct folder structure so that the automatically generated archive file will have the correct folder structure. This folder structure is different from the one required for manual installation. Some people can't use Boards Manager due to network security settings, etc. so it's nice to still offer the option of easy manual installation. So it may be more convenient to set up the repository folder structure for manual installation and then create your own Boards Manager archive file with the correct folder structure at each release.

LukeZ:
Let's say I want our board to compile with avr-gcc 4.8.1. Do I need to maintain and update my own version/download of the compiler or can I simply refer to this somewhere at Arduino.cc?

The way you have it set up now does work but unfortunately there was a bug in previous versions of the Arduino IDE that will cause installation of your hardware package in Arduino IDE 1.6.10 to break the Arduino AVR Boards package. For more information see:

So the options are to just state that the package is not compatible with Arduino IDE 1.6.10 or to change the JSON file so that your package uses its own avr-gcc tool instead of using the arduino tool. You can still have the file downloaded from their server.

LukeZ:
what is the difference between 4.8.1-arduino2 and 4.8.1-arduino5?

Arduino has their own custom builds of avr-gcc. so the 4.8.1 is the official avr-gcc release version and the -arduino5 is the revision of the Arduino build. I'm not sure exactly what the changes are, I think you can see them at GitHub - arduino/toolchain-avr: The AVR toolchain used by the Arduino IDE.

LukeZ:
I imagine it is easier to help someone correct a thing they have already created, than to help someone create something that doesn't yet exist.

I think that's a good approach. I find GitHub to be an excellent tool for facilitating collaboration.

LukeZ:
Then I open Boards Manager and get a java.lang.NullPointerException error

For anyone else following along, I think we have this issue fixed now.

5

Once again thank you so much Pert for all your help, you're a one-man support banshee and I really appreciate it.

LukeZ:
documentation is not great

Well it is better than it at first appears, one big problem is that any Google search of this topic brings up primarily lots of information on the old method of modifying boards.txt, or if you do find a tutorial on the new method it is actually a tutorial on how to paste someone's JSON URL into your preferences, rather than creating your own archive. Once I did find your pages I was able to get pretty far all things considered, but there are so many options and possibilities that are not expounded upon, nor probably easily summarized. Most of these I suspect are basically optional but that is not clear to the newbie.

Also page headings that say things like "this is for Arduino 1.5.3" tend to throw people off, or anyway they did me, since I was leery of using instructions from a prehistoric version in my quest to make my code compatible with the latest version.

Nevertheless, my brain tumors have not been too painful yet, and it's only day 1 and I have a working board package (with your help), so it was not as bad as I feared.

pert:
This folder structure is different from the one required for manual installation. Some people can't use Boards Manager due to network security settings, etc. so it's nice to still offer the option of easy manual installation. So it may be more convenient to set up the repository folder structure for manual installation and then create your own Boards Manager archive file with the correct folder structure at each release.

Looking at Sparkfun and others it doesn't appear most actually do provide the correct folder for manual installation? Sparkfun's layout, which I copied, has this folder structure in their GitHub repository:
sparkfun\avr\stuff...

Whereas I would imagine for manual install it should be something like this instead:
sparkfun\hardware\avr\1.0.0\stuff...

...with instructions that the sparkfun folder should be dumped into the ..\Arduino15\packages\ folder. I suppose maybe SparkFun is only keeping the most recent version of the individual files but the version number still needs to be part of the path (I think) - at least to judge from the way Arduino installs things. Am I understanding this correctly?

The way SparkFun has their files arranged seems to be in a way that would be convenient for creating the installation archive. But presumably they are the only ones creating the archive, and anyone else who wanted the individual files would only need them for manual install. I like having a single repository for all this the way SparkFun does (and I'm not brave/motivated enough to try the automatically generated GitHub release), but perhaps I can align my folder structure to what people will need for manual install, if indeed I have divined that structure correctly.

pert:
So the options are to just state that the package is not compatible with Arduino IDE 1.6.10 or to change the JSON file so that your package uses its own avr-gcc tool instead of using the arduino tool. You can still have the file downloaded from their server.

Laziness tells me to do the former, but it wouldn't be hard to change the JSON I guess. My concern is related to getting the correct link to Arduino's stash and how to maintain that if they change/upgrade it? But this may be an issue anyway, eg:

pert:
Arduino has their own custom builds of avr-gcc. so the 4.8.1 is the official avr-gcc release version and the -arduino5 is the revision of the Arduino build.

Would one be safe to assume that Arduino is not going to release a version -arduino6? If they did no one would remember to go back and change their JSON files. What I would prefer is to have the JSON tell the IDE "I want to use avr-gcc 4.8.1 and you take care of getting me the best version of that without me having to think about it." But it seems that is perhaps not possible/practical regardless of whether I specify my own tool or use Arduino's.

Amazingly I haven't even got to my real list of questions yet but I'll give you a temporary rest here. The board does install correctly in Arduino so it's not so much an issue of getting it to work as of better understanding what I'm doing.

6

LukeZ:
Nevertheless, my brain tumors have not been too painful yet

For a couple months after Boards Manager was added to the Arduino IDE there was absolutely no documentation. I had to figure it out by trial and error. Talk about brain tumors! Finally there were enough other people asking questions about it that I decided to write a tutorial. The next day Arduino came out with the first draft of the JSON specification page and requested that I incorporate the content from my tutorial into the specification. I commented that I should have waited a day to write my tutorial but they said it was seeing my tutorial that made them realize they really should get on writing some official documentation. The playground page you linked above was the left over content when I finished moving the rest over.

LukeZ:
Looking at Sparkfun and others it doesn't appear most actually do provide the correct folder for manual installation? Sparkfun's layout, which I copied, has this folder structure in their GitHub repository:
sparkfun\avr\stuff...

Whereas I would imagine for manual install it should be something like this instead:
sparkfun\hardware\avr\1.0.0\stuff...

That is the manual installation folder structure. Manual installation doesn't have the version folder. The Boards Manager folder structure would be sparkfun\stuff because the architecture and version folders are automatically generated during Boards Manager installation based on the information in the JSON file. It's quite annoying that they have to be slightly different. You will see that some hardware packages have completely switched to Boards Manager installation only and no longer have any instructions for manual installation. As I said before, I do think there is good reason to leave this option available to the users. Of course if you know what you're doing it's easy enough to just create the architecture folder during the manual installation but that's not really very user friendly.

LukeZ:
but perhaps I can align my folder structure to what people will need for manual install, if indeed I have divined that structure correctly.

Yes, your repository currently has the correct folder structure for easy manual installation. The manual installation instructions would be:

Seems pretty simple, right?

LukeZ:
My concern is related to getting the correct link to Arduino's stash and how to maintain that if they change/upgrade it?

You can just grab the URL from their JSON file:
http://downloads.arduino.cc/packages/package_index.json
I doubt they'd change the location anytime in the near future because that would require them to edit their JSON file but you could always download the file and host it yourself to be sure. I'm not sure what the etiquette is around using Arduino's download server to host your tools. Obviously it's not something they're terribly worried about since Boards Manager already has a mechanism for referencing their tools, as your JSON file currently does.

LukeZ:
Would one be safe to assume that Arduino is not going to release a version -arduino6?

No, I don't think there's any chance of that because they've already moved on to avr-gcc 4.9.2. They will likely release some new revisions of 4.9.2 in the future.

LukeZ:
What I would prefer is to have the JSON tell the IDE "I want to use avr-gcc 4.8.1 and you take care of getting me the best version of that without me having to think about it." But it seems that is perhaps not possible/practical regardless of whether I specify my own tool or use Arduino's.

You could probably write some sort of script that parses through Arduino's JSON file to fine the latest revision of 4.8.1 and then inserts that version in the JSON file or downloads it to your own hosting location and renames it but that would be done independent of the Arduino IDE as it has no mechanism for this sort of thing.

If you want a good example of a hardware package that is easy to do a manual or Boards Manager installation of check out:

(or any of MCUdude/hansibull's other packages). The master branch has the package in the manual installation folder structure. The JSON file and Boards Manager installation archive files for each release are located in the gh-pages branch. Putting these in the separate branch makes downloads for manual installation smaller and removes any files that would be unnecessary for that purpose. The reason the gh-pages branch has that name is it makes the Boards Manager URL a little shorter due to the special properties GitHub gives branches of that name:

https://mcudude.github.io/MightyCore/package_MCUdude_MightyCore_index.json

rather than:

https://github.com/MCUdude/MightyCore/raw/gh-pages/package_MCUdude_MightyCore_index.json

as it would need to be in a branch of any other name.

The only downside of this system is that the folder structure must be changed for Boards Manager compatibility and the archive file manually generated but that's not so bad. I've considered writing a script to automate this as well as generating the JSON file entry (something like this) but it's easy enough to do it manually and the releases aren't so frequent.

7

pert:
That is the manual installation folder structure. Manual installation doesn't have the version folder. The Boards Manager folder structure would be sparkfun\stuff because the architecture and version folders are automatically generated during Boards Manager installation based on the information in the JSON file.

Ok, after many hours re-reading your posts and looking at other's GitHub pages I understand where my confusion was coming from with regards to the installation folders.

I knew that Boards Manager installed the package in the Arduino15 folder:

C:\Users\username\AppData\Local\Arduino15\packages\

Although the constructed package that Boards Manager downloads only needs the folder projectname\stuff, I also knew that when Boards Manager unpacked it to the Arduino15 folder, it adds all kinds of other intermediate folders automatically -

Arduino15\packages\projectname\hardware\avr\version\stuff

My assumption was that manual install must do the same thing. I see now that not only does manual install require a different folder structure, it doesn't even go into the same folder. Perhaps the average user is not supposed to know about the Arduino15 folder and maybe I only knew enough to be dangerous.

So for those reading this later with the same confusion, ignore what Boards Manager does in Arduino15. For manual install the files go into the user's sketchbook directory in the hardware subfolder (create it if it doesn't exist):

sketchbookfolder\hardware\projectname\avr\stuff

Ok, so anyway I have taken various of your suggestions and created a new repository from scratch. I've edited my prior posts to point to the new, hopefully final locations for anyone else following along.

GitHub: OpenPanzerProject\OpenPanzerBoards

This repo has two branches. The master branch (also the default) has the manual installation files. The gh-pages branch (I'd never heard of such a thing before), has our JSON and packages for the IDE Boards Manager. I've modelled this after MCUDude's setup as you mentioned.

I think MCUDude's method would be a good example to use on the Boards Manager Support File Locations on GitHub example page, perhaps replacing Adafruit. Adafruit is a little unusual, they actually use two different repositories entirely, one for manual install files (Adafruit_Arduino_Boards) and the other for the Boards Manager packages (arduino-board-index). They do use a gh-pages branch on this latter one but that is also essentially the only branch in that repo. It took me a while to dig up both these repos and understand what they had done, as neither one is linked from the example page.

I'm also still a relative newbie when it comes to GitHub, so to me the pros/cons of adding another branch aren't really apparent. What I detect is that branches are perhaps typically intended eventually to be merged and GitHub wants to keep reminding me of that with my gh-pages branch, although in this application I will want the master and gh-pages branch to remain separate for all eternity and basically represent completely different things.

Finally, I've tested my package in several versions of the IDE. My case is a little unusual than most in that I am trying to force the use of avr-gcc-4.8.1 even in later versions of the IDE that would otherwise use the newer 4.9.2. I found that when I tried anything beyond 1.6.9 I would get compile errors, which I was finally able to correct by copying the platform.txt from AVR Boards 1.6.11 (the last AVR Boards version to use avr-gcc-4.8.1, and not to be confused with the IDE version). Now it seems all to be working.

In my JSON I've specified the avr-gcc specifically in the "Tools" section, copying the URLs from Arduino's JSON. I'm not going to host them myself, I'll just use theirs. Hopefully this avoids the bug in IDE 1.6.10. It seems to work in my testing but of course I'm only testing my own board... I did not add avrdude to the "Tools" section, I don't know if that will cause issues in IDE 1.6.10 or not, I'm assuming not.

8

LukeZ:
I think MCUDude's method would be a good example to use on the Boards Manager Support File Locations on GitHub example page, perhaps replacing Adafruit. Adafruit is a little unusual

That's a good idea. At the time I wrote that page MightyCore didn't exist. There were only a few existing 3rd party packages available to use as examples.

LukeZ:
I'm also still a relative newbie when it comes to GitHub, so to me the pros/cons of adding another branch aren't really apparent.

In this case the pros are:

  • Someone wanting to do a manual installation won't have to download an be confused by the Boards Manager support files, which are irrelevant to their usage.
  • The commit history for each branch is kept separate, which makes sense because they are not really relevant to each other.
  • People wanting to contribute work to the project won't be confused by the presence of these mysterious Boards Manager files.
  • gh-pages branch allows a shorter Boards Manager URL.

The only con I can think of is it makes the repository structure a little more confusing but if that's a problem a little documentation will make things clear. I think the README.md you added to your gh-pages branch does a good job of this.

LukeZ:
What I detect is that branches are perhaps typically intended eventually to be merged

Certainly the standard flow for contributing commits is fork, branch, commit, pull request but that's not the only use for branches. GitHub added the gh-pages branch feature specifically with the intent that branch would never be merged to master so using as you are is quite standard.

LukeZ:
In my JSON I've specified the avr-gcc specifically in the "Tools" section, copying the URLs from Arduino's JSON. I'm not going to host them myself, I'll just use theirs. Hopefully this avoids the bug in IDE 1.6.10.

I'll take a look at this later to see if I notice any issues. It can be very tricky and time consuming trying to support backwards compatibility with a reasonable range of Arduino IDE versions but people do tend to stick with whatever IDE version they've become comfortable with so it is good to at least make an attempt.

LukeZ:
I did not add avrdude to the "Tools" section, I don't know if that will cause issues in IDE 1.6.10 or not, I'm assuming not.

That shouldn't be a problem. I've seen mixed results as to whether the new avrdude version gets used by Arduino AVR Boards in old IDE versions but even if it does it shouldn't break anything. Using avr-gcc 4.8.1 with Arduino AVR Boards 1.6.12 or newer doesn't work because it uses LTO, which is not supported by Arduino's avr-gcc 4.8.1 versions.

9

Thanks again Pert for all your help, and I hope this thread and your comments will be of use to others looking for more information as they create their own custom boards. The best advice I can give is to study how others have done it, but of course one needs to be pretty well educated on a lot of topics to begin with, which I suppose anyone should be who has gotten to this point. But I myself have gotten to this point and I don't feel hardly educated at all.

I hope I have not trespassed any etiquette but I took the liberty of adding our humble board to the Unofficial List of 3rd Party Boards page. If I should not have done or did it incorrectly of course please let me know.

10

So I figured out my other problem with the compiler which I will post about in the other thread, but the result is that I am now free to use the later version of the avr-gcc compiler.

I can go ahead and just specify "4.9.2-atmel3.5.3-arduino2" in my toolsDependencies, which I see is the latest version as of today. But then when Arduino releases 4.9.2-xx-arduino3 or whatever, I will need to update my package once again.

Browsing through other peoples JSON files, I note two things:

  • Big companies like SparkFun and Adafruit have not specified 4.9.2 yet; if they list anything it's 4.8.1. Could this just be a consequence of the hassle of updating your JSON every time Arduino decides to upgrade which seems to be about every five minutes? Or could it be that they prefer the older compiler for some reason I don't know about?
  • I also notice that many JSONs don't list any toolsDependencies. Could it be that when this value is left blank the IDE will simply use whatever version is active for the currently loaded AVR Boards, which would be dependent on the IDE version?

If #2 is true then it would seem the easiest thing to do from a maintenance standpoint is to leave the entry blank and let the IDE take care of it. But I don't know if that's necessarily the case.

I will do some testing but I wondered if there was a best-practice or guidance on this.

11

LukeZ:
Big companies like SparkFun and Adafruit have not specified 4.9.2 yet; if they list anything it's 4.8.1. Could this just be a consequence of the hassle of updating your JSON every time Arduino decides to upgrade which seems to be about every five minutes? Or could it be that they prefer the older compiler for some reason I don't know about?
[/list]

There's a good chance it's just that they haven't released a new package version since the new compiler became available. It might also be that they tend to just blindly copy and paste the previous entry for each new release. It could also be that they know things work with the old version and don't want to spend time testing and dealing with any possible issues upgrading could cause.

LukeZ:
I also notice that many JSONs don't list any toolsDependencies. Could it be that when this value is left blank the IDE will simply use whatever version is active for the currently loaded AVR Boards, which would be dependent on the IDE version?

Yes, whatever tool versions Arduino AVR Boards is using will be used when the toolsDependencies field is left blank. The toolsDependencies entries were removed from those JSON files because that was the easiest way of dealing with Boards manager install of older avr-gcc breaks compilation for all AVR boards in 1.6.10. · Issue #142 · arduino/ArduinoCore-avr · GitHub. That bug has been fixed since but there are likely still some people using Arduino IDE 1.6.10. The downside is that you don't have any control over which tool versions are used with your package but it does make the JSON file more simple and you don't need to worry about checking to see what the latest tool versions are at each package release. It also means that the installation of your package will always be super fast since the tools will never have to be downloaded.

12

Thanks Pert. The next question that naturally arises is, since we are allowed to leave toolsDependencies blank and let the IDE choose the compiler for us, are we therefore also allowed to omit platform.txt from our package? Since platform.txt varies depending on the compiler version and we don't know which version will be used, it would seem this must be so.

I've done a test package with toolsDependencies left blank and platform.txt omitted, and all seems to work as it should. But just wanted to make sure this is technically correct, since I don't see others doing this.

Looking at MCUDude's MightyCore repo for example, he leaves toolsDependencies blank, but then he has a platform.txt file that appears to be rather old (it doesn't for example specify LTO compiler flags that were introduced back with IDE 1.6.10). Because toolsDependencies is blank the IDE is going to use the latest version of avr-gcc, but his platform.txt file is going to preclude him from taking advantage of its new features. Maybe he doesn't want those new features, or maybe this is a case where he's letting the IDE take care of one decision while not recognizing he should let the IDE take care of the other decisions as well.

13

LukeZ:
Thanks Pert. The next question that naturally arises is, since we are allowed to leave toolsDependencies blank and let the IDE choose the compiler for us, are we therefore also allowed to omit platform.txt from our package? Since platform.txt varies depending on the compiler version and we don't know which version will be used, it would seem this must be so.

I've done a test package with toolsDependencies left blank and platform.txt omitted, and all seems to work as it should. But just wanted to make sure this is technically correct, since I don't see others doing this.

Yes, because your package references the arduino:arduino core the Arduino AVR Boards platform.txt will be used if your core doesn't have one or if the platform.txt in your package is incomplete (for example, sometimes a core will have a platform.txt only so the package name can be defined but the rest is left blank since the referenced package's platform.txt works fine).

LukeZ:
Looking at MCUDude's MightyCore repo for example, he leaves toolsDependencies blank, but then he has a platform.txt file

MightyCore uses its own core so it must have a complete platform.txt file.

LukeZ:
Maybe he doesn't want those new features, or maybe this is a case where he's letting the IDE take care of one decision while not recognizing he should let the IDE take care of the other decisions as well.

Because MightyCore must have its own platform.txt and avr-gcc 4.8.1 doesn't support LTO adding that feature would break backwards compatibility with older IDE/Arduino AVR Boards versions. We had a lot of discussion regarding adding an avr-gcc tool to MightyCore but I think the main issue was the added complication this would cause for manual installation.

14

Actually I forgot, MightyCore does have a custom tools menu to turn LTO on or off e.g.:

This allows the feature to by used by people who have a recent version of avr-gcc installed while retaining backwards compatibility.

15

pert:
Yes, because your package references the arduino:arduino core the Arduino AVR Boards platform.txt will be used if your core doesn't have one or if the platform.txt in your package is incomplete (for example, sometimes a core will have a platform.txt only so the package name can be defined but the rest is left blank since the referenced package's platform.txt works fine).

Thanks for confirming that, and also the tidbit about the package name! I was wondering why my header disappeared in the Boards list.

The very word "core" starts to make me feel like a headache is approaching, so I am going to please myself with the thought I don't need to worry about such things yet (I'm not sure I even comprehend what all a core entails - thankfully). In the end my package is quite simple and I am letting the IDE choose most things for me. This means our code needs to be compatible with many versions of the IDE but for us that is a good goal anyway. If something breaks again like we started off with, I now have a much better grasp of how to create a more tailored package.

I appreciate again all your help and I do hope this thread is useful to some other poor sap out there who has graduated to this level...

16

LukeZ:
The very word "core" starts to make me feel like a headache is approaching, so I am going to please myself with the thought I don't need to worry about such things yet (I'm not sure I even comprehend what all a core entails - thankfully).

When I say core I mean the files inside the cores folder in the hardware package, which contains the source files for the standard Arduino API, basically just a collection of libraries and the main function definition. You will also see the term core used to refer complete hardware packages (e.g. "ESP8266 core for Arduino") but I think that's confusing because a hardware package may not even contain a cores folder and instead reference the core from a different package, as does the OpenPanzer package. So I use the term package to refer to a collection of files used to add boards (or programmers in rare cases) to the Arduino IDE.

LukeZ:
I appreciate again all your help and I do hope this thread is useful to some other poor sap out there who has graduated to this level...

Glad I've been able to help. I know it's been a bit of work for you to get this set up but I do think it will make your project significantly easier for the users to install and update and will reduce the number of support requests. I've seen authors of other packages be a bit skeptical at first about the value of adding Boards Manager support but they definitely came around on the subject over time.

17

This thread is the most helpfull thing i've found in the whole internet, you guys rock.

If someone is still there i just want to ask...
(where did you learned how to develop/code your own bootloader)

18

You could start with the datasheet. For example, in the Nov 2016 datasheet for the Atmega328P:

30.8.13. Simple Assembly Code Example for a Boot Loader

which I won't copy here as it's a bit lengthy with comments and all.

19

Yvanox:
This thread is the most helpfull thing i've found in the whole internet, you guys rock.

That's pretty much how I felt when Pert helped me through this molasses.

Yvanox:
If someone is still there i just want to ask...
(where did you learned how to develop/code your own bootloader)

The best way to understand the bootloader is to examine closely ones that already exist, and figure out what part that you need to modify for your own version. The other thing is to question yourself as to whether a custom bootloader is really needed or necessary, in most cases it is not, but there are exceptions.

The readme on this GitHub page has a link to the bootloader we based ours off of (applicable only to the ATmega2560), and information about compiling and flashing a custom bootloader, maybe some of it will be useful.

20

Well two years have passed and during that time I've been trucking along like a little song, but then I had to upgrade my Arduino IDE from 1.8.1 to 1.8.5 due to some Teensy stuff that didn't compile in 1.8.1, and that has unleashed new problems on this old (but still ongoing) Open Panzer project.

The issue now is that, although the IDE loads my custom boards file just fine, and it shows up in Board Manager, I can't compile my project nor flash the custom bootloader.

When I try to compile my code using my custom board definition, I get the error:

recipe.preproc.macros pattern is missing

If I try to burn the bootloader, I get the message:

Could not find tool avrdude

Now our firmware is running on an ATmega2560 processor and the firmware compiles just fine if I select the Arduino-standard ATmega2560 board (and runs fine). I am also still able to flash my custom bootloader by using avrdude from the command line, so for now I have assumed that my firmware and bootloader hex are fine. Since the package is also loaded without error by the IDE I'm also assuming that my json file is also still fine and working as it should.

I've tried rolling back to IDE 1.8.1 but the damage was done, now I get the same errors regardless. Tried also installing 1.8.1 fresh on a different computer and same deal, so either I was really lucky that it worked in the old days, or something has changed about Arduino's boards requirements and my package is no longer up to spec.

What I am going to guess is that the problem is related to my boards.txt file and/or my platform.txt file, which can be seen in the "avr" folder of this GitHub repo, but the contents of which are also copied below:

boards.txt

# Open Panzer TCB Board Configuration
#
# For more information see:
# - https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5-3rd-party-Hardware-specification
# - https://github.com/arduino/Arduino/wiki/Arduino-Hardware-Cores-migration-guide-from-1.0-to-1.6
# - http://forum.arduino.cc/index.php?topic=457405


##############################################################

optcb2560.name=Open Panzer TCB (Mega 2560)

optcb2560.bootloader.tool=avrdude
optcb2560.bootloader.extended_fuses=0xFD
optcb2560.bootloader.high_fuses=0xDA
optcb2560.bootloader.low_fuses=0xF7
optcb2560.bootloader.unlock_bits=0x3F
optcb2560.bootloader.lock_bits=0x0F
optcb2560.bootloader.file=openpanzer_boot/optcb2560_boot.hex

optcb2560.build.mcu=atmega2560
optcb2560.build.f_cpu=16000000L
optcb2560.build.core=arduino
optcb2560.build.variant=mega
optcb2560.build.board=AVR_MEGA2560

optcb2560.upload.protocol=wiring
optcb2560.upload.maximum_size=258048
optcb2560.upload.maximum_data_size=8192
optcb2560.upload.speed=115200
optcb2560.upload.tool=avrdude

platform.txt

# Open Panzer TCB Platform Configuration
# --------------------------------------------------------------------------->>
# We are leaving it up to the IDE to decide what platform specifics to use,
# but we still create this file so we can have the header name in the boards 
# list (name value below)
# --------------------------------------------------------------------------->>

name=Open Panzer Boards
version=1.0.3

You can see these are rather simple files. We discussed this earlier in the thread, and it was my understanding then (and it worked then) that by specifying for example "upload.tool=avrdude" or "build.core=arduino" that we would simply tell the IDE to use whatever it already knew about such things. Perhaps we need to be more explicit now?

From my preliminary searching it looks like the documentation on these topics is the same now as it was two years ago, and it all refers to 1.5 or 1.6. Could there be un-documented changes (or documents I have not found) related to more recent developments?

I'm happy to do as much work myself as I can but after pulling my hair out for the past couple days I realize I need some pointers about where to be looking, and I'm hoping Pert or someone will be able to assist.

Thank you in advance for what I know will be a very helpful response!