How to replace the default Wire library with non-hangup version?

Using Arduino Networking, Protocols, and Devices
1

Like many others on this forum, I too have had problems with I2C, and after much experimentation it was clear the root problem is the blocking while() loops in twi.c.

Just like the other posters here, I have tried to figure out why the original Wire library (with the broken twi.c) hasn't been fixed. Here's what I've found so far:

I made a local modification to the Wire library for my own use that merges the code from the 'I2C Master Library' and the twi.c/h changes posted by unaie, along with changes to the Wire class to allow the 'lockup reset counter' to be reset to zero and to be displayed by the user's arduino program. This version is available on my 'Paynter's Palace' blog site at Integrating Time, Memory, and Heading Capability, Part V | Paynter's Palace. I tested this code on a project that would regularly lock up after a few hours at most, and the same hardware ran for 3 days without lockups before I had to shut it down for other reasons.

So, it appears we have all the code and modifications we need to eliminate the I2C lockups - but nobody has figured out how to get that version into the basic Wire library that ships with Arduino. Does anyone have a clue who 'owns' this library and how we can get these modifications made?

I'm perfectly willing to put my modified version up on GitHub, and allow others to beat on it, but eventually we need to figure out how to replace the Wire library distributed with Arduino's basic utilities with the resultant 'approved' version.

Anyone out there know how this is done - who 'owns' the Arduino distributions?

TIA,

Frank

2

paynterf:
Does anyone have a clue who 'owns' this library

The first thing you need to understand is that the Wire library is architecture-specific. A different Wire library is used depending on whether you are compiling for an Uno, Due, Zero, ESP8266, ESP32, etc. Ideally this isn't noticeable to the user because the Arduino IDE automatically uses the correct Wire library for the board you have selected and the APIs are all compatible. Typically each hardware package will bundle its own Wire library (though it is possible to share the Wire library of a different package). Arduino owns the Arduino AVR Boards, Arduino SAM Boards, Arduino SAMD Boards, and Intel Curie Boards. The other 3rd party hardware packages are owned by others.

paynterf:
Anyone out there know how this is done

  • Search to determine whether someone has already submitted a pull request for your proposed change. Search open and closed pull requests to make sure the proposal wasn't previously rejected. Arduino AVR Boards and Arduino SAM Boards used to be part of the Arduino IDE repository so you also need to search that repository: GitHub - arduino/Arduino: Arduino IDE 1.x
  • Fork the repository.
  • Create a branch in your fork to make the changes to.
  • Make the changes.
  • Commit the changes.
  • Submit a pull request to the parent repository. Make sure you completely explain the reason for the change.
  • Be responsive to any requests for information or changes.

Thanks for your interest in contributing to the Arduino project!

3

Thanks for the input.
 Searched the Arduino links you mentioned, but didn't find anything relevant (may be there and I just don't know what to look for, but…
 Searched all of GitHub for "Arduino Wire Library" and got 132 hits. Among them were

  1. DSSCircuits I2C-Master-Library,
  2. thexeno/Hardware-Arduino-Library
  3. the streamfire/WSWireLib with enuaie's timeouts added
  4. czukowski's Wire.h - a 2012 fork of the Wire library
  5. stickbreaker's Wire, posted in Aug 2015
  6. mikaelpatel's 'Arduino-TWI' updated June of this year
  7. SterlingPeet's 'OpenServo' library from 2012
  8. a 2017 fork of the Arduino Wire Library by SodaqMoja
  9. Lahorde's Wire library from 2015
  10. jmparatte's 'jm_Wire' from May 2017
  11. signorettae's AS3935_arduino_wire_library from Nov 2017
  12. gion86's 'Wire' for ATmega and ATtiny from Nov 2017
  13. zoubworldArduino's 'WireUtility' for I2C debug from May 2018
  14. BrainSpawnInfoSphere's 'Wire' library for unix from 2012
  15. kein0r's 'WirePlus' "replacement of Arduino Wire Library" from Oct 2015
  16. vad7’s ‘Arduino-DUE-WireSam for Due with EEPROM, FRAM, RTC DS3231 DS3232, DS2482 updated April 2018
  17. Beirdo’s Arduino-Wire1 for the ATMega 168PB
  18. LukeC8’s ‘WireCommand’ from April 2017
  19. 3Lxx’s ‘WirePort’ port of the Arduino wire library for Linux
  20. Eecharlie’s ‘WireMaster’ “Arduino drop-in replacement fo pfleury’s I2CMaster library
  21. Mhct’s ‘newWire’ modified Arduino two-wire library(1.6.2) from May 2015
  22. Freespace/SBWire “modified Arduino wire library with basic timeout” from Apr 2017
  23. Spirilis’ TWire variation of Wire library employing timeouts, from May 2012

I didn’t find anything in this search that said “this is the official Arduino Wire Library”. Instead, it looks like literally dozens of different folks rolled their own Wire libraries, but none of these variations ever made it into the ‘official’ Arduino distribution, regardless of the hardware-specificity issues. Mikael Patel seems to be actively maintaining his ‘Arduino-TWI’ library, but I don’t know enough about it to comment on its suitability for designation as ‘the’ Arduino library, or even if Mikael would be interested in this idea: (Mikael, if you are reading this, I would love to hear your input)

I’ve only been in the Arduino game for a few years, but it seems to me that updating the ‘official’ Wire library implementations for at least the most common hardware might cut down at least a little bit on all the duplicated effort, and maybe even reduce the volume of the “I2C hangs up” posts on the forum. I’m perfectly willing to help this effort in any way I can, but I’ll need some guidance from the real experts.

Thoughts everyone/anyone?
TIA,

Frank

4

paynterf:
I didn’t find anything in this search that said “this is the official Arduino Wire Library”.

Each hardware package bundles its own Wire library. You will find it under the libraries subfolder.

The official Arduino Wire library for AVR (Uno, Mega, Leonardo, etc.) is here:

The official Arduino Wire library for SAM (Due) is here:

The official Arduino Wire library for SAMD (Zero, etc.) is here:

The official Arduino Wire library for ARC32 (Arduino 101) is here:

The official Wire library for ESP8266 is here:

The official Wire library for ESP32 is here:

And so on with all the other 3rd party hardware packages...

You might start with a pull request submitted to the Arduino AVR Boards repository to see what sort of feedback you get before proceeding with the other hardware packages.

5

Thanks for the input - now at least I know where to find the 'official' library repositories ;-).

I drilled down in the ArduinoCore-avr/libraries/Wire at master · arduino/ArduinoCore-avr · GitHub repo to twi.c, and then clicked on 'pulls' to see if there were any pull requests for the while() loop fixes. However, the 'Pulls' link takes me to Pull requests · arduino/ArduinoCore-avr · GitHub, which AFAICT has nothing to do with twi.c. It is completely unbelievable to me that there haven't been numerous attempts to fix this in the past, but I can't find anything- am I missing something?

As an exercise, I reviewed some of the 'avr' pull requests, but realized I don't know how to interpret them. Some appear to be very old, and appear to have never been reviewed, let alone merged into the master repo. I saw only one 'closed' request, but can't figure out whether this meant the pull request was rejected or accepted/merged. I looked through some 'pull request' tutorials, but couldn't find anything about how to determine pull request status - can you point me to any documentation for this?

I'll be happy to fork the avr repo and submit a pull request, as soon as I figure out how to do it ;-).

Regards,

Frank

6

paynterf:
I drilled down in the ArduinoCore-avr/libraries/Wire at master · arduino/ArduinoCore-avr · GitHub repo to twi.c, and then clicked on 'pulls' to see if there were any pull requests for the while() loop fixes. However, the 'Pulls' link takes me to Pull requests · arduino/ArduinoCore-avr · GitHub, which AFAICT has nothing to do with twi.c.

That page shows you all pull requests to the repository. There's no way to see only pull requests for a specific file. In this case, considering there are only 13 pull requests total, it's easy enough to just scan through the whole list.

paynterf:
It is completely unbelievable to me that there haven't been numerous attempts to fix this in the past, but I can't find anything- am I missing something?

As I said before. Arduino AVR Boards used to be part of the Arduino IDE repository. It was only recently moved to its own repository. So you need to search the pull requests in the Arduino IDE repository also:
https://github.com/arduino/Arduino/pulls?utf8=%E2%9C%93&q=is%3Apr+wire

paynterf:
I reviewed some of the 'avr' pull requests, but realized I don't know how to interpret them. Some appear to be very old

10 months old at most. If you think that's old, wait until you look at the ones in the Arduino IDE repository!

paynterf:
and appear to have never been reviewed, let alone merged into the master repo.

Not every pull request will be merged. There are only a few Arduino employees with the power to merge pull requests and they have a lot of work on their plate. Unfortunately Arduino has recently been adding a lot of new projects of questionable value like cloud based closed source tools. These take up the developers time instead of fixing known problems and accepting improvements to the existing software. At the same time, there is always progress, even if it's slower than we would like. Contributions from the community are valuable. Sometimes a PR or bug report will sit, seemingly forgotten for years, then out of the blue it's resolved. I have submitted obvious, non-controversial, pull requests that sit in Arduino repositories for long periods of time but I've also had pull requests merged. Even pull requests that aren't merged can be valuable because other users may find them and use them as a reference for their own code.

paynterf:
I saw only one 'closed' request, but can't figure out whether this meant the pull request was rejected or accepted/merged.

You can see this at a glance from the little icon on the left of the pull request listing. The green icon means it's open and unmerged. The purple icon means it was merged and closed. The red icon means it was closed and unmerged. Unmerged closed pull requests don't always mean the PR was completely rejected. Sometimes a different approach to achieve the same goal is considered preferable but maybe the original proposal got the conversation started.

paynterf:
I'll be happy to fork the avr repo and submit a pull request, as soon as I figure out how to do it ;-).

It's a very good skill to learn. Once you learn how to submit pull requests it makes it so easy to contribute to open source software projects. I really like that I can actually do something so direct about bugs I notice in software. I can now submit a fix for a minor issue in only a couple minutes.

There is a lot of tutorials for how to do this. The basic outline is:
Fork the repository. This creates an online copy of the repository that you own.

Clone your fork. This makes a copy of the repository on your computer so you can edit it locally. GitHub does allow you to edit files via the web interface but this is pretty limiting. For example, you can only edit a single file per commit. It's ok for something minor like fixing a typo but for more significant work it's really better to work on local clone of the repository. The Git software is used to work with repositories. If you like, you can just use Git directly from the command line. You can also use a Git client software that provides a GUI for common Git operations. For a beginner, I recommend GitHub Desktop. It's pretty nice but also easy to use. Once you get more experienced and find GitHub Desktop doesn't have the advanced features you need you might decide to move to a different client or just start using Git from the command line. I now use Git Extensions client as well as Git directly but I got started with GitHub Desktop and really liked the interface.

Make a branch for your proposed changes. This is useful because you can only submit one pull request per branch so if you want to submit multiple pull requests you just need to make multiple branches. Each will be branched from the master branch, which you keep in sync with the parent repository.

Make the desired changes to the files.

Commit your changes to the repository. Make sure to add a descriptive commit message.

Push your commit from the local clone to your fork on GitHub.

Submit a pull request from the branch of your fork to the parent repository.

https://help.github.com/articles/creating-a-pull-request/

It seems a bit complicated at first but if you just jump in and start playing around it makes sense pretty fast. GitHub and GitHub Desktop makes everything as easy as possible. If you have any questions just let me know.

7

OK, so I forked the Arduino/avr repo and cloned it to my PC. Then I merged my changes to the twi & Wire files into the Arduino ones, and committed them to my local repo. So far, so good.

However, when I tried to test the changes, I ran into a problem; I'm using the I2CDev library, which in turn uses the Wire library. The only way I can figure so far to test everything is to simply replace the 'src' folder in my Arduino folder tree with the 'src' folder from my local clone. This works, but seems a bit clunky. Is there any clean way in Win 10 to tell the compiler to use my local repo rather than the installed Arduino folder tree?

Next step is to push the local commit to my GitHub account and create a pull request.

TIA,

Frank

8

paynterf:
OK, so I forked the Arduino/avr repo and cloned it to my PC. Then I merged my changes to the twi & Wire files into the Arduino ones, and committed them to my local repo. So far, so good.

Good work!

paynterf:
Is there any clean way in Win 10 to tell the compiler to use my local repo rather than the installed Arduino folder tree?

It's probably possible to do it using symlinks or directory junctions but I don't have experience with that. I would just replace the stock Wire library installation with your repository and work on it directly right there.

9

Pull request created :wink:

10

Let me know if you need any help resolving the issues matthijskooijman mentioned. Rewriting commit history and doing force pushes are some of the more advanced Git features that you can't do directly from GitHub Desktop but really they're not so difficult. They're also really useful! I tend to do my development work in small steps, committing and testing as I go, but then I rewrite the history to combine those fragmentary commits into atomic commits before publishing it to the master branch on GitHub. So I use these features constantly.

11

Well, I gave up on trying to get a pull request through the Arduino maintainers. It seems that rather than worrying about the technical aspects of the proposed changes, I was going to have to become a 'whitespace' expert. Now I know why the Wire library hasn't been updated in almost ten years, despite well-known and well-publicized defects.

Instead, I found Shuning (Steve) Bian's SBWire library (GitHub - freespace/SBWire: Modified Arduino Wire library which implements basic timeout.), which includes the lockup prevention code, and collaborated with him to add the parameter access functionality I was trying to put into the 'official' Wire library.

In total contrast to my efforts to create a pull request for the Arduino Wire library, the whole process went very smoothly, and my pull request was merged with the mainline code within a day of its creation.

So, for anyone running across this thread - forget the Arduino Wire library, as it will probably never be changed to fix the hangup bugs. Instead, use SBWire.

Frank

12

You need to understand that the Arduino project is a collaboration between hundreds of people. This is why whitespace, atomic commits, and commit messages are important. If every person making a commit decided to make a bunch of arbitrary unrelated whitespace changes the codebase would look like a war zone. In fact it already does from a history of sloppy development work. This makes it extremely difficult for the other collaborators. If you had taken a little more care with your commits, the Arduino developers wouldn't need to waste their valuable time slogging through a bunch of pointless changes to see what you changed. That doesn't only apply to the review process. Those commits are part of the history of the project. People will be looking at them for years to come. I frequently find myself searching back through commit histories and cursing sloppy work that makes the task unnecessarily difficult.

From your comment, it sounds like you've been involved in collaborative software development in the past. I'm really surprised you haven't learned the importance of these things by now. This is the kind of work and attitude I'd expect from a newbie.

I see in your SBWire pull request you avoided making widespread unrelated whitespace changes but still failed to follow the established indentation style by using true tabs instead of spaces. I never would have merged that PR and I consider it sloppy work that freespace let that slide. A conscientious collaborator always follows the style conventions of the codebase they're working in. So now the next person to work in that code will have big dilemma. We now have code with a mixture of spaces indentation and tabs indentation. So which one is correct? This sort of thing often takes way more of my time than the actual code I wanted to change. People love to argue about tabs vs. spaces. We all have our personal preference but when you're working in existing code the correct answer is always to follow the established style conventions.

When I was a youngster first getting involved with programming in a team environment I rolled my eyes at the uptight other employees who yelled at me for not using the correct indentation style and comments. I thought I was hot stuff writing code at a blinding rate. I didn't care about the other people who had to work with my code. I look back at that younger me now in embarrassment.