Go Down

Topic: OWBus: New Library for 1-wire probes (Read 2443 times) previous topic - next topic

destroyedlolo

Hi folks,

I'm using intensively 1-wire probes for my home automation.
Unfortunately, I wasn't able to find an high level library to access to all probes models I'm using, only individual codes leading to code duplication and API incoherence.

So I created my own one : OWBus

It was made/tested on ESP8266 but it's expected to be compatible with all Arduino like platforms.
It support DS18b20, DS28EA00 (both temperature and PIOs), DS2406 and DS2413. Alarms are on way ...

To be honest, it's my first Arduino large project, so if I focused on features, easiness (though OOP approach) but some work are to be done to comply with Arduino "top of the art" way of building libraries.

So

  • I would be happy if some people can do some tests, especially on other platform
  • I need some advises/help to ensure only the required code is added : as example, if working on DS18B20, there is no need to have DS2406's code compiled. Normally, it's supposed to be done automatically by the linker but I'm not sure it's the case
  • Anybody obviously welcomed to join this project  :)


Thanks & regards,

Laurent

pert

I think it's problematic that you use D1 in your example sketches. The Dn pin notation is specific to only certain ESP8266 boards (I think only the Wemos boards) so this will cause the example sketches to not compile for the majority of users. Use of integer constants for pin numbers will cause the sketches to compile for any board. Yes, the numbers on the WeMos board silkscreen don't match the GPIO numbers but I think the owners of those boards will probably have realized that long before they start messing with your library.

I don't see the purpose of the call to ESP.getFlashChipRealSize() in NumberOfProbes example sketch. This just makes it not compile on non-ESP8266 boards.

The folder that contains your example sketches is named "Examples", while the Arduino  Library Specification says:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#library-examples
Quote
Library examples must be placed in the examples folder. Note that the examples folder must be written exactly like that (with lower case letters).
Yes, it works as it is now but you have no guarantees it will continue to into the future. I always think it's best to follow the specification unless there is good reason not to.

I find the placement of some of the source files in a subfolder named OWBus a bit strange. Is that to avoid conflict with other libraries that contain files of the same name? Have you tested that for backwards compatibility with older IDE versions? Typically if library source files are placed in a subfolder it will be named either utility (in the case of the 1.0 library format) or src (in the case of the 1.5 library format).

Multiple_Temperature example sketch compilation fails with ambiguous errors related to probe2.isParasitePowered(), probe2.getAddress(), and   probe2.getFamilly(). This seems specific to DS28EA00, does not occur for DS18B20.

Misspelling of getFamilly() (should be getFamily()).

destroyedlolo

Thanks Pert for your return.

I'll apply all the changes you've spoke about.

I find the placement of some of the source files in a subfolder named OWBus a bit strange.  Is that to avoid conflict with other libraries that contain files of the same name?
I need some "sub include" for devices, and I did use the same habit as with others non Arduino C or C++ library projects. And yes, it's avoiding confusion with other includes that may have the same name (as probe's name may be quite common in such project, isn't it ?).

My goal is to finally having only the code needed by the probes I'm using and one task is . With leads to your next point :
Typically if library source files are placed in a subfolder it will be named either utility (in the case of the 1.0 library format) or src (in the case of the 1.5 library format).
I did only some early tests (as I said, my goal until now was mainly to move ahead on functionalities I need) to put sources in src. I've tried to split the monolithic OWBus.cpp to several sub .cpp files, but the compiler complained it can't found other piece of code :(
But I know it's something I have to do.

Multiple_Temperature example sketch compilation fails with ambiguous errors related to probe2.isParasitePowered(), probe2.getAddress(), and   probe2.getFamilly(). This seems specific to DS28EA00, does not occur for DS18B20.
As the number of examples is increasing (and will increase) is it a way to batch compile in IDE compatible way ?
(makefile, option to the IDE to only launch the compilation, whatever ...  ?). If yes, I'll automatize regression testing to prevent such isues.

Bye

pert

I did only some early tests (as I said, my goal until now was mainly to move ahead on functionalities I need) to put sources in src. I've tried to split the monolithic OWBus.cpp to several sub .cpp files, but the compiler complained it can't found other piece of code
I'd be happy to look into that if you can provide the version of the library that had the problem. If the way you have it now is how you want it and it works then I guess it's no problem. I was just surprised to see it since I'd never seen any other Arduino library do that. I expected that it wouldn't compile but it did. I only checked with Arduino IDE 1.8.5 and the 1.9.0 beta build though so I don't know about backwards compatibility.

As the number of examples is increasing (and will increase) is it a way to batch compile in IDE compatible way ?
(makefile, option to the IDE to only launch the compilation, whatever ...  ?). If yes, I'll automatize regression testing to prevent such isues.
You can use a script with the Arduino IDE command line interface to do test compilations, or there are some makefiles available. Another option is to set up continuous integration testing online using Travis CI or something similar. The advantage of that is it will automatically test pull requests that are submitted to your repository. The disadvantage is that to run the tests on your own work you need to push them to GitHub. Once you have Travis CI set up for your repository it will automatically run the configured tests on every commit. When I switched to using Travis CI I found that kind of annoying because usually I like to test my work locally and only push to the public repository once the work is finalized but now I just work in a development branch and then rebase the commits to master once the work has passed the testing. I wrote a script that's intended to make it very easy to use Travis CI on Arduino projects:
https://github.com/per1234/arduino-ci-script
I'd be happy to help out with getting Travis CI set up on your repository if that's something you think would be useful.

destroyedlolo

So, I corrected the issues. The tricky one, "ambiguous definition" was due to "diamond inheritance"  (I duno if it's the correct english wording, meaning that a class inherits from 2 others one which both inherit themselves from another unique one ...).

I'd be happy to look into that if you can provide the version of the library that had the problem.
It was only a quick and dirty test : will redo it more "seriously" :)
If the way you have it now is how you want it and it works then I guess it's no problem.
I still need to ensure that unneeded code is not included in the final binary. For example, if I'm not using DS2406, the result shouldn't contain any part of its code.


I definitively needs something to avoid regression and Travis CI looks the good candidate as probably more secure than Makefile if someday I'm not the only one working on the code.

I'd be happy to help out with getting Travis CI set up on your repository if that's something you think would be useful.
I'll first trying to split the code to really start in good condition and avoid deep change afterward, but yes, it's a good idea :)

Bye

destroyedlolo

Hi pert,

In the branch reorg, I split my code in several .cpp source code and it's compiling without trouble.
If you don't have other comments, I'll merge it to the "master" one and starts with Travis CI. You're help will be very welcome for this setup :)

Bye

Laurent

pert

I'm putting together a demonstration of my proposals for using Travis CI on your repository now. I'll post a link to it as soon as it's ready and we can go from there.

pert

OK, I have set up testing with Travis CI on my fork of your repository with the help of my script.

Here's the build of the more basic version of my proposal:
https://travis-ci.org/per1234/OWBus/builds/308749348

Which is the test of this branch of my fork: https://github.com/per1234/OWBus/tree/travis-ci

It compiles all the example sketches for Uno and WeMos D1 Mini using a selection of milestone IDE versions. It's easy enough to add other boards or IDE versions if you want. That does add more time to the build process, especially when adding other IDE versions, which requires them to be downloaded. The current build time is around 10 minutes

It also compiles using the hourly build of the Arduino IDE but compilation failures with that IDE version do not cause the build to fail. This is intended to give you advance notice of upcoming changes to the Arduino IDE that may introduce incompatibilites with your library.

You may notice that, even though I have the verbosity set to the minimum level, the build log exceeds the length that can be displayed in Travis CI's standard "folded" output and so you need to sift through the raw log. This is quite inconvenient so I have added the option for my script to publish a tab separated summary report of the compilation results to a GitHub repository. I have demonstrated that feature in my "bells and whistles" version of the proposal:
build: https://travis-ci.org/per1234/OWBus/builds/308749500
branch: https://github.com/per1234/OWBus/tree/travis-ci-report
report: https://github.com/per1234/CI-reports/blob/OWBus/build_00011/travis_ci_job_report_00011.001.tsv
it has the optional behavior of commenting on the commit with a link to the report:
https://github.com/per1234/OWBus/commit/3f33988e768106c56faef2699590d26fca1bdbb2#commitcomment-25913971

I find the report to also be helpful for comparing the results of a commit to previous reports. Unfortunately I have not been able to generate a consolidated report but it's easy enough to just merge together the report files and sort the data however you like in a spreadsheet program.

The downside to the report feature is that it requires you to add a GitHub personal access token to allow the script to commit to the repository and optionally to comment on the commit. For security reasons this means it's best to store a copy of the script in your own repository since the token will be passed to the script and otherwise you would have no guarantee that I have not modified the script to do something malicious with your token. You can see how I have set this up in my branch here:
https://github.com/per1234/OWBus/tree/travis-ci-report/extras/arduino-ci-script
Let me know what you think.

destroyedlolo

Hi Pert,

So, I corrected all the bull*its I made during nightly coding, like forgetting return statements on some aliases ...  >:(
It's why I'm always coding with -wall enabled with my other projects ... but didn't made the change yet on the IDE (because I haven't searched yet what I have to modify  :smiley-confuse: ).

Anyway, my code being now almost clean, I'm starting with Travis.
As one of my goal is to understand what I'm doing, I didn't merged your changes as they are but set the link with Travis by myself following some tutorials I found on the web and apply your .travis.yml

And problems started : Travis is complaining about

Code: [Select]
$ cd destroyedlolo/OWBus

$ git fetch origin +refs/pull/2/merge:

fatal: Couldn't find remote ref refs/pull/2/merge

The command "eval git fetch origin +refs/pull/2/merge: " failed. Retrying, 2 of 3.

fatal: Couldn't find remote ref refs/pull/2/merge

The command "eval git fetch origin +refs/pull/2/merge: " failed. Retrying, 3 of 3.

fatal: Couldn't find remote ref refs/pull/2/merge

The command "eval git fetch origin +refs/pull/2/merge: " failed 3 times.

The command "git fetch origin +refs/pull/2/merge:" failed and exited with 128 during .

Your build has been stopped.


Do you know where I was wrong ?

[edit]
Hum, some minutes afterward, I got a "build successful" email ... so, was it a temporary error I can ignore ?
[/edit]
For the moment, the raw text is enough. I'll switch later to tsv if needed :)

The next step will be to had another target to build on low end ESP's, I mean with only 512Kb of flash.
What will be the FlashSize I have to set ?

So I'll have 3 targets :
  • D1 which is close to the ESP-201 I'm working on
  • a 512K one
  • the Uno one to test on non-ESP

to test on low-end configuration and my projects one.

And finally, I would like to automatically rebuild another project (in very very early stage yet) which is for my very own chicken coop automation : https://github.com/destroyedlolo/ESP8266/tree/master/Poulailler

  • Do I have to create a dedicated repo for it or can I have keep it part of ESP8266 one ? (Travis doesn't have to be triggered for changes outside "Poulailler"
  • How to integrate OWBus as decencies of Poulailler ?
  • How to make Poulailler tested if its own code is changed or OWBus one ?


Thanks again for all your help

Bye

Laurent

pert

I'm glad to hear you got Travis CI going on your repository!

It's why I'm always coding with -wall enabled with my other projects ... but didn't made the change yet on the IDE (because I haven't searched yet what I have to modify  :smiley-confuse: ).
File > Preferences > Compiler warnings > More will give you -Wall. Confusingly enough File > Preferences > Compiler warnings > All gives -Wall -Wextra. The latter is what I have my script set to use.

For the moment, the raw text is enough. I'll switch later to tsv if needed :)
There actually is a tsv report at the very end of the build log (you have to view the raw log). It's just kind of inconvenient and slow to get to it and it doesn't render well at all until you copy and paste it to a spreadsheet program. This is done by this line:
https://github.com/destroyedlolo/OWBus/blob/master/.travis.yml#L60

The next step will be to had another target to build on low end ESP's, I mean with only 512Kb of flash.
What will be the FlashSize I have to set ?
The "esp8266:esp8266:d1_mini:CpuFrequency=80,UploadSpeed=921600,FlashSize=4M3M" you see used in the sketch is the "Fully Qualified Board Name" (FQBN) of the the WeMos D1 R2 and Mini board of the ESP8266 core for Arduino with the default settings from the other custom Tools menus added by that board definition. You can find the FQBN for the board you want to compile for by doing a compilation in the Arduino IDE with File > Preferences > Show verbose output during > compilation checked. You will see it in the -fqbn option passed to arduino-builder in the first line of the output. For example, the default settings for the "Generic ESP8266 Module" board gives:
Code: [Select]
esp8266:esp8266:generic:CpuFrequency=80,FlashFreq=40,FlashMode=dio,UploadSpeed=115200,FlashSize=512K64,ResetMethod=ck,Debug=Disabled,DebugLevel=None____

So to your .travis.yml you could add the lines:
Code: [Select]

  - build_sketch "${SKETCHBOOK_FOLDER}/libraries/OWBus/examples" "esp8266:esp8266:generic:CpuFrequency=80,FlashFreq=40,FlashMode=dio,UploadSpeed=115200,FlashSize=512K64,ResetMethod=ck,Debug=Disabled,DebugLevel=None____" "false" "oldest" "newest"
  - build_sketch "${SKETCHBOOK_FOLDER}/libraries/OWBus/examples" "esp8266:esp8266:generic:CpuFrequency=80,FlashFreq=40,FlashMode=dio,UploadSpeed=115200,FlashSize=512K64,ResetMethod=ck,Debug=Disabled,DebugLevel=None____" "true" "hourly"

That will compile with all installed IDE versions except the hourly with failure not allowed and compile with the hourly build with failure allowed for the "Generic ESP8266 Module" board with 512 kB flash.


Do I have to create a dedicated repo for it or can I have keep it part of ESP8266 one ? (Travis doesn't have to be triggered for changes outside "Poulailler"
Travis is triggered for every commit or pull request. Maybe there's a way to add some logic to immediately finish the build with success if the files in the Poulailler folder were not affected. The first thing that comes to mind is pulling that information from the output of a Git command. The tricky thing I foresee with that is if you push multiple commits at the same time Travis only builds the latest commit. So that wouldn't work with just getting a changed file list for the latest commit.

How to make Poulailler tested if its own code is changed or OWBus one ?
I think the best way to trigger a Poulailler Travis build on changes to OWBus would be to make the OWBus Travis build do an API trigger of the Poulailler build. There's information on that here:
https://docs.travis-ci.com/user/triggering-builds/
I've never done this before. I think you might need to use a Travis CI API token for that, which I think is something you want to keep secret, so be sure to store it as a secret Environment Variable so it doesn't show in the public logs, following these instructions:
https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings
and leaving the "Display value in build log" switch in the off position.

The other option would be to clone the Poulailler repository and compile the sketch in your OWBus build. That could be better if you wanted to do Poulailler builds using the OWBus tag but wanted to also test Poulailler using the latest version of the OWBus library in the OWBus build.

Thanks again for all your help
You're welcome. It took me a while to figure this stuff out when I got started with it since using Arduino with Travis CI isn't so easy as other projects that Travis has built in tools and documentation for so I like to pass that on to others. That's been my motivation for writing arduino-ci-script and trying to make it useful for testing any sort of Arduino project. I think the Arduino community could really benefit from more people using continuous integration on their projects.

destroyedlolo

#10
Dec 02, 2017, 12:09 pm Last Edit: Dec 02, 2017, 12:33 pm by destroyedlolo
Thanks for the compiler info and for other boards definitions : exactly what I was looking for  :smiley-lol:

For Poulailler, I will put it in an external repository because ... it will not be an example but a full project (that I have to finish before Xmass holidays :) ).

Quote
I think the best way to trigger a Poulailler Travis build on changes to OWBus would be to make the OWBus Travis build do an API trigger of the Poulailler build.
It's not only that, I need to tell to Travis to include OWBus as a library. But as it isn't (yet ?(*)) part of the IDE known ones, the question is "how".

To be more precise : my need to test Poulailler compilation for OWBus update, but CI on poulailler itself is not mandatory, as I'm always compiling/testing locally before committing changes :)

Quote
I think the Arduino community could really benefit from more people using continuous integration on their projects.
Clear, and it's a great help for a decent regression testing for EVERY changes ! As well as compilation with -Wall enabled ... when I see the zillion of warnings coming from other libraries :(

(*)Do you think this library may be useful for others ? Because I'm a bit surprised that no one else comment ? Is it useless (but for me obviously) ?

Bye

Laurent

pert

For Poulailler, I will put it in an external repository
I think that's the best solution. It's a good idea to make a different repository for each distinct project and it's not as if GitHub limits how many you can have.

I need to tell to Travis to include OWBus as a library. But as it isn't (yet ?(*)) part of the IDE known ones, the question is "how".
I distinctly remember answering that in my last reply but now I see it's not there, I must have accidentally deleted it before posting. In the case of the OneWire library I was able to just use the command:
Code: [Select]
- install_library OneWire
because that library is in the Library Manager and I have written the script so that it can accept the library name of libraries in the Library Manager. However, install_library will also work with a URL. If the URL ends in .git then it will do a Git clone of the repository. So if you add this line to your Poulailler repository's .travis.yml file:
Code: [Select]
 - install_library https://github.com/destroyedlolo/OWBus.git
it will install the OWBus library as a clone of the OWBus repository at the tip of the default branch (master). install_library also has an optional branchName parameter if you wanted to specify a different branch or a tag (AKA GitHub release). install_library does not currently support the option to automatically checkout the newest tag (it's on my to-do list) but if you want that you can adapt the commands I used to checkout the newest tag of arduino-ci-script:
https://github.com/destroyedlolo/OWBus/blob/master/.travis.yml#L16-L22

when I see the zillion of warnings coming from other libraries :(
Yes, that's annoying. I wish everyone would enable warnings and fix them whenever possible. I try to write my own code without warnings so it's frustrating to have to skim through a lot of warnings from other people's code when I'm watching for ones I might have made.

Do you think this library may be useful for others ?
I've never used a 1-wire device so I haven't even made an attempt to evaluate your library. I think if you found it useful there are likely to be others who will also. I think it's great to have a lot of options available for Arduino libraries. You've made a good effort at documenting the library and responding to my feedback and now with the CI testing it's even better so as far as the administration of your project goes I certainly think you've acted commendably.

Whenever I write some code that will be useful for multiple projects I turn it into a library. If there is any chance of that being useful to others I publish it on GitHub. None of them are terribly impressive and they haven't gotten much traffic but I feel I've done my duty in giving back to the open source community to the extent of my abilities so I think it was worth doing. It pushes me to make a bit more of an effort with documentation and API design which seems like extra work but in the end I directly benefit from it even if no one else does. I'm certainly grateful to read my own documentation when I come back to a project after some months or years of not having used it.

Go Up