New LiquidCrystal library - LCD library

fm:
Hi Bill el al.,

sorry I didn't answer before. You have raised very fair points in your previous post, I will do a cut and past from the previous post to answer and keep in one place the library discussion.

I can also see that people will not like to replace the current LCD library and when they compile find out that it doesn't. To be fair, I think that I will rename this library and try to create a "compatibility define".

Mind you, this is not an easy task, since to be fair, the LiquidCristal class would have to be the base class to all 4 bit objects and also to be used as a virtual class.

If the intent is to replace the LiquidCrystal library, then, IMHO, I think it is important
that it be a drop in replacement and still allow existing LiquidCrystal sketches
to continue to compile without having to make any changes, even if it is as simple
as updating the constructor names in the sketch

I think that the easiest thing to do is:

  • Rename the base class to something . I would appreciate all your help here. Something that is easy to remember and descriptive of what it does (I had "LCD" in mind).
  • Rename the current LiquidCrystal_4bit class to LiquidCrystal.

This would simply give you full compatibility with the current library and be able to use the extensions differently. In this case, the library would be a simple replacement without having to change a single line of code.

I think I'd try to keep the separation and not rename anything LiquidCrystal.
Not sure what the real answer is for a new name because even the name LCD may not be clear enough
I've seen people get confused and call both text and graphic devices a "LCD".
(I'll use the name hd44780 in my examples below)

I think the key to providing backward compability.
is how to transparently map the existing LiquidCrystal constructor name in their source code
to the new libraries constructor name.
i.e. you don't necessarily have to actually rename the hd44780_4bit constructor to "LiquidCrystal".
You just have to recognize that constructor name or transparently map it to your new name.
After that, since all the API function names
in the new class match the old class it is all transparent.

So, I think you could have a a set of headers that all use something like hd44780 and then
have a LiquidCrystal.h header file that might look like:

#include <hd44780.h>
#include <hd44780_4bit.h>
#define LiquidCrystal hd44780_4bit

An alternate method might be to have a constructor named LiquidCrystal in the hd44780_4bit.cpp module
that can be enabled with a special define.
In that case the LiquidCrystal.h header would include the 2 hd44780 header files as above
and turn on the special define to enable the LiquidCrystal named constructor.

I don't believe that it can be done using only a simple define in the new headers
to turn on a 100% backward compatibility mode because
to have full 100% backward compatibility, the header file LiquidCrystal.h has to exist because that is what
the existing code that uses the LiquidCrystal library includes.
And if a file called LiquidCrystal.h exists in this new library,
then this new Library cannot co-exist with the current LiquidCrystal library as it
will confuse the IDE or worse the include path points to both directories and its not clear which LiquidCrystal.h header
is included. (the original one, or the one from this new library)
It is a total catch-22.

I don't see that this issue can ever be resolved.
There are essentially 2 choices

  • total replacement of existing LiquidCrystal library and have a new fully backward compatible library
    (which requires removing existing LiquidCrystal library)
  • A new library named something like hd44780 that lives side by side with LiquidCrystal

The 2nd case is probably easier to swallow for people. This also removes any backward compatibility issues
for the new library as old/existing code continues to use the existing LiquidCrystal library and new applications
will simply include the new hd44780.h and then the appropriate hd44780_xxx.h header for the particular hardware.
Also, any existing application that is to be converted to using the new library will simply require a few small edits
to get the includes and the constructor names set up correctly.

Another possibility might be still going with option #2 above but also
to ship the new library with something like a dummy/masked LiquidCrystal.h
backward compatibility header that is named something like LiquidCrystal.h.compat
(to avoid any name collisions with the current LiquidCrystal library and hide it from the IDE)
that contains the includes and mapping defines kind of like what was shown above.
Then provide instructions to people that want use this new library as a full replacement for the existing LiquidCrystal library with
how to do that.
(By simply renaming the LiquidCrystal.h.compat to LiquidCrystal.h and removing the current LiquidCrystal library.)

Just some ideas...


One big thing I noticed that I think is a real deal killer is that the library code currently only works with the Arduino 1.0
pre-release code and does not work with the pre Arduino 1.0 environment.
There are few places that need to check the value of ARDUINO < 100 and then include <wiring.h> rather
than "Arduino.h> and then the write() function needs the same conditional and either return vode or a size_t

BTW, very cool that the shift register code is now in there.

--- bill

I see that we have followed very much the same thinking process.

There is no simple way to have full compatibility without having to replace the LiquidCrystal library as such.

Adding defs to the Library will simply end up in a compiler/linker problem where symbols are multiply defined. This is how the Arduino compilation environment is. On a different environment it would be simpler in the sense that you would point your include and linker libraries to different locations.

In any case, what I am going to do is to have it a replacement library since as far as the 4bit and 8bit control are identical.

Regarding the arduino SDK version, it is just a question of time that all users will migrate to the new version. I don't like messing with compilation flags to make things compatible, but I can understand the need for it. So for the first versions of the library I will use conditional compilation flags to make the library compliant with the 22 version and the 1.0.

I was also thinking about doing a port of the SPI control library but at this point I think the package can go as is.

Thanks for your feedback it has been very useful.

LCD library V1.1.0 - New LiquidCrystal library available for download:

The library adds the following new features:

I haven't tested on the Arduino 22 SDK, so feedback would be most welcome on how it behaves.
Remark - library examples have not been ported to the 22 SDK.

fm:
Regarding the arduino SDK version, it is just a question of time that all users will migrate to the new version. I don't like messing with compilation flags to make things compatible, but I can understand the need for it. So for the first versions of the library I will use conditional compilation flags to make the library compliant with the 22 version and the 1.0.

I'm glad to see the support for 0022 (I'm off to try it out on my 1602 display - will report back)

I do think the adoption rate of 1.0 is a great unknown.
I would not underestimate the confusion and frustration that 1.0 is going to create, particularly
given that it is going to break so much existing code.
Think about this:

  • 100% of all the existing library code will not work with 1.0 until it is modified
  • 1.0 uses more RAM which will cause some sketches to no longer work (and they will fail in crazy ways since it will likely be stack overflows)
  • Some APIs have slightly changed (Print class being the most notable) which causes existing sketches to no longer work or compile.

I believe that some folks will simply choose to stay with 0022 instead or even revert back to it from 1.0,
particularly if they as the start to find existing libraries or sketches that don't work with 1.0 or that things
simply are not as stable under 1.0 as they were under 0022.

From a high level perspective, for all it breaks, it really doesn't offer much.
A few IDE GUI changes and overlapped hardware serial output in some situations
at the cost of eating up extra ram, a few new events and timer control is about it.
Yes it does allow some new things going forward but has a very high cost in terms of
backward compatibility.

Adding to the future pain & confusion is that it has not been particularly easy to find the 1.0 beta & release candidates.
To get the best feedback, it should have been prominently offered along side the latest (0022) release so that
people of all skill levels could be trying out the release and reporting feedback.
As it is, I think that 1.0 is getting very little testing by the real end user target audience.
As an example, I find it quite telling that the GLCD v3 library that I maintain
has been downloaded over 2000 times,
and while I have added 1.0 support into that code, I have not released it yet and have not
received a single request for 1.0 support.

So, IMHO, 1.0 will be a mess and create quite a bit of frustration for quite some time once
it is released.

--- bill

I'm glad to see the support for 0022 (I'm off to try it out on my 1602 display - will report back)

Fantastic, thanks for trying it out. Sounds brilliant - unfortunately I was not able to run it with the 0022 version.

I think you have very fair point and agree with you that the version 1.0 is going to cause a lot more headaches than what it is trying to solve. Only time will tell.

You are more than right, finding or downloading version 1.0-RC2 is not an easy feat. For some reason it is not where the rest of the downloads for version 0022. You do need to go into the entire layout to find it. Not very practical for early adopters and start rolling it out around the community.

Nice to see that you are the maintainer of the GLCD Library. Great job there!

Initial testing is not good.
BTW, I highly suggest you install a 0022 package for testing.
0022 can coexist with the 1.0 packages
so that testing can be done on both as I believe that 0022 really needs to work
and is going to be around for quite some time.

ARG!!!! backward compatibility for 2wrire stuff is also broken in 1.0!
Why oh why didn't the arduino guys provide a wrapper function to map read() to receive()
and write() to send(). Just another example of and how far reaching the 1.0 damage is
and how messed up the 1.0 transition is going to be.
These guys really need to have a "backward compatibility" mode (define) that can be set
from the IDE to allow all the old sketches to continue to compile under 1.0 - I'm going to add this to the Arduino issues list.

Back to the new lcd library.

There are some issues with the 1.0 ifdefs on the 0022 side.
Look at the declarations and code for write() in the LCD.h and LCD.cpp files:

needs to be something more like:
LCD.h

#if (ARDUINO <  100)
   virtual void write(uint8_t value);
#else
   virtual size_t write(uint8_t value);
#endif

LCD.cpp

#if (ARDUINO <  100)
void LCD::write(uint8_t value) {send(value, HIGH);}
#else
size_t LCD::write(uint8_t value) 
{
   send(value, HIGH);
   
   return 1;           // assume OK
}
#endif

Ok a few typos, those are easy to fix.
Then there is the issue of using .ino for the sketch extensions.
That doesn't work for 0022. .pde will work for both 0022 and 1.0
Ok so thats an easy one to fix as well.

There is also the 2wire issues mentioned above for 0022.
I tried to use ifdefs to replace write with send and read with receive but it wasn't enough.
I got some strange undefined references:

undefined reference to `__cxa_pure_virtual'

perhaps due to the function prototypes changing or perhaps a name collision
with the Wire library send() function.

So I never was able to get it to link on 0022. (it did build, link and work on 1.0rc2)

You will see the errors/issues when you compile the code on 0022 after you fix the write/send read/receive issue.

Now for the Grand Daddy of all problems.
"As is" the new library modules will not compile on 0022 or 1.0 unless <Wire.h> is included by the users
sketch. Existing code does not include this header so it breaks backward compatibility.

This is caused by the way the way the IDE handles its include paths.
IMHO it is a HUGE issue that caused much heartburn during the ks0108/glcd library
development. In fact it still creates issues to this day.

Here is the issue. The IDE sets the compiler include path to the Arduino core directory
plus it determines additional paths path based on the included files discovered/used in the sketch.
The sketch is compiled using that include path.
However, and here is the HUGE issue, when compiling the files for a library that the IDE detected the sketch needs,
the IDE uses the same include path as the sketch plus it also adds any directories in the library that contain headers,
to the include path when it compiles all the modules for the library.

This now creates some horrible issues. Like a library has no guaranteed way to include the header files
for another library because the base level library directory is not part of the include path and the
users sketch may not include a header from that library.

So things like
#include <Wire.h> won't work in a library unless the user sketch also includes <Wire.h> because
the necessary path is not part of the include path unless the sketch includes it.
Now you can sort of cheat and use "../Wire/Wire.h", because the library modules being compiled
are all relative to base level library directory.
However, this will not work if the user installs the library
under his personal libraries directory under his sketchbook because in that case the other
libraries may not be relative to the library and may be over in the Arduino libraries directory wherever the
IDE was installed.
You can't use <../Wire/Wire.h> because it looks like gcc doesn't allow relative paths off directories
in the include path.

The Arduino sketch building methodology simply falls apart when libraries need to be able to directly call other libraries
transparently from the users sketch.

The real answer is to always add both base level library directories to the include path.
That way you can do things like:
#include <Wire/Wire.h> or #include <glcd/glcd.h> etc... It is very clean and it is the way it should have been
done from the beginning rather than this goofy method of adding all the include paths for just the libraries that are detected as being used.

But I think convincing the Arduino team to make any changes is difficult at best.

--- bill

I noticed is that this new library is 104 bytes larger than the current
LiquidCrystal library on 1.0rc2 when using the 1.0rc2 LiquidCrystal supplied HelloWorld.ino example sketch.

2648 bytes (new library on 1.0rc2) vs 2544 bytes (LiquidCrystal library on 1.0rc2)

The HelloWorld.pde sketch with the old LiquidCrystal library on 0022 is 2522 bytes.

So switching from the current LiquidCrystal library on 0022 to 1.0 with the new cd library
costs an extra 126 bytes of flash, and there is also some additional RAM requirements as
well since there are some new virtual functions.

Just be conscious of these additional resource demands.

--- bill

Hi Bill,

thanks for the feedback. Very much appreciated. I will look into the 0022 issue. The problem is I work on a Mac and having two instances of the Arduino IDE is a bit of a pain since you have to install one and remove the other one. I will take a look and see what the problem is during linkage.

As for the Wire.h, every single sketch that uses the Wire library has to have the include in the sketch. It is a real pain. How did you sort this out in the GLCD? Relative includes?

Size, size. This is a pending subject. I haven't taken a closer look at the elf file, but I assume that some of the penalties are from having 2 classes more than just 1, additional constructors, a bit more code during initialization to make the library a bit faster.

As far as RAM, there are 2 new virtual methods, so the overhead is there, not much, but a bit.

OK, first hurdle sorted out. I can use both versions of the IDE on the Mac.

During the installation, click on keep the old one installed.

fm:
As for the Wire.h, every single sketch that uses the Wire library has to have the include in the sketch. It is a real pain. How did you sort this out in the GLCD? Relative includes?

Wire.h isn't a problem for GLCD is currently it currently doesn't use or support the 2wire stuff. :smiley:

With this new LCD library the issue is that the sketch is not using the Wire library.
The LCD library is the the one actually using the Wire library.
And so the complexity is that when using older LiquidCrystal sketches, the 2wire stuff is not used at all
(no include for Wire.h) even though it is required for the new LCD library modules to compile.

This has to be resolved if you want the library to be a drop in replacement with no edits on older source.
To get it work, I changed the includes in I2CIO files to use "../Wire/Wire.h" instead of <Wire.h>
It works, but it means that the library must live in the arduino libraries directory vs under the uses sketch area.
This probably isn't that big of an issue since in order to use it they have to remove the originally supplied
LiquidCrystal library because this new library cannot coexist with the current LiquidCrystal library.

Have you thought any more about just not providing 100% backward compatibility by default
by say having a hd44780 library and then provide a LiquidCrystal class or thin library
that can be turned on for those that want full compatibility at the expense of removing their current LiquidCrystal library?

Or perhaps maybe never offering 100% backward compatibility and always require them to
edit their code to use the new library?
The edits are quite minimal and are limited to header file names and constructors names.
That way you don't require them to make any changes to their existing Arduino install and they can install
this new library in their sketchbook area.
They can use the new library if they use the new library headers and constructors
and get the stock LiquidCrystal library if they use the LiquidCrystal headers and LiquidCrystal constructors.
It would keep a very clean separation and allow them to install or remove the library in their own
sketchbook area.

--- bill

Ummmm! I need to reconsider after seeing how the entire compilation chain works (people wonder why I like to develop my stuff with eclipse).

For the linker problem, I got round it by not declaring the "begin" and "send" as pure virtual methods. For some reason, which I still need to figure out, the thing doesn't like pure virtual methods on the base class. If I change it to an empty method it is cool.

The nice thing I like about the library is that the I2C implementation on the example files are almost 700 bytes less that the 4bit library (not bad).

I've uploaded version 1.1.1 up to the repository, all tested in 0022 and 1.0-RC2.

Tomorrow, a bit more thinking and considerations regarding the library as a drop-in or not...

RELEASE ANOUNCEMENT

Rel. V 1.1.2 of the LCD library has been released with performance enhancements.

Release highlights

  • Version V 1.1.2 is 3.25 times faster than the original LiquidCrystal library
  • Minor comments added to source code
  • Documentation only available in .html format

Download
Project wiki - https://bitbucket.org/fmalpartida/new-liquidcrystal/wiki/Home
Project donwload - https://bitbucket.org/fmalpartida/new-liquidcrystal/downloads

Version V 1.1.2 is 3.25 times faster than the original LiquidCrystal library

Which APIs?

Does this measurement take into account the slow down due to calling via the use of virtual methods?

Sounds very interesting. I will try to download next week.

Iain

All the operations apart from the home and clear which are slow by controller definition. However, I am running these down to see if I can get them to go a bit faster.

The comparison is 1 to 1, i.e. same sketch running with the LiquidCrystal Lib and the LCD Library, no source code changes, just replace one library by another.

The test, writes 16x2 characters to the display, clears the LCD and starts again. Each cycle the performance measured is about 3.25 times faster.

That is awesome

sixeyes:
That is awesome

Thanks for that.
Try it out and let me know how it behaves with your LCD/s. I've got only a few models to try it on.

Very nice. I like it.
It now compiles and runs using existing Liquidcrystal code on 0022, and 1.0rc2
I've tested this on this shield: Keypad LCD Shield V2.0 - emartee.com
Unfortunately, I can't tell you which LCD it is as it is soldered on the shield.

I think the challenge will be for things like I2c and trying to be compatible with other i2c
implementations and to not break other lcd libraries.
For example, your i2c backpack and backpacks
like this: i2c / SPI character LCD backpack - STEMMA QT / Qwiic : ID 292 : $9.95 : Adafruit Industries, Unique & fun DIY electronics and kits
or this: Error, Electronic & Electronics Components Depot United States
don't use the same pinout for the bits from the i2c register output going into the hd44780 interface.
Many Arduino users are quite hardware ignorant and may really struggle with these types of differences.
I'm not sure how to best support these different hd44780 I2C backpacks and yet make it easy for the Arduino user.
Its a real tough challenge.

Some vendors use a separate lcd library for their board. This is quite simple and avoids any compatibility
or library collisions as it is a separate library and the user simply copies
over the .cpp and .h files into the libraries directory and thats it. They include the vendors "xxx_i2c.h" header
and then it all works just like the LiquidCrystal library.

While adafruit has a seperate custom i2c lcd library for their backpack: https://github.com/adafruit/MCP23008-library
Adafruit also now has released a multi headed LiquidCrystal library that works for both 4 bit parallel and their
own i2c or SPI: GitHub - adafruit/STEMMA_LiquidCrystal: Liquid Crystal Library for Arduino
Their implementation is slower than a real i2c implementation as it uses an "arduino like" interface across the i2c. i.e. they
do pinMode(), DigitalWrite() type operations through the i2c to control the lcd pins.
So it does independent i2c operations for setting every single individual hd44780 signal.
It does have the advantage that if they would update their i2c constructor they could configure which
i2c output register bit controls which lcd signal. Today they have it hard coded to their own i2c backpack
but this could allow it work with any backpack.

I like your implementation as it can be made to support any hd4480 backpack.
I think the challenge is how to deal with all these different interfaces when replacing the LiquidCrystal library.
i.e. Your library cannot coexist with the latest AdaFruit library. Both want to replace the LiquidCrystal library
and both have hardcoded i2c implementations that are incompatible with each other.

From an end users perspective, it gets tricky at best and more than likely frustrating with the current state
of libraries in that if you have different i2c backpacks, it may or may not be possible to control them with just
sketch changes. You may have to replace LiquidCrystal libraries between sketch builds for the different
boards. And that is pretty ugly/painful.

As it is today, the i2c lcd libraries look a bit problematic and painful to use if you have more than a single
vendor's i2c lcd backpack.

I think with your library model, it can be resolved.
It could be through either adding additional supported "boards", or by updating the i2c constructors to support
the ability to configure which hd44780 pin is connected to which i2c register bit. kind of the way the
existing LiquidCrystal constructors work with the 4 bit interface. In the case of i2c, the i2c register bit number
would be specified rather than an arduino pin number.
I think the latter, while not the absolute fastest way to run the interface, offers the greatest flexibility and
ease of use for the users especially those that want/need to support different i2c/spi lcd backpacks
that are wired up differently.


On performance tuning, I would caution you to be careful when using some of the techniques that the code is
currently using. For example, in functions like pulseEnable() some of the delays have been removed.
This is making an assumption about a particular API implementation [ digitalWrite() ] that
offers no such guarantee. ie. digital does not promise to take any particular amount of time.
So it theoretically could take zero time. While 0 really isn't reality, much smaller than 1us or even
the needed 450ns is a potential reality. Consider if this library is to be ported to say the chipKit or the Maple
then you may have issues. Also if the arduino code is compiled differently by compiling all the code together
vs seperately and building core libraries you can get optimizations that can dramatically affect
the timing. If you replace the poor Arduino core code with smarter code, then it could
potentially optimize the digitalWrite() functions down to single cycle instructions which on a standard arduino board
is only 62.5ns

If you still want to go down the path of trying to squeeze out the performance by removing or reducing delays like this,
I'd encourage using some sort of ifdef that can turn them back on to insert the proper delays for environments
that have faster digitalWrite() implementations.
BTW, if you want to step just a bit outside of the Arduino core library functions, there are other delay functions
that can offer shorter than 1us delays with much better accuracy than the Arduino delayMicroseconds() function
which has terrible accuracy.

--- bill

Hi Bill,

thanks again for your feedback. It seams that you are reading my mind... I was just polishing up the I2C class, with new constructors to enable configuration of the control and data pins of the LCD. This is something that I had in mind for the I2C class to enable people to configure the pin mapping of their "backpacks".

Should the backpack, controller chip differ from what is already there, it is a simple question of just extending the class hierarchy.

As far as optimizations are concerned, I am not too concerned about a different digitalWrite coming into the scene, in the sense that people would have to rewrite parts of the SW. However, I do think it is a nice have compilation flag to compile in our out the FAST mode doesn't hurt at all. In those lines, I was also going to write a fast digital write to find the limits of the LCD. How much more will I be able to squeeze out? Perhaps a few usecs per write operation, but it will be very specific to AVRs.

Keep you all posted, stay tuned for version 1.1.3 of the library.

fm:
As far as optimizations are concerned, I am not too concerned about a different digitalWrite coming into the scene, in the sense that people would have to rewrite parts of the SW.

Well, in some cases that re-writing has already been done.
There are already Arduino compatible s/w core libraries for the Maple and the Chipkit
which use 32 bit ARMs running at 60-80+mhz.
I have not looked at their digitalWrite() implementation but the processor is already
5-6 times faster. So even if they used the same crappy Arduino core code digitalWrite()
implementation, they would likely already be sub microsecond for digitalWrite() given
that the current digitalWrite() on a 16Mhz AVR is about 4us.

Paul's Teensy for example is AVR based but does not use the same core code.
It's implementation is much better and much faster that the stock Arduino core

  • in some cases it can reduce to a single clock cycle for digitalWrite().
    The Arduino team refuses to use it.

There is also the digitalwritefast project:
http://code.google.com/p/digitalwritefast/
(not the way I'd do it, and not the way the Paul did it for Teensy) but it does
already exist today.

The method used in the dwf project gives you very fast i/o operations when all the constants
are known at compile time. This method of optimization is quite simple and a little too simple minded.
While it works great for a sketch where all parameters can be known at compile time,
for a library that allows configuration of pins, this would be of no benefit since it would revert
back to the existing digitalWrite().

What really needs to be done is to throw out the current digitalWrite() implementation
and re-write how digitalWrite() works so it not only does the same things that
dwf is doine but also reduces the number of lookups since much
of the information being looked up is known at compile time.
And that is what Paul's teensy core code does.

If the Arduino team hadn't defined its interface using naked constants (i.e used #defines like DPIN_0 for digital 0 etc
instead of allowing naked constants like 0 or 1 etc)
many other things could be done to speed up the implementation even for pins that are not constants.

BTW, you are looking for additional support of your library, I'd bet you'd be greeted fairly well by
the chipKit and Maple guys as they are more open than Arduino and are also looking to beef up their
library support.

--- bill