ArduinoBLE scan() will eventually fail to yield any new events

This post identifies a fundamental issue with the ArduinoBLE library when attempting to run an indefinite scan(). It relates to an earlier post on 8th June 'No longer receives connections after a while', but I've started a new topic as the earlier one was simply a manifestation of the problem.

If you implement a solution which runs scan() indefinitely, filtered or unfiltered, accessing events using either poll() or available(), then there will come a time after which no further advertising events are ever returned and the application is effectively dead. To start receiving again, it's necessary to stop and re-start the scan.

The reason runs as follows: Some BLE advertising events are introductory (eg types 0 and 2), and one marks completion (type 4). The ArduinoBLE library remembers introductory event objects identified by address in an internal list: it does this to build in additional data before passing the object to the application. An introductory event adds the event object to the list, and a type 4 event removes it again. The problem is that if you fail to get a type 4 event for a previous type 2, then then the related object stays in the list. The list finite and is configured to be 5 long, so after 5 event objects with missing type 4's the list gets full: the code then rejects any future events (strictly except type 3, which don't go via the list at all) and that's it until you restart the scan. I have observed this in practice, where after 5 missed type4 events which are of no interest to me, the kind of event that I'm specifically looking for stops coming through.

I don't follow the intricacies of BLE at all, so I don't know how common lost packets are, but any approach which relies on receiving specific events like this is doomed, because you can never absolutely guarantee that a message will get through. In my case, I've identified the Android/Apple message sent as part of Covid tracking as being a particular source of grief. This changes its address every 15 minutes; if I have the BLE device near the phone, I see matched type2/type4 packets streaming continuously. If I move the phone away, I observe the number of events drop, and in particular there are often type 2 events without a matching type 4. If I go far enough away, there are obviously no events at all. This variation by circumstance and proximity makes for a problem which is hard to track down. [Although the Covid data is the worst case, it is not the only one, but that simply prolongs the time before things fail].

The ArduinoBLE library interface is effectively undocumented, so there's little to say about how it should behave in practice. If I'm right that there should be no impediment to running a single scan() indefinitely, and if I'm right that the reason for the maintained list is to enable the event to be augmented until it is complete, then I don't see a rigorous solution without adding more interface functionality to allow control over the list behaviour. For my own requirements I can hack GAP.cpp to either not use the list, or to skip events which aren't of interest before adding them to the list, but neither would be a general solution which could be applied to the canonical source.

So I'm interested to know what others with more experience of BLE and the Arduino library think. Is my understanding in the previous paragraph correct? Is what I'm trying to do (simply take action whenever a specific device sends a message) an unreasonable application? Is it unreasonable to seek a solution which doesn't contain a fundamental gotcha of this sort, and if so what might be the best approach?

Finally, a separate but related question: does anyone know how the BLE chip itself deals with culling duplicates, since it presumably has the same problem as above?

The BLE peripherals I’m using - W4 Beacons - use the advertisement to publish the accelerometer data. So, it changes over time.

The library doesn’t deal with this. I’ve updated it so that it does, and once I’m happy it works efficiently etc, I’ll submit a PR.

Ian