CO-AUTHOR: non-blocking Async modem class - COMPLEX.

Hi all,
I should be doing this myself, but events have recently left me at a disadvantage to tackle this alone.
I'm willing to help with some payment (I'm broke, but this is important to me), and when sufficient interest and icommitment is shown, I could provide the hardware to test & finalise the code. I'm located in Melbourne, Australia
Ultimately, I think this would be a really worthwhile contribution to the C/C++ enthusiast community.

WHAT I WANT TO CREATE
An async modem class that runs 'in the background' of watever primary app is being developed.
Primarily to manage SMS text traffic withou blocking the 'foreground', and other modem housekeeping tasks needed.

Sometime after the initial solution has been developed, it would be the right platform to extend into GPRS/data functions as well.

At the current time, I use a modified Adafruit FONA library - with a simple state-machine wrapped around the various functions needed to perform the required sequences without blocking. Modems in use at the moment are SIMCOM 5320 and 7600 families, but the command set is 'almost' standard across other brands & models..

Most modem AT commands take a finite time to respond (up to a couple of seconds max.).
Ideally, all these will be implemented as state-machines so the ‘execution’ command can return quickly, and a call-back function will notify the host application when that function is complete. A frequently called ‘ticker’ function will be used to check & update the state sequencing.

What it does...

  • Power on / shutdown / off
  • Connect & Report connection status
  • Queueing of messages (and commands / replies ?)

I use a FIFO stack to queue multiple SMS text messages, so messages can be pushed to the stack in real-time to release the caller, and the (ticker) state machine checks that stack to pull the top-most message and send/retry when the modem is free.
After retries or repeated failures, messages are popped off the queue, and nottified to the caller.

  • Checking the clock and RSSI have smaller, simpler mechanisms to return quickly
  • Ideally all the functions will work in the same way through a consistent interface
  • but CAUTION…*
    Some messages arrive asynchronously (e.g. +CMT notifications for incoming SMS messages)
    These should be queued for the host program to pull when it's free.

Of course, this will be a struggle to run in the limited RAM of a 328P (that's the biggest challenge - use of RAM for the FIFO and buffers, and should fit quite well in a MEGA... My personal choice for 'Arduino' 8-bitters are MEGA1284P based.
PM if you're interested and have strong experience in library / class development.

-bump-
Helpers wanted !

Hi

I'm not sure I perceive the exact need for a class ? As you describe it, a simple state-machine wrapped around the FONA library is not that complex to craft.

what do you have in mind ?

Thanks for the question...
The FONA library doesn’t do a lot of things, and is quite clumsily written, I deleted maybe 20% of it to get it trimmed down)

The state machine wasn’t that hard (it was fiddly), but it would be nice to pack it into a single library that updates itself asynchronously on a ticker heartbeat, without blocking the ‘foreground’ tasks.

Particularly sending SMS, takes up to a couple of seconds per ‘part’, I often send several messages at a time to different recipients, but don’t want to hold up the primary tasks of the program... I also handle queueing, which would be nice inside a wrapper that handles the messy stuff.

Even simpler functions like +CCLK, and +CSQ take a modest period to complete.

It would also be nice if incoming SMS were parsed in the library (along with other messages) and also buffered in a queue, then delivered to a callback function, rather than having to poll/checkin the main program - 99% of the time for nothing.

As you’ve probably guessed, my current project has a lot going on, while modem tasks are relatively slow.
(multiple 20mS events running continuously uptime well over a year.)

Hey

sorry for not answering earlier. don't have much time at the moment

Attached (too big to fit on screen) is a something I had been working on in the past

Indeed Your explanation reminded me that I had somewhere an async Stream communication class (templated) that I had developed for handling multiple Serial communications on a MEGA (send a command, expect an answer ending with an arbitrary key phrase - ie an endMarker that is a cString, not just one char).

It's probably not super clean (was not meant to be published and I needed something quick) but should work with anything that inherits from the Stream class as long as you can concurrently read and write from that stream and others (if you have multiple). I used it with a MEGA and Hardware Serial. SoftwareSerial would be an issue as you can't listen from many Streams at the same time.

This is how it works:

You instantiate the StreamQueue class through a template by indicating how long the queue needs to be (max numbers of pending commands) and what's the max size for the answer. Then you provide the Stream you want to be attached to.

For example

StreamQueue<3, 20> serialQueue(Serial); // on Serial port, 3 commands max in queue, response buffer has 20 bytes max (+ a trailing NULL). 
StreamQueue<10, 60> serial1Queue(Serial1); // on Serial1, 10 commands max, response buffer has 60 bytes max (+ a trailing NULL)

serialQueue could be used for end user terminal menu GUI for example whilst serial1Queue could be used for AT commands with a SIMxxx modem

a Command is a structure:

struct t_AsyncCommand
{
  t_commandID    _commandID;  // your unique ID for this command, used in callBack
  const char*   _command;     // the text of the command. needs to be persistant text or NULL if just waiting for an answer
  const char*   _endMarker;   // the text of the end Marker. needs to be persistant text or NULL if just sending a command not waiting any anwser
  uint32_t      _maxWait;     // the max delay you are willing to wait before you timeOut on this command
  t_callback    _callback;    // a pointer to the function you want to execute after the command is complete (either got the end marker or timed out or was just sent if no answer was needed)
};

For example for a ESP-01 initialization at 9600 bauds you could do something like (typed here so for illustration not sure it's a fully correct sequence)

t_AsyncCommand espCommands[] = { // {commandID, command, endMarker, maxWait, callBack}
  {101, "AT", "OK", 1000, okCallback1},          // AT requires an OK answer, 
  {102, "AT+RESTORE", NULL, 5000, NULL},       // reset to factory default
  {103, "AT", "OK", 1000, okCallback2},          // AT requires an OK answer
  {104, "AT+UART_DEF=9600,8,1,0,0", "OK", 5000, swicthBaudsCallBack}, 
};
const uint8_t nbCommands = sizeof(espCommands) / sizeof(espCommands[0]);

each command has

  • an ID (101, 102, 103, ...) you arbitrarily select (passed in the callback)
  • the text of the command you want to send
  • the answer you want to wait for
  • how long you are willing to wait before timeout
  • the callback function you want to call upon completion

you can make your magic happen in the callbacks through a state machine. it's up to you there.
The callback can be NULL if you don't have anything to do (for example when I sent the restore)
but ideally you would want to trap timeout at least.

To add a command to the queue there is a registerAsyncCommand() method, for example if you want to load the 4 commands we defined above, you would do:

 for (uint8_t i = 0; i < nbCommands; i++)
    serial1Queue.registerAsyncCommand(&(espCommands[i]));

Then in the loop you need to ping the StreamQueue to check for update (It would not make much sense to have that in a timer ISR as most Streams depends on interruptions being available to receive data).

void loop()
{
  serialQueue.updateQueue();  // ping each of your StreamQueues
  serial1Queue.updateQueue(); // ping each of your StreamQueues
}

there is one extra feature, when you doStreamQueue<10, 100>  serialQueue(Serial);
you are using the default end of line validation so every command will be sent with a trailing CR+LF and every end Marker will be expected to have also CR+LF. (most common way with AT commands and UI stuff)

The constructor for the StreamQueue class actually takes a second parameter if you want something differentStreamQueue(Stream& commChannel, uint8_t tail = asyncCRLF_TAIL)

[b]tail[/b] comes from an enumenum : uint8_t {asyncNO_TAIL = 0b00, asyncLF_TAIL = 0b01, asyncCR_TAIL = 0b10,  asyncCRLF_TAIL = 0b11};
where you have the choice of NO_TAIL (then you have to provide your own exact characters in your commands and markers), only LF, only CR or both CR+LF (which is the default).

The Stream is under the responsibility of the caller, so your main code has to ensure it exists before attaching it to a StreamQueue and of course open it at the right speed before asking the commands to be sent.

Between two consecutive commands there is a bit of code that will try to empty the Stream input (whatever is there) so that we try as much as we can to start a command with a fresh empty incoming buffer. This is not perfect as you can't really predict when things arrive in an async protocol but proved to be good enough for my use (in case of timeouts or buffer overflows)

You need to install the StreamQueue.h (which embeds the full code...) next to your Sketch and start your .ino with#include "StreamQueue.h"

This is not great but if I remember well I faced some challenges with the IDE trying to second guess what needed to happen at compile time when using templates in a .h and .cpp and I was ending up with cryptic double definitions of functions.

I can't say it was heavily debugged nor memory optimized, if you or anyone in the forum want to have a look and explore - I'm happy to provide comments on the ideas behind the code.

it probably deserve a full rewrite though :slight_smile:


the example attached runs a simple quizz test on Serial, open the console at 115200 bauds with CR+LF as validation line. you should see the commands being sent in sequence and you can type your answer in the Serial monitor. The call back function verifies if your input corresponds to the right answer of the quizz.

after you completed the quizz, the callback reload the queue with one new question to see if you want to play again. that will call a different callback, if you answer 'Y' it will reload the (same) set of commands to run the quizz again.

that should help demonstrate how things work. You can let the answer timeout or answer something wrong as well as this is handled in the callback.

AsyncCommandQuizz.zip (8.08 KB)

I must admit I'm disappointed that there is zero download 10 days later. I guess that you went for your own exploration.

Not at all, The current global calamity has me distracted, and my own medical circumstances tie into that to some extent...

I was just thinking exactly that.. I should grab it before something unexpected happens !

Sorry for scaring you ! :slight_smile:

good to hear you are safe !

Yeah. Hope you all stay safe.

I've found that this has provided me with some time to focus on a number of my home projects. Been Zooming with the kids too.

J-M-L:
I must admit I'm disappointed that there is zero download 10 days later.
I guess that you went for your own exploration.

Explained - I got it, will take a bit to digest and aply your examples.
Thanks