Array of Integers Read in Runtime?

Hello all,

I am relatively new to C and coding for Arduino, so please excuse any amateurish mistakes.

I am currently doing a project which is used to move some motors based on data sent to the Arduino from a bluetooth device.

As of now, I have this working sketch which uses an integer array in a way that is similar to keyframes in animation, with each row acting as a keyframe for the position and containing the Δt to the next keyframe.

The tricky part is how I should go about filling this int array during runtime from bluetooth.

I plan to have the loop read the keyframes from bluetooth (serial) continuously and replace the current array of keyframes with the newly read information, until a button on the device is pressed at which time the motors will move according to the latest array. After the move is done, the device would return to the waiting for button state.

Data comes from bluetooth in the form of a string (Serial2.readString()):
{time, pos1, pos 2}, {time, pos1 pos2}, …

At first I was under the impression that I could just set the array of integers to contain this string, which is always formatted exactly like the keyframe information I would put into the sketch pre-compiling. However, after some research and a few failed attempts I found this will not work.

Then, I thought I could simply append the integer array like one appends a string, and add extra rows with a simple command. For this I’d send the data in a bunch of messages from serial instead of just one, in the order of the array. However, I don’t think that is how it works upon further research. I read up on dynamically sized arrays, but people on the Arduino forums seemed to advise against them due to memory issues.

Right now, I am thinking I can read a string from serial with , and } used to separate values, and then write that data into the array as integers, but how I would go about doing the conversion from String to array of Int and how I would approach the array of unknown size (well, known in one axis) without any data until runtime, I do not know.

Is there an approach I haven’t thought of, or a correct way to do this? If so, please let me know :slight_smile:

Attached is a sketch which shows how the code reads the int array for movement

Stage_2_-_Arrays___Coords.ino (1.4 KB)

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

Don't use Serial.readString() as it is a blocking function and will interfere with performance.

It is not clear from this

{time, pos1, pos 2}, {time, pos1 pos2}, ...

whether a single message is "{time, pos1, pos 2}," or whether a single message is longer. Can you provide an example of a single complete message.

...R

Thanks for the link, I’ll be sure to read it over!

Apologies for the unclear explaination of the messages that the device receives.

There are 2 modes for the data being sent:

Mode 1: one long string:
{1000, 0, 0, 0}, {1000, 1024, 1024, 1024}, {100, 1366, 1366, 1366}, {500, 0, 0, 0}, {-1, 0, 0, 0}

Mode 2: individual strings or “keyframes”
Message 1:
{1000, 0, 0, 0},
Message 2:
{1000, 1024, 1024, 1024},
Message 3:
{100, 1366, 1366, 1366},
Message 4:
{500, 0, 0, 0},
Message 5:
{-1, 0, 0, 0}

The reason for the -1 at the end is that -1 duration signifies the end frame, for which the program stops the loop when it’s read.

An example sketch of how the arduino interprets this information is attached to the original post.

There can be an arbitrary number of “keyframes” so the string in mode 1 can contain any number of {time, pos, pos, pos} and mode 2 can send an arbitrary number of messages.

Hopefully this explaination is helpful.

Thank you!!

Have you the option to change the style in which the data is sent?

The reason I ask is that for the long message it would be good to have start- and end-markers. For the short messages the {} and can used.

On the other hand, if you just use {} as start- and end-markers for the long message the Arduino will probably just see it as a series of separate messages which may make the whole process easier.

Separately, why are some of the frames sent individually and some as a group?

How will you let the Arduino know which mode is in operation?

...R

Robin2:
Have you the option to change the style in which the data is sent?

The reason I ask is that for the long message it would be good to have start- and end-markers. For the short messages the {} and can used.

On the other hand, if you just use {} as start- and end-markers for the long message the Arduino will probably just see it as a series of separate messages which may make the whole process easier.

Separately, why are some of the frames sent individually and some as a group?

How will you let the Arduino know which mode is in operation?

…R

As of now, I have the option to send in either way, I can go with whatever works more easily with the Arduino coding. The other side of the project can either have the user set the keyframes and store them all in one string, then send all at once as the big string, or it can send one at a time. Its not set in stone. I can add end markers to it as well, the reason I designed it the way I did was to make the data sent as similar to the data in the array as possible.

I read over the serial communication page, and it was helpful in regards to understanding start and end markers. I’ve somewhat figured out the receiving with end and start markers, using the more proper way from the serial communication post.

What I don’t understand is how to adapt that to an array and string of an unknown length.

Let’s suppose I adapt the string to be of the form <{1000, 0, 0, 0}, {1000, 1024, 1024, 1024}, {100, 1366, 1366, 1366}, {500, 0, 0, 0}, {-1, 0, 0, 0}> with <> as start and end markers

And let’s suppose I have the Arduino read this string

I suppose the final question would be:

How could I parse the integers from this string into an int array, if I do not know exactly how long the array or string will be until runtime?

Thank you!

if I do not know exactly how long the array or string will be until runtime?

You don't need to know that, as long as your program consumes the incoming characters rapidly enough not to exceed the default serial buffer size.

Simply read, decode and act on data chunks as they come in, for example using { and } as the start and end markers.

Is there some compelling reason to send multiple chunks in one long string, like this? To me this just creates additional problems to solve.

{1000, 0, 0, 0}, {1000, 1024, 1024, 1024}, {100, 1366, 1366, 1366}, {500, 0, 0, 0}, {-1, 0, 0, 0}

jremington:
You don't need to know that, as long as your program consumes the incoming characters rapidly enough not to exceed the default serial buffer size.

Simply read, decode and act on data chunks as they come in, for example using { and } as the start and end markers.

The issue is the the program is not acting as the data is received in realtime.

The function of the program is as follows:

Receive information from serial
Store that information
Given a signal (button press), act on the stored information

So the store the information part is the part I am asking about - I am basically trying to treat the array like a spreadsheet or table, where data gets stored and later referenced.

I would ideally do something like this, and this isn't code by any means, just an English explanation:
Read string from serial

Chunk the string up into individual integers, and store them into the array such that they follow the correct order in the array

Stop reading the string when the end marker is hit

Wait for the button to be pressed and move according to the info (this part is all done)

jremington:
Is there some compelling reason to send multiple chunks in one long string, like this? To me this just creates additional problems to solve.

The rationale behind the long string method is for ease of use on the other end. The other side is a mobile app which I can make either send realtime info as information is inputted on the phone or it can be made to store the information and be re-editable and send it all at once as a sort of “move file” to the Arduino. I can make it send the information in either way, but it’s more straight forward on that end if it is all sent as one string.

For the sake of getting the firmware on the Arduino working right, I can make it work either way, and figure out improving it later on after I’ve made the hardware a bit more refined.

So in summary, I prefer using the single string, but if that’s way harder I am okay with sending the data in chunks as well.

I can also start and end the long string with <> as start and end point markers to help the serial reading.

Consider the consequence of any synchronization issues you might experience if bytes are lost during transmission (e.g. low RSSI causes signal interruption.)

You should think about wrapping your data up into a "packet" that contains synchronization, packet info, packet type and even an integrity check which could be in the form of a simple checksum.

For example, your message might be structured something like as follows:

[MSGHDR][MSGTYPE][NUMBYTES][DATA0]...[DATAN][CS]

where:
MSGHDR - a fixed byte that never changes; could be 0xA5 and demarcates the start of a packet
MSGTYPE - you mentioned two "modes"; this byte can tell the receiver what mode this message is
NUMBYTES - the number of bytes to follow in this message, including the CS byte
DATAx - data structured any way you like
CS - a checkum of the entire message; the transmitter computes the CS value so that when all bytes in the message, from MSGHDR to CS inclusive are added together the lower 8 bits of the sum are 0x00.

You could structure the data part of your message as, say, 8 bytes per "quad" -- 4 ints x 2 bytes per int -- and reconstruct them at the receiver. You will be told the number of bytes in the message and thus can know how many quads to expect. As long as the transmitter and receiver use the same algorithm for packing and unpacking the bytes forming each int (e.g. the "endian-ness") you'd be good to go.

You could also add additional message types, in the form of commands or resets or ... This type of structure is really helpful in keeping things aligned, flexible and ensuring message integrity.

why not allocate your array size big enough to handle the biggest set of data you expect to handle?

Then you fill it from item [0] to [n]

Keep n so you know how much of the array is valid.

After the button press, set n back to zero so new data starts filling in from the start again...

ArduinoBoiSashaAyyy:
Let’s suppose I adapt the string to be of the form <{1000, 0, 0, 0}, {1000, 1024, 1024, 1024}, {100, 1366, 1366, 1366}, {500, 0, 0, 0}, {-1, 0, 0, 0}> with <> as start and end markers

On that basis I would simplify the data format to this

<1000, 0, 0, 0} 1000, 1024, 1024, 1024} 100, 1366, 1366, 1366} 500, 0, 0, 0} -1, 0, 0, 0}>

A single character such a } is sufficient to delineate the blocks and it will make parsing easier.

I have some sympathy for @Blackfin’s comments although s/he seems to over complicate the concept IMHO - but maybe that is more to do with the way it is described. Certainly you need more clarity on how the system is to operate, especially if there are multiple modes.

Personally I would try to avoid multiple modes, even at the cost of some redundancy.

If the idea is to send data to the Arduino so that it always has some data “in store” to work with then I reckon the simpler system would be to get the Arduino to request a new packet whenever it has space for it. Such a system would work nicely if single packets are sent because the Arduino could request a single packet or several single packets in quick succession. It also avoids the risk that too many packets are sent.

Equally simple would be a system in which the Arduino uses up all the existing data and then signals that it is ready for another message containing several packets.

I have a system for controlling a small CNC lathe. Messages from the PC are designed to be less than 64 bytes so they fit in the Serial Input Buffer. Before the Arduino starts a movement it signals to the PC to send the next message. By the time the movement is finished the message has arrived and can be read and acted on very quickly (and another message requested to be sent while that move is in progress).

…R

There is no difference between the long string and the short string. If you leave the commas out between the individual 'records', a long string is the same as N short strings.

In that case you can simply use Robin's 'receive with start and end marker'

{1000, 0, 0, 0}{1000, 1024, 1024, 1024}{100, 1366, 1366, 1366}{500, 0, 0, 0}{-1, 0, 0, 0}

It will give you back the first set first that you can save, next the second set that you can save etc.

sterretje:
There is no difference between the long string and the short string. If you leave the commas out between the individual 'records', a long string is the same as N short strings.

I did refer to that in Reply #3, but perhaps not clearly enough. And my code would just ignore the commas between the records.

On the other hand, the OP has not explained how the different message types should be handled and IMHO that is essential for an effective design. And it may show to him as well as us that two modes are not needed.

...R

Robin2:
I did refer to that in Reply #3, but perhaps not clearly enough. And my code would just ignore the commas between the records.

Apologies, must have missed that or misunderstood.

But now that approach has 2 votes :wink:

sterretje:
Apologies, must have missed that or misunderstood.

Nothing to apologise for. If you missed it then it is even more likely that the OP did. I should have written it more clearly.

But now that approach has 2 votes :wink:

At this stage I am only giving it a half-vote because I don't know how the OP plans to store the data when received. :slight_smile:

...R

Thank you all for all the input on this issue, I’ve learned a lot and it’s greatly appreciated.

Now, to address some of the ambiguity from the original post:

Robin2:
I don’t know how the OP plans to store the data when received. :slight_smile:
…R

I’m sorry, I should have included a better description of exactly what the device is to do, so here’s my best attempt at a short summary:

Sender-side: The user inputs position and time “keyframes” to a simple app that can be made to work in 2 different ways: 1: The user makes a keyframe and sends it, sending these one by one and putting them into the array. 2: The phone stores the keyframes and puts them all into one string, I prefer this method because it lets the user edit previously made keyframes before sending them over, and lets them save the move phone-side so it can be repeated later.

Receiver side: The Arduino initializes the motors and serial ports, and then it waits for the paired bluetooth device to send it keyframes. These are stored in the array in the code I have attached to the original post, and when a “go button” is pushed on the device, the data stops being overwritten and the move begins, when it’s finished the devices goes back to listening for new information.

What happens when new information comes in?
This was largely planned around method 2, with one long string, so the way I had envisioned the program running is that the received string gets translated to an int array and stored in MoveMatrix (see the attached .ino on the original post), and every new string that’s received overrides the previous MoveMatrix completely.

Robin2:
On the other hand, the OP has not explained how the different message types should be handled and IMHO that is essential for an effective design. And it may show to him as well as us that two modes are not needed.

…R

Sorry if I caused misunderstanding with the two message types, I mean that I can make the sender-side send the data in 2 different ways, but I have control over that and I only need one method to work. I mention the two ways because I don’t know which one will be more effective when it comes to coding the Arduino side of the project. By all means, if the multiple messages way or the single long message way is better, I will get rid of the other one, I don’t intend to concurrently use both in the final program.

My idea for the multimessage one was to “append” the array with each new string, and signalling the Arduino to stop appending and clear the matrix when the final frame is sent (time = -1). This is purely because I am inexperienced with serial communication and I don’t know which method is better.

sterretje:
There is no difference between the long string and the short string. If you leave the commas out between the individual ‘records’, a long string is the same as N short strings.

In that case you can simply use Robin’s ‘receive with start and end marker’

{1000, 0, 0, 0}{1000, 1024, 1024, 1024}{100, 1366, 1366, 1366}{500, 0, 0, 0}{-1, 0, 0, 0}

It will give you back the first set first that you can save, next the second set that you can save etc.

Again, I mention the two methods because I don’t know which one will work best, not because I will use both concurrently. Sorry about the misunderstanding. I think that using Robin’s “receive with start and end marker” is probably the best solution for the receiving of the data after reading over the “serial communication basics”, but where I am lost is how to convert the received string of unknown length into an int array of unknown height.

Robin2:
On that basis I would simplify the data format to this

<1000, 0, 0, 0} 1000, 1024, 1024, 1024} 100, 1366, 1366, 1366} 500, 0, 0, 0} -1, 0, 0, 0}>

Personally I would try to avoid multiple modes, even at the cost of some redundancy.

If the idea is to send data to the Arduino so that it always has some data “in store” to work with then I reckon the simpler system would be to get the Arduino to request a new packet whenever it has space for it. Such a system would work nicely if single packets are sent because the Arduino could request a single packet or several single packets in quick succession. It also avoids the risk that too many packets are sent.

Equally simple would be a system in which the Arduino uses up all the existing data and then signals that it is ready for another message containing several packets.

I have a system for controlling a small CNC lathe. Messages from the PC are designed to be less than 64 bytes so they fit in the Serial Input Buffer. Before the Arduino starts a movement it signals to the PC to send the next message. By the time the movement is finished the message has arrived and can be read and acted on very quickly (and another message requested to be sent while that move is in progress).

…R

The system I am making has some similarity to your CNC lathe, but I think it’s simpler in that it doesn’t simultaneously move and communicate, rather it is sent a list of complete coordinates and moves according to the stored list. There are 2 states - receiving and moving, and they do not occur simultaneously.

DaleSchultz:
why not allocate your array size big enough to handle the biggest set of data you expect to handle?

Then you fill it from item [0] to [n]

Keep n so you know how much of the array is valid.

After the button press, set n back to zero so new data starts filling in from the start again…

I was thinking of that as well, but the reason I don’t want to go with that is that I don’t see it as the most elegant possible solution because it will waste memory on storing an array that’s empty a lot of the time. It can work for sure, but this system is likely to sometimes just have 2 coordinates, and sometimes 30+, this is because I am using these keyframes for what I call “ghetto acceleration” where I will cluster a bunch of keyframes with the same change in position, with progressively shorter or longer time in between to get the device to controllably accelerate and decelerate. I was also considering using malloc(), but I’ve read on the Arudino forum that it does not work well for Arduinos due to memory allocation issues.

Blackfin:
You should think about wrapping your data up into a “packet” that contains synchronization, packet info, packet type and even an integrity check which could be in the form of a simple checksum.

For example, your message might be structured something like as follows:

[MSGHDR][MSGTYPE][NUMBYTES][DATA0]...[DATAN][CS]

That’s definitely an elegant solution for the data transmission, and it’s almost certainly the most reliable and complete solution, however for testing purposes I think I’ll start with something a bit more rudimentary. That’s more of something I’d want to add after I get the communication working and established in a functional way. The question I have right now is about the process of actually converting the string from serial into the int array, with the catch that I don’t know how long the string and array will be. Thanks for the input, much appreciated!

Sorry about the long post, there was a lot to respond to. Hopefully there were no important details I left out.

TL:DR:

The two modes are not going to be used concurrently, I mention them because I have the option to use whichever mode is easiest.

The device stores the data from the string until a new string is received, at which it’s overwritten, and it will not listen for new info while it’s moving.

Thank you all for the input, it’s been extremely informative and helpful!!

Your Reply #15 is long (but thorough) so I apologise in advance if I have missed something.

My take on the situation is as follows …

On the Arduino you need to allocate space to store the data items from the largest message you plan to send (whether it is sent all in one piece or otherwise).

I think my strategy would be to send a message with a variable number of parts (not exceeding the max you have allocated space for. If you send data like this (as posted earlier)

<1000, 0, 0, 0} 1000, 1024, 1024, 1024} 100, 1366, 1366, 1366} 500, 0, 0, 0} -1, 0, 0, 0}>

then you can have different number of groups. You just need to parse the data in two phases - first split on the } and within that group split on the comma. As you discover an extra group (when parsing on the { you can increment the outer index for your 2D data array.

With this system it is probably not necessary to include the number of parts in the message but you should also have a variable that keeps count of how many groups have been received so that if you received 5 the last time and only 3 this time you would not accidentally interpret items 4 and 5 a second time.

If every group will have 4 data items then another approach is to send the data like this

<5, 1000, 0, 0, 0, 1000, 1024, 1024, 1024, 100, 1366, 1366, 1366, 500, 0, 0, 0, -1, 0, 0, 0>

where the first value is the number of items and then the Arduino knows to interpret the remainder as 5 groups of 4

Yet another approach is always to send the maximum number of groups with the “unwanted” groups just containing 0s. That has the benefit of always over-writing all of the data elements.

And, of course, it is possible to send the data in binary form and just copy it straight into an array of ints. But that makes debugging harder and I would only do it if it was essential for performance reasons.

…R

Have you mentioned the maximum number of 'records' you want to store? And what board you are using?

What are the maximum numbers in each of the fields of a record; are they basically all (unsigned) integers (0..65535) or do some fit in a byte as well (0..255).

Where to store depends on that. On a 328P you might be able to store 100 'records' in RAM (depending on the rest of the code), on a 2650 based board it will be more.

Other option is internal eeprom; be aware that each cell in an eeprom can only be written 100,000 times so if you update often, you will have a problem sooner or later.

Last option is to store received records in external storage; external eeprom (limited write-cycles like internal eeprom), FRAM (has basically unlimited write cycles) or SD card (limited write-cycles, not sure how much; big storage capacity).

Robin2:
Your Reply #15 is long (but thorough) so I apologise in advance if I have missed something.

My take on the situation is as follows …

On the Arduino you need to allocate space to store the data items from the largest message you plan to send (whether it is sent all in one piece or otherwise).

I think my strategy would be to send a message with a variable number of parts (not exceeding the max you have allocated space for. If you send data like this (as posted earlier)

<1000, 0, 0, 0} 1000, 1024, 1024, 1024} 100, 1366, 1366, 1366} 500, 0, 0, 0} -1, 0, 0, 0}>

then you can have different number of groups. You just need to parse the data in two phases - first split on the } and within that group split on the comma. As you discover an extra group (when parsing on the { you can increment the outer index for your 2D data array.

With this system it is probably not necessary to include the number of parts in the message but you should also have a variable that keeps count of how many groups have been received so that if you received 5 the last time and only 3 this time you would not accidentally interpret items 4 and 5 a second time.

If every group will have 4 data items then another approach is to send the data like this

<5, 1000, 0, 0, 0, 1000, 1024, 1024, 1024, 100, 1366, 1366, 1366, 500, 0, 0, 0, -1, 0, 0, 0>

where the first value is the number of items and then the Arduino knows to interpret the remainder as 5 groups of 4

Yet another approach is always to send the maximum number of groups with the “unwanted” groups just containing 0s. That has the benefit of always over-writing all of the data elements.

And, of course, it is possible to send the data in binary form and just copy it straight into an array of ints. But that makes debugging harder and I would only do it if it was essential for performance reasons.

…R

sterretje:
Have you mentioned the maximum number of ‘records’ you want to store? And what board you are using?

What are the maximum numbers in each of the fields of a record; are they basically all (unsigned) integers (0…65535) or do some fit in a byte as well (0…255).

Where to store depends on that. On a 328P you might be able to store 100 ‘records’ in RAM (depending on the rest of the code), on a 2650 based board it will be more.

Other option is internal eeprom; be aware that each cell in an eeprom can only be written 100,000 times so if you update often, you will have a problem sooner or later.

Last option is to store received records in external storage; external eeprom (limited write-cycles like internal eeprom), FRAM (has basically unlimited write cycles) or SD card (limited write-cycles, not sure how much; big storage capacity).

Apologies for the lack of activity in the last few days, I’ve been trying to get the program running based on previous feedback. I have gotten it to a state that reads and processes the string how I want, converting it correctly to a 2d int array of variable size. The problem I have run into now is that there is no way to return this array from within a function.

Here’s a link to my project’s current Arduino code. I don’t know if it’s customary to use Github here, but I don’t see a way to attach a file to a reply so I hope this is a suitable alternative.

What works:
Reading the string consistently, without data loss, and reliably
Converting a properly formatted string to an int array
Moving the motors given an int array

What doesn’t work:
When the array is declared in the while statement in loop as
Void Loop(){ while(condition){ make array } if(button) {read array}}
Anything outside of the while statement will read the globally declared blank array.

The reason I put everything in loop, instead of organizing the app into functions, is that I cannot return the array, my thinking was that within void loop, any variables I set will be consistent, regardless of whether they are right in the loop or nested within an if or while statement. It seems I was wrong.

Now my question is - and I am not sure if this will warrant another post or not - how can I “return” an array from a function in C? I read online that I can use pointers, but I can’t find a clear source of documentation on pointers as it pertains to coding for Arduino, or any clear info for that matter. Is there a way to do it without using pointers?

sterretje:
Have you mentioned the maximum number of 'records' you want to store? And what board you are using?

What are the maximum numbers in each of the fields of a record; are they basically all (unsigned) integers (0..65535) or do some fit in a byte as well (0..255).

Where to store depends on that. On a 328P you might be able to store 100 'records' in RAM (depending on the rest of the code), on a 2650 based board it will be more.

Other option is internal eeprom; be aware that each cell in an eeprom can only be written 100,000 times so if you update often, you will have a problem sooner or later.

Last option is to store received records in external storage; external eeprom (limited write-cycles like internal eeprom), FRAM (has basically unlimited write cycles) or SD card (limited write-cycles, not sure how much; big storage capacity).

The number of records will vary during use, the smallest being a 2x2 array for one motor moving between 2 positions and the largest is realistically a 6 (3 axes of movement + 3 lens controls) x 100 (a large and complex move with accelerations). The "X" axis will be known for both instances, and be changed before compiling. The y axis will be unknown until runtime. In the latest version of the code, I made a blank array and filled it, with the catch that when I declared it the second time to override the previous one, I set the height of the array as an integer instead of being unbounded like the first time it was set (see GitHub link in reply #18).

The board I am using is the OpenCM 9.04 from Robotis, it's an Arduino compatible board running on ARM. I don't think I will need an SD card or any external storage for the dataset.

Thank you for your input!