Sending large array between 2 arduinos

Using Arduino Project Guidance
1

Hi all!

I'm currently working on a project that requires an arduino based system to communicate with slave units that will each have another arduino in them.

I'm using two arduinos to host a live web-page that updates the status of the slave units online. One arduino polls the slave units whilst the other hosts the web page to allow for a reliable connection.

The problem I have is passing 2 arrays between the 2 arduinos from the poll uit to the webpage unit. the first array (state[33]) contains the states of all slaves being 0 for offline A for available and B for busy. The second array contains the student id of the person using the slave units (stu_id[33][6]) a 6 number unique ID and one for each machine. Both of these arrays need to pass from poll unit to web page unit using an interrupt what would be the best way of doing this? Ive tried the soft easy transfer library as well as trying to write my own protocol but without much luck. Ive attached both poll and web unit code for reference,

Any help is appreciated,
Thanks for your time

Andy :slight_smile:

POLL_SERVER.ino (4.5 KB)

ETH_SERVER.ino (5.69 KB)

2

The second array contains the student id of the person using the slave units (stu_id[33][6]) a 6 number unique ID and one for each machine.

Storing the ID as a string is a waste. Store the data as a long. Then, you have an array of 33 elements, not an array of 198 elements.

Both of these arrays need to pass from poll unit to web page unit using an interrupt what would be the best way of doing this?

Why "using an interrupt"?

3

Thanks for the quick reply,

Good point about the ID will change that to long int, it was done as a string because of the ID's requiring leading zeros but that can easily be dealt with. The main problem with using long int is how i go about reading the data out of the serial buffer at the other end as serial.read() only takes 8 bytes at a time

As for the interrupt, the polling unit needs to continually communicate with the slave units to get real time updates therefore the interrupt means the poll unit only has to stop communication with the slaves when requested by the webserver unit

4

I will be polite and say this is most unsuitable

void interruptRoutine()
 {

digitalWrite(gnLed, HIGH);
digitalWrite(rdLed, LOW);

 ET.sendData();

 delay(5000);

}

You can't send data from within an ISR because interrupts are off AND because you don't want to spend that length of time in the ISR

You do not want any delay() in an ISR. You should plan for your ISR being as short as possible and not more than about 100 microsecs.

Apart from these technicalities an Interrupt is intended for something that is very brief or requires microsecond precision. If you are measuring time in millisecs there is no need for an interrupt.

I would also like to develop what @PaulS has said by suggesting that you devise a communication protocol that minimizes the amount of data you need to send. The state of 8 devices can be recorded in a single byte - which means 5 bytes for all 33 devices. That has left 7 bytes unused and they could be used to record the ID of 127 different students.

However if you have the time to "waste" a byte on each device it would make the programming and debugging easier. Why not use a much higher baud rate. And why not incorporate the userID and the on/off status in a single byte ? For example 'a' means off and 'A' means on for user A.

Have a look at the examples in serial input basics. They are simple, reliable, non-blocking and don't need any ISR. It is a good idea to design the receiving system first and create a sending system that complies with it.

...R

5

Hi,

I am aware thats an unsuitable solution, I should have made clear Ive tried several methods to pass the arrays between the two arduinos, that just happened to be the latest thing I tried,

I understand what youre saying about using a byte for the machine status however part of the specification is the slave units have 4 states, online, available, busy, offline. This can be narrowed to 3 states as available and busy imply the machine is on-line however this is not a binary relationship so cant be expressed in 1's and 0's hence the use of chars.

The interrupt documentation also makes clear that receiving serial data is not possible during the interrupt as receiving serial data relies on an interrupt itself. However its fully possible to write serial data during an interrupt as this does not rely on interrupts. The difficulty I have had is on the receiving end as when the received serial data is printed directly to serial monitor it appears just as it was sent with no issue. The problem I have is using control characters such as STX, ETX, EOT to separate the 2 arrays such that the state data is stored in one array and stu_id is stored in another.

The smallest I can see I could get the sent 'message' would be:

Using byte / char for state = single byte per slave = 32 bytes
Using long int for stu_id = 4 bytes per slave = 128 bytes

giving total 160 bytes

I understand how the serial read functions work as Ive said the library I used was just a ditch attempt at getting the thing to work. Ive attached some previous code I wrote that uses the normal serial read write

Due to the specification I have to use an interrupt routine as the poll system must continually be polling. The system works in this way:

WEB SERVER:

Look for Ethernet clients
If client - serve page
If more that 10 seconds elapsed - Get update from poll system

POLL SYSTEM:

Incrementally get states of each slave unit storing to array
On interrupt serial write data to software serial port

ETH_SERVER.ino (5.44 KB)

POLL_SERVER.ino (4.65 KB)

6

andyowenwest:
However its fully possible to write serial data during an interrupt as this does not rely on interrupts.

Have you tested this? It will not work and sending this amount of data will block it in a very bad way - the serial buffer will fill and it will wait for the interrupt signalling the next character can be sent but that will never arrive when interrupts are off and then it waits forever for the buffer to empty.

I don't think you need highly compressed data. You aren't trying to send 100 updates per second. Spread the data out exactly as you planned with characters 'A', 'B' indicating whatever you want. Student IDs with leading zeros definitely must be strings, not integers.

So what is hard about picking a start and end marker for your blocks of data? If "ZZZ" can never appear in your real data, then use "ZZZ" as the start marker for one of your data blocks. If the blocks are fixed-length then you can just fill up your receiving buffer by counting characters. It is also going to be a good idea to put a checksum on the end of each block. Serial data can easily get corrupted and it only takes one incorrect bit to turn an 'A' into a 'B'. With a checksum, you only release the data from the receive buffer after the checksum is checked, that way you don't corrupt the previously-good data.

7

The 4 (or 3) states of a unit can be stored in 2 bits: 00, 01, 10, 11.

Assuming the id is always a positive number, and its max value is 999999, it can be stored in 20 bits.

So with some bit shifting, you can store state and id of a unit in 22 bits.

8

Thanks again for all your help,

I definitely think using the 2 bits to describe the states is worth using as i can get all 32 units in 8 bytes

And I think you're completely right about the serial blocking as in my tests the more data I send the more scrambling I get

I think my hurdle with the stat end end markers was trying to use them within the ISR, I think this may be a better approach from your comments:

POLL UNIT:

Incrementally poll slave units & get state + student id's
When data received from slave units compress into 2bit per unit and long ints for student id's
When interrupted set a boolean to mark and interrupt has occurred
Resume loop function (exit ISR)
If boolean is set listen for send command from web server arduino (timeout will be used just incase)
If send command is found calculate checksum send 4 packets then wait for another send command (and so on until all sent)
When all packets sent reset boolean resume polling

WEB UNIT:

Listen for Ethernet clients
If more than ten seconds elapsed and no clients interrupt poll unit
Send interrupt and set boolean to mark interrupt
Resume loop function (exit ISR)
If boolean is set, send command to Poll unit to commence data transmission
listen for 4 packets and calculate checksum when received
If checksum matches send command to poll unit for next for and so on until delivered
When all packets sent reset boolean and continue polling

I don't think a resend protocol is needed as the interrupt is every ten seconds and so incorrect packets could just be discarded

I think this seems a more sensible way round doing this as I'm not relying on the ISR to deliver the serial data preventing all the horrible scrambling problems plus i have the ability to use flow control :slight_smile:

Ill post the code and results tomorrow one the I've written and tested it on the hardware at work

Please let me know if i'm on the right track now or any other improvements to this i can make :slight_smile:

thanks again for your help

Andy

9

I don't have a clear image in my mind of the relationships between your units. You describe a Poll Unit and a Web Unit. But your Poll Unit seems to be getting data from a 3rd type - a Slave Unit.

What is triggering your interrupt?
As I said earlier, I can't see any need for any interrupts as nothing seems to require instant attention. Using interrupts always makes code harder to debug.

You might look at how millis() is used to manage timing without blocking in several things at a time

...R

10

The slave units are arduinos connected to the logic board of a washing machine and a RFID tag reader. The washing machines are to be fitted in public areas such as universities so the system works like this:

If washing machine is turned on and no one using it report via an xbee radio its available
If washing machine is in use report via xbee it is busy and the student number or the RFID tag using it
If a washing machine is turned off nothing will be reported and so it will be assumed offline

The specification says the system is to support up to 32 machines from one web server

The hardware consists of only arduino uno's

the poll unit is an arduino with an xbee attached that receives the data from the slave units by incrementally and continually sending a command to each unit numbered 1 through 32 so the status is updated in real time

the web server unit then displays a HTML table of the machine address, current status and if busy the student id of the person using the machine

the web server unit is also to write the date time and student number of a machine when it changes state to busy to record who is using it

so the interrupt is such that the web server can listen for incoming clients and a timer can be used to get the data from the poll unit every ten or so seconds as its not possible for a single Uno to poll the slave units and serve the webpage and get an NTP time update all at the same time. Therefore the interrupt is just to trigger the polling unit to stop polling and give the poll data to the webserver so the slave units can be continually polled

So in short the interrupt is triggered by the web unit when it wants an update from the poll unit

hopefully that explains it a bit more :slight_smile:

11

Sounds like a complex project with a lot of separate parts. Which of the project parts do you have working at this point?

12

Im actually working on version 2 of the project as i have an existing system working, it was just that i was using a single arduino for the web server and poll unit but the polling of the slave units is only reliable when continually being polled hence using one arduino for polling and one for hosing the web server :slight_smile:

So other than sending the arrays between the 2 (which Im working on again today) it is all working fine

13

Now that you have described the project more clearly it should be simple to implement. It is conceptually very similar to the system I am using for radio control of model trains.

It may be worth considering using a Yun as the master as it has Wifi built in and all the web code can be done on the Linux side using Python.

In my system all of the slaves are listening and receive all messages but a specific slave only reacts to a message that contains its own ID number. When it detects its ID it first sends a reply to the master and then implements whatever it was told to do. The reply to be sent to the master has already been composed before it gets its slot to send.

The master just waits a short time for the reply from a slave. If it gets no reply it assumes the slave is offline. After the reply, or the short wait, the master sends a signal to the next slave.

No part of my system requires interrupts.

I wonder if you were to use ESP8266 WiFi modules in place of the Xbees maybe the entire master system could work from Linux - perhaps on a RaspberryPi which would be cheaper and more capable. But that would require a reversal of the logic - the slaves (clients) would have to poll the master (server) at intervals.

...R

14

You're right both systems do sound quite similar but unfortunately i'm restricted by hardware, I'm a junior engineer working for a small washing machine company and so I've been given a specification of the hardware I have to used so I'm restricted in that sense but otherwise WIFI modules would make the entire project easier however in my case the web server is connected through a 3G modem :slight_smile:

Have managed to play around with a code and even ended up removing the interrupt which made the entire data transmission 90% easier!!

Ill attach the new modified code here for reference for anyone else with the same problem (although i only had time to program the code to send one array the same method should be able to be used for multiple arrays)

I also need to sort out packeting when i get time but for now Id like to thank you all for your help, thanks to your suggestions I'm fairly confident in the direction I need to go in :slight_smile:

ETH_SERVER.ino (5.67 KB)

POLL_SERVER.ino (4.06 KB)

15