How to optimize my code for future use?

Hi All,

In 2010 I saw several projects on using arduino’s to monitor a garden. As an excercise I developed a similar project to monitor and control a greenhouse and send data wirelessly using Xbee to another xbee-equipped arduino, which connected to the internet and put the received data in a database.

That worked fine, until the box which I thought was waterproof, turned out not to be waterproof. Fried electronics and demotivated I threw the project aside.

Now, I want to pick it up again and change and expand it, but I am running into issues with managing my code. For example, for whatever reason I cannot seem to get more than 35 or so characters in an Xbee package between two Xbee (series 2) radio’s. Another is the gateway device that I would like to extend to be able to deal with more nodes and thus more different types of data. For example: a weathernode may send temp, humidity, wind, rain, etc values. The garden node may send soiltemp, airtemp, soil moisture levels, etc while a node in the water tank reports water levels.

The problem with that is that I am not sure how to develop a protocol that can be used by the nodes and expanded later. I started using a string in each xbee message like this key:value,key:value,key:value but considering the 35 character limit of an xbee packet, I cannot send all the readings in one package.

I think I will be able to program a solution (I did manage to build the controller node and gateway node after all), but not sure how to “future proof” my code.

I know this is not a very concrete question, but is there anybody with similar experiences that has already learned some similar lessons and is willing to share?

If you want to have a look at the old code, I put it up on github some time ago: GrOW/GrOW_Gateway.pde at master · sindono/GrOW · GitHub
GrOW/GrOW_Controller.pde at master · sindono/GrOW · GitHub

All help appreciated!

For example, for whatever reason I cannot seem to get more than 35 or so characters in an Xbee package between two Xbee (series 2) radio’s.

Were you able to, before? You can get a lot of data in 35 bytes.

I started using a string in each xbee message like this key:value,key:value,key:value but considering the 35 character limit of an xbee packet, I cannot send all the readings in one package.

Are key and value strings? They don’t need to be.

Is there any reason why you can’t send just key and value in a packet, and send multiple packets?

but not sure how to “future proof” my code.

That’s easy. Compile, link, upload, delete. No future changes possible.

PaulS:
Were you able to, before? You can get a lot of data in 35 bytes.

Yes, I am not sure why but in the code I linked to, it works up to 35 characters, but breaks when I try to send 36. But, I believe others have been able to send a lot more in one packet?

Are key and value strings? They don't need to be.

Yes, I quite literally put in 'dev:grow,tmp:23.2,hum:86,mst:1021,lgt:1001,d1:1,d2:0,d3:1,d4:0'. At least thats the kind of information I want to send, but its too long. How would you suggest sending the data if it should not be strings?

The reason to send key and value pair (instead of only value and decoding it somewhere down the line) was to be able to easily expand whatever is sent without having to update the gateway unit to decode it before sending it to a webserver? Come to think of it, I could of course just send the whole payload to the webserver and use the gateway only as a device to change tranmission medium (xbee to ethernet).

Is there any reason why you can't send just key and value in a packet, and send multiple packets?

The reason for sending it all in one go was to save energy and have the radio on only as short as possible since I want to run the future update of this project on a seeedstudio stalker with solar panel and battery.

That's easy. Compile, link, upload, delete. No future changes possible.

I think if I rephrase my question/idea: how to go about dreaming up a way to easily expand what is sent from xbee nodes without having to change code everywhere. And do it in a fashion that consumes as little energy as possible.

[/quote]

it works up to 35 characters, but breaks when I try to send 36.

In the XBee.h file, this statement exists:
#define MAX_FRAME_DATA_SIZE 110

Theoretically, you should be able to send much larger packets. Your use of the String class may be fragmenting memory. You lack of use of the F() macro also results in strings in SRAM that needn't be there, reducing the amount of memory available for other uses. You may be facing a memory overuse condition.

The reason to send key and value pair (instead of only value and decoding it somewhere down the line) was to be able to easily expand whatever is sent without having to update the gateway unit to decode it before sending it to a webserver?

Reasonable, but the value could be binary (a float is 4 bytes; an int is two bytes) instead of the (sometimes much) longer string representation of the value.

The reason for sending it all in one go was to save energy and have the radio on only as short as possible since I want to run the future update of this project on a seeedstudio stalker with solar panel and battery.

The time it takes to send data is a function of the amount of data, not the number of packets. While there is some overhead in packeting the data, that should be minimal.

I think if I rephrase my question/idea

I knew what you meant. 8)

I still think that the idea of a packet containing one key and one value is the simplest. A given node may want to send or receive many packets, because it has/needs many key/value pairs. The energy and time required to send 5 packets, each containing one key/value pair is not going to be significantly different from that required to send one packet containing 5 key/value pairs.

Changing one node to send an additional key/value pair as a new packet will be easier than changing the code to send the key/value pair in the existing packet.

Proper design of the code on the receiver (and server) would allow a node to send more data with no changes required to the receiver.

In your logMessage() function, Serial.print(logMessage); is trying to send a
string of GINORMOUS length. Serial.print() has a limited buffer size. Try
sending the string one char at a time, as in your code,

if (debugging) {   
  for (int i = 0; i < logDataString.length(); i++) {     
    Serial.print(payload[i]);    
  }   
    Serial.println("]");
}

and see if that helps.

Ah thanks! I will give that a try. I did not notice any crashing of the program or something, so how would this improve the sketch? By saving/not using memory?

If your goal is to save energy, put all data into a struct and send this struct. So all is packed quite well. There is, of course, a downside... reading the data at the receiver is harder if this part is not also written in C:

struct {
    float humi;
    float temp;
} payload;

Thanks for the tip, but I do not quite understand what benefit I get from putting it in a struct compared to sending just the two floats?

The receiver is also an arduino with an ethernet shield and an xbee shield, so it will also be programmed in C (Arduino IDE)

In your logMessage() function, Serial.print(logMessage); is trying to send a string of GINORMOUS length. Serial.print() has a limited buffer size. Try sending the string one char at a time, as in your code,

The Serial.print() method, in 0023 and earlier, does not have a buffer. So, it blocks as it calls Serial.write() to send each character.

In 1.0+, there is a buffer. Because of that buffer, the Serial.write() method no longer blocks, IF there is room in the buffer. If there is not room, Serial.write() blocks until there is room.

There is no problem with the way that the logMessage() function is written.

The problem is with how it is called. The String uses in SendLogData() is atrocious. Get rid of the String object, and call logMessage() for each part that is appended to the String instance, instead.

Changing the logMessage() method that takes a String instance to take a char * instead will eliminate the creation of a String object each time the function is called. Of course, you can only make this change after banishing the String class from your other code.

tht:
If your goal is to save energy, put all data into a struct and send this struct. So all is packed quite well. There is, of course, a downside... reading the data at the receiver is harder if this part is not also written in C:

struct {

float humi;
    float temp;
} payload;

Note, if you are sending data over the wire in binary form rather than textual (i.e. you are not doing a print), you need to worry about whether the AVR and your host computer store bytes in a different order. Now, both the x86 and avr store things in little endian order, but ARM processors can be run in either endian format. While it is rare nowadays, you also aren't guaranteed that both sides represent floating point using the same representation. Finally the compiler for each system can decide to put arbitrary padding bytes in between structure elements.

In general, what you need to do is marshall the data into an agreed upon binary representation, send over the bytes, and then unmarshall the data on the other side. Even so, it still will be faster to do this than doing prints and scans. However, it is more code to write.
:sweat_smile:

Just out of curiosity, remove a few of those log messages and see if you can send a larger packet.

Thanks! Will do, although I built in a parameter to switch logging on and off, so i hope that did not influence it. But I will work all these ideas in (once I am finished moving my "tinkering" room to the other side of the house) :slight_smile:

tht:
If your goal is to save energy, put all data into a struct and send this struct. So all is packed quite well. There is, of course, a downside... reading the data at the receiver is harder if this part is not also written in C:

struct {

float humi;
    float temp;
} payload;

Also if your receiver does not use the same chip for its computer, you will run into problems where different architectures/abis/etc. have different rules for how structures are packed, what the sizes of the various types are, the representation of the data within types (including whether bytes are stored most significant first, least significant first, or mixed).