Hi all - I'm new to arduino programming but I have a good bit of C experience. I've also searched the forums and haven't found any answers to what I'm asking, so sorry if I've missed it. Anyway.
I'm using an arduino mega interfaced with a WiFly GSX. For now, believe me that the hardware is hooked up and configured correctly, as it successfully sends single bytes to and from the server application I've written.
What needs to happen is I need to send a device ID to the server. This device ID will be of type long, something like 123456789. Since it seems that if I try to use Serial.print(id), it'll convert 123456789 into a string and send 9 bytes, whereas only 4 bytes are necessary for a long. Here's the pseudocode (c/arduino mix) for how I'd expect to do this (also, there are multiple types of messages, so the first byte sent will determine the type of message to follow)
unsigned long ID = 123456789;
unsigned char msg[1024];
msg[0] = ID_MSG_TYPE; // in this case, ID_MSG_TYPE is the number 101
memcy(&msg[1], &ID, sizeof(long));
msg[sizeof(long)+1] = '\0'; // does print print everything up til the \0, or will it keep going out to msg[1024]?
Serial.print(msg);
And the opposite sort of memcpy on the server side, but it produces a very wrong number for the ID at output. The server has an intel chip which should be little endian, and my research shows arduino is also little endian. (Does arduino even support htonl()??)
Really not sure if I'm doing this the right way at all. Any help would be greatly appreciated!!
sendBinaryMsg(byte *f, int len)
{
while (len--) {
Serial.write(*f++);
}
}
Otherwise you're at the mercy of bytes that might be null in the middle of your numeric data.
How are you assembling bytes received at the other end back into a message ?
What data are you actually seeing? (send hex and look at hex for debugging; it's much easier to see what's going on!)
As far as standard length, that really isn't an option. Certain messages will be a single byte, other messages may be 50-60 character strings (variable).
I'll check on that output the next time I have a chance - the hardware is in a lab so I can't look into it right now.
I will add, though, that I wrote a test client in C to check the protocol, and I package the character array up like my original example (except I can use "send" and specify number of bytes to send), and it IS interpreted and displayed correctly by the server.
As far as standard length, that really isn't an option. Certain messages will be a single byte, other messages may be 50-60 character strings (variable).
Then if that is not an option, then binary data transfer becomes a lot more difficult -- not impossible -- but sometimes close to that. So, then you need to have a message length parameter included in the message which lengthens the very short blocks considerably. Or ID strings.. or and they all waste space.
There is another way to do it if you understand parsers and compilers -- maybe -- depends on the data formats and patterns.
Of course if you do succeed without using any of those mechanisms you well end up rich. $)
Then if that is not an option, then binary data transfer becomes a lot more difficult -- not impossible -- but sometimes close to that. So, then you need to have a message length parameter included in the message which lengthens the very short blocks considerably. Or ID strings.. or and they all waste space.
I disagree, somewhat. What's mostly necessary is a well defined protocol, and to put a bit of what you said into practice. For example, here are some of the messages we're implementing (and I can't really explain why they're all necessary at this point, just trust me that they are even though gps and address might seem redundant):
register: includes ID (long)
gps position: includes two coordinates (both floats)
street address: big long string. first byte will be number indicating length of string
So, as long as the server knows how much data to expect based on the msg ID (first byte of the message) there doesn't need to be any extra bytes to indicate message length, assuming protocol is followed exactly. It would be necessary for the case of variable length data (like the string) in which case a byte indicating string length would be sent first, and the protocol would dictate to examine that byte for length.
Yes, it adds a bit of data and complexity, but if the "gps position" message is sent out every 5 seconds and "street address" is received from another client every 10 minutes, you're saving a TON of space in the long run, because a fixed length message would need 50-60 bytes and only 3 of them would be used for the most frequent message.
register: includes ID (long)
gps position: includes two coordinates (both floats)
street address: big long string. first byte will be number indicating length of string
Sounds simple enough, are you still having a problem?
a byte indicating string length would be sent first
The string can still be null terminated.
None of this is particularly robust though, I hope you're not making important decisions based on the received data.
decoding network packets is much more efficient if you have the length of strings explicitly, rather than needing to scan all the data to find a terminator (or internal special characters. The "telnet" protocol, for instance, is pretty difficult to optimize because the receiver has to look at every byte received in case it might be a "command" character.)
None of this is particularly robust though,
How so? Network wise, it is being sent over a TCP connection, so additional error checking shouldn't be necessary.
OTOH, there's that "unreliable" serial link between the Arduino and the thing that creates the TCP packet, that is not included in any of the TCP's error checking.
decoding network packets is much more efficient if you have the length of strings explicitly
But we're not decoding escape sequences here, I'm happy with a length or a null, I see no real difference between decrementing a counter and testing for null.
there's that "unreliable" serial link between the Arduino and the thing that creates the TCP packet,
That's all I'm interested in. If the "thing" is a shield plugged onto an Arduino then I'm probably being over cautious, but if it's at the end of a 30' wire then maybe not.
Granted TCP or any other established protocol will be reliable and if that's being used as the transport mechanism that's good. But it doesn't make the underlying data robust. Whether or not it's worth hardening that up though depends on theseankelly's application.
About to start working on this again for the afternoon and will try to apply what some of you have suggested - I'll let you know results.
In the meantime, my biggest concern is I can't figure out how the wifly will actually encapsulate data. For instance, if I use the approach of making a little loop to send the character array byte by byte, won't the wifly send a unique TCP packet for each byte?? As best I can tell whenever the wifly receives data on rx, it opens a connection and sends it off. Seems terribly inefficient.
But my thought was if I can package all the info up onto a single string and send the string all at once using print or something, maybe the wifly won't start sending until after the uART end signal is received.
This device ID will be of type long, something like 123456789. Since it seems that if I try to use Serial.print(id), it'll convert 123456789 into a string and send 9 bytes, whereas only 4 bytes are necessary for a long.
Right -- and a long could easily contain a null. Mine do so often. Why are yours different? ...or maybe they are not.
Make a set of packet protocols with packets of convenient lengths and carry on.
...Or else send text. It's your call you are the designer.
Now which is it? Binary or text? Then maybe you can get some help.
This device ID will be of type long, something like 123456789. Since it seems that if I try to use Serial.print(id), it'll convert 123456789 into a string and send 9 bytes, whereas only 4 bytes are necessary for a long.
Right -- and a long could easily contain a null. Mine do so often. Why are yours different? ...or maybe they are not.
Make a set of packet protocols with packets of convenient lengths and carry on.
...Or else send text. It's your call you are the designer.
Now which is it? Binary or text? Then maybe you can get some help.
Binary.
And I did get it working using the sendBinaryMsg idea. I swear that's what I did yesterday and it didn't work. Not really sure what changed, but I can successfully send floats. Any advice on how the WiFly will package this up and send it? One byte per TCP message, or might it buffer for some period of time?
I apologize for ambiguity, I'm completely new to micro-controllers.
This device ID will be of type long, something like 123456789. Since it seems that if I try to use Serial.print(id), it'll convert 123456789 into a string and send 9 bytes, whereas only 4 bytes are necessary for a long.
Right -- and a long could easily contain a null. Mine do so often. Why are yours different? ...or maybe they are not.
Make a set of packet protocols with packets of convenient lengths and carry on.
...Or else send text. It's your call you are the designer.
Now which is it? Binary or text? Then maybe you can get some help.
Binary.
And I did get it working using the sendBinaryMsg idea. I swear that's what I did yesterday and it didn't work. Not really sure what changed, but I can successfully send floats. Any advice on how the WiFly will package this up and send it? One byte per TCP message, or might it buffer for some period of time?
I apologize for ambiguity, I'm completely new to micro-controllers.
And you think a lot of us aren't? lol
You really need to design a protocol. But if you don't want to do It I can't stop you from sending byte by byte. Really.
I see no real difference between decrementing a counter and testing for null.
On "large" CPUs, a bytecopy(src, dst, length) will frequently optimize the copy so that it is doing memory fetches/stores at the full memory width (4 bytes, 8 bytes, etc), while a strcopy(src, dst) that has to check each byte for null will be restricted to operating one byte at a time... Data that is beyond a string (say, a second string) will have to "parse" the first string (one byte at a time) to find out where the second string begins, whereas a counted string could be simply skipped over...
Any advice on how the WiFly will package this up and send it?
The "packet boundry" problem is significant when you have a "dumb" (serial) connection between your data source and the device actually sending the packets. And there is no standardization. You'll have to dig into the wifly documentation to see if they have any specific features to address the issue. One common scheme is to wait for a "gap" in the received serial data...
When in doubt, see what the giants have done, then stand on their shoulders. And stick to standards known to work.
If I understand the problem correctly, it is of sending binary data through the network - in this case a WiFly unit.
One approach would be to think of it as an email: x-mime, base-64 encoding.
Then all the packet-level trivia falls away.
On the receiving side, you'll get an email or at least something decodable that way.
If you need help - keep asking. The gang on this board may have a library already. If not, let's write one an get it out there.
PS: A lot of the standard internet protocols (HTTP, SMTP, FTP, etc) ended up electing to send numbers as text. Some of that is because the underlying protocols (TCP) give the application a byte stream with the same lack of packet boundries that you're having problems with. But it also improves debuggability and removes some other worries (endianness.)
