Pages: [1] 2   Go Down
Author Topic: Mega project with large arrays Help  (Read 1278 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I have a project where I need a Arduino Mega 2560 to act as a serial data controller. Here's my issue, it needs to send command strings of bytes in the manufactures format BINR. The device will then send back a stream of bytes formatted (10 startbyte, message type byte, data, 10 03 end byte combo). As an example however if I request GPS empheritis I will have at the least 12 messages 84 bytes long. I then need to pull most of the bytes out and reformat for transmission in RTCM SC10403.1 format. I believe I can get the code written to put these into arrays however I am thinking I will be out of SRAM right away. Does that sound right?

If so how can I add external RAM to the device to use for my array storage. Also is there a limitation to the addressing size the ATMEGA can use?
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 548
Posts: 46026
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I will have at the least 12 messages 84 bytes long.
12 x 84 = 1008 bytes, of the 8K you have available on the Mega.

Quote
I then need to pull most of the bytes out and reformat for transmission in RTCM SC10403.1 format.
Potentially, then, you'll need 2 or 3 times that much space for the output arrays. Still plenty of room.

Quote
I believe I can get the code written to put these into arrays however I am thinking I will be out of SRAM right away. Does that sound right?
Which part? The part about being able to write the code? I can't comment on that. The part about running out of memory? I wouldn't think so.

The thing that I'd be concerned about is often you need to read 1000 bytes of data, at some speed, and send out 1000 +/- bytes of data, at some speed. Can you accomplish that in the window between data being available and processing needing to be complete?
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I will actually have up to 24 arrays for gps, 24 arrays for glonass, and 16 for galileo eventually, as its a 32 channel reciever. There will also be a few other data arrays to send regarding the antenna location and elevation.
Most messages need to be sent every 1 second. Others only every half hour or hour.

Which leads me back to needing external ram I think. How do you access it? Are there any examples out there? I have not found any easily available. I did find a post in the forum that never recieved any answers about external ram.
Logged

Leeds, UK
Offline Offline
Edison Member
*
Karma: 71
Posts: 1641
Once the magic blue smoke is released, it won't go back in!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Have you seen this:
http://andybrown.me.uk/wk/2011/08/28/512kb-sram-expansion-for-the-arduino-mega-build/
Logged

~Tom~

New Hampshire
Offline Offline
God Member
*****
Karma: 13
Posts: 779
There are 10 kinds of people, those who know binary, and those who don't.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

http://ruggedcircuits.com/html/quadram.html

That expansion card provides some samples to show the various methods of accessing the additional memory.
Logged


Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks

Those look like good leads
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This is a piece of the code to read one large message from the device. I send 4 bytes to request and it should answer back,
It will respond with a 50 byte hex( 10 E4 ....misc message data...10 03) indicating end of segment, then it will send out shorter 18 byte messages for each active channel tracking a satelite each ending in 10 03.

Within the 50 byte nessage it will say how many channels will follow. How do I limit how many times it should look for an 18 byte message based on this. Can I create only enough arrays to fill with the number of messages comming? If so how do you do it? any examples would be great.

Is there a simpler way to create the needed arrays to store the data in?

How does one "look" for a 2 byte combination as a trigger to save data or see the end. All of these messages start with 10 xx (message type), and end with 10 03.

Code:
byte packetE4h[50];
byte svData1[18];
byte svData2[18];
byte svData3[18];
byte svData4[18];
byte svData5[18];
byte svData6[18];
byte svData7[18];
byte svData8[18];
byte svData9[18];
byte svData10[18];
byte svData11[18];
byte svData12[18];
byte svData13[18];
byte svData14[18];
byte svData15[18];
byte svData16[18];
byte svData17[18];
byte svData18[18];
byte svData19[18];
byte svData20[18];
byte svData21[18];
byte svData22[18];
byte svData23[18];
byte svData24[18];
byte svData25[18];
byte svData26[18];
byte svData27[18];
byte svData28[18];
byte svData29[18];
byte svData30[18];
byte svData31[18];

void setup(){
  Serial2.begin(19200);
}


void loop(){

  // retreave orbiter data and store
  while (Serial2.available () >0){
    byte inbyte = Serial2.read();
    if (inbyte == 0xE4){  //check for message type
      for(int n = 0; n < 50  ; n++){ //save bytes in array
        packetE4h[n] = Serial2.read();
      }
      if (packetE4h[48] == 0x10 && packetE4h[49] == 0x03){ //end bytes of first section of data stream
        for(int n=0; n<18; n++){
          svData1[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData1[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData2[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData3[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData4[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData5[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData6[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData7[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData8[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData10[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData11[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData12[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData13[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData14[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData15[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData16[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData17[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData18[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData19[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData20[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData21[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData22[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData23[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData24[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData25[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData26[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData27[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData28[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData29[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData30[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData31[n] = Serial2.read();
        }     
      }
    }
  }
}
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 548
Posts: 46026
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Is there a simpler way to create the needed arrays to store the data in?
Whenever you start numbering variables, you should consider using arrays. Since the variables you are numbering are already arrays, then you should be thinking 2D arrays.
Code:
byte svData[32][18];
Be aware that this array, or your 32 arrays are taking up a lot of space.

Code:
  while (Serial2.available () >0){
    byte inbyte = Serial2.read();
    if (inbyte == 0xE4){  //check for message type
      for(int n = 0; n < 50  ; n++){ //save bytes in array
        packetE4h[n] = Serial2.read();
      }
Ooh. Bad idea. If there is at least one byte available to read, read it. If it is 0xE4, read 50 more bytes that likely haven't arrived yet. Not good at all.

If, by some miracle, this is true:
      if (packetE4h[48] == 0x10 && packetE4h[49] == 0x03){ //end bytes of first section of data stream
You now read 576 bytes that likely have not all arrived yet, either (especially considering that the Arduino only has a 64 byte serial buffer).

You have a lot to learn about how serial data is sent and received. and when you can read it.
Logged

SF Bay Area (USA)
Offline Offline
Tesla Member
***
Karma: 106
Posts: 6371
Strongly opinionated, but not official!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I will have at the least 12 messages 84 bytes long.
Just because your message is 84 bytes long doesn't mean that you'll need to store 84 bytes of data.  If the analysis is fast enough, you can distill your message down to just the relevant information.
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

PaulS, I know I have alot to learn.

Here's what I was thinking:
Code:
if (packetE4h[48] == 0x10 && packetE4h[49] == 0x03){ //end bytes of first section of data stream
I was wanting to use that line to verify that I had a complete first section before reading the next section of the stream. If its not true then the byte data is misaligned and I would not be able to call specific sections of bytes back out to resend out.

Should I add a if, available before each subsequent read?
Code:
     if (Serial2.available() >0){
        svData1[n] = Serial2.read();
        }
        for(int n=0; n<18; n++){
          svData1[n] = Serial2.read();
        }
}

I appreciate the 2D array advice. As far as the size I am looking at adding exernal RAM for that.
I have at least 3 more message types to store and send besides this one.

Any examples of what works better would be appreciated.
Logged

Global Moderator
UK
Offline Offline
Brattain Member
*****
Karma: 238
Posts: 24321
I don't think you connected the grounds, Dave.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Any examples of what works better would be appreciated.
That works better than what?

It seems to me that storing ASCII (I assume) messages may not be the most efficient use of RAM.
Rather, as westfw suggested, you store the data the messages represent as, say, binary, which may well be considerably smaller, without the overhead of also storing the message identifiers.
For example, north or south could be encoded in a single bit, rather than a whole byte if represented as a character.
Assuming the other seven bits can be used for other things (including ASCII characters), that can be quite a saving.
Logged

"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

That could make sense except on this project the data is in binary already. I also have to pull out the relative byte segments and send them out in another binary format to an NTRIP server program or else incorperate a http ethernet connection in this same project.
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset




Is this a workable meathod of identifing a 2 byte sequence to start of message? My device will send out a 0x10 (used as tuffer bytes) followed by the message identifier in this case 0x60. Id there a better way to code this?


Code:

void packet60(){
  // packet60h no. sats used and DOP
  while (Serial2.available () >0){
    byte inbyte = Serial2.read();
    if (inbyte == 0x10){ //check start byte
      inbyte = Serial2.read();
      if (inbyte == 0x60){  //check for message type
        for(int n = 0; n < 12  ; n++){ //save bytes in array
          if (Serial2.available() > 0){
            packet60h[n] = Serial2.read();
          }
          if (packet60h[11]==0x10 && packet60h[12] == 0x03){
            satDOP = true;
          }
          else {
            satDOP = false;
          }
        }
      }
    }
  }
}
Logged

Global Moderator
Dallas
Offline Offline
Shannon Member
*****
Karma: 176
Posts: 12283
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Is this a workable meathod of identifing a 2 byte sequence to start of message?

No.  But before correcting the mistakes ... Can packet60 wait for data to arrive?
Logged

Offline Offline
Jr. Member
**
Karma: 0
Posts: 65
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yes, there id time to collect before processing. I want to collect the complete binary string then read segments from the arrays to convert to an RTCM message format. The device messages have fixed lengths however I have 1- 79 byte array and I will need 24- 134 byte arrays. So I don't know how to do that with the 64 byte serial buffer.
Logged

Pages: [1] 2   Go Up
Jump to: