Go Down

Topic: MEGA 2560, receiving large data from serial port (Read 4714 times) previous topic - next topic

cbodio

Hi
I have a problem with receiving large data from serial port on Arduino MEGA 2560.
I need to read about 7000 bytes via serial port from electric meter and send them to server
using GPRS SIM800 module.
I coundn't create so big buffer to read all data in one time and after that send them to server.
I try to read data from Serial2 to a little buffer, if buffer is full send data to server and continues read data from Serial2. But when data come's to server I have a lot of miss bytes.
I made a lot of tests and found the problem. When my buffer is full I try to send data to server and when data is sending in SendData() functions Serial2.read() does not read bytes from serial buffer.
It's mean that incoming data overwrite old data in uart buffer because Serial2.read() is not called  when SendData() is wokring. I call WriteDataToSerial3_TCP_CLIENT(5) function in serialEvent2().

How to fix my problem?
May be there is other way to send data via gprs module and read from serial in non blocking way?

Code: [Select]

#define BUF_SIZE_OUT 64
byte out_bufer[BUF_SIZE_OUT];
void WriteDataToSerial3_TCP_CLIENT(int timeout_byte){
 index_out_buffer = 0;
 memset(out_bufer, '\0', BUF_SIZE_OUT);
 unsigned long start_time = millis();
 do{
 while (Serial2.available() > 0)
 {
 if (index_out_buffer < BUF_SIZE_OUT)
 {
 out_bufer[index_out_buffer] = Serial2.read();
 index_out_buffer++;
 start_time = millis();
 }
 else
 {
 //Serial.println(F("out_bufer is FULL"));
 SendData(out_bufer, index_out_buffer);
 index_out_buffer = 0;
 memset(out_bufer, '\0', BUF_SIZE_OUT);
 start_time = millis();
 }
 }
 } while((millis() - start_time) < (unsigned long)timeout_byte);
 SendData(out_bufer, index_out_buffer);
}
void SendData(byte const *data, int data_size)
{
 
 char at_str[20] = "AT+CIPSEND=";
 char str[5];
 sprintf(str, "%d", data_size);
 strcat(at_str, str);
 if (SendAtCommands(at_str, "ERROR", 5, 1) == false)
 {
 Serial3.write(data, data_size);
 WaitResp("ACCEPT", 10, 200);
 }
}

doughboy

you can lower the baud rate of the incoming data or increase the baud rate of the outgoing data or both.
also use buffered read instead of reading one character at a time.

serial driver will continue to overwrite old data if you do not read it fast enough and it fills up the serial buffer. So the idea is to read and process it faster than it arrives.

as a last resort, if the sender supports hardware handshake, you can use arduino platforms (such as teensy 3.2) that supports hardware handshake to pause incoming data while you process received data.

cbodio

Baud rate of serial ports:
Serial2 - 9600 - is port for incoming data and I coundn't change it
Serial3 - this port for SIM800 module, I try different speed from 9600 to 115200.
But speed of sending data by SIM800 module depend only from signal strengh.
I also try to use AT+CIPQSEND=1. This command set quick send mode.
If I send data to serial of SIM800 module, it only accept my bytes to his buffer
and send them later.
I will try to use buffered read but don't understand how it can help me?
serialEvent2 it interrupt event or it called each time in loop?

SurferTim

#3
Apr 22, 2016, 07:54 am Last Edit: Apr 22, 2016, 08:07 am by SurferTim
How often do these 7000 byte packets arrive? Is it one right after the other, or do you get a break in between them? If there is a long enough break, you might want to consider a SD card module. Read the serial onto the SD card, then when complete, transfer to the GPRS module.

edit: Does the meter have any hardware or software flow control available? It may be possible to pause the transmit while you send.

cbodio

I have receive some query from server via GPRS module. When I receive one, I send it to meter.
Then I wait data from meter and if something come from it I must them to server via GPRS module.
All incoming data from server is small packets, max 100bytes.
Outgoing data from meter can be different from 100 to 10kbytes.
For example
in AF 3F B4 39 30 B1 35 B2 B1 21 8D 0A
out AF C5 4D 48 35 5C C0 5C B2 30 B1 CC 5A D1 CA CC 30 30 B1 36 C6 8D 0A
in 81 D2 35 82 50 2E 30 B1 28 B2 B1 36 30 B4 B1 30 B2 B1 30 30 BB B2 B1....
out (packet 7kbytes)
in 81 D2 35 82 50 2E 30 B1 28 B2 B1 36 30 B4 B1 B1 B2 B1 30 30 BB B2 B1 .......
out (packet 7kbytes)

Meter is not my device and I coundn't change it configuration.
And meter have RS485 port

SurferTim

It may not need a configuration change. It may have flow control already there.
http://www.moxa.com/resource_file/509820091121333.pdf
If you can get a make and model off the meter, the meter's user manual or datasheet may have that information available.

Koepel

#6
Apr 22, 2016, 06:16 pm Last Edit: Apr 22, 2016, 06:21 pm by Koepel
That is ASCII readable hexadecimal code with a space in between ?
Could you store it as binary data ?
"AF " is three bytes, but stored as 0xAF takes one byte. You need 3333 bytes to store 10kbyte of  incoming characters.
The Arduino Mega 2560 has 8kbyte sram.

Or use code / library for a ring buffer. But that ring buffer could overflow as well. A fixed array of 3333 bytes is safer.

Is about 10kbyte the maximum amount ? Are you sure ?

You have to do both encode and decode of course. When the data is transmitted, it must be converted back to "AF " and so on. It will slow things down, but at 9600 baud it is no problem.

Perhaps you need to use a Arduino with ARM Cortex processor.

CrossRoads

Use a chip with more SRAM. '1284P has 16K of SRAM, enough for two 7000 byte buffers.
I offer boards in several form factors, DIP and SMD. One example:
http://www.crossroadsfencing.com/BobuinoRev17
Designing & building electrical circuits for over 25 years.  Screw Shield for Mega/Due/Uno,  Bobuino with ATMega1284P, & other '328P & '1284P creations & offerings at  my website.

cbodio

It may not need a configuration change. It may have flow control already there.
http://www.moxa.com/resource_file/509820091121333.pdf
If you can get a make and model off the meter, the meter's user manual or datasheet may have that information available.

For test I use Converter RS232-RS485 Moxa A52, but can be another convertor.
Meters can be different too. For test I use itron (Actaris) SL7000 and EMH lzqj-xc.
SL7000 use binnary portocol - dlms.
LZQJ use text protocol - IEC1107.
I don't know what meters will be connected to my device.

Koepel

Binary, you mean that the data is not readable ASCII with a space between them, but binary data, like 0xAF ? The data over RS-232 is often readable ASCII.

cbodio

I send data to server via stantard tcp socket. Server can be different program and I cound make any changes in data is sending. I want to make device that help to read data from a lot of meters via gprs.
As I understand my problem is that MEGA 2560 have hardware buffer only with 2bytes.

When I test with binary protocol, all is work good.
But when I start to read meter with text protocol, somethings is wrong.
I coundn't understand why I have problem with buffer when work with second meter.
I attach 2 files with logs.
In

SurferTim

Last I checked the Mega 2560 has a 64 byte software buffer also, but that is not really your problem. Have you considered migrating your code to a Due? I think 96K of SRAM would solve your problem.

cbodio

Yes, MEGA 2560 have 64kbytes of software buffer.
Is mean that serial class is reading bytes from hardware bufer to software.
Software buffer sizes are in HardwareSerial.h and I already try to increase buffer to 128 and 256kbytes.
But have no good rezult because I have overwrite bytes in hardware buffer.

I will think about Due but my problem is speed of reading serial buffer.
More RAM can help only if I want to read all data to buffer and then send them to server.


-dev

1)  Indent your code:

Code: [Select]
#define BUF_SIZE_OUT 64
byte out_bufer[BUF_SIZE_OUT];

void WriteDataToSerial3_TCP_CLIENT(int timeout_byte)
{
  index_out_buffer = 0;
  memset(out_bufer, '\0', BUF_SIZE_OUT);
  unsigned long start_time = millis();

  do{
    while (Serial2.available() > 0){
      if (index_out_buffer < BUF_SIZE_OUT){
        out_bufer[index_out_buffer] = Serial2.read();
        index_out_buffer++;
        start_time = millis();
      } else {
        //Serial.println(F("out_bufer is FULL"));
        SendData(out_bufer, index_out_buffer);
        index_out_buffer = 0;
        memset(out_bufer, '\0', BUF_SIZE_OUT);
        start_time = millis();
      }
    }
  } while((millis() - start_time) < (unsigned long)timeout_byte);

  SendData(out_bufer, index_out_buffer);
}

void SendData(byte const *data, int data_size)
{
  char at_str[20] = "AT+CIPSEND=";
  char str[5];
  sprintf(str, "%d", data_size);
  strcat(at_str, str);
  if (SendAtCommands(at_str, "ERROR", 5, 1) == false){
    Serial3.write(data, data_size);
    WaitResp("ACCEPT", 10, 200);
  }
}

Space characters are free; use them to make your code easy-to-read.  If nothing else, hit ctrl-T in the IDE editor too auto-format your code.  -_-

2)  Did you know that "int timeout_byte" is actually two bytes?

3)  Why don't you always send a 64-byte buffer?    Pad shorter packets with bytes you can ignore at the receiver.  Then you can write the characters as you receive them, and there won't be a big pause in reading while you send.

4)  serialEvent is not called from the interrupt.  It is equivalent to calling available/read from loop.    Even if your function were called from the interrupt, you're just buffering the data.  You might as well just increase the 64-byte rx_buffer in the HardwareSerial.* files.  Also, you can't do a send from inside an interrupt function.  And what does this mean:

Quote
But have no good result because I have overwrite bytes in hardware buffer.
5)  Your program structure should probably get turned inside-out.  Look at Serial Input Basics for how you should be reading constantly and, when a newline comes in, do something.  loop gets called many times before it finally does something.  That lets other things have a chance to run.  Right now, you're staying inside a while loop until you're done reading.  Nothing else can happen, which is why you have a "timeout".  That's ok on a PC with a nice multi-tasking OS, but that's not ok on the Arduino.  Useful Links has lots of good info, not just "Serial Input Basics".

6)  Post or attach your entire sketch, but not the libraries... just insert a link for those.  You really have a program structure problem, and we can't make a suggestion without seeing the entire program.  Snippets (small code excerpts) force us to answer "Y?", when the problem is really "X!"  Read this.

Cheers,
/dev
Really, I used to be /dev.  :(

CrossRoads

"Yes, MEGA 2560 have 64kbytes of software buffer."

64 kbytes?  Seems to be a little on the high side, when there is only 8 kbytes of SRAM.

Maybe 64 bytes.


Designing & building electrical circuits for over 25 years.  Screw Shield for Mega/Due/Uno,  Bobuino with ATMega1284P, & other '328P & '1284P creations & offerings at  my website.

Go Up