Serial array size not linear?

Projects Programming
1

I'm trying to pass a dynamic number of characters from a Python program on my Raspberry Pi to my Adafruit Feather 32u4 radio. After discovering that using the available() function truncates my character array to 64 bytes I searched the web and found this link below.

http://forum.arduino.cc/index.php?topic=396450.0

I'm using Example #2 and according to the link above I can change the array length to what I choose but I'm finding that it's working in a non-linear fashion.

This is my modified test using Example 2:

// Example 2 - Receive with an end-marker
#include <SPI.h>
const byte numChars = 1500;
char receivedChars[numChars];   // an array to store the received data

boolean newData = false;

void setup() {
    Serial.begin(74880);
    Serial.println("<Arduino is ready>");
    Serial1.begin(74880);
}

void loop() {
    recvWithEndMarker();
    showNewData();
}

void recvWithEndMarker() {
    static byte ndx = 0;
    char endMarker = '\n';
    char rc;
    
    while (Serial1.available() > 0 && newData == false) {
        rc = Serial1.read();

        if (rc != endMarker) {
            receivedChars[ndx] = rc;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
        }
        else {
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            newData = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        newData = false;
    }
}

Note that the baud rate is the highest I can use without getting garbage characters.
Serial1 is the serial coming from the Pi. My test string from the Pi is

Changing the size of the numChars to 500 yields the most but increasing it yields fewer characters. From where I see, the number of characters does not act in a linear fashion. I'm sure there's something else going on, but I'm not seeing it.

numChars = 650
This just in ... INSERT INTO weathers(currentWindSpeed, currentWindGust, totalRain,  bmp180Temperature, bmp180Pressure, bmp180Altitude, bmp180SeaLevel, ou

numChars = 500 //(closest to all the characters)
This just in ... INSERT INTO weathers(currentWindSpeed, currentWindGust, totalRain,  bmp180Temperature, bmp180Pressure, bmp180Altitude, bmp180SeaLevel, outsidetemperature, outsideHumidity, currentwinddirection, currentWindDirectionVoltage) VALUES(0.000, 0.000,

numChars = 400
This just in ... INSERT INTO weathers(currentWindSpeed, currentWindGust, totalRain,  bmp180Temperature, bmp180Pressure, bmp180Altitude, bmp180SeaLevel, outsidet

numChars = 1000
This just in ... INSERT INTO weathers(currentWindSpeed, currentWindGust, totalRain,  bmp180Temperature, bmp180Pressure, bmp180Altitude, bmp180SeaLevel, outsidetemperature, outsideHumidity, currentwinddirection, currentWindDirectionVoltage) VALUES(0

numChars = 1500
This just in ... INSERT INTO weathers(currentWindSpeed, currentWindGust, totalRain,  bmp180Temperature, bmp180Pressure, bmp180Altitude, bmp180SeaLevel, outsidetemperature, outsideHumidity, currentwinddirection, currentWindDirectionVolta

Looking at this reference seems to suggest that I can create a large array? In this test my string length is 312 characters but the actual number is likely longer and dynamic.

https://www.arduino.cc/en/Reference/Array

Clearly, there's something going on that I'm not understanding and a pointer to what I'm doing wrong would be helpful.

2

It's possible you are waiting for the buffer to fill up before accepting data from it. The size of the buffer is normally not that important, as it is just a safety measure to allow for delays in reading. When you use the available() function properly, there will only be a few characters in the buffer because you didn't wait for it to fill up.

On the other hand, the declared size of the Cstring array that your code puts characters in is only limited by the size of available memory. The 32u4 only has 2.5k of internal RAM.

The solution is to process the data serially as it arrives, rather than waiting for the entire string to arrive. This is really a "must" if the maximum size of the string is unknown.

3

After discovering that using the available() function truncates my character array to 64 bytes

Nonsense. The available() function does not truncate anything. It simply tells you how much data is in the 64-byte incoming serial buffer.

There is rarely any reason to make the incoming serial buffer bigger, since you can remove data from the buffer faster than the buffer can be filled.

What you do with the data after you extract it from the serial buffer is a different matter.

Why ARE you sending a novel to the Arduino, anyway? Why doesn't the Pi put the data into the database?

4

You need to change this line

const byte numChars = 1500;

to

const int numChars = 1500;

because a byte cannot have a number greater than 255

Same thing for this line

static byte ndx = 0;

...R

5

Excellent! Changing the type from byte to int shows the entire set of characters.
The reason this is not going into a database on the Pi is that while that works, the goal is to get past the Pi's poor wifi performance. This is a remote weather station and wlan keeps breaking down. The goal is to use the radio from a significant distance to push data into the database remotely which is more likely to be running. Thx again!

6

But then it remains, why are you sending novels? 12000 bit is A LOT of information from a weather station...

7

But then it remains, why are you sending novels? 12000 bit is A LOT of information from a weather station...

It should NOT be necessary to send all the column names. They aren't going to change, are they?

It should not be necessary to use such long column names. For the purposes of the database, it is not necessary to define that the temperature and humidity came from a bmp180.

8

PaulS:
For the purposes of the database, it is not necessary to define that the temperature and humidity came from a bmp180.

It looks like he is sending a complete SQL statement.

sam452:
the goal is to get past the Pi's poor wifi performance. This is a remote weather station and wlan keeps breaking down. The goal is to use the radio from a significant distance to push data into the database remotely which is more likely to be running. Thx again!

Do you mean that the RPi is collecting the data, then sending it by Serial to the Arduino which then sends it on somewhere else?

If so, where is it sending it and how is it sending it?

...R

9

Ultimately since the Pi's wlan is not reliable, I intend to take the values transmitted by this radio to a receiver radio a good distance away and build a curl command to my web app which is a front for my postgresql db.
I'm transitioning from creating a SQL statement from my python script on the Pi to sending the values via the radio.
For the purposes of this question the SQL statement is long and dynamic enough (longer than the 64 bytes the buffer uses) to use as a proxy for the dynamic values to come soon once this step is complete.
Thx for your help, sam

10

Why not connect that unknown radio to the Pi?

11

sam452:
Ultimately since the Pi's wlan is not reliable, I intend to take the values transmitted by this radio to a receiver radio a good distance away and build a curl command to my web app which is a front for my postgresql db.

That probably means something to you because you know all about your project but it is meaningless to me.

Please answer the 3 simple questions in Reply #7

...R

12

Robin2:
Do you mean that the RPi is collecting the data, then sending it by Serial to the Arduino which then sends it on somewhere else?

If so, where is it sending it and how is it sending it?

...R

That is affirmative. The Pi is connected to the weather sensors and is transmitting it to a Feather 32u4 using Serial.

Where is it sending? To a sister Feather 32u4 that is located some distance away.

How is it sending it? Using the rfm69 library located here GitHub - LowPowerLab/RFM69: RFM69 library for RFM69W, RFM69HW, RFM69CW, RFM69HCW (semtech SX1231, SX1231H)

13

septillion:
Why not connect that unknown radio to the Pi?

14

How about just sending the data and let receiving Arduino create the sql statement.

I haven't counted the number of bytes that you need to send.

15

sterretje:
How about just sending the data and let receiving Arduino create the sql statement.

+1

sam452:
How is it sending it? Using the rfm69 library located here

I have no experience with those wireless devices.

...R