Trouble transferring and logging data between arduinos using the serialTransfer and SD libraries

Hello there, first post so hopefully I don't bungle any rules here.

I'm working on a telemetry logging project and the specific problem I'm running into is a confusing interaction seemingly between the serialTransfer library and the combination of the SPI and SD libraries.

The problem is that gps data that I have gathered on one Arduino Uno, and then transferred to the second Arduino Uno, doesn't get written to an SD card attached to the second Arduino.

I've put together as minimally reproducible of an example as I could make and attached the code in 3 scripts.
1 would be running on the first Arduino, while 2 or 3 would be running on the second Arduino.

The wiring setup for all 3 scripts is commented in the 3rd script

  1. GPSsend.ino collects the gps data and sends the data off utilizing the serialTransfer library.
#include <TinyGPSPlus.h>
#include <SoftwareSerial.h>
#include <SerialTransfer.h>

static const int RXPin = 4, TXPin = 3;
static const uint32_t GPSBaud = 9600;

float lon=0;
int h = 0,m=0,cs=0;

// The TinyGPSPlus object
TinyGPSPlus gps;

// The serial connection to the GPS device
SoftwareSerial ss(RXPin, TXPin);

SerialTransfer myTransfer;

struct STRUCT {
  int s;
  float lat;
} testStruct;

void setup()
{
  Serial.begin(9600);
  ss.begin(GPSBaud);
  myTransfer.begin(Serial);

  testStruct.s = 6;
  testStruct.lat = 4.5;

  Serial.println("Starting GPS test");
  delay(1000);
}

void loop()
{
  // This sketch displays information every time a new sentence is correctly encoded.
  while(ss.available() > 0){
    if (gps.encode(ss.read())){
      if (gps.location.isValid()){
        testStruct.lat = gps.location.lat();
      }


      if (gps.time.isValid()){
        testStruct.s = gps.time.second();
      }
    }
  }   

  myTransfer.sendDatum(testStruct);
  delay(10); 
 
}
  1. GPSrecieve.ino accepts the data from the arduino running GPSsend.ino and then displays the results.
#include "SerialTransfer.h"

SerialTransfer myTransfer;

struct STRUCT {
  int s;
  float lat;
} testStruct;


void setup()
{
  Serial.begin(9600);
  myTransfer.begin(Serial);
}


void loop()
{
  if(myTransfer.available())
  {
    myTransfer.rxObj(testStruct);
    Serial.print(testStruct.s);
    Serial.print(" | ");
    Serial.println(testStruct.lat);
  }
}

Code 1 and 2 work together as intended, after a brief gps warmup the serial monitor displays the lat and s values.

  1. GPSrecieveAndLog.ino is the code that is acting up. It is the same as code 2 but has the added SD logging functionality. I have tested this functionality and it logs data that is generated on a single arduino just fine, however now that it has data coming in through serialTransfer, it does not ever seem to make it past the void setup chunk as I never see anything written to the serial monitor, and no Telem.txt file is ever created on the sd card.
#include <SPI.h>
#include <SD.h>
#include "SerialTransfer.h"

// Wiring Setup-----------------
// 5v everything running through dom arduino 5v
// grounds Everything back to dom arduino grounds
//0(RX)dom to 1(tx)sub
//1(tx)dom to 0(rx)sub
//sub 4 to tx gps
//sub 3 to rx gps
// dom 6 to cs sd
// dom 11 to mosi sd
// dom 12 to miso sd 
// dom 13 to sck sd

SerialTransfer myTransfer;

struct STRUCT {
  int s;
  float lat;
} testStruct;

File myFile;

void setup()
{
   Serial.begin(9600);
  
  //sd setup
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  Serial.print("Initializing SD card...");

  if (!SD.begin(6)) {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");

  myFile = SD.open("TELEM.txt", FILE_WRITE);
  myFile.println("Lat, Sec");
  //close the file:
  myFile.close();

  myTransfer.begin(Serial);
 
}


void loop()
{

  myFile = SD.open("TELEM.txt", FILE_WRITE);

  // if the file opened okay, write to it:
  if (myFile) {
    Serial.print("Writing to TELEM.txt...");
     if(myTransfer.available()){
      myTransfer.rxObj(testStruct);
      myFile.print(testStruct.s);
      myFile.print(", ");
      myFile.println(testStruct.lat);
      } else {
        myFile.print("NA");
        myFile.print(",  ");
        myFile.println("NA");
      }

    // close the file:
    myFile.close();
    Serial.println("done."); 
  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }

  delay(10);  // delay in between reads for stability
  
}

If anyone has any ideas of how to fix this and use the existing methodology, great! Otherwise, I'm sure that there's a better method of doing this, but as it's my first attempt at sending info from one Arduino to another I'm rather overwhelmed with options that I don't fully understand. I've looked at the serialTransfer library's spi_tx_datum/spi_rx_datum examples but I'm unclear on how I would wire the connection between Arduino 1 and Arduino 2 and don't see documentation on how to do so.

I know I haven't given specific info on the gps or sd module but I don't believe the issue is dependent on those specs. I'm happy to provide them if someone believes otherwise though.
Thank you for any help!

Why the second arduino? All that can run on one.

Ah yes, sorry, I could have been more clear there. 2 reasons.

  1. When I run the gps and sd logging on a single Arduino it "works" but the gps readings are rather inconsistent. Meaning that it'll log one gps second tick 3-6 times (fine whatever) but then sometimes it'll miss 5 seconds worth of gps signal. I'm guessing that this is due to the while(ss.available() > 0) call that I took straight from the tinygps library. This is only true, at least on my cheapo gps module, for a couple tenths of a second every second or so. So I'm thinking it goes through the loop and maybe keeps missing that timing, when it's logging the rest of the sensors I'm telling it to log. Unclear. regardless, that leads to 2
  2. I've cut out a fair bit of other stuff that the SD logging Arduino is doing here. There's a lot of other sensors hooked up to it that are all working properly and by testing each one individually I've seen that the rest of them don't seem to be causing the failure to log the gps data.

What's interesting is even when I just have the gps module and the sd module on a single Arduino, no other sensors at all. It still gives me that choppy signal being logged to the sd card, whereas, when I just read the GPS or read the gps and use serialTransfer to send it to a second Arduino that comes through very smooth and consistent about 2hz ish.

So mostly this is in the name of keeping a smooth refresh rate to the time. If there's a better way to just do that on one Arduino I'm all ears, but given that I'm planning to add a fair few sensors, I thought it best to spread out some of the processing load.

I would be surprised that if there is an issue using one Arduino without using tasking, that it would be any better communicating to a second.
If I had the time I might play with it but I am fairly sure you need to implement at least millis() based tasking if not proper tasking.

So, I'd agree with you except for the fact that the second is taking away the whole reason that the single would have any trouble without tasking. Regardless, thank you for the input, but I think we're getting a little off in the weeds.

To reiterate though, my main problem is that, while I'm able to read and transfer well better than 1hz gps readings (which honestly the fact that it's gps shouldn't matter at all, it's just sending as an int and a float in the struct object), when I attempt to log those transferred values the system does not seem to get past the void setup step. Hence my main question being, what about this interaction of serialTransfer, SD, and SPI is causing this failure, or is it something that I've written wrong?

Some suggestions:

  • Use a Teensy or other MCU that has more horsepower than a normal Arduino (plus they usually come with built-in, high-speed microSD interfaces). This would eliminate the need for two MCUs
  • Either way, you should try and do data acquisition (either from receiving from the GPS or another Arduino) in an ISR and the actual logging in the main loop like this: Teensy 3.6, SD Card, missing time in milliseconds | Teensy Forum
  • SerialTransfer demands quite a bit of memory for what some of the lower-end Arduinos provide and may struggle with it when paired with other memory intensive libs

I don't know why the third code doesn't work, but opening a file, writing a line of data and closing a file brings out the very worst of the already poorly performing Arduino SD library.

Doing so vastly increases the time it takes to write the data, the error rate and the current consumption, and is more or less guaranteed to trash the SD card in a fairly short time. Furthermore, you open the file every time loop() runs, but don't close it unless some data were written, which is also a bad idea.

Open the output file once in setup() then close the file only when you are all done collecting data. Many people use a button to stop data collection and close the file.

For long term data collection, be sure to execute the myFile.flush() command once every hour or day to update the file pointers, then you will not lose all the data if the power fails.

PS: this is nonsense:

  delay(10);  // delay in between reads for stability

You might also find it easier to use a different NMEA parser lib

You might try assigning the SD's CS to D10 instead of D6.

I don't know if the OP is still here, but I'm trying to figure out how even the second sketch (no SD) works. Is what's shown not the complete sketch? Where is softwareSerial set up? Or does SerialTransfer do that for you? Or is he using D0/D1 for both the incoming GPS data and the output to the serial monitor. Can someone explain what's going on? I have no experience with serialTransfer, and can find no documentation on how to use it.

First off, Sorry for the flailing here was, trying to get proper reply formatting.

Hey there @ShermanP ,
The second script is a complete script. It didn't have softwareSerial set up because in that script it isn't doing anything with it. That second script works as I expected and receives the testStruct object and prints the s and lat elements being sent to it by the sub arduino. Also yes, these arduinos are connected between the 0 and 1 pins. The wiring setup for all 3 scripts is shown in the header of script 3.

I recognize that using those 0 and 1 pins may be part of the problem, but as I'm pretty new to this and the documentation in the serialTransfer library doesn't give any explanation as to where wires are supposed to be connected I was just making an educated guess.

@jremington yes, thank you, I'd since updated it to not open and close the file on each loop, but hadn't updated the post as it didn't seem to have any effect on my main issue of the data not transferring and logging when using both serialTransfer and SD together.

As for that last point. I've seen that at the end of oodles of scripts, and had just left it in. Do you have an explanation as to why it's "nonsense" when it's so ubiquitous in demo scripts?

Statements and comments like that are put into code by beginners who have no idea when it is required or appropriate to use delay(), and left in place by beginners who copy the code and are likewise uninformed.

In fact, delay() is almost never necessary. Use of delay() can be convenient, for example when all you want to do is slow down some printout, so you can read it.

When you get to the point where you want the Arduino to be doing several different things in the same program, you can't afford to use delay().

Okay gotcha, I'd thought it was an odd thing to be there as my understanding was that it simply delayed the next operation by that many milliseconds, but figured with it being so ubiquitous there must be some reason it was actually helpful, that I wasn't aware of.

Correct. delay() just wastes some processor time, which could actually be used to do something useful instead.

1 Like

@Power_Broker A couple things.

First Thing:
given nothing is showing up when 3 and 1 are run together does this all just relate back to using the 0,1 pins for the serialTransfer, and should I instead be using your I2C or SPI transfer? If so would you mind pointing me to a little explanation on how I would wire that, because in your I2C and SPI examples I'm not seeing anything about that (and my hardware and wiring knowledge is pretty turrible)

The rest of the things:
The reason for looking to use two arduinos rather than 1 did start off with me thinking it was just too much work to put on a single Arduino, but I believe the fix of not opening and closing the sd card file each loop has gotten my gps logging to a point where it's pretty fine on 1 arduino.

The main reason for using 2 at this point is that the logging for everything except the gps is conducted at well better than 1hz, however my el-cheapo gps module only receives a signal at about 1hz. Therefore I wanted to use the structure I have in my example code so that the dom (master/control whatever you want to call it) arduino can be logging various other sensors quickly, and pull the latest gps info I've saved into the testStruct without having to wait for a new GPS signal. I do agree that I might need something more powerful than an uno to act as the dom here, given this message I get when I upload the sketch

In a final version there'd definitely be more global variables so I'll need to get around that. However, right now at 86% it still doesn't look maxed out, and I'd expect to see something, as opposed to nothing, when script 3 is run on the dom and script 1 is run on the sub, as again script 2 on dom and script 1 on sub works perfectly and doesn't look massively different in terms of available space.

Additionally, I'm hesitant to try your NMEA parser. There seems to be almost no documentation for it and the one example doesn't have any explanation. I think it'd just lead me back to being confused and making another forum post.

So you have the SD logging working on another sketch, but using the same SD wiring as on sketch #3 above, particularly using D6 as the CS? I just want to be sure the SD part works on its own.

I don't know if using D0/D1 is the problem. It would be nice if that worked because they use the UART peripheral in the 328P, which saves a lot of time because you don't have to bit-bang the transfers as softwareSerial would do. And you could use the second Uno's Rx to receive the GPS data, but use Tx to transmit to the serial monitor. I just don't know if that works. We really need @Power_Broker to explain better how his library works.

That said, you do have the option of using I2C or SPI, both of which are also internal peripherals. But that might depend on how you will be communicating with all the sensors and such. The problem would be that your second Arduino would need to be the sub when receiving data from the first Arduino, but the dom when saving to SD. I don't know if that's practical.

I just wonder if it might all be easier to just use one Arduino for everything. And if that's too much, maybe something like a Mega could be used. Well, I guess I'm just going over what's already been said.

Most often the 1hz location update rate is the default setup, but even el-cheapo GPSs can normally be configured for location update rates of 5hz or more.

hey @ShermanP, yes I do have a single Arduino solution that gathers data and logs it, with the same wiring setup, and yes in particular using the D6 for CS.

Agreed on the rest of those parts. However, the reason that I'd like to get this two arduino solution working properly, is mostly to use it as a proof of concept when adding more and more sensors. As I'd like to be able to effectively plug a cluster of sensors into the system and tell the SD focussed arduino to log it, but that's well beyond the scope of this post and question. I guess I should look deeper into creating a faux CAN bus system or something like it.
I guess, whether @Power_Broker is right and it's too much load to use serialTransfer and a struct object on an Arduino Uno, or there's something not playing nice between serialTransfer and SD, I'll be trying out a teensy board or trying out a different Arduino to Arduino communication method.

@srnet if you've got any examples/documentation to point me to there I'm all ears. I've done the obligatory google on the subject, but my understanding is that those signals are only being sent out at about 1hz, so unless my GPS module had an internal IMU (which it doesn't) to do some sweet sensor fusion, I'm not actually going to be getting any new information.

We really need @Power_Broker to explain better how his library works.

When you look at the documentation and especially the multitude of given examples, you shouldn't need any more explanation on how it works.

Users of the lib should already know how to wire up a UART connection between two Arduinos and read example sketches - it's basic knowledge.

I'm hesitant to try your NMEA parser.

That's fair, I wrote it fairly recently for a personal project and never "punched it up". However, this would be a good example if you're using RMC messages:

#include "NMEA_Parser.h"

nmea_parser gnssParser;
nmea::rmc_struct rmc;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200); // Use the serial port and baud of YOUR GPS receiver/Arduino connection
  gnssParser.begin(Serial1);
}

void loop()
{
    if (gnssParser.available() && (gnssParser.getCurrMsgType() == nmea::msg_types::RMC))
    {
        rmc = gnssParser.getRMC_struct();

        Serial.print(rmc.hour);        Serial.print(",");
        Serial.print(rmc.min);         Serial.print(",");
        Serial.print(rmc.sec, 5);      Serial.print(",");
        Serial.print(rmc.pos_status);  Serial.print(",");
        Serial.print(rmc.lat_dms, 5);  Serial.print(",");
        Serial.print(rmc.lat_dms_dir); Serial.print(",");
        Serial.print(rmc.lat_dd, 5);   Serial.print(",");
        Serial.print(rmc.lon_dms, 5);  Serial.print(",");
        Serial.print(rmc.lon_dms_dir); Serial.print(",");
        Serial.print(rmc.lon_dd, 5);   Serial.print(",");
        Serial.print(rmc.sog_kn, 5);   Serial.print(",");
        Serial.print(rmc.cog_true, 5); Serial.print(",");
        Serial.print(rmc.year);        Serial.print(",");
        Serial.print(rmc.month);       Serial.print(",");
        Serial.print(rmc.day);         Serial.print(",");
        Serial.print(rmc.mag_var, 5);  Serial.print(",");
        Serial.print(rmc.mag_var_dir); Serial.print(",");
        Serial.print(rmc.mode_ind);    Serial.print(",");
        Serial.print(rmc.checksum);    Serial.print(",");
        Serial.print(rmc.usTimestamp); Serial.println();
    }
}

Overall, my suggestion is to get a Teensy 4.1 plus a Ublox NEO-M9n and do everything you want in one combined setup.