Is SD card module not working properly with SIM800l

Hello,

Im a newbie to arduino. Im working on a project with arduino nano. I have to make a http post to a server and then store data on to the SD card. for the sdcard module im using SdFat library, and to communicate with the sim800l module im using Software serial. when testing both modules independently , everything works fine. Once integrated them to one code, the problem starts. The problem is the AT command replies read by the software serial are partial strings for Longer AT commands. And the the replies which are partial have variable lengths. I have tried changing the Software serial RX buffer size,still no use. The constraint is that i have to make it work in the nano,because of the convienient size. If there any suggestions, much appreciated. Thank you

(deleted)

Hi Peter
Im good thanx :slight_smile: .

Yes i did, the sketch uses 63% ,20538 bytes. when keeping SdFat sd,SdFile myFile as global variables,the global variable memory spiked upto 63%,so i put them into my function and it came down to 32%. does the SdFat library use the same RX buffer as SoftwareSerial ?

(deleted)

hi :slight_smile: ,

I did check with that, that uses SPI.h and SD.h libraries. Im not using them,they turned out to be quite heavy, im only using SdFat.h only,im really sure there some interference in the RX buffer only

The SD card im using is 16gb, does that have any effect on the memory utilized in the arduino?

(deleted)

(deleted)

Hi peter,
Thank you so much for your support. I really do appreciate it.

void writeToSDCard(String date,String data){
 
SdFat sd;
SdFile myFile;

String filename = date+".txt";
  // Initialize SdFat or print a detailed error message and halt
  // Use half speed like the native library.
  // change to SPI_FULL_SPEED for more performance.
if (!sd.begin(chipSelect, SPI_FULL_SPEED)) sd.initErrorHalt();

  // open the file for write at end like the Native SD library
  //O_RDWR | O_CREAT | O_AT_END
  if (!myFile.open(filename.c_str(), O_WRITE | O_CREAT | O_AT_END)) {
    sd.errorHalt("opening test.txt for write failed");
  }

  // if the file opened okay, write to it:
  Serial.print("Writing to datalogger");
  myFile.println(data);
  myFile.flush();

  // close the file:
  myFile.close();
  Serial.println("done.");
  
}

This is the function im using for SD card write.
And yes, i had to use String data type, but i was able to solve the memory issue for that with F() function which stores the static Strings in the Flash memory.

String writeATCmnd(String cmd,String positivereply,int retry_count,int del_t=0){
  
float timeout_start=0;
float timeout_elapse=0;
int count=0;
bool reply_correct=false;
bool command_sent=false;
String reply="";

  //Serial.println("At writeATcmnd");
   timeout_start =0;
//this will loop until the retry count finishes if the reply is incorrect
   while(count<retry_count){
    reply="";
     //Serial.print("command :" );
    // Serial.println(cmd);
     mySerial.println(cmd);
     mySerial.flush();
     //Serial.println(timeout_start);
     delay(del_t);
     timeout_start = millis();
 //this will loop after the the command is sent in one try,and wait for the response till the time out happens
     while(timeout_elapse<1200000){
      timeout_elapse+=(millis()-timeout_start);
      if(mySerial.available()){
          reply = mySerial.readString();
          
          if(reply.indexOf(positivereply)>0){
            Serial.print("reply success:");
            Serial.println(reply);
            return reply ;
            
          }
          else{
            if(reply.indexOf("ERROR")>0){
              Serial.print("reply fail:");
            Serial.println(reply);
            return "ERROR";
            }
            else{
              Serial.print("Invalid reply:");
              Serial.println(reply);
              continue;
            }
          }
      }
      
     }
 
     timeout_elapse=0;
     timeout_start=0;
     count++;
   }
  Serial.println("Timeout fail!!");
  return "OK";
}

And this code is used to write AT commands

As Peter said, you need to avoid using the String datatype.

The String datatype dynmically allocates memory from the heap whenever a new String is required, or a String is changed in length etc. There is no garbage collection for the released memory which means if the Strings are of different length the heap memory gets fragmented eventually resulting in a failure to allocate memory and apparently random problems such as one String not being allocated (and is therefore empty) so the data is lost and the next String working (because it is smaller or another String was freed).

As usual, post your whole sketch that is not working, along with the exact error messsages (if any) that you get.

Hi countrypaul,

Yes thats very true what peter mentioned. But i have about 13 AT Commands to send,because i need use authorization Keys in the HTTPPARA. So strings made it really easy for me to do. I also tried to save those strings in a char array as well, both seem to consume same memory.

Guys i was able to solve it. The method i used is to use thread.h library and run the sd card funtion using a thread, now it works fine.

What is the alternative to write AT commands without using strings.

Rather than using Strings you could use character arrays - means you have to do a little more coding. If those character strings (or parts of them) can be predefined you could store them in flash memory using the PROGMEM keyword so that they don’t take up the working sram (and use the F macro to extract). Using character arrays also avoids the use of dynamically allocated memory from the heap which therefore ensures there is no fragmentation.

Using Strings does make it appear easier but in the long run it becomes a major nuisance and learning how to use character arrays from early on pays dividends. On a full operating system than includes garbage collection and has plenty of memory using Strings is not usually a problem.

Hi countrypaul,

Thank you for your suggestions. I shall try in char arrays and see. You are right about the long run.

Thank you peter and countrypaul for your valuable help. I really do appreciate it :slight_smile: .

(deleted)

Hello Peter,

The way i got it to work now is by using thread.h library, and using thread method to run my function which is responsible to write on the SD card module. I am now currently experimenting on the suggestions by you and countrypaul about Strings,and trying to convert those to char arrays and store them in the nano's flash memory.

GitHub - ivanseidel/ArduinoThread: ⏳ A simple way to run Threads on Arduino, this is the thread library i used to solve my problem. Thank you guys. If i get good progress on the char array methods instead of strings, i will definitely update in this forum :slight_smile:

HI, I am using ESP32 with sim800L. I get data from the SD card and upload it via FTP over the server. I dont know if my code will help you in any way, but do read it and see if you can make any changes to yours.

#include <SPI.h>
#include <Wire.h>
#include <SD.h>
#include <SoftwareSerial.h>
SoftwareSerial MySerial(26, 27); // RX, TX

// TTGO T-Call pins
#define MODEM_RST            5
#define MODEM_PWKEY          4
#define MODEM_POWER_ON       23
#define MODEM_TX             27
#define MODEM_RX             26
#define I2C_SDA              21
#define I2C_SCL              22


#include "FS.h"
#include "SD.h"

#define IP5306_ADDR          0x75
#define IP5306_REG_SYS_CTL0  0x00

//char* file_name1 = "/basic.png";
//char* file_name1 = "/xl.xlsx";
char* file_name1 = "/data1.txt";
char char_buffer;
String string_buffer = "";
int buffer_space = 1000;
File File1;

SPIClass SPI1(HSPI);
#define MY_CS       33
#define MY_SCLK     18
#define MY_MISO     19
#define MY_MOSI     32

void setup() {
  Serial.begin(19200);
  MySerial.begin(19200);
 // pinMode(53, OUTPUT);

  delay(10);

  // Keep power when running from battery
  bool isOk = setPowerBoostKeepOn(1);
  Serial.println(String("IP5306 KeepOn ") + (isOk ? "OK" : "FAIL"));
  
setupSDCard();

  File1 = SD.open(file_name1);
  if (File1) {
    Serial.println("Opening the file: " + String(file_name1) + " done.");
  }else {
    Serial.println("Error opening " + String(file_name1));
    while(true);
  }
 setupModem();

 Wire.begin(I2C_SDA, I2C_SCL, 400000);
 
  Serial.println("Starting...");
  gprs_modem_function ();
  Serial.println("The end...");
}

void setupSDCard()
{
    SPI1.begin(MY_SCLK, MY_MISO, MY_MOSI, MY_CS);
    //Assuming use of SPI SD card
    if (!SD.begin(MY_CS, SPI1)) {
        Serial.println("Card Mount Failed");
    } else {
        Serial.println("SDCard Mount PASS");
        String size = String((uint32_t)(SD.cardSize() / 1024 / 1024)) + "MB";
        Serial.println(size);
    }
}

bool setPowerBoostKeepOn(int en){
  Wire.beginTransmission(IP5306_ADDR);
  Wire.write(IP5306_REG_SYS_CTL0);
  if (en) {
    Wire.write(0x37); // Set bit1: 1 enable 0 disable boost keep on
  } else {
    Wire.write(0x35); // 0x37 is default reg value
  }
  return Wire.endTransmission() == 0;
}

void setupModem()
{
    // Set modem reset, enable, power pins
  pinMode(MODEM_PWKEY, OUTPUT);
  pinMode(MODEM_RST, OUTPUT);
  pinMode(MODEM_POWER_ON, OUTPUT);
  digitalWrite(MODEM_PWKEY, LOW);
  digitalWrite(MODEM_RST, HIGH);
  digitalWrite(MODEM_POWER_ON, HIGH);
}

void loop() {
  // put your main code here, to run repeatedly:
}

byte gprs_modem_function (){
  byte reply = 1;
  int i = 0;
  while (i < 10 && reply == 1){ //Try 10 times...
    reply = sendATcommand("AT+CREG?","+CREG: 0,1","ERROR", 1000);
    i++;
    delay(1000);
  }
  if (reply == 0){
    reply = sendATcommand("AT+SAPBR=3,1,\"Contype\",\"GPRS\"","OK","ERROR", 1000);
    if (reply == 0){
      reply = sendATcommand("AT+SAPBR=3,1,\"APN\",\"airtelgprs.com\"", "OK", "ERROR", 1000);
      if (reply == 0){
        //reply = sendATcommand("AT+SAPBR=3,1,\"USER\",\"entelpcs\"", "OK", "ERROR", 1000);
        if (reply == 0){
          //reply = sendATcommand("AT+SAPBR=3,1,\"PWD\",\"entelpcs\"", "OK", "ERROR", 1000);
          if (reply == 0){
            reply = 2;
            i = 0;
            while (i < 3 && reply == 2){ //Try 3 times...
              reply = sendATcommand("AT+SAPBR=1,1", "OK", "ERROR", 10000);
              if (reply == 2){
                sendATcommand("AT+SAPBR=0,1", "OK", "ERROR", 10000);
              }
              i++;
            }
            if (reply == 0){
              reply = sendATcommand("AT+SAPBR=2,1", "OK", "ERROR", 1000);
              if (reply == 0){
                reply = sendATcommand("AT+FTPCID=1", "OK", "ERROR", 1000);
                if (reply == 0){
                  reply = sendATcommand("AT+FTPSERV=\"ftp.example.com\"", "OK", "ERROR", 1000);
                  if (reply == 0){
                    reply = sendATcommand("AT+FTPPORT=21", "OK", "ERROR", 1000);
                    if (reply == 0){
                      reply = sendATcommand("AT+FTPUN=\"username\"", "OK", "ERROR", 1000);
                      if (reply == 0){
                        reply = sendATcommand("AT+FTPPW=\"password\"", "OK", "ERROR", 1000);
                        if (reply == 0){
                          reply = sendATcommand("AT+FTPPUTNAME=\"" + String(file_name1) + "\"", "OK", "ERROR", 1000);
                          if (reply == 0){
                            reply = sendATcommand("AT+FTPPUTPATH=\"/\"", "OK", "ERROR", 1000);
                            if (reply == 0){
                              unsigned int ptime = millis();
                              reply = sendATcommand("AT+FTPPUT=1", "+FTPPUT: 1,1", "+FTPPUT: 1,6", 60000);
                              Serial.println("Time: " + String(millis() - ptime));
                              if (reply == 0){
                                if (File1) {
                                  int i = 0;
                                  while (File1.available()>0) {
                                    char_buffer = File1.read();//Takes one by one character in char_buffer from File.read()
                                    Serial.println("char_buffer: " + String(char_buffer));
                                    string_buffer.concat(char_buffer);//Concats every single character from char_buffer and makes string_buffer
                                    Serial.println("string_buffer: " + String(string_buffer));
                                    i++;
                                    if (i == buffer_space) {
                                      sendATcommand("AT+FTPPUT=2," + String(buffer_space), "AT+FTPPUT=2,10", "ERROR", 1000);
                                      sendATcommand(string_buffer, "OK", "ERROR", 5000);
                                      string_buffer = "";
                                      i = 0;
                                    }
                                  }
                                  if (string_buffer != ""){
                                    sendATcommand("AT+FTPPUT=2," + String(i), "AT+FTPPUT=2,10", "ERROR", 1000);
                                    sendATcommand(string_buffer, "OK", "ERROR", 5000);
                                    sendATcommand("AT+FTPPUT=2,0", "OK", "ERROR", 1000);
                                  }
                                  File1.close();
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  return reply;
}

byte sendATcommand(String ATcommand, String answer1, String answer2, unsigned int timeout){
  byte reply = 1;
  String content = "";
  char character;
 
  //Clean the modem input buffer
  while(MySerial.available()>0) MySerial.read();

  //Send the atcommand to the modem
  MySerial.println(ATcommand);
  delay(100);
  unsigned int timeprevious = millis();
  while((reply == 1) && ((millis() - timeprevious) < timeout)){
    while(MySerial.available()>0) {
      character = MySerial.read();
      content.concat(character);
      Serial.print(character);
      delay(10);
    }
    //Stop reading conditions
    if (content.indexOf(answer1) != -1){
      reply = 0;
    }else if(content.indexOf(answer2) != -1){
      reply = 2;
    }else{
      //Nothing to do...
    }
  }
  return reply;
}

Hi bhupiister,

Thank you for sharing :). Ill have a run through:)