I am trying to record data from a nearby cellular base-station using Arduino and and GSM/GPSR board. The data is written to a connected SD card reader and onto a file in the SD card.
This process has been without a doubt the most aggravating thing ever. After making sure data was being recorded using the serial port reader, I leave the sketch and Arduino running for a few days so i can collect a large amount of data for analysis. I had two Arduinos running.
This morning I checked one of them and after removing the SD card and inserting it I discovered the file it should have been recording on was missing. I later found in the dreaded FOUND.000 flder divided into CHK files, meaning the data was corrupted somehow. I checked the second one later today and found that it too was corrupted, the difference being the files were empty and gobbledegook was there instead as seen in the included file.
So in a vain hope of reaching an answer, I have come here, this is my source code, is there anything wrong with it? A connection is established with the SD card and cell station, an AT command is passed so I get signal strength and then it is written to the SD card. The cards were pre-formatted to FAT32 before the tests were run. A README file was put on the SD card outlining the start time of the tests.
#include "SIM900.h"
#include <SD.h>
#include <SoftwareSerial.h>
#define MEM_PW 8
const int chipSelect = 4;
void setup()
{
Serial.println("Initializing SD card...");
// // make sure that the default chip select pin is set to
// // output, even if you don't use it:
pinMode(10, OUTPUT);
//Serial connection.
Serial.begin(9600);
Serial.println("GSM Shield testing.");
// see if the card is present and can be initialized:
if (!SD.begin(chipSelect)) {
Serial.println("Card failed, or not present");
// // don't do anything more:
return;
}
Serial.println("Card initialized.");
Serial.println("Card initialized.");
//Start configuration of shield with baudrate.
//For http uses is reccomended to use 4800 or slower.
if (gsm.begin(9600))
Serial.println("\nstatus=READY");
else Serial.println("\nstatus=IDLE");
};
void loop()
{
serialhwread(); //for inputting commands
serialswread(); //for reading in data from the Arduino
delay(5000);
};
void serialhwread()
{
gsm.SimpleWriteln("AT+CSQ");
delay(10);
}
void serialswread()
{
String reply = "";
char datain;
while(gsm.available()>0) {
datain=gsm.read();
if(datain>0) {
reply.concat(datain);
}
reply.replace(":",",");
reply.replace("+CSQ","");
reply.replace("OK","");
reply.trim();
}
reply.concat('\r');
// open the file. note that only one file can be open at a time,
// so you have to close this one before opening another.
File dataFile = SD.open("RSS.csv", FILE_WRITE);
Serial.println(reply);
// if the file is available, write to it:
if (dataFile) {
dataFile.println(reply);
dataFile.close();
// print to the serial port too:
}
// if the file isn't open, pop up an error:
else {
Serial.println("error opening file");
}
}
int find_text(String needle, String haystack) {
int foundpos = -1;
for (int i = 0; (i < haystack.length() - needle.length()); i++) {
if (haystack.substring(i,needle.length()+i) == needle) {
foundpos = i;
}
}
return foundpos;
}
The String library has a reputation for causing problems on the Arduino due to the small amount of memory available which may be exacerbated by leaving the program running for several days during which a lot of String manipulation takes place every 5 seconds..
reply.concat(datain);
How many bytes will the reply String contain in the worst case ?
Valten1992:
So in a vain hope of reaching an answer, I have come here, this is my source code, is there anything wrong with it?
Except that you are using String-objects in your code which I would never ever do on an Arduino (I'd use C-Strings only) and that your writing interval with opening/closing the file every 5 seconds is very short, which will lead to defective sectors on the SD card which must be replaced by the SD controller with spare sectors every couple of days or so, I cannot find a big problem with the code.
So please describe your circuit a little bit more.
Which SD-module are you using?
Please link to the exact model of SD-module or SD-shied you are using, perhaps something is wrong with your SD card circuit.
This morning I checked one of them and after removing the SD card and inserting it I discovered the file it should have been recording on was missing. I later found in the dreaded FOUND.000 flder divided into CHK files, meaning the data was corrupted somehow. I checked the second one later today and found that it too was corrupted, the difference being the files were empty and gobbledegook was there instead as seen in the included file.
SD cards are somewhat finicky about clean, 3.3V power. The fact that you have a cellular radio running off the chip may indicate RF interference. Even if you only read, the data-comm is two-way.
SD cards are not all the same. The author of the SD library recommends "industrial' type cards. You can read about it here: HERE
The SD library requires a significant amount of SRAM. As UKHeliBob said, Strings could be an issue. Now, I have never had any issue with 'em, but some folk do. There are other ways to code without String, but not as convenient.
Personally, I never like to run an application along-side the SD library. I believe in off-loading the SD to a dedicated processor running at 3.3V from a well designed regulator with plenty of filtering. When the SD writes, it can draw a goodly amount of current which could cause issues if the bulk capacitor is not properly installed and sized. Other issues could be a small amount of resistance in the Vcc line which would cause a voltage drop during high-current writes. Example of off-loading.
Shielding is important around RF. This cannot be overstated.
Bill Greiman is ever improving the SD library. The one bundled with the Arduino install has a lots of hair.
and that your writing interval with opening/closing the file every 5 seconds is very short, which will lead to defective sectors on the SD card which must be replaced by the SD controller with spare sectors every couple of days or so
Any quality card designed for camera use will have 'weal leveling' in the card controller. I have opened and closed a card every second for weeks without any issues. The SD library ensures that a full sector is read or written and wear leveling is a sector-based technology. Unlike a hard disk that can read and update the same track/sector/cluster, SD cards (typically) do not work that way. I say "typically" because manufacturers are responsible for the wear-leveling algorithms and there is always a possibility that some 'junk" could find its way into the consumer stream.
Pro of this SD-module:
It comes with a 5V==>3.3V voltage regulator, so cabling the VCC power is easy:
Just connect Arduino 5V connector to SD-module 5V VCC and done.
Contra of this SD-module:
It comes with NO LEVELSHIFTER for the I/O pins.
So I/O pins of the SD-card work with 3.3V.
But I/O pins of the Arduino work with 5V.
You are NOT ALLOWED to connect the I/O pins of the Arduino directly to this SD-module.
Let me make a guess: You connected the I/Os of the SD-Module directly to your Arduino, didn't you?
This whole thing is giving me a headache. I am new to Arduino just so you know.
The SD card reader is directly connected by pins to their respective sockets, is doing it through a breadboard preferable?
EDIT: Saw the last post just now, yep, thats what I did. Crap.
I had no idea radio waves could affect direct writing to an SD card, is the link you supplied the best way to protect the module?
Ill try using bytes instead of strings, only used them as I am used to editing strings for file writing.
Would an alternative SD library like SDFat give me less problems? I am getting quite desperate.
Valten1992:
Ill try using bytes instead of strings, only used them as I am used to editing strings for file writing.
Would an alternative SD library like SDFat give me less problems? I am getting quite desperate.
First step: Keep cool!
Second step: Try using your SD card in your PC!
If you can format it in your PC, copy files to the SD card and read back from it, most likely the SD card is not demaged and can be used later again.
Third step: Look though your craft stuff what you have!
Do you have some SPI-levelshifter chip that can shift voltage levels 5V-3.3V in both directions?
Or do you have a handful resistors in the Ohm range of about 1K to 3.3K Ohms or so?
Do you have anything like that?
You just need the correct circuit to connect your Arduino with that type of SD-module, then it will work.
I'll be perfectly honest with you. This is a University project I started and my only experience is with coding the hardware and wiring part is beyond me.
I was just given two Arduino's, a GSM board and a big box of bits and bobs I had no idea what you used for. Guess I better roll up my sleeves and start working on it. I honestly thought most of the work would just be coding the thing, I figured one you wired it up and it didnt cook itself it would work ok. My mistake.
If you have very little experience with the hardware side you may be better off buying an SD card shield that will work without any changes being required.
Valten1992:
I'll be perfectly honest with you. This is a University project I started and my only experience is with coding the hardware and wiring part is beyond me.
I was just given two Arduino's, a GSM board and a big box of bits and bobs I had no idea what you used for. Guess I better roll up my sleeves and start working on it. I honestly thought most of the work would just be coding the thing, I figured one you wired it up and it didnt cook itself it would work ok. My mistake.
Perhaps you can buy another SD-module with these two features:
5V->3.3V voltage regulator on board
5V->3.3V level shifter circuit on board
If you want to use the SD-module you currently have with an Arduino UNO, you can do so (99% success guarantee) using the following voltage dividers on some connections using a few resistors (Image included as file attachment):
If you have very little experience with the hardware side you may be better off buying an SD card shield that will work without any changes being required.
I know a guy who can help me set up a breadboard so wiring it should not be impossible. I wish I saw that SD card shield linked in that topic earlier though. Is it safe to stack shields on top of each other on an Arduino like that?
FYI, looking through the resistors I have, I have 1 1K resistors, 10 10K Resistors and a couble of 220K's. I take it its unsafe to use a resistor that high in this case.
Valten1992:
FYI, looking through the resistors I have, I have 1 1K resistors, 10 10K Resistors and a couble of 220K's. I take it its unsafe to use a resistor that high in this case.
You don't need the exact values to create a working voltage divider, but
the resistors used should be in the 1K ... 10K range
and the voltage division should match
When using the 1.8K/3.3K combination, you get
5V * 3.3 / (1.8+3.3) = 3.24V
When using the 1K/2.2K combination, you get
5V * 2.2 / (1+2.2) = 3.43V
I think all combinations that lead to a voltage in the range 3.0V ... 3,6V should be OK for the SD card.
Valten1992:
FYI, looking through the resistors I have, I have 1 1K resistors, 10 10K Resistors and a couble of 220K's. I take it its unsafe to use a resistor that high in this case.
IMO:
I never recommend a resistor voltage divider for SD. Why? Because the stray and inherent capacitance of the wiring + SD module and the resistors create an RC network which will degrade the digital signaling: that is, the square wave will have slope on the leading and falling edges. This slope causes timing concerns and while many experimenters "get away with it" you are setting your project up for potential weird and difficult to diagnose problems.
