What is my Arduino doing beside executing MY code?

Projects General Guidance
1

HI,

I am building a datalogger using a GPS and writing it as a csv file on an SD card.
The whole program executes in the loop() , where it:

  • Reads button state
  • Reads serial data from the GPS (9600 baud) via Softwareserial
  • Writes a line to the csv file (only every 200ms)

But I am having a bit of an issue with dropped chars on the GPS serial, so I decided to see how long I was using in the write csv function as I suspected that was the culprit.
But the timings for the csv write function varies quite a bit, but in a pattern. Following is the ms it takes to call the function (5 executions pr. second):

166
165
165
165
165
165
164
164
160
149
148
142
136
132
128
125
115
107
105
104
100
21
20
19
20
20
19
22
20
19
52
21
20
19
19
20
19
22
20
20
19
19
20
19
19
20
20
22
19
20
19
19
24
20
19
87
155
Dropped bad nmea frame: $GPGGA,211652.600,5700.7072,N,01002.3501,E,1,9,0.99,-3.9,M,42.5
136
Dropped bad nmea frame: $GPGGA,211652.800,5700.7072,N,01002.3501,E,1,9,0.99,-3.9,M,42.54,031213,,,A*68

135
Dropped bad nmea frame: $GPGGA,211653.000,5700.7072,N,01002.3501,E,1,9,0.99,-3.9,M,42.54,031213,,,A*61

164
164
164
165
164
166
165
169
166
165
165
165
165
155
156
148
140
137
132
131
124
117
115
113
107
102
101
94
92
95
20
19
20
19
19
20
23
19
19
20
19
19
20
20
19
19
23
19
19

The above pattern repeats itself, taking ~160ms, then falling to ~20ms, then rising again resulting in dropped characters on the serial interface.
So it seems that something is slowing my code down and making me miss characters on the serial interface.
The csv function should be quite static in its timing, it just concats some strings together (using char buffers), and writes a line to the SD card.

Any ideas on why the timing is varying to much? Is the CPU off doing something else other than executing my code?

2

Any ideas on why the timing is varying to much? Is the CPU off doing something else other than executing my code?

Not without seeing your code, I'm afraid. It's unlikely that it is a result of your Arduino "doing something else".

3

There are various things going on in the Arduino framework that rely on interrupts. Since the Arduino can only execute one interrupt handler at a time, if there is conflict then something is going to be delayed. Since the SoftwareSerial handling is time-sensitive, that might cause bit errors and lost data.

I haven't checked the implementation but I'd expect SoftwareSerial to have an internal buffer (probably of a few dozen bytes) so it would not be possible to simply overflow the serial buffer unless you were consistently leaving chars unread. If you recorded the amount of unread data then you could tell how much use you were making of that buffer and whether that was a potential problem. Without seeing your code I can't tell whether you're likely to leave bytes unread, but it should not be necessary to.

The SD write execution time will vary because the amount of work that has to be done to allocate space on the card and find the end of the current file block will vary.

4

I didn't really notice when I first read your question, but I see you are using 9600 baud for the GPS. Most GPS units will handle 115200 baud, which is about 12 times faster than 9600 (87 microseconds/character vs. 1042 microseconds/character). If the Software Serial does not handle 115200, you can always use 57600 and leave more time, or go to a unit with more than 1 hardware serial port.

You are also running at 5Hz, and a GPGGA sentence is usually around 70 characters, or about 73 milliseconds. Depending on what other sentences you have coming in, you aren't leaving a lot of time for processing the line(s) and the (high overhead) SD write.

5

lar3ry:
It's unlikely that it is a result of your Arduino "doing something else".

I dunno, the NSA is getting into everything these days :wink:

6

It might just be sulking.

7

lar3ry:
Not without seeing your code, I'm afraid. It's unlikely that it is a result of your Arduino "doing something else".

The sketch is getting quite large, so I hoped that I had given you enough information to make a guess. The code is available here Bitbucket for those of you that have the time to read through it :slight_smile:

PeterH:
I haven't checked the implementation but I'd expect SoftwareSerial to have an internal buffer (probably of a few dozen bytes) so it would not be possible to simply overflow the serial buffer unless you were consistently leaving chars unread. If you recorded the amount of unread data then you could tell how much use you were making of that buffer and whether that was a potential problem.

The problem definitely looks like a buffer overrun in SoftwareSerial, so I suspect you are right here. I am just looking at why the write method varies so wildly in execute time.

PeterH:
The SD write execution time will vary because the amount of work that has to be done to allocate space on the card and find the end of the current file block will vary.

Aahhh that might be. At least it is easy for me to check...will try tonight :slight_smile:

lar3ry:
I didn't really notice when I first read your question, but I see you are using 9600 baud for the GPS. Most GPS units will handle 115200 baud, which is about 12 times faster than 9600 (87 microseconds/character vs. 1042 microseconds/character). If the Software Serial does not handle 115200, you can always use 57600 and leave more time, or go to a unit with more than 1 hardware serial port.

I have tried raising the baud rate, but could not really get it to work. It is this gps: Adafruit Ultimate GPS Breakout - 66 channel w/10 Hz updates [PA1616S] : ID 746 : Adafruit Industries, Unique & fun DIY electronics and kits
So I am not sure it can handle more than 9600...

lar3ry:
You are also running at 5Hz, and a GPGGA sentence is usually around 70 characters, or about 73 milliseconds. Depending on what other sentences you have coming in, you aren't leaving a lot of time for processing the line(s) and the (high overhead) SD write.

It am getting both GPGGA and GPRMC, so the problem is even worse :slight_smile: However, as far as I understand the program is not paused during the full time of the read, but only a very short time while receiving each character.

I might very well be, I know I would if I had to do it's job :grin:

8

bomadsen:
The sketch is getting quite large, so I hoped that I had given you enough information to make a guess. The code is available here Bitbucket for those of you that have the time to read through it

OK, I downloaded it, and had a look. It won't compile for me, because it's getting an error in the SD library. Error is:
(x86)\Arduino\libraries\SD\src\utility\Sd2Card.cpp:35: error: 'SPI' was not declared in this scope

I'll see if I can figure out what's wrong. I had trouble with this library before, and ended up using the SDFat library.

The problem definitely looks like a buffer overrun in SoftwareSerial, so I suspect you are right here. I am just looking at why the write method varies so wildly in execute time.

I've been using AltSoftwareSerial, for reasons that I can't remember right now, but you might want to try that.

I have tried raising the baud rate, but could not really get it to work. It is this gps: Adafruit Ultimate GPS Breakout - 66 channel w/10 Hz updates [PA1616S] : ID 746 : Adafruit Industries, Unique & fun DIY electronics and kits
So I am not sure it can handle more than 9600...

Are you using the MTK3329 or the MTK3339? One of them has internal flash for data logging. When you say you tried raising the baud rate, but could not get it to work, do you mean that the baud rate changed, but it still failed in the same way, or were you unsuccessful in raising the baud rate? I'm 100% sure that GPS module will handle a higher baud rate. I am using an SKM53 module, and have run it as high as 115200 baud. I think it uses the same chipset as yours.
Here's a little sketch to check it out. It allows you to set the baud rate. Don't forget to change the baud rate in the sketch after you set it higher, and re-upload it.

You'll need to change your SoftwareSerial pins.

/*
Enter a packet starting with $PMTK, and ending with a *
In this sketch the * indicates the end of transmission.
The sketch takes the packet, adds a checksum and sends it to
the GPS.

Note: If you change the baud rate of the GPS, you will have to
close the Serial monitor, change the baud rate in Serial1
(or whatever Serial port you use) and restart it.

Set your Serial Monitor to No Line Endings. The sketch supplies those too,
and if you send them, they will get you out of sync.

This sketch assumes you are running an Arduino Mega2560, and
have a serial GPS on Serial1. Change this to suit your own
configuration. (Software Serial for Uno, etc.)

One packet type not mentioned in the document at
http://api.ning.com/files/L30xp1HTxtEUhhmBj4yYWIB31IDKs*xJNecJYvcEKZofuHJcZ3wnTMuZL5FeJC535I6DJbBZZE7FpJwnQPxgT9yKLCibZaZj/NMEA_Packet_Userl.pdf
is packet type 220. Here are a couple of samples, as you
would enter them into the Serial Monitor

$PMTK220,200*
$PMTK220,500*
$PMTK220,1000*
$PMTK220,2000*

These will set the reporting interval to 200, 500, 1000,
and 2000 milliseconds, or 5, 2, 1, .5 Hz,  respectively.
*/
//#include <AltSoftSerial.h>
//AltSoftSerial gpsSerial;

#include <SoftwareSerial.h>
SoftwareSerial gpsSerial(8,9);  // Rx,Tx

char sentence[80];
int idx = 0;

void setup() {
  // initialize both serial ports:
  Serial.begin(57600);
  gpsSerial.begin(9600);
}

void loop() {
  // read from Serial1, send to Serial:
  if (gpsSerial.available()) {
    Serial.write(gpsSerial.read());
  }

  if (Serial.available())  {
    char inbyte = Serial.read();
    sentence[idx++] = inbyte;
    sentence[idx] = 0;
    if (inbyte == '*') {
      for (int i=0; i < strlen(sentence); i++) {
        Serial.print(sentence[i]);
      }
      Serial.println();
      sendConfig ();
      sentence[0] = 0;
      idx = 0;
    }
  }
}

void sendConfig () {
  unsigned char CS = 0;
  for (int i=1; i< strlen(sentence)-1; i++) {
    CS ^= sentence[i];
  }
  gpsSerial.print(sentence);
  if (CS < 10) gpsSerial.print("0");
  gpsSerial.print(CS,HEX);
  gpsSerial.println();
}
9

Error is:
(x86)\Arduino\libraries\SD\src\utility\Sd2Card.cpp:35: error: 'SPI' was not declared in this scope

Try adding to the main sketch:

#include <SPI.h>
10

lar3ry:
OK, I downloaded it, and had a look. It won't compile for me, because it's getting an error in the SD library. Error is:
(x86)\Arduino\libraries\SD\src\utility\Sd2Card.cpp:35: error: 'SPI' was not declared in this scope

Hmmm, I don't understand this. I run it in a totally vanilla Arduino 1.0.5 environment on windows. (sorry for not mentioning this in the first post).

lar3ry:
I've been using AltSoftwareSerial, for reasons that I can't remember right now, but you might want to try that.

I might give this a try, however it would be nice to able to use the standard libraries. However having read the description of SoftwareSerial on the AltSoftwareSerial page , it gave me some doubts about SoftwareSerial :slight_smile:

lar3ry:
Are you using the MTK3329 or the MTK3339? One of them has internal flash for data logging. When you say you tried raising the baud rate, but could not get it to work, do you mean that the baud rate changed, but it still failed in the same way, or were you unsuccessful in raising the baud rate? I'm 100% sure that GPS module will handle a higher baud rate. I am using an SKM53 module, and have run it as high as 115200 baud. I think it uses the same chipset as yours.

I am not sure if it's the MTK3329 or the MTK3339, but it definitely have an on-chip buffer for logging.
And I am sorry for not being clear about my baud problem. When I try to use higher baud rates than 9600 I cannot get reliable communication with the GPS...or any communication at all.
I tried using different baud rates yesterday, and get it up and running on 57600 a couple of times. At that baud rate the performance impact on the rest of the code was much less than at 9600 baud, so it seems promising.

I think tonight's task must be to get GPS communication up and running at 57600 baud!

11

Nick's suggestion to include SPI.h did the trick. Thanks Nick!

A little testing, using the code I posted, shows me that AltSoftwareSerial, while it handles 115200 baud to a degree, will drop characters at 10Hz reporting rate. I'm testing it now at 57600. I'll let it run a while. One possible route is to use a Mega2560, which has three hardware serial ports. Hmm... just thought of something. That 115200 test may be because I was running Serial to the PC at 57600. Will check that later.

I am not sure if it's the MTK3329 or the MTK3339, but it definitely have an on-chip buffer for logging.

Does have or does not have?

Question.. are you going to be hooking this up to a CanBus? OBDII perhaps?

I'll be looking at this later (have to go out for a while), and I think I'll try hooking up the switch and LEDs and such, and give it a good test.

12

As your not taking input from the PC you can make use of the RX pin and use hardware serial on an Uno. Just set serials baud rate to the correct value for the GPS and connect the GPS TX to the Arduino RX.

Mark

13

lar3ry:

I am not sure if it's the MTK3329 or the MTK3339, but it definitely have an on-chip buffer for logging.

Does have or does not have?

The chip I have definitely does have an on-chip buffer :slight_smile:

lar3ry:
Question.. are you going to be hooking this up to a CanBus? OBDII perhaps?

Yup, that was the plan. I have been looking at datalogging for my trackdays, and usable commercial systems start at around $1500. I thought the Arduino way would make for an interesting challenge, and perhaps even save me a $ or two :slight_smile:

holmes4:
As your not taking input from the PC you can make use of the RX pin and use hardware serial on an Uno. Just set serials baud rate to the correct value for the GPS and connect the GPS TX to the Arduino RX.

I had been thinking about that, but would I still be able to 1) Upload a new version of the program, 2) Get diagnosting output on the computer via Serial.println()?

14

I had been thinking about that, but would I still be able to 1) Upload a new version of the program, 2) Get diagnosting output on the computer via Serial.println()?

  1. Yes just unhook the pin to the arduinos RX.

  2. Yes - there is no connection between the Rx and Tx of the Uart it's really two uni-directional links not one bi-directional link. Just don't hook the Arduinos TX to the GPS.

Mark

15

Using the hardware serial RX is a great idea! The only downside I see (and it isn't much of a downside), is that the GPS and PC comm program need to be at the same baud rate.

I tried the AltSoftwareSerial with 115200, and Serial also set at 115200, and even increased the AltSoftwareSerial 80 character buffer to 128 characters, but it still overran the AltSoftwareSerial buffer (I think that's what was causing the garbled and/or dropped characters).

Something I've always wondered about OBDII. I have a 2000 Ford F-150, so I don't think it's CanBus. Could I, with either straight OBDII, or Canbus through the OBD connector, send commands to control something like the throttle? As well, could I read things that are not usually read, like actual fuel usage? (assuming, of course, that there is such a thing to be read).

I've done some work on a CanBus system, though it was for a bed designed for quadriplegics, and still have some prototype boards with CanBus chips on them.

16

I generally off-load SD processing to a separate uC
http://forum.arduino.cc/index.php?topic=154864.0
If you want to go more professional, an under $5 3.3 volt Mini Pro from Hong Kong will work great, too.
http://www.ebay.com/itm/New-Pro-Mini-atmega328-3-3V-8M-Replace-ATmega128-Arduino-Compatible-Nano-/130905365831. The 3.3V saves the hassle of the level conversion, too.

Are you really needing the $GPGGA sentence. Since you are capturing $GPRMC, you may not and therefore remark it from the Adafruit library which will open some headroom.

Ray

17

mrburnette:
I generally off-load SD processing to a separate uC
off load serial data recording to dedicated 328P and SD card - Storage - Arduino Forum
If you want to go more professional, an under $5 3.3 volt Mini Pro from Hong Kong will work great, too.
http://www.ebay.com/itm/New-Pro-Mini-atmega328-3-3V-8M-Replace-ATmega128-Arduino-Compatible-Nano-/130905365831. The 3.3V saves the hassle of the level conversion, too.

That makes perfect sense. I had been looking at the megas, but thought it a bit much to upgrade just to get another uart.

mrburnette:
Are you really needing the $GPGGA sentence. Since you are capturing $GPRMC, you may not and therefore remark it from the Adafruit library which will open some headroom.

Good catch! Only thing I need from the GPGGA that I can't get from GPRMC is HDOP, but the value of that is questionable. That might actually free up quite a bit, specially at 9600 baud.

lar3ry:
I tried the AltSoftwareSerial with 115200, and Serial also set at 115200, and even increased the AltSoftwareSerial 80 character buffer to 128 characters, but it still overran the AltSoftwareSerial buffer (I think that's what was causing the garbled and/or dropped characters).

I think AltSoftwareSerial is plan B, hardware serial must be A :slight_smile:

lar3ry:
Something I've always wondered about OBDII. I have a 2000 Ford F-150, so I don't think it's CanBus. Could I, with either straight OBDII, or Canbus through the OBD connector, send commands to control something like the throttle? As well, could I read things that are not usually read, like actual fuel usage? (assuming, of course, that there is such a thing to be read).

You can't control anything interresting such as throttle or breakes. I know some of some systems that allows you to control stuff like language and units of the in-car displays, but nothing more than that. If I where in doubt about the ODB capabilities of my car, I would start by buying a cheap ELM327 bluetooth dongle and see if I could get some info on my phone. Cheap and effective.

18

Yes, definitely the Mega2560 for plan A.

Thanks for the info on the OBDII/CanBus stuff. I didn't think anything important would be controllable. Some things in cars really drive me nuts., and so many of the annoyances are microprocessor controlled. It's just too bad I can't get access to them. On my wife's previous car, the seat belt alarm was not only LOUD, but if I wanted to back the car out of the garage and park it 100 ft. away, it was a constant DIG DING DING.. Wife wouldn't let me put a switch in. :roll_eyes:

I wanted to ask you about your sketch. As posted, did it work to put data onto the SD card? I added a switch, but not the LEDs. I am having a little trouble following the logic in the code, and have yet to figure out how to get any Serial diagnostics out.

Good plan, getting rid of the GPGGA.

19

I wanted to ask you about your sketch. As posted, did it work to put data onto the SD card? I added a switch, but not the LEDs.

Yes - the code works... I have tested with a number of "old' SD cards from 512MB to 4GB but newer high-speed card are likely to not work. I used this Ladyada sketch as a starting point for my logger:

The FLAT-duino and the SD-card recorder were Instructables articles that I once published and then republished in this forum:
http://www.instructables.com/id/Build-A-FLAT-duino/
http://www.instructables.com/id/Under-8-Arduino-Serial-Data-Logger-Record-to-SD/

DO NOTE that I slaved the atmega328P-PU off if the 3.3V controller for the SD card. The supply to the LC Studio Card is 5.0V but all logic is working at 3.3V. The article mentions it, but you may need to use a voltage divider network to temper 5V write signals to the mega328P or you may simply use the same trick that V-USB uses which is a 68 Ohm resistor and 3.6V zener such as:
http://www.newark.com/vishay-semiconductor/1n5227b-tap/zener-diode-500mw-3-6v-do-35/dp/18M3528

I have not tested beyond 9600 BAUD input signals which in testing and use are taken from the log output of an Arduino Mega2560.

I am having a little trouble following the logic in the code, and have yet to figure out how to get any Serial diagnostics out.

The code is time-sensitive. Putting serial.print statements in the code will very likely cause it to fail. But, you may get lucky if you blast the output out at 115K BAUD or higher. The LED is there for diagnostics and to show activity. The proof of success is the file being created and written-to on the SD card.

If you are having trouble, try to locate some 'older' 1, 2, or 4G Kingston or MicroCenter "generic' cards. I found a bunch recently here at home in an old pouch that I carried around for an old MP3 players from years gone by.

Ray

20

Ahh... that probably explains it. I don't have any of the older SD cards any more. I have given them all away. No matter. As I mentioned earlier in that thread, I had trouble with the SD library, and ended up using the SDFat library. It doesn't matter too much, as I only wanted to be able to compile and run the sketch in order to help bomadsen. When I get around to doing my own logger, I'll try the SDFat library and write my own.

Thanks for the info, Ray.