Final graduation project facing RTC, RFID and HTTP request issues.

EDIT: issues are resolved. Please read Edit section.

Firstly, I am very sorry for the long post. And then...

I have been working on this project for quite some time now. It is part of a bigger final project that I am working on with my three co-students for our graduation. The project consists of a hardware part (Arduino Ethernet Rev.3 + a lot of components) and a .NET solution containing a MSSQL database, ASP.NET Web API 2 and an ASP.NET MVC website. In its whole, the project is a system to register and bill travellers in public transportation services like bus, train etc.

The hardware part is almost done, but I am struggling with a rather obscure programming issue which I desperately need help to overcome. Therefore I turn to you my fellow Arduino gurus hoping that some of you might help me. Just as a disclaimer, I should say that I am not particularly advanced in Arduino/C/C++ and my knowledge in microcontrollers is still on a rather novice level, although I am deeply interested in this. The sketch I am working with is my cook up of different web-client, RFID and RTC examples, which I have adapted to our purpose.

Following, I will explain the way my setup works (or at least how it should be working), and then I will describe the impending issues. I have provided the full source code in the link below.

Arduino source code GIST (Updated with new code revision.)


How it works
On start up, we will receive an IP-address from DHCP and indicate success by turning on the green RGB LED. After getting the IP-address we send a HTTP request (CheckApiConnection()) to a controller on my test API, which returns nothing but a "200 OK" response. This is done solely to verify that we can connect to the API and on success we indicate by switching the green RGB LED to blue. After this process the system is in "Ready" state.

When reading a RFID tag the process is as follows:

  1. The tag is read and converted (ReadRFIDTag()) into a string containing the tag ID in HEX format.

  2. A JSON string is built (BuildJSONObject()) by concatenating different values in the following format: {"TerminalId":10, "TagId":"", "Timestamp":"<timestamp from GetTimeStamp()>"}.

  3. While building the JSON string we call GetTimeStamp() which is returning a string containing date and time in MSSQL DateTime datatype format (YYYY-MM-DD hh:mm:ss[.nnn]).

  4. Once the JSON string is built, we send a HTTP request (SendRequest()) to a controller on the test API. This request is containing the JSON string which the API then acts upon.

  5. After sending the HTTP request we wait for the response (ReadResponse()). The received response is then decoded for content which is just a string returned from the API controller ("NEW", "CHECKIN" or "CHECKOUT") indicating which signal (sound and LED) the Arduino is supposed to give.

  6. After the above process, the Arduino is again in "Ready" state, waiting to read another tag.


The problem
NOW here comes the issues which is driving me crazy:

On start up, when we call CheckApiConnection() to check if we can connect to the API:

  1. First of all, sometimes (maybe 3 times out of 10) I don't get a response from the API. And by the nature of Arduino it is hard to debug this process, so I am not sure whether this is caused by the Arduino not sending the request properly OR the API not responding. This is a minor issue though.

  2. The issue which concerns me the most is when CheckApiConnection() was run on start up, and I get a succesful response, the system goes into a really obscure state. Now when I scan a tag and the JSON string is being built something is breaking. I have circled in on the issue and found that when GetTimeStamp() is called from within BuildJSONObject() the program hangs right there, printing out inexplicable text (like maybe half of the JSON string or nothing at all) to the Serial Monitor instead of the JSON string I expect.

Now if I Serial.println() GetTimeStamp() in the Setup() function after calling CheckApiConnection(), this works fine. The problem occurs only when GetTimeStamp() is called from within BuildJSONObject() AFTER running CheckApiConnection(), which I think is really strange and unexpected behavior. If I comment out the CheckApiConnection() the whole program runs as it should and I am able to scan multiple tags and sending requests to the API without problems.

WHAT could possibly cause this strange behavior? The only thing that comes to my mind is memory register failure (data overwriting each other unexpectedly), which should not be happening since my sketch is only using 64% of program storage space and 61% of dynamic memory. Could this be it? Does anyone have a clue. Any help is highly appreciated.

Thank you in advance.


EDIT (10-26-2016):
After studying the replies to this thread and getting help from other parts, I have been doing a major refactoring of the source code (The GIST above is updated). Besides from renaming a lot of functions and variables I have been doing a little memory optimizing. This actually solved the described issues I was facing.

To make it short: what did the trick was to move around with some String variables to make sure they are falling out of scope when not needed anymore as well as getting rid of some other variables. I managed to reduce the usage of dynamic memory by 13%. This apparently did the trick. Though I still don't understand the underlying problem. I wouldn't have thought that using 61% of dynamic memory could make the Arduino crash.

P.S. Be aware that a lot of function and variable names has changed so some of the names mentioned earlier in the thread are no longer applicable.

I believe that your use of the String class in, for example, getTimeStamp() will lead to a problem. You have defined a local object timeStamp and return a handle to it to the calling function, but the object has gone out of scope when the function getTimeStamp() returns.

but the object has gone out of scope when the function getTimeStamp() returns.

The copy that getTimeStamp() returns hasn't, though, so that is not the issue.

The overuse of Strings may be, though.

OP: Which Arduino are you using? I don't see an Arduino is your picture. I don't see an ethernet shield in your picture.

Nothing that your code is doing requires Strings. Everything that it does CAN be done with NULL terminated arrays of chars.

Your GET request is wrong.

    client.println("GET http://************.azurewebsites.net/api/status HTTP/1.1");

The Arduino only know how to use the http protocol, so the http:// part of the request is NOT supposed to be there.

You can NOT connect to one server and execute a script on another server, so the server part of the request is wrong.

    client.println("GET /api/status HTTP/1.1");

Is what you should have.

ALL string literals are, unless you take action otherwise, copied to SRAM at run-time. That is usually NOT necessary, so stop it from happening by using the F() macro:

client.println(F("GET /api/status HTTP/1.1));

      if (client.available()) {
        char c = client.read();
        httpResponse += c;
      }

Executing the outer while loop over and over is silly, when most of what you want to do in that while loop is execute this block of code. Make this a while statement, instead.

The copy that getTimeStamp() returns hasn't, though, so that is not the issue.

Oops. Yes. That is true. I had confused this behaviour of String with the use of String().c_str() in a function which gives a pointer to buffer which vanishes.

The only other comment I would have about the code is that pin 1 is an odd choice for softwareSerial since it is used by the hardware serial interface.

Hi both.

Thank you very much for your input.

OP: Which Arduino are you using? I don't see an Arduino is your picture. I don't see an ethernet shield in your picture.

I am using the retired Arduino Ethernet Rev.3 which is basically just an Arduino Uno with an Ethernet shield in one board.

Arduino Ethernet

The Arduino is located on the backside of the perfboard which doesn't show on the first picture.


Nothing that your code is doing requires Strings. Everything that it does CAN be done with NULL terminated arrays of chars.

Thanks for the tip. I will look into refactoring this. But as I understand this could not lead to the issues I have described?

The Arduino only know how to use the http protocol, so the http:// part of the request is NOT supposed to be there.

You can NOT connect to one server and execute a script on another server, so the server part of the request is wrong.

Will look into this too.

ALL string literals are, unless you take action otherwise, copied to SRAM at run-time. That is usually NOT necessary, so stop it from happening by using the F() macro:

client.println(F("GET /api/status HTTP/1.1));

I did this several places in the code, but I might have missed some places.

Executing the outer while loop over and over is silly, when most of what you want to do in that while loop is execute this block of code. Make this a while statement, instead.

I see your point. I will look into this as well.

Thanks again for your input. I will refactor my code and then update the GIST. I am still hoping that someone can help me fix the described issues.

But as I understand this could not lead to the issues I have described?

I think that your understanding is wrong then. Running out of memory can cause all kinds of problems. Misusing memory, by using Strings, can cause you to run out sooner than you think.

I did this several places in the code, but I might have missed some places.

So, you changed the code, but didn't post the changed code. Chances of getting help with code we can't see are somewhere between fat and not-a-snowball's.

Hi guys.

I have solved these issues. Please read the Edit section of the initial post.

I think that your understanding is wrong then. Running out of memory can cause all kinds of problems. Misusing memory, by using Strings, can cause you to run out sooner than you think.

My understanding of memory management and such are still somewhat poor. But I would never have thought that this could be the problem. But now I know.

So, you changed the code, but didn't post the changed code. Chances of getting help with code we can't see are somewhere between fat and not-a-snowball's.

Actually the source code you saw was up to date. What I meant was that I already had the F() macro in several places, but that I might had missed some places.

I have updated the GIST. If anyone has any additional tips on how I can optimize my code, I would be happy :slight_smile:

Thanks for your help.