Problem with writing SD card files

Hello,
First of all, I’m not sure if I’ve posted this in the correct sub-section of the forum - apologies if not !

As part of a larger sketch, I need two “Temporary” log files (one for General messages, G.tmp and one for Error messages, E.tmp) I am using a Mega2560 Rev 3 and an Ethernet card, by the way ! I wrote a function to manage updating these two files, but I seem to be getting interaction between one file access and the other. (I have ‘poured’ over the code for days now and it is driving me nuts !!) In the attached code (can’t find the # button so going to attach files instead), the function concerned is TempLogMsg(), and parameters indicate whether it is General file or Error file, and whether to create a new file or append to an existing one. I have included lots of debugging statements to try to check what’s going on, but the most valuable ones are the characters that I prepend the message with.
I have included two functions as part of my debug activity to display the General and Error files to aid my debug
I’m posting the results displayed on the Serial Monitor in the second attachment

The significant issue is that when creating the General file (G.tmp), the Error file “seems” to get updated with the same message (prepended by G1). Then when creating the Error file, the General file “seems” to get updated with the same message (prepended by E2) but truncated !!!

I hope that I have given sufficient explanation, but if not, let me know and I’ll try to make it clearer !

No doubt, someone will find a glaring error in my code and I shall be suitably embarressed, but I’ve got to share this problem with someone, or I’ll start going round the bend !!

TempLogMsgSketch.txt (7.46 KB)

TempLogMsg - SM output.txt (770 Bytes)

 ============ Creating G.tmp ==================
Writing to G.tmp
chFileMode = C
File removed - G.tmp
Record is - Some text

------- Start of display of G.tmp - 1 -------
00:00  -  G1Some text
------- End of display of G.tmp --------

------- Start of display of E.tmp - 2 -------
00:00  -  E2And some other text
------- End of display of E.tmp ------


 ============ Creating E.tmp =============
Writing to E.tmp
chFileMode = C
File removed - E.tmp
Record is - And some other text

------- Start of display of G.tmp - 3 -------
00:00  -  G1Some text
------- End of display of G.tmp --------

------- Start of display of E.tmp - 4 -------
00:00  -  E2And some other text
------- End of display of E.tmp ------
Finished creating temporary logs

E.TMP00:00  -  E2And some other text

G.TMP00:00  -  G1Some text

Only change I made to your sketch, was to the SD cs_pin which in my case is 53, works as shown. Only comment I would make is you are seemingly using pin 4, but failed to change pinMode(53, OUTPUT); to pinMode(4, OUTPUT); on line 35?

Regards,

Graham

I can't help you much, but the Networking section [EDIT: oops, Storage section] handles SD issues. Also, the IDE SD library is 5 years out of date, and you should try the newer one on Bill Greiman's github page. He's the expert.

oric_dan:
I can't help you much

So why bother?

oric_dan:
Also, the IDE SD library is 5 years out of date

Yes it is but still works adequately for this purpose when configured correctly.

oric_dan:
and you should try the newer one on Bill Greiman's github page. He's the expert.

While this is true, and a good recommendation it was not the cause of the problem, and while I may not be a Faraday member, I did identify it for him!!

The SD_cs pin needs to be set as an output, however he is using pin 4, but set pin 53 as output..... Your answer while not especially helpful appears to have the intent of belittling my reply! Why? If you specifically wish to be pedantic, I would suggest the Storage section as a better place to seek help with storage issues than the Networking section!

Regards,

Graham

"the Error file "seems" to get updated"

So what is actually in the files when you finish? Is it this?

G.tmp - "And some ot"
E.tmp - "And some other text"

or is it this?

G.tmp - "Some text"
E.tmp - "And some other text"

What I would suggest for now is taking out the display parts of your program. Write to G, then E, then confirm they work after the fact. If that works, display both, but only at the end. Simplify till it works, then slowly add functions until it breaks, then you'll have your answer.

I second the use of a new SD library. SDFat is much better; small footprint and more stable.

ghlawrence2000:
E.TMP00:00  -  E2And some other text

G.TMP00:00  -  G1Some text

When the pin mis-match problem is rectified, and the sketch runs as intended, the contents of the files was shown by myself a few posts prior..... Do people read anymore?

Graham

ghlawrence2000:
So why bother?

Yes it is but still works adequately for this purpose when configured correctly.

While this is true, and a good recommendation it was not the cause of the problem, and while I may not be a Faraday member, I did identify it for him!!

The SD_cs pin needs to be set as an output, however he is using pin 4, but set pin 53 as output..... Your answer while not especially helpful appears to have the intent of belittling my reply! Why? If you specifically wish to be pedantic, I would suggest the Storage section as a better place to seek help with storage issues than the Networking section!

Regards,

Graham

Sheesh, lighten up. I didn't tear your solution apart, did I!!

I just informed OP of other information. Sheesh. What did he say:

I'm not sure if I've posted this in the correct sub-section of the forum

And I'm sure he wasn't aware that the SD library is 5 years out of date. Few people are. I only discovered this myself a couple of weeks ago, when the old library crapped out on doing some very trivial things. For instance, if you combine the listfiles demo with the read/write demos, which is the first obvious thing to try, then listfiles no longer works after a read/write. DUH!

For my part, I've spent lightyears of time fixing crap in 3rd party and other libraries, which the authors were too d*mn lazy to test properly.

What's the other thing he said:

(I have 'poured' over the code for days now and it is driving me nuts !!)

If I can help save someone some time by giving them some helpful information, then you'll just have to live with it. Sheesh.

ghlawrence2000:
When the pin mis-match problem is rectified, and the sketch runs as intended, the contents of the files was shown by myself a few posts prior..... Do people read anymore?

Graham

I was just asking what the OP was getting in his file. If your solution works for him, he'll tell you thanks and ignore my post as he should. If not, I wanted to see what was there vs. what his serial print was saying. Not trying to offend you, you don't need to get angry at me.

No matter what pin I use for SPI chip select, the hardware SS must be set to output.

In this case if he did not set pin 4 to output or isn't using it then that is a problem itself.

I really don't want to go poring through a big sketch looking for 'a' problem not just because it's big but because I'm likely to come out with a list. That's just the nature of so much code and I'm not caffeinated atm.

One possible solution is to write a single log file with time stamps and line identifiers up front then pass that through some other code that generates the separate files. It will actually write faster.

Hi all,
Thanks for all the comments - a shame that it caused a bit of agro !!
First of all, the CS issue ! This code was extracted from the main sketch which can access the SD card and ethernet successfully in other areas, so these settings DO work (I originally found this whole subject VERY confusing, the different pins being quoted here there and everywhere ! The fact is that the current settings DO work on my Mega.
I checked the contents of the SD card in my card reader and can confirm that the files are as shown in the Serial Monitor by the Display functions in my reduced sketch. (I have attached them for reference) As I said in my original post, the fact that BOTH files have an E2 prefix shows that the record was written in an Error file write access but the General file was updated also with the same record but truncated. I hope that I’ve made this more understandable. THIS is my problem !
Graham, did you say that you had managed to make the sketch run with the correct result (“Some text” in the G.tmp file and “And some other text” in the E.tmp file) ?? (Can’t scroll up to your posting whilst editing this).

P.S. Just found that I can’t upload files with a .tmp extension, so I have renamed them as .txt

G.txt (23 Bytes)

E.txt (33 Bytes)

Are you using the same buffer to write both files?

GoForSmoke:
Are you using the same buffer to write both files?

As you will see from the sketch file that I posted originally, the text to be written is set in one of the function parameters as literal data.

As I said in my original post, the fact that BOTH files have an E2 prefix shows that the record was written in an Error file write access but the General file was updated also with the same record but truncated.

You might want to dig a bit more into how that literal data gets into the file(s).

Same data does not necessarily mean same write. Truncated data leaves me suspecting the buffer was not clean when that operation was made. You may have to dig into the library to find out.

I started my original comments here with an indication that I'm not very experienced with using SD, and couldn't really do a decent job troubleshooting code without writing and testing it myself, which is one reason I mentioned the other forum section.

However, despite possible secondary repercussions, I'll make a couple more comments, whether these directly affect the present matter or not. SD has a lot of issues that can contribute to problems.

  1. as I understand it, the "older" libraries had issues with opening multiple files. I would definitely
    update to the newer SdFat library.

  2. SD libraries do a "lot" of background buffering, since apparently you can only write entire
    512-byte blocks to the card. So, even if writing only one byte, it'll read in an entire block,
    patch in the new data, and then write the entire block back to the card. This can be an issue.

  3. this I do know for sure - there are possible problems when using SD with other devices
    on the SPI buss, such as ethernet and radios, etc. SD does not release the MISO line
    properly in the old libraries. I discovered this myself when trying to use SD with an RFM12
    radio [also SPI interface]. Bill Greiman has fixed the MISO problem in the new library
    updates. It may or may not be a problem in different situations.

  4. people also have CS problems when using multiple devices on the SPI buss. In my own
    programs, I had all kinds of issues with this on my mega2560 board which has SD,
    ethernet, RFM12 radio, and LCD shield, until I wrote a routine that would "specifically"
    de-assert every CS line prior to using any of the 4 SPI peripherals.

  5. SD uses a lot RAM for buffering. On my UNO-328 board, there is not enough RAM to
    handle both SD and the RFM12 radio libraries, and the program would crash. But
    this shouldn't be a problem with a mega2560 board.

So, whether these items are relevant to the current problem or not, there's all kinds of stuff hidden in the bushes with SD.

Hi GoForSmoke and oric_dan,
Thanks for your comments this morning - some REALLY interesting stuff in there !
To follow up on a previous suggestion, I have altered the sketch (in CreateTemporaryLogs) to follow this sequence:-

  1. Write the G.tmp file
    Display the G.tmp file (all as expected)
  2. Write the E.tmp file
    Display the E.tmp file (all as expected)
  3. Re-display the G.tmp file (data stored exhibits the same ‘aliasing’ property as seen before)

Have a look at NewSketchSM for the Serial Monitor output !

What REALLY is interesting is the prefix to the stored message (The G or E show the type of write being performed whereas the following numeral shows the 1st, 2nd, … use of the TempLogMsg function. So the first prefix is G1 showing a write to the General file in the FIRST call to the function. The second reference is E2 - Error file, 2nd use. BUT, redisplaying the G.tmp finally, ALSO shows E2 which to my mind implies that THAT data has corrupted the General file.

My next step WILL be to download SDFat library and see how that affects things !

NewSketch.txt (7.54 KB)

NewSketchSM.txt (737 Bytes)

Remittub:
My next step WILL be to download SdFat library and see how that affects things !

I have now downloaded the SdFat library from GitHub and converted my sketch. Fortunately/Unfortunately (??) this has not altered ANYTHING !! Same symptoms as earlier this morning

Could someone review my coding of the function TempLogMsg() ?? Better still, if you have a Mega + Ethernet card, could you try running it ?

As indicated way back in my "very first" [and contested] post, I would try to get Bill Greiman, the SD expert, involved in your issues. On the Storage section. He might be able to spot your problem right away.

I’ve asked Bill to have a look at this thread, (http://forum.arduino.cc/index.php?topic=286112.0)

… but, in the mean time, I thought that I would try a different microSD card, just for the hell of it !
And the code ran fine, with the expected results !! (This was a Samsung 512MB) Back to the original Nokia 512MB - failure as reported all along. I really don’t understand this !!!

Bill will know. He did say that even 'official' looking SD cards may be bogus items. I tried running CardInfo example on my 2 cards, both bought at big box stores, and one printed out weird info. Bill said it was probably an illicit knockoff.

See my post in storage.