Go Down

Topic: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library (Read 19865 times) previous topic - next topic

Koni

Slowly it's getting better, the code almost does what I want it to do. A *.BIN file is written, and when renaming that *.BIN file to *.TXT file or *.CSV file on my windows machine then the result looks like this:

Time: 21713,A-B: -0.071,A: 0.393,B: 0.464,Additional_Data...
Time: 21733,A-B: 0.026,A: 0.416,B: 0.390,Additional_Data...
Time: 21753,A-B: -0.026,A: 0.412,B:NUL 0.438,Additional_Data...
Time: 21773,A-B: 0.113,A: 0.516,B: 0.403,Additional_Data...
Time: 21793,A-B: 0.016,A: 0.354,B: 0.338,Additional_Data...
Time: 21813,A-B: 0.071,A: 0.435,B: 0.364,Additional_Data...
Time: 21833,A-B: 0.122,A: 0.274,B: 0.151,Additional_Data...
Time: 21853,A-B: -0.084,A: 0.116,B: 0.200,Additional_Data...
Time: 21873,A-B: -0.016,A: 0.293,B: 0.309,Additional_Data...
Time: 21893,A-B: 0.216,A: 0.538,B: 0.322,Additional_Data...
Time: 21913,A-B: -0.061,A: 0.416,B: 0.477,Additional_DaNULta...
Time: 21933,A-B: 0.229,A: 0.480,B: 0.251,Additional_Data...
Time: 21953,A-B: -0.039,A: 0.361,B: 0.400,Additional_Data...
Time: 21973,A-B: -0.132,A: 0.416,B: 0.548,Additional_Data...

There is somewhere a NUL character added every 512 bytes, but I couldn't figure out how to avoid that. Except of that the logged data looks ok, and apparently it's quite fast. The code that I logged the data above is as follows:

Code: [Select]


  dataString += String("Time: ");
  dataString += String(int(time));
  dataString += String("\t");
  dataString += String("A-B: ");
  dataString += String(ADCvalue_0-ADCvalue_1,3); 
  dataString += String("\t"); 
  dataString += String("A: ");
  dataString += String(ADCvalue_0,3);
  dataString += String("\t");   
  dataString += String("B: ");
  dataString += String(ADCvalue_1,3);
  dataString += String("\t"); 
  dataString += String("Additional_Data... ");
  dataString += String("\n");  //Create a new line on the SD card

  int length = dataString.length();

  if (length > 512)
  {
    noInterrupts();

    SDbufferString = dataString.substring(0,511);         //Create a string with 512 bytes for writing on the SD card.
    dataString = dataString.substring(511,length);        //Remove the 512 bytes for the SD card from the main string.

    char charBuf[512];
    SDbufferString.toCharArray(charBuf,512);              //Convert String to Char Array
     
    memcpy(pCache, &charBuf, 512);                        //Write binary data to cache

    if (!sd.card()->writeData((uint8_t*)pCache)) ; 

    bn++; //increment block number
    count = 0;
    memset(pCache, '0', 512);
    interrupts();
  }



The binaryToCsv conversion code does not work, there is only garbage to see later on the computer. But as already mentioned above, when renaming the BIN file to TXT on my computer then the result is as above.

The binaryToCsv code I used is the following:

Code: [Select]
void binaryToCsv() {
  uint8_t lastPct = 0;
  int buf[512];
  uint32_t t0 = millis();
  uint32_t syncCluster = 0;
  SdFile csvFile;
  char csvName[13];
  BufferedWriter bw;

  if (!myFile.isOpen()) {
    Serial.println(F("No current binary file"));
    return;
  }
  myFile.rewind();
  //if (!binFile.read(&buf , 512) == 512) error("Read metadata failed");
  // create a new csvFile
  strcpy(csvName, binName);
  strcpy_P(&csvName[BASE_NAME_SIZE + 3], PSTR(".CSV"));

  if (!csvFile.open(csvName, O_WRITE | O_CREAT | O_TRUNC)) {
    error("open csvFile failed"); 
  }
  Serial.println();
  Serial.print(F("Writing: "));
  Serial.print(csvName);
  Serial.println(F(" - type any character to stop"));
  bw.init(&csvFile);
  bw.putStr_P(PSTR("Time,")); //Write header at top of file
  bw.putStr_P(PSTR("A-B,"));
  bw.putStr_P(PSTR("A,"));
  bw.putStr_P(PSTR("B"));
  bw.putCRLF();

  uint32_t tPct = millis();
  while (!Serial.available() && myFile.read(&buf, 512) == 512) {
    uint16_t i;
    //read 512 bytes of data here i.e. 512 samples
    for (i = 0; i < 512; i++) {
     bw.putChar(buf[i]);
    }
   
    //bw.putCRLF();

    if (csvFile.curCluster() != syncCluster) {
      bw.writeBuf();
      csvFile.sync();
      syncCluster = csvFile.curCluster();
    }
    //looks like code to print out % done
    if ((millis() - tPct) > 1000) {
      uint8_t pct = myFile.curPosition()/(myFile.fileSize()/100);
      if (pct != lastPct) {
        tPct = millis();
        lastPct = pct;
        Serial.print(pct, DEC);
        Serial.println('%');
      }
    }
    if (Serial.available()) break;
  }
  bw.writeBuf();
  csvFile.close();
  Serial.print(F("Done: "));
  Serial.print(0.001*(millis() - t0));
  Serial.println(F(" Seconds"));
}

Kinnishian

I'm not sure if it is more appropriate for me to make a new thread, or to post on your thread since you appear to have relevant experience.

My question: Did you modify any of the SPI controls to account for the teensy 3.1 vs teensy 3.0?

I am struggling with getting sdfat to work in my system. After literal days, from hardware to software problems, I can read, but not write. Since there are some issues with entering spi idle mode I'm trying to investigate how the spi stuff works.

What I did notice was that there's a file called "SdSpimk20dx128.h" which is for the teensy 3.0 presumably. However the teensy 3.1 is defined as mk20dx256, so I'm wondering if I don't need to modify things to account for that.

Any guesses? Did you have this problem?

Thanks.
-Tomek

fat16lib

Are you using this version of SdFat?  https://github.com/greiman/SdFat

There is a mod in SdSpiMK20DX128.cpp by Paul Stoffregen that fixes SPI problems on Teensy 3.0 and 3.1 with newer versions of the Teensy IDE.  This fix may help.

Another option is to edit SdFatConfig.h and use the standard SPI.h library.  Change this line.
Code: [Select]

/**
* Force use of Arduino Standard SPI library if USE_ARDUINO_SPI_LIBRARY
* is nonzero.
*/
#define USE_ARDUINO_SPI_LIBRARY 0


I tested this version on Teensy 3.1 using the bench example with 32 KB reads and writes.  about 2200 KB/sec write and 2500 KB/sec read.
Quote


Use a freshly formatted SD for best performance.

Type any character to start
Free RAM: 28315
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0X3752B221
Manufacturing date: 10/2013

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2260.89,21620,13472,14475
2285.79,21563,13417,14323
2287.89,20618,13423,14306
2292.10,20659,13448,14280
2285.79,20639,13418,14316
2292.10,20628,13449,14279
2278.47,27660,13419,14371
2286.84,20569,13422,14310
2292.10,20587,13448,14279
2286.84,20682,13429,14316

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2550.30,13285,12824,12847
2551.61,13205,12780,12844
2550.30,13205,12778,12843
2551.61,13207,12778,12843
2551.61,13206,12778,12844

Done


Here is the LowLatencyLogger example logging four analog pins per sample at 2500 samples/sec.

Quote

Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: DATA03.BIN
Max block write usec: 200
Record time sec: 9.747
Sample count: 24365
Samples/sec: 2499.74
Overruns: 0
Done



Kinnishian

Super cool to get a reply from [who I believe to be] the SdFat creator. Thanks!

Ok, to answer the questions:
(1) Yes, I am using the most recent github pull, authored 5 days ago.   Originally I was using old code, because the links on the web are a bit messy due to the old google code and people not catching on the fact its on git now.   But I've been using the new code for the last two days.

(2)
Quote
There is a mod in SdSpiMK20DX128.cpp by Paul Stoffregen that fixes SPI problems on Teensy 3.0 and 3.1 with newer versions of the Teensy IDE.  This fix may help.

I might be simply unaware, but that's the file that confused me a little when I looked at it. It seems to only offer a fix for teensy 3.0 since it only specifies the arm processor with the 128 suffix, not the 256 that the teensy 3.1 uses. So I thought maybe something was going on there. When you say there is a mod in the file, do you mean there's a mod, elsewhere, that I need to add to the git file? Or is it already included?

(3) I'll try using the default arduino library. Actually the first time I tried that once something got confused, because the compiler said "spi.h" not found. I didn't pursue that, since I was on a different troubleshooting bent at the time.


(4) Question to sdfat creator or anyone else: Is it common that I can read from my card, open a file and such, but not write to the file? I was thinking that might suggest a lot about what my error is.  I can do card.init, vol.init, those basics. but card.begin() fails, in the past I've narrowed it to failing to enter SPI idle mode in sdfat.begin() [which calls card.begin, i believe]. I am doing this on a teensy 3.1.

fat16lib

SdSpiMK20DX128.cpp works for both Teensy 3.0 and Teensy 3.1.  It is Paul's version.  Ignore the file name it dates from when only Teensy 3.0 existed.

Quote
I can do card.init, vol.init, those basics. but card.begin() fails,


Why are you doing this?  You should not be using these calls.

You should only need sd.begin() in your code.

Does the bench example work?

The results above are running the GitHub version on a Teensy 3.1 with no mods.

Have you tried any of the SdFat examples ?

Kinnishian

Hi! I'm working from memory before I run out the door, but I will double check everything later this evening. I hope it all makes sense, I have spent a lot of newb-hours into this.  Getting pretty exhausted, though it's also meaningful and I learn a bit about something (either code, or troubleshooting as a concept.) Hard to see the positive big picture though, when I can be pretty confused why things aren't working.

SO:
The other inits are just part of my process of trying to understand what works, and what doesn't.  Even if I don't call them, I can't call sd.begin [but I need to call them if I just want to read the file, which is the best I can do right now. I can read a test .cvs file from the reading file example, and I can run quickstart and I can run sdfatinfo.].

Early on I could get the example SdFatInfo to work properly for multiple cards. yay! I thought that was going to be it, but wasn't.

I noticed SdFatInfo *does not call sd.begin* and rather just calls the other init functions. So I had something like "ok, these inits work, but Sd.begin doesn't?" Over time, I realized I think Sd.begin fails at the point where it has the option of entering SPI idle mode, except it fails instead. However, if I comment out that if statement, I still get [a different] failure.

I couldn't get example Quickstart to work, and went line by line (at least the lines i understood, newb-me), to see the differences between QuickStart and SdFatInfo. If I recall, they follow the same init process. So this was different, it was not failing on Sd.begin.  

In the end, I can get Quickstart to work now. The solution?? I have to explicitly hardcode the chipSelect pin  in the init call....I don't know why... but when i choose it through the serial monitor it just fails. Really weird,, right?

So SdFatInfo, and Quickstart work. Not so much other examples, though I'll try a few more tonight.


TO OP: I'm sorry! I've gotten off-topic. It looks like my initial on-topic question has been answered. There was nothing in the 20mdx128 part vs 256, and the code optimized for teensy 3.0 is also set for teensy 3.1. Perhaps I should make a new thread or request these posts be moved. Not sure what is the best option.

fat16lib

QuickStart has this warning:
Code: [Select]

// Normally the SdFat class is used in applications in place
// of Sd2Card, SdVolume, and SdFile for root.
// These internal classes are used here to diagnose problems.
Sd2Card card;
SdVolume volume;
SdFile root;


SdInfo uses these because it is also intended to diagnose problems.  It must work with an SD card that is not formatted.

Look at something like the ReadWriteSdFat example.

Kinnishian

But....what....how?

I culled all the diagnostic code I had mucked up, and restarted with the readwrite example. I only added the minimum info to run my card on my board (involves turning on some power-saving mosfets.) I then got the basic read-write code working, and added in a few more functions and still have it working. 

...This is really, really, really, really, nice.

Thank you. I'm not sure how I got caught up in everything or where I went wrong...but now I get to move forward, which is great.

Go Up