Storing sensor data to Winbond chip

It occurred to me after posting the code that there's a problem with it. Each time you power on your system, the code will start trying to log data and write it to flash memory starting from address zero.

You probably want to implement some sort of mode switch. In the avionics world we used to call it ground test mode where you could do different things to the normal flight mode. It only needs 1 discrete input pin that is read once during start() to determine the mode of operation.

The Sparkfun library example gives you a simple menu where you can erase the chip etc. You could run similar code in your ground test mode to extract the logged data, erase the chip and anything else you need to do. Maybe you might want to alter the rate at which data is logged whilst out in the field. In ground test mode you could alter the number of milliseconds between samples and store that in the internal EEPROM. You can then read that value out when your system is in flight mode.

Just some thoughts .....

@markd833 wow, I can't thank you enough, i've managed to log the data to the chip, thank you again for taking your time to help.

I've got a rough layout of how I'd like to store the data
(With headings)
Test 1
data......

(Next reboot)
Test2
data.......

(Next reboot)
Test(x)
data

and i'll include options to erase the chip and store the vales for the measurements delay and the value of test x in the EEPROM (Now will look into how to program it) and have x reset each time I erase the chip.

Only issue I currently have is converting the raw data back, I can't seem to find any examples, and so for my test code I have it looped so that I take 20 sensor readings, id then like to save those readings to my SD card and print them to the serial both in a readable manner.

I seriously can't thank you enough for taking time out of your day to help, thank you :slight_smile:

Getting the raw data back is done by reading back the stored data from the flight. Obviously you know that the first record will be at address 0, but how to know when you've got to the end of the data. Well, flash memory, when erased reads as 0xFF in each byte. You can use this fact when reading in each record from the flash. Check the value of the record number against 0xFFFFFFFFas you go.

Something like this bit of code may work. It prints out the recorded data as comma separated variables. You can capture and save the data using one of the Serial Terminal programs that you can find on the net.

void dumpData() {
  uint32_t flashReadAddress = 0;

  // read the very first record from the flash chip
  myFlash.readBlock(flashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
  flashReadAddress = flashReadAddress + sizeof(oneRecord);

  // quick sanity check - the record number for this entry should be zero
  if (oneRecord.recordNumber == 0) {
    // print out the headings
    Serial.print(F("Rec Num,Timestamp,Temp,Press,Accel"));    

    // erased flash memory reads as 0xFF to use that to check if the end of the data has been reached
    while (oneRecord.recordNumber != 0xFFFFFFFF ) {
      Serial.print( oneRecord.recordNumber );
      Serial.print( "," );
      Serial.print( oneRecord.timeStamp );
      Serial.print( "," );
      Serial.print( oneRecord.temperature );
      Serial.print( "," );
      Serial.print( oneRecord.pressure );
      Serial.print( "," );
      Serial.println( oneRecord.acceleration );

      // read the next record from the flash
      myFlash.readBlock(flashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
      flashReadAddress = flashReadAddress + sizeof(oneRecord);
    }
  }
}

Including the ability to log more than 1 flight in the flash chip needs a little bit of extra code.

You need to decide if you want to include the flight number as part of each record, or as a separate record. If you want to include it as part of each record, then you need to add another parameter to the struct like:

struct oneRecordType {
  uint8_t flightNum;
  uint32_t recordNumber;
  uint32_t timeStamp;
  float temperature;
  float pressure;
  float acceleration;
} oneRecord;

BTW, I changed recordNumber to an unsigned int as it makes no sense for it to be -ve.

If instead you want to have a one line entry to indicate the flight number, then you could manipulate the recordNumber entry. Maybe write out a blank record with the record number set to 0xFF000000 for the 1st flight, 0xFF000001 for the 2nd flight etc.

Either way, you need a piece of code similar to the one that dumps the recorded data out. It's called during setup() and it's job is to read through the flash chip from the start until it gets a record with a record number of 0xFFFFFFFF. It then knows it has reached the next free location in the flash and that is where the next set of recorded parameters starts to be written to. You would also need to keep an eye out for your chosen method of indicating the start of another flight so that you can increment the flight number by 1 and write that out to your newly discovered end of file as it were.

@markd833 I seriously can't thank you enough for helping me with my project.
The write and read code works perfectly, however there seems to be a repeating issue at the 10th recording

9,5753,25.02,99955.38,0.00
10,5856,25.01,nan,nan
11,5960,25.01,99953.30,0.00

(Pressure and Acceleration)
(Also the acceleration is zero as I haven't added the code for my IMU just yet)

and I spent some time and have made an EEPROM program where I can write and save data in the memory, which was made so much easier thanks to your code.

Also thinking does that mean I should write the current address the data is writing to each data log and store that in the EEPROM so that on the next flight I can start to write from that address, and set it back to 0 each time I erase the chip.

just thought as well, what if the address is greater than 255, would I need to store the value over 8 bytes, or is it easier to read the values until they equal 0 and just start form there :slight_smile:

@harveyn4444 - good progress.

First a word of warning, whilst using the on-board EEPROM may seem like a good idea, there's one main gotcha that you need to be aware of. The internal EEPROM (well, any EEPROM actually) has a limited number of writes that you can do to each location. I seem to recall that the 328P micro has a limit of around 100,000 write/erase cycles. What this means is that if you were using EEPROM locations 0-3 to store a 32-bit unsigned integer that happened to be the next address to write to, then one of those locations is going to change every time you write a new record to the flash and update the address in EEPROM. After 100,000 records (roughly - it's not an exact science), you will start to see corruptions in the EEPROM memory as the locations start to fail.

You can avoid storing the address in EEPROM and just hold it in internal RAM whilst the code is running. You just need that bit of code in startup() that reads the flash memory from address zero onwards (called a linear search) till it finds an unused memory location. That's the very simple and long winded way of doing it, and it obviously takes a bit longer after each flight as more records are added.

That might not be an issue if you expect to store a few 1000 records but obviously is going to take a while when you start to hit 1,000,000 records!

Another option for you (possibly, because I guess weight is an issue on rocketry) is to use a battery backed real time clock module. Some of these have 50+ bytes of RAM that is battery backed and therefore remembered between flights (assuming you use a backup CR2032 coin cell).

Or you can use what's called a binary search to quickly locate the next unused memory record in the flash chip.

Anyways, on to your numbers corruption. In case you didn't know, NAN is short for Not A Number and is associated with floating point values. It means that whatever bit pattern is stored in that floating point memory location is not a valid floating point number.

A couple of things here. You mention address over 255 - that's guaranteed in your scenario. Your address might fit in 16-bits so you can go up to 65,535 memory locations (not records). I'd specify your address as a 32-bit unsigned int and you'd be covered.

Secondly, and I don't know the record structure that you've finally chosen here (maybe post your code?) but flash memory is stored in pages and there are things to be aware of when writing across page boundaries. I've not looked too hard at the flash library code so I don't know if it handles that for you.

Maybe post you current code so we can see what might be happening.

The code I have now is
<

void loop() {
Serial.println("Enter y to start - Enter e to erase chip");
Serial.println("EEPROM size : ");Serial.print(EEPROM.length());Serial.println("");
while (Serial.available()) Serial.read(); //Clear the RX buffer
while (Serial.available() == 0); //Wait for a character
byte choice = Serial.read();

if (choice == 'y'){
for (j = 0; j<20; j++) {
if (! bmp.performReading()) {
Serial.println("Failed to perform reading :(");
return;
}
// read the temperature from whatever device you are using
oneRecord.temperature = bmp.temperature,3;
Serial.println(oneRecord.temperature);
// read the pressure from whatever device you are using
oneRecord.pressure = bmp.pressure,3;
Serial.println(oneRecord.pressure);
// read the acceleration from whatever device you are using
//oneRecord.acceleration = ;

// add in the time stamp so you know roughly when the reading happened
oneRecord.timeStamp = millis();
Serial.println(oneRecord.timeStamp);
// write the readings to the flash chip

//Flight num = read address 1? and thta value is the flight no.

myFlash.writeBlock(flashWriteAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );

// increment the record number - you don't really need this but it will help
// detecting the end of the data when retrieving the data from the flash chip 
oneRecord.recordNumber++;

// update the flash write address
flashWriteAddress = flashWriteAddress + sizeof(oneRecord);

// delay for a specific period of time - say 100ms
delay( 100 );
}
dumpData();
choice == "o"; 

}

else if (choice == 'e')
{
Serial.println("Erasing entire chip");
myFlash.erase();
Serial.println("Chip erased");
Serial.println(" ");
}
else{
Serial.print("Unknown choice: ");
Serial.write(choice);
Serial.println();
}

}

void dumpData() {
uint32_t flashReadAddress = 0;

// read the very first record from the flash chip
myFlash.readBlock(flashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
flashReadAddress = flashReadAddress + sizeof(oneRecord);

// quick sanity check - the record number for this entry should be zero
if (oneRecord.recordNumber == 0) {
// print out the headings
Serial.println(F("Rec Num,Timestamp,Temp,Press,Accel"));

// erased flash memory reads as 0xFF to use that to check if the end of the data has been reached
while (oneRecord.recordNumber != 0xFFFFFFFF ) {
  //Outputs the flash data
  Serial.print( oneRecord.Flightnum );
  Serial.print( "," );
  Serial.print( oneRecord.recordNumber );
  Serial.print( "," );
  Serial.print( oneRecord.timeStamp );
  Serial.print( "," );
  Serial.print( oneRecord.temperature );
  Serial.print( "," );
  Serial.print( oneRecord.pressure );
  Serial.print( "," );
  Serial.println( oneRecord.acceleration );




  // read the next record from the flash
  myFlash.readBlock(flashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
  flashReadAddress = flashReadAddress + sizeof(oneRecord);
}

}
}

I don't have any code to handle going between the pages.

Maybe I could store the flight number to my SD card and save it as a variable before flight, because the main reason i'm not storing it on the SD is for the in-flight portion.
I think i've run into the page issue and I currently haven't had time to go over the linear search, the maximum number of flights at a time would be maybe like 5-10 MAX so I think it would be suitable.
Im not sure why it can store the other values and not the values at 10, so i'm assuming its a memory error and not a data type error

It took a while but this may get you started :grinning:

// I've got a Winbond 25Q64 device.

#include <SPIMemory.h>     // get it from https://github.com/Marzogh/SPIMemory

// sampling interval in milliseconds - seems to work at 10 milliseconds
const uint32_t sampleInterval = 10;

const byte PIN_FLASH_CS = 7; // Change this to match the Chip Select pin on your board

SPIFlash myFlash( PIN_FLASH_CS );

// edit this structure to hold one set of parameters you want to log
struct oneRecordType {
  uint32_t flightNumber;
  uint32_t recordNumber;
  uint32_t timeStamp;
  float temperature;
  float pressure;
  float acceleration;
} oneRecord;

uint32_t prevFlightNumber = 0;

// the maximum number of records in the 64Mbit (8Mbyte) flash
const uint32_t flashMemTop = 8388608L;

// this variable holds the next address to write to in the flash chip
uint32_t nextFlashWriteAddress = 0;

// this variable holds the next address to read from in the flash chip
uint32_t nextFlashReadAddress = sizeof(oneRecord);

// recording flag
uint8_t isRecording = false;

uint8_t isFlashDataValid = false;
uint32_t prevMillis, currentMillis = 0;
uint32_t rawDumpAddress = 0;

// used purely for fake data generation
float fakeTemperature = 12.0;
float fakePressure = 1013.25;
float fakeAcceleration = 1.0;

void setup() {
  Serial.begin(115200);
  Serial.println(F("\n\nRocket SPI Flash Data Logger."));

  if (myFlash.begin() == false)
  {
    Serial.println(F("SPI Flash not detected. Check wiring. Maybe you need to pull up WP/IO2 and HOLD/IO3? Freezing..."));
    while (1);
  }
  flashInfo();

  // check to see if the flash holds our recordings or not
  if ( getMyFlashSignature() == false ) {
    Serial.println(F("Flash doesn't appear to hold valid recordings - may need erasing first."));
    isFlashDataValid = false;
  }
  else {
    isFlashDataValid = true;

    // scan the flash chip for the next free memory lcoation to write to
    nextFlashWriteAddress = getNextFlashStartAddress();
    Serial.print(F("Next free memory location is: 0x"));
    Serial.println( nextFlashWriteAddress, HEX );
    Serial.print(F("Next flight number will be: "));
    Serial.println( prevFlightNumber + 1 );
  }

  showMenu();
}

void loop() {
  // grab the latest elapsed time
  currentMillis = millis();

  // see if there are any serial commands received
  checkForSerialCommand();

  // are we recording?
  if (isRecording == true) {
    // is it time for another sample?
    if ( currentMillis - prevMillis > sampleInterval ) {
      prevMillis = currentMillis;

      // prepare the record for writing to the flash chip
      // read in the data from your real sensors here!!!!
      oneRecord.recordNumber++;
      oneRecord.timeStamp = currentMillis;
      oneRecord.temperature = getFakeTemperature();
      oneRecord.pressure = getFakePressure();
      oneRecord.acceleration = getFakeAcceleration();

      // write the record to the flash chip
      myFlash.writeByteArray( nextFlashWriteAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
      nextFlashWriteAddress = nextFlashWriteAddress + sizeof(oneRecord);
    }
  }
}

// Handle serial port commands
void checkForSerialCommand() {
  if (Serial.available() != 0) {
    char choice = Serial.read();

    switch ( choice ) {
      // ignore CR and LF characters
      case '\n':   // LF
      case '\r':   // CR
        break;

      // dump a single recording
      case 'D':
        // D = start with 1st recording
        nextFlashReadAddress = sizeof(oneRecord);
      case 'd':
        // d = just dump the next recording
        isRecording = false;
        dumpOneRecording();
        showMenu();
        break;

      // erase the flash - takes about 20 sec for my 25Q64 device
      case 'e':
      case 'E':
        isRecording = false;
        prevFlightNumber = 0;
        eraseFlash();
        showMenu();
        break;

      // dump raw memory from the flash
      case 'R':
        // R = start at address 0x00000000
        rawDumpAddress = 0;
      case 'r':
        // r = dump next 1k block
        isRecording = false;
        dumpRawMemory();
        break;

      // start recording
      case 's':
      case 'S':
        Serial.println(F("START RECORDING"));
        fakeTemperature = 12.0;
        fakePressure = 1013.25;
        fakeAcceleration = 1.0;
        oneRecord.flightNumber = prevFlightNumber + 1;
        oneRecord.recordNumber = 0;
        isRecording = true;
        break;

      // stop recording
      case 'p':
      case 'P':
        Serial.println(F("STOP RECORDING"));
        prevFlightNumber = oneRecord.flightNumber;
        Serial.print(F("Logged "));
        Serial.print( oneRecord.recordNumber );
        Serial.println(F(" records for that flight."));
        Serial.print(F("Next free memory location is: 0x"));
        Serial.println( nextFlashWriteAddress, HEX );
        isRecording = false;
        break;

      default:
        showMenu();
        break;
    }
  }
}
// Display the menu of supported commands
void showMenu() {
  Serial.println(F("\nMENU OPTIONS:"));
  Serial.println(F("d : dump 1 recording (D to reset to 1st recording)"));
  Serial.println(F("e : erase flash (~20sec for 64mbit device)"));
  Serial.println(F("r : raw dump 1K blocks (R to restart at 0x0000)"));
  Serial.println(F("s : start recording"));
  Serial.println(F("p : stop recording"));
}

// generate a CSV formatted output of one flights worth of recordings
void dumpOneRecording() {
  Serial.println(F("\nDUMP ONE RECORDING"));

  // read the very first record of the flight
  myFlash.readByteArray(nextFlashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
  nextFlashReadAddress = nextFlashReadAddress + sizeof(oneRecord);

  // quick sanity check - the flight number should NOT be 0xFFFFFFFF
  if (oneRecord.flightNumber != 0xFFFFFFFF) {
    // remember which flight this is
    uint32_t thisFlightNumber = oneRecord.flightNumber;

    // print out the headings
    Serial.println(F("Flt Num,Rec Num,Timestamp,Temp,Press,Accel"));

    // erased flash memory reads as 0xFF so use that to check if the end of the data has been reached
    // also stop if the flight number has changed
    while ( (oneRecord.flightNumber != 0xFFFFFFFF ) && (oneRecord.flightNumber == thisFlightNumber ) )
    {
      Serial.print( oneRecord.flightNumber );
      Serial.print( "," );
      Serial.print( oneRecord.recordNumber );
      Serial.print( "," );
      Serial.print( oneRecord.timeStamp );
      Serial.print( "," );
      Serial.print( oneRecord.temperature );
      Serial.print( "," );
      Serial.print( oneRecord.pressure );
      Serial.print( "," );
      Serial.println( oneRecord.acceleration );

      // read the next record from the flash
      myFlash.readByteArray(nextFlashReadAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
      nextFlashReadAddress = nextFlashReadAddress + sizeof(oneRecord);
    }

    // reverse by 1 record as we may have read 1st record of the next flight
    nextFlashReadAddress = nextFlashReadAddress - sizeof(oneRecord);
  }
  else
  {
    Serial.println(F("No more flights."));
  }
}

// raw hex dump of the contents of the flash chip.
// each call dumps 1K.
void dumpRawMemory() {

  for (uint16_t x = 0 ; x < 0x0400 ; x++)
  {
    if (rawDumpAddress % 16 == 0) {
      Serial.println();
      Serial.print("0x");
      if (rawDumpAddress < 0x1000) Serial.print("00");
      if (rawDumpAddress < 0x100) Serial.print("0");
      if (rawDumpAddress < 0x10) Serial.print("0");
      Serial.print(rawDumpAddress, HEX);
      Serial.print(": ");
    }

    byte val = myFlash.readByte( rawDumpAddress++ );
    if (val < 0x10) Serial.print("0");
    Serial.print(val, HEX);
    Serial.print(" ");
  }
  Serial.println();
}

// erase the flash and write our unique signature in the 1st record
void eraseFlash() {
  Serial.println(F("ERASING FLASH CHIP"));
  myFlash.eraseChip();
  Serial.println(F("Writing unique signature"));
  oneRecord.flightNumber = 0x524F434B;   // = "ROCK" in ASCII
  oneRecord.recordNumber = 0;
  oneRecord.timeStamp = 0;
  oneRecord.temperature = 0.0;
  oneRecord.pressure = 0.0;
  oneRecord.acceleration = 0.0;
  myFlash.writeByteArray( 0, (uint8_t *)&oneRecord, sizeof(oneRecord) );
  nextFlashWriteAddress = sizeof(oneRecord);
  Serial.println(F("Done"));
}

// print out some details about the flash chip
void flashInfo() {
  Serial.print(F("Manufacturer: "));
  Serial.println(myFlash.getManID(), HEX);
  Serial.print(F("Capacity: 0x"));
  Serial.println(myFlash.getCapacity(), HEX);
}

// Scans the flash chip to locate the address of the next free unused memory location
// using a crude linear search from the start of memory upwards.
uint32_t getNextFlashStartAddress() {

  // start at first record past our signature header
  uint32_t testAddress = sizeof(oneRecord);

  myFlash.readByteArray( testAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );
  if ( oneRecord.flightNumber != 0xFFFFFFFF ) {
    // there's some data recorded so find the end of it
    while ( oneRecord.flightNumber != 0xFFFFFFFF ) {
      prevFlightNumber = oneRecord.flightNumber;
      testAddress = testAddress + sizeof(oneRecord);
      myFlash.readByteArray( testAddress, (uint8_t *)&oneRecord, sizeof(oneRecord) );

      if ( testAddress > flashMemTop ) break;
    }
  }
  return testAddress;
}

// reads record 0 from flash memory to see if it contains our signature
// phrase - may indicate that there's actual data rather than data from another app.
bool getMyFlashSignature() {
  // read the first record in flash memory
  myFlash.readByteArray( 0, (uint8_t *)&oneRecord, sizeof(oneRecord) );

  // does the flight number field of the record hold 0x524F434B => ASCII for ROCK !!!!
  if ( oneRecord.flightNumber == 0x524F434B ) return true;

  return false;
}

float getFakeTemperature() {
  fakeTemperature = fakeTemperature + 0.01;
  return fakeTemperature;
}

float getFakePressure() {
  fakePressure = fakePressure - 0.01;
  return fakePressure;
}

float getFakeAcceleration() {
  fakeAcceleration = fakeAcceleration + 0.01;
  return fakeAcceleration;
}

It took a while as each flash library I tried seemed to have issues writing a block of data across a page boundary. I think the library I've used here handles that ok. There's a load of comments in the code to get you going. Have fun!

So I figured that there may be a slightly less gruesome way of doing this. I had a go with the SerialFlash library written by Paul Stoffregen. It implements a simple file system on SPI flash chips. The main downside is that you have to pre-declare how big your file going to be beforehand. That's not too bad in this case as if you declared a file of 1 Mbyte, then writing fixed records of 32 bytes gives you 32,768 records. Writing 1 record every 10ms gives you 327.68 seconds of recording - just under 5min 30sec.

I've no idea how long a rocket flight lasts (assuming a model rocket here!) but I guess a 2 Mbyte file will give you 10 minutes worth of data assuming you wanted to log the descent as well.

Anyways, here's another version of roughly the same code but using Paul's library:

// Uses the SerialFlash library from Paul Stoffregen
// Get it from: https://github.com/PaulStoffregen/SerialFlash
//
// I've got a Winbond 25Q64 device.

#include <SerialFlash.h>

#define FILE_SIZE_512K 524288L
#define FILE_SIZE_1M   1048576L

SerialFlashFile file;

// sampling interval in milliseconds - seems to work at 10 milliseconds
const uint32_t sampleInterval = 10;

const byte PIN_FLASH_CS = 7; // Change this to match the Chip Select pin on your board

// edit this structure to hold one set of parameters you want to log
struct oneRecordType {
  uint32_t recordNumber;
  uint32_t timeStamp;
  float temperature;
  float pressure;
  float acceleration;
} oneRecord;

// use this array to create a filename
char filename[16];

uint8_t dumpFileNumber = 1;

// recording flag
uint8_t isRecording = false;

uint32_t prevMillis, currentMillis = 0;

// used purely for fake data generation
float fakeTemperature = 12.0;
float fakePressure = 1013.25;
float fakeAcceleration = 1.0;

void setup() {
  Serial.begin(115200);
  Serial.println(F("\n\nRocket SPI Flash Data Logger - File Sys Version."));

  if (!SerialFlash.begin( PIN_FLASH_CS )) {
    Serial.println(F("SPI Flash not detected. Check wiring. Maybe you need to pull up WP/IO2 and HOLD/IO3? Freezing..."));
    while (1);
  }
  
  flashInfo();

  // how do we know that there is a filesystem on the chip?
  // there doesn't seem to be anything obvious in the library!
  if ( !SerialFlash.exists( "dummy.txt" )) {
    Serial.println(F("Flash doesn't appear to hold a file system - may need erasing first."));
  }
  else {
    Serial.println(F("Files currently in flash:"));
    SerialFlash.opendir();
    while (1) {
      uint32_t filesize;
      if (SerialFlash.readdir(filename, sizeof(filename), filesize)) {
        Serial.print(filename);
        Serial.print(F("  "));
        Serial.print(filesize);
        Serial.print(F(" bytes"));
        Serial.println();
      }
      else {
        break; // no more files
      }
    }
  }

  showMenu();
}

void loop() {
  // grab the latest elapsed time
  currentMillis = millis();

  // see if there are any serial commands received
  checkForSerialCommand();

  // are we recording?
  if (isRecording == true) {
    // is it time for another sample?
    if ( currentMillis - prevMillis > sampleInterval ) {
      prevMillis = currentMillis;

      // prepare the record for writing to the flash chip
      // read in the data from your real sensors here!!!!
      oneRecord.recordNumber++;
      oneRecord.timeStamp = currentMillis;
      oneRecord.temperature = getFakeTemperature();
      oneRecord.pressure = getFakePressure();
      oneRecord.acceleration = getFakeAcceleration();

      // write the record to the flash chip
      file.write( (uint8_t *)&oneRecord, sizeof(oneRecord) );
    }
  }
}

// Handle serial port commands
void checkForSerialCommand() {
  if (Serial.available() != 0) {
    char choice = Serial.read();

    switch ( choice ) {
      // ignore CR and LF characters
      case '\n':   // LF
      case '\r':   // CR
        break;

      // dump a single recording
      case 'D':
        // D = star with 1st recording
        dumpFileNumber = 1;
      case 'd':
        // d = just dump the next recording
        isRecording = false;
        dumpOneRecording();
        dumpFileNumber++;
        showMenu();
        break;

      // erase the flash - takes about 20 sec for my 25Q64 device
      case 'e':
      case 'E':
        isRecording = false;
        eraseFlash();
        showMenu();
        break;

      // start recording
      case 's':
      case 'S':
        Serial.println(F("START RECORDING"));
        getNextFlashFilename();
        Serial.print( F("Writing to file: " ));
        Serial.println( filename );
        SerialFlash.create( filename, FILE_SIZE_512K );
        file = SerialFlash.open( filename );
        fakeTemperature = 12.0;
        fakePressure = 1013.25;
        fakeAcceleration = 1.0;
        oneRecord.recordNumber = 0;
        isRecording = true;
        break;

      // stop recording
      case 'p':
      case 'P':
        Serial.println(F("STOP RECORDING"));
        Serial.print(F("Logged "));
        Serial.print( oneRecord.recordNumber );
        Serial.println(F(" records for that flight."));
        file.close();
        isRecording = false;
        break;

      default:
        showMenu();
        break;
    }
  }
}
// Display the menu of supported commands
void showMenu() {
  Serial.println(F("\nMENU OPTIONS:"));
  Serial.println(F("d : dump 1 recording (D to reset to 1st recording)"));
  Serial.println(F("e : erase flash (~20sec for 64mbit device)"));
  Serial.println(F("s : start recording"));
  Serial.println(F("p : stop recording"));
}

// generate a CSV formatted output of one flights worth of recordings
void dumpOneRecording() {
  Serial.println(F("\nDUMP ONE RECORDING"));

  sprintf( filename, "flt%02d.bin", dumpFileNumber );
  file = SerialFlash.open( filename );

  if (file) {
    Serial.print( F("Contents of file: ") );
    Serial.println( filename );
    
    // print out the headings
    Serial.println(F("Rec Num,Timestamp,Temp,Press,Accel"));
    
    do {
      file.read( (uint8_t *)&oneRecord, sizeof(oneRecord) );

      // library doesn't know where end of actual written data is so we have
      // to look for it ourselves!
      if ( oneRecord.recordNumber != 0xFFFFFFFF ) {
        Serial.print( oneRecord.recordNumber );
        Serial.print( "," );
        Serial.print( oneRecord.timeStamp );
        Serial.print( "," );
        Serial.print( oneRecord.temperature );
        Serial.print( "," );
        Serial.print( oneRecord.pressure );
        Serial.print( "," );
        Serial.println( oneRecord.acceleration );
      }
    } while ( oneRecord.recordNumber != 0xFFFFFFFF );
    file.close();
  }
  else {
    Serial.println(F("No more files."));
  }
}

// erase the flash and write our unique signature in the 1st record
void eraseFlash() {
  Serial.println(F("ERASING FLASH CHIP"));
  SerialFlash.eraseAll();

  // create a dummy file so we can look for it
  // this seems the only way of knowing if there is an actual
  // filesystem present when we startup ...
  SerialFlash.create( "dummy.txt", 16 );
  file = SerialFlash.open("dummy.txt");
  file.write( "Hello", 6 );
  file.close();
  Serial.println(F("Done"));
}

// print out some details about the flash chip
void flashInfo() {
  uint8_t id[5];
  
  SerialFlash.readID(id);
  Serial.print(F("Capacity: "));
  Serial.println(SerialFlash.capacity( id ));
}

// Scans the flash chip to determine the next filename to use.
uint8_t getNextFlashFilename() {
  uint8_t fn=0;

  do {
    // create the filename and see if it exists
    sprintf( filename, "flt%02d.bin", ++fn );
    Serial.println( filename );
  } while ( (SerialFlash.exists( filename )) && (fn<10) );
  
  Serial.println( fn );
  return fn;
}

float getFakeTemperature() {
  fakeTemperature = fakeTemperature + 0.01;
  return fakeTemperature;
}

float getFakePressure() {
  fakePressure = fakePressure - 0.01;
  return fakePressure;
}

float getFakeAcceleration() {
  fakeAcceleration = fakeAcceleration + 0.01;
  return fakeAcceleration;
}
1 Like

@markd833 I can't thank you enough for spending your time helping me with my project.

The SerialFlash library works perfectly, the SPIMemory didn't seem to agree with my chip, but the Paul Stoffregen library example is flawless and I really can't tell you how much I appreciate.

I just need to break down the code and transfer it to my SD card, for which i'm going to read all the addresses that aren't empty and transfer the data to a csv file one row at a time.

I'll try and work on it tomorrow, i've just bene a busy lately, but will hopefully post the outcome tomorrow. I really can't say thank you enough for helping with my project I would be completely lost without your help.
thank you again - Harvey :slight_smile:

A slight mod to the dump routine to redirect the prints to a file on the SD card should be all you need to do. You could use the same filenames.

You might want to add in a bit more error checking. For example the code doesn't check if the fixed length file is full of not. On the other hand you could declare a larger file. I don't know off the top of my head if the write routine stops writing if the file is full of not.

@markd833 wow it's perfect.
I made the change in the dump data part, as its just a copy of the serial print but to the file.
I was wondering about naming the files on the SD card, is there a way to implement the naming of the flash files into the SD files as that part of the code seems slightly complex
<
do {
// create the filename and see if it exists
sprintf( filename, "flt%02d.bin", ++fn );
Serial.println( filename );
} while ( (SerialFlash.exists( filename )) && (fn<10) );

Serial.println( fn );
return fn;
}

I'm confused with the %02d, is this the part with increases each time, and can I name a file on the SD card with a variable name which creates a new text file for each flight and named after that flight (E.g FlightData1 & FlightData2) which i'm going to start looking into.

The next part is hopefully implementing it all into my flight software which can now FINALLY be completed :slight_smile: Again I truly can't thank you enough, i'm a huge step closer to finishing the first iteration of my project all thanks to you.
-Harvey

The %02d bit is part of a format string. Have a look at the C or C++ documentation for printf(). Basically, the % symbol indicates the start of a format specification. The d says that I want a decimal integer. The 2 says I want a 2-digit number, and the 0 says pad the number with leading zeros. The whole lot says take the value in the variable fn and pre-increment it by one. Then convert the new value of fn into a 2 digit ASCII character representation. Store the complete string in the char array called filename.

That bit of code does exactly what the comment says. It loops round creating filenames, flt01.bin, flt02.bin etc checking to see if they already exist and gives up if fn gets to 10. That bit of code works out what the next filename should be without having to remember anything between reboots etc.

If you wanted to use FlightData1, FlightData2 etc then the format string would be "FlightData%d".

@markd833 I've managed to increase the SD file names now, and have it run during the setup of the program.

In the future if I were to have multiple flash chips on a board is it as simple as only talking to one chip at a time, obviously having different CS pins.

Hopefully I should just be able to implement the SD and flash file saving into my flight software which ill should hopefully try soon, in terms of the writing when full, i'm going to run a test of having a 16 byte file size and writing say 100 values to it and seeing at which number it stops. Im hoping by making a greater file size than needed it should be fine, I may do 2Mb files just to be safe. Again I truly can't thank you enough for helping me, you've been so helpful I can't thank you enough. :slight_smile:
-Harvey

Harvey, multiple CS pins is how SPI works so that's the way to go. I just checked and you can get Winbond W25Q128 devices for less than 2GBP (or USD...). That'll give you 16Mbytes on one 8-pin SOIC chip.

If you work out your storage needs per launch, that should give you a good idea of the amount of flash you will need. For example if you were to log 32bytes every 10ms for 10 minutes, that would take up 32 x 100 x 60 x 10 bytes - 1,920,000 bytes - say 2Mbytes. A single W25Q128 would give you 8 flights.

A lot depends on how much data you are logging and how fast you are doing it. Oh, and how much of the flight you are interested in recording. I guess the interesting bit is from launch till all the fuel is expended!

If you divide up a W25Q128 into 2Mbyte files, then you would know that flights 1-8 would be on the chip connected to CS1 and flights 9-16 on the chip connected to CS2. That's a bit simplistic as there is likely to be a slight overhead due to the "filesystem" so you may need to back off the file sizes to 1.9Mbytes but you get the idea.

You would need to see if Pauls SPI library supports more than one flash device at once.

Edit: I think you may be able to simply call SerialFlash.begin() again with the new CS pin to select a second or third SPI flash device.

@markd833 I think to keep it simple i'm just going to upgrade to a 128Mbit flash chip, simpler is better.
I've integrated your code into my flight software and it works, however I only store the data on the flash chip for now (Need to code the landing section and SD transfer) so as of now the data is just written to the chip, when I upload your code to dump the file it doesn't read the file, i'm guessing its because the file existed before the program is uploaded, is there a simple modification to the code to be able to simply just read the data of the flight data (Only 1 flight at a time at the moment, possible could add a feature to choose the flight to read from)

Apart from this my project is 99% complete, once this last bit is finished im going to post the complete code here for anyone with the same issues

I truly can't thank you enough for spending your time helping me with my project and being patient with me, I can't emphasise how helpful you've been

Many thanks -Harvey :slight_smile:

Harvey, when you run the code to dump the files, 1 of 2 things should happen - hopefully ..... !!

You should get either a message saying "Flash doesn't appear to hold a file system - may need erasing first.", or you should get a list of files that are on the flash.

If you see the "Flash doesn't appear to hold a file system..." message, then that is most likely that there isn't a file on the flash called "dummy.txt". My code creates it as part of the chip erase process in eraseFlash(). The only reason it's there is so I knew if the chip had a filesystem on it or not. I couldn't see anything obvious in the library that indicated whether the chip was "formatted" or not, so I simply created that 16 byte file so I could check for it in the future.

If it's not that, then something strange is going on.

Let me know how you get on.

EDIT: Just a thought but when you try and dump the file, does it say "no more files"? If you changed the names of your files to FlightDataxx, then you need to change the name in dumpOneRecording() otherwise it will be looking for the wrong files!

I realised i'd changed the name of the files, so it was searching for the wrong files, stupid mistake :slight_smile: It works perfectly, it was me that managed to mess it up.

I think now its complete :slight_smile: thanks to you
-Harvey

@markd833
The full code for anyone in the future

// Uses the SerialFlash library from Paul Stoffregen
// Get it from: https://github.com/PaulStoffregen/SerialFlash
//
// I've got a Winbond 25Q64 device.
//Created by Markd833     2021

#include <SerialFlash.h>
#include <SPI.h>
#include <SD.h>

#define FILE_SIZE_512K 524288L
#define FILE_SIZE_1M   1048576L

#include "Adafruit_BMP3XX.h"
Adafruit_BMP3XX bmp;

//File system on the flash chip
SerialFlashFile file;
//File system on the SD card
File DataFile;

// sampling interval in milliseconds - seems to work at 10 milliseconds
const uint32_t sampleInterval = 10;

const byte PIN_FLASH_CS = 4; // Change this to match the Chip Select pin on your board

// edit this structure to hold one set of parameters you want to log
struct oneRecordType {
  uint32_t recordNumber;
  uint32_t timeStamp;
  float temperature;
  float pressure;
  float acceleration;
  float height;
} oneRecord;

// use this array to create a filename
char filename[16];
char SDfilename[16];

uint8_t dumpFileNumber = 1;

// recording flag
uint8_t isRecording = false;

uint32_t prevMillis, currentMillis = 0;

// used purely for fake data generation
float fakeAcceleration = 1.0;

void setup() {
  Serial.begin(115200);
  delay(1000);
  Serial.println(F("\n\nRocket SPI Flash Data Logger - File Sys Version."));
  //This is used for the BMP388 module used, use the setup of each sensor
  if (!bmp.begin_I2C()) {   // hardware I2C mode, can pass in address & alt Wire
    Serial.println("BMP Error");
    while (1);
  }
  bmp.setTemperatureOversampling(BMP3_OVERSAMPLING_8X);
  bmp.setPressureOversampling(BMP3_OVERSAMPLING_4X);
  bmp.setIIRFilterCoeff(BMP3_IIR_FILTER_COEFF_3);
  bmp.setOutputDataRate(BMP3_ODR_50_HZ);
  if (!bmp.performReading()) {
    Serial.println("Failed to perform reading :(");
    return;
  }
  //Just apart of the bmp module to print some base recordings
  Serial.println(bmp.temperature);
  Serial.println(bmp.pressure / 100);
  //Change the 10 to the cs of the SD card
  SD.begin(10);
  if (!SD.begin(10)) {
    Serial.println("SD Error");
    while (1);
  }

  if (!SerialFlash.begin( PIN_FLASH_CS )) {
    Serial.println(F("SPI Flash not detected. Check wiring. Maybe you need to pull up WP/IO2 and HOLD/IO3? Freezing..."));
    while (1);
  }
  //Prints the info of the flash chip
  flashInfo();

  // how do we know that there is a filesystem on the chip?
  // there doesn't seem to be anything obvious in the library!
  //This is recreated each time the chip is erased
  if ( !SerialFlash.exists( "dummy.txt" )) {
    Serial.println(F("Flash doesn't appear to hold a file system - may need erasing first."));
  }
  else {
    Serial.println(F("Files currently in flash:"));
    SerialFlash.opendir();
    while (1) {
      uint32_t filesize;
      if (SerialFlash.readdir(filename, sizeof(filename), filesize)) {
        Serial.print(filename);
        Serial.print(F("  "));
        Serial.print(filesize);
        Serial.print(F(" bytes"));
        Serial.println();
      }
      else {
        break; // no more files
      }
    }
  }

  showMenu();
}

void loop() {
  // grab the latest elapsed time
  currentMillis = millis();

  // see if there are any serial commands received
  checkForSerialCommand();

  // are we recording?
  if (isRecording == true) {
    // is it time for another sample?
    if ( currentMillis - prevMillis > sampleInterval ) {
      prevMillis = currentMillis;


      // prepare the record for writing to the flash chip
      // read in the data from your real sensors here!!!!
      if (!bmp.performReading()) {
        Serial.println("Failed to perform reading :(");
        return;
      }
      oneRecord.recordNumber++;
      oneRecord.timeStamp = currentMillis;

      oneRecord.temperature = bmp.temperature, 3;
      oneRecord.pressure = bmp.pressure / 100;
      oneRecord.acceleration = getFakeAcceleration();

      // write the record to the flash chip
      file.write( (uint8_t *)&oneRecord, sizeof(oneRecord) );
    }
  }
}

// Handle serial port commands
void checkForSerialCommand() {
  if (Serial.available() != 0) {
    char choice = Serial.read();

    switch ( choice ) {
      // ignore CR and LF characters
      case '\n':   // LF
      case '\r':   // CR
        break;

      // dump a single recording
      case 'D':
        // D = star with 1st recording
        dumpFileNumber = 1;
      case 'd':
        // d = just dump the next recording
        isRecording = false;
        getNextSDFilename();
        dumpOneRecording();
        dumpFileNumber++;
        showMenu();
        break;

      // erase the flash - takes about 20 sec for my 25Q64 device
      case 'e':
      case 'E':
        isRecording = false;
        eraseFlash();
        showMenu();
        break;

      // start recording
      case 's':
      case 'S':

        Serial.println(F("START RECORDING"));
        getNextFlashFilename();
        //getNextSDFilename();
        Serial.print( F("Writing to file: " ));
        Serial.println( filename );
        SerialFlash.create( filename, FILE_SIZE_512K );
        file = SerialFlash.open( filename );
        fakeAcceleration = 1.0;
        oneRecord.recordNumber = 0;
        isRecording = true;
        break;

      // stop recording
      case 'p':
      case 'P':
        Serial.println(F("STOP RECORDING"));
        Serial.print(F("Logged "));
        Serial.print( oneRecord.recordNumber );
        Serial.println(F(" records for that flight."));
        //
        Serial.println(file);
        file.close();
        isRecording = false;
        break;

      default:
        showMenu();
        break;
    }
  }
}
// Display the menu of supported commands
void showMenu() {
  Serial.println(F("\nMENU OPTIONS:"));
  Serial.println(F("d : dump 1 recording (D to reset to 1st recording)"));
  Serial.println(F("e : erase flash (~20sec for 64mbit device)"));
  Serial.println(F("s : start recording"));
  Serial.println(F("p : stop recording"));

}

// generate a CSV formatted output of one flights worth of recordings
void dumpOneRecording() {
  Serial.println(F("\nDUMP ONE RECORDING"));

  sprintf( filename, "flt%d.bin", dumpFileNumber );
  //Opens the flash chip file
  file = SerialFlash.open( filename );
  //Opens and created the file on the SD card
  DataFile = SD.open(SDfilename, FILE_WRITE);
  if (file) {
    Serial.print( F("Contents of file: ") );
    Serial.println( filename );

    // print out the headings
    Serial.println(F("Rec Num,Timestamp,Temp,Press,Accel,Height"));

    do {
      file.read( (uint8_t *)&oneRecord, sizeof(oneRecord) );

      // library doesn't know where end of actual written data is so we have
      // to look for it ourselves!
      if ( oneRecord.recordNumber != 0xFFFFFFFF ) {
        Serial.print( oneRecord.recordNumber );
        Serial.print( "," );
        Serial.print( oneRecord.timeStamp );
        Serial.print( "," );
        Serial.print( oneRecord.temperature );
        Serial.print( "," );
        Serial.print( oneRecord.pressure );
        Serial.print( "," );
        Serial.print( oneRecord.acceleration );
        Serial.print( "," );
        Serial.println( oneRecord.height );
        //Writes to SD
        DataFile.print( oneRecord.recordNumber );
        DataFile.print( "," );
        DataFile.print( oneRecord.timeStamp );
        DataFile.print( "," );
        DataFile.print( oneRecord.temperature );
        DataFile.print( "," );
        DataFile.print( oneRecord.pressure );
        DataFile.print( "," );
        DataFile.print( oneRecord.acceleration );
        DataFile.print( "," );
        DataFile.println( oneRecord.height );
      }
    } while ( oneRecord.recordNumber != 0xFFFFFFFF );
    //Saves the files
    file.close();
    DataFile.close();
    Serial.println("Data written to SD");

  }
  else {
    Serial.println(F("No more files."));
  }
}

// erase the flash and write our unique signature in the 1st record
void eraseFlash() {
  Serial.println(F("ERASING FLASH CHIP"));
  SerialFlash.eraseAll();

  // create a dummy file so we can look for it
  // this seems the only way of knowing if there is an actual
  // filesystem present when we startup ...
  SerialFlash.create( "dummy.txt", 16 );
  file = SerialFlash.open("dummy.txt");
  file.write( "Hello", 6 );
  file.close();
  Serial.println(F("Done"));
}

// print out some details about the flash chip
void flashInfo() {
  uint8_t id[5];

  SerialFlash.readID(id);
  Serial.print(F("Capacity: "));
  Serial.println(SerialFlash.capacity( id ));
}

// Scans the flash chip to determine the next filename to use.
uint8_t getNextFlashFilename() {
  uint8_t fn = 0;

  do {
    // create the filename and see if it exists
    //Creates the filename with increasing numbers
    sprintf( filename, "flt%d.bin", ++fn );
    Serial.println( filename );
  } while ( (SerialFlash.exists( filename )) && (fn < 10) );

  Serial.println( fn );
  return fn;
}
//Scans SD and checks for next file name
uint8_t getNextSDFilename() {
  uint8_t sd = 0;

  do {
    // create the filename and see if it exists
    sprintf( SDfilename, "flt%d.txt", ++sd );
    Serial.println( SDfilename );
  } while ( (SD.exists( SDfilename )) && (sd < 10) );

  Serial.println( sd );
  return sd;
}

//Just used to create the fake acceleration values
float getFakeAcceleration() {
  fakeAcceleration = fakeAcceleration + 0.01;
  return fakeAcceleration;
}
1 Like