Logging onto SD while maintaining high hz

Hey guys!

I want to log attitude data gathered from a gyro. In addition to that I want to measure, calculate and log some other data and I am having trouble doing all this while maintaining a high enough speed in order to grab and use the gyro's data.

Could I use an interrupt function to get the latest info from the gyro every 4ms or something? What happens when an interrupt gets called during a write process to th SD? Does it continue writing after finishing the interrupt? gets the data on the sd corrupted?

Is there a way to speed up the SD logging process? I am already basically just the things I have to.

If there is no way to accomplish this with programming tricks would it be feasible to have one arduino handeling the gyro and then sending the attitude data to a second arduino for logging purposes via i2c?

Hi,
Try logging the raw data, and doing the calculations after the event, that is if you don't need the calculated info in real time.

Tom... :slight_smile:

That’s a good idea but still brings me no where near compared where I need to be. I am currently looking at ~2600 microsec cycletime for the gyo. Pretty sure there is no way I can write what I need to write onto the SD in that time.

Furthermore I am reading preasure from an I2C device which requires me to first write something to it and then wait for about 10milliseconds beore I can read the result. This is why I was so interested in the interrupt option

What happens when an interrupt gets called during a write process to th SD?

The process is suspended.

Does it continue writing after finishing the interrupt?

Yes.

gets the data on the sd corrupted?

No.

The Arduino is handling interrupts all the time. Every clock tick, an interrupt happens. You can't really believe that writes to an SD card take less than a clock tick, can you?

Could I use an interrupt function to get the latest info from the gyro every 4ms or something?

Maybe. It depends on HOW you get the gyro data. If you are using the Wire library, which needs interrupts enabled, then, no, because when your interrupt handler, to handle the timer event, is called, interrupts are disabled.

Is there a way to speed up the SD logging process?

Write less data.

Of course, this is all speculation without seeing your code.

Thank you very much PaulS for your help!

PaulS:
The Arduino is handling interrupts all the time. Every clock tick, an interrupt happens. You can't really believe that writes to an SD card take less than a clock tick, can you?

I didn' know that an interrupt was called every clock tick. Is this in order to increment the timer?

PaulS:
Maybe. It depends on HOW you get the gyro data. If you are using the Wire library, which needs interrupts enabled, then, no, because when your interrupt handler, to handle the timer event, is called, interrupts are disabled.

Unfortunately I am using the Wire libary to talk to the gyro. Can't I call an interrupt within an interrupt? Is that why that won't work?
Is there a way to use the i2c interface without the Wire library so I could use an interrupt?

Thanks again for your great and detailed support (=

Is this in order to increment the timer?

It is in order to increment the value that millis() returns (among other things).

Can't I call an interrupt within an interrupt?

You can enable interrupts from an interrupt handler IF you know what you are doing, and what all the interrupts that will happen are, and what their handlers do. I don't, and wouldn't even consider that approach.

Is this in order to increment the timer?

Yes.

Can’t I call an interrupt within an interrupt?

Not normally.

Is there a way to use the i2c interface without the Wire library so I could use an interrupt?

Yes, you can access the hardware I2C module directly and wait for the output in a software loop. There are other I2C libraries that do this.

Here are the function calls I use (the “read multiple bytes” are custom routines and won’t work for some devices).

/*

Simple I2C routines
ATmega328 @ 16 MHz  Atmel Studio IV/ avr-gcc

*/

#define F_CPU 16000000UL
#include <math.h>
#include <util/delay.h>
#include <avr/io.h>

// ---------------------------------------------------------------------------
// I2C (TWI) ROUTINES
//
// The standard clock rate is 100 KHz, and set by I2C_Init
// FIXED I2C_Stop() sjr

#define F_SCL 100000L // I2C clock speed 100 KHz
#define READBIT 1  //low bit of device address for read
#define TW_START 0xA4 // send start condition (TWINT,TWSTA,TWEN)
#define TW_STOP 0x94 // send stop condition (TWINT,TWSTO,TWEN)
#define TW_ACK 0xC4 // return ACK to slave
#define TW_NACK 0x84 // return NACK to slave
#define TW_SEND 0x84 // send data (TWINT,TWEN)
#define TW_READY (TWCR & 0x80) // ready when TWINT returns to logic 1.
#define TW_STATUS (TWSR & 0xF8) // returns value of status register

// I2C register definitions


void I2C_Init(){

// at 16 MHz, the SCL frequency will be 16/(16+2(TWBR)), assuming prescalar of 0.
// so for 100KHz SCL, TWBR = ((F_CPU/F_SCL)-16)/2 = ((16/0.1)-16)/2 = 72
    TWSR = 0; // prescalar to zero
    TWBR = ((F_CPU/F_SCL)-16)/2; // set SCL frequency in TWI bit register
}

void I2C_Stop(void) {

	TWCR=TW_STOP;
	 // wait for stop condition to be executed on bus
  	 // TWINT is not set after a stop condition!
 	while(TWCR & _BV(TWSTO));
}

unsigned char I2C_Detect(unsigned char addr){

// look for device at specified address; return 1=found, 0=not found
    TWCR = TW_START; // send start condition
    while (!TW_READY); //wait
	TWDR = addr; // load device's bus address
    TWCR = TW_SEND; // and send it
    while (!TW_READY);
    return (TW_STATUS==0x18); // return 1 if found; 0 otherwise
}

void ShowDevices(void){

// search all 127 addresses, report those present on the I2C bus

    for (unsigned char addr=1; addr<128; addr++) {
        if (I2C_Detect(addr<<1)) // I2C detected?
           printf(" .%02X",addr<<1);
		I2C_Stop();
    	}
}

void I2C_Start (unsigned char slaveAddr) {

    I2C_Detect(slaveAddr);
}

unsigned char I2C_Write (unsigned char data) {

// sends a byte to slave

    TWDR = data; // load data to be sent
    TWCR = TW_SEND; // and send it
    while (!TW_READY); // wait
    return (TW_STATUS!=0x28); //0 if successful, 1 if not
}

unsigned char I2C_ReadACK () {

// reads a byte from slave

    TWCR = TW_ACK; // ack = will read more data
    while (!TW_READY); // wait
    return TWDR;
    //return (TW_STATUS!=0x28);
}

unsigned char I2C_ReadNACK () {

// reads a byte from slave

    TWCR = TW_NACK; // nack = not reading more data
    while (!TW_READY); // wait
    return TWDR;
	//return (TW_STATUS!=0x28);
}

void I2C_WriteByte (unsigned char busAddr, unsigned char data) {

// write byte to slave

    I2C_Start(busAddr); // send bus address
    I2C_Write(data); // then send the byte
    I2C_Stop();
}

void I2C_WriteRegister(unsigned char busAddr, unsigned char deviceRegister, unsigned char data){

    I2C_Start(busAddr); // send bus address
    I2C_Write(deviceRegister); // first unsigned char = device register address
    I2C_Write(data); // second unsigned char = data for device register
    I2C_Stop();
}

unsigned char I2C_ReadRegister(unsigned char busAddr, unsigned char deviceRegister) {

// read single byte of data in register

    unsigned char data = 0;
    I2C_Start(busAddr); // send device address
    I2C_Write(deviceRegister); // set register pointer
    I2C_Start(busAddr+READBIT); // restart as a read operation
    data = I2C_ReadNACK(); // read the register data
    I2C_Stop(); // stop
    return data;
}

// Read a two-byte word, low order first

signed int I2C_ReadWord(unsigned char busAddr, unsigned char deviceRegister) {

    unsigned int data = 0;
	unsigned char l;
    I2C_Start(busAddr); // send device address
    I2C_Write(deviceRegister | 0x80); // set register pointer, autoincrement
    I2C_Start(busAddr+READBIT); // restart as a read operation
    l = I2C_ReadACK(); // read the register data
	data |= I2C_ReadNACK(); //read next unsigned char
    I2C_Stop(); // stop
    return (signed int) ((data<<8)|l);
}

// read a 3 byte value, lowest order byte first

signed long I2C_ReadPressureRaw(unsigned char busAddr, unsigned char deviceRegister) {

    unsigned char pxl,pl,ph;
    I2C_Start(busAddr); // send device address
    I2C_Write(deviceRegister | 0x80); // set register pointer, autoincrement
    I2C_Start(busAddr+READBIT); // restart as a read operation
    pxl = I2C_ReadACK(); // read ls byte
    pl = I2C_ReadACK(); // read middle
    ph = I2C_ReadNACK(); // read high
    I2C_Stop(); // stop
    return (int32_t)ph << 16 | (uint16_t)pl << 8 | pxl;
}

MarkGoingToSpace:
Hey guys!

I want to log attitude data gathered from a gyro. In addition to that I want to measure, calculate and log some other data and I am having trouble doing all this while maintaining a high enough speed in order to grab and use the gyro's data.

Could I use an interrupt function to get the latest info from the gyro every 4ms or something? What happens when an interrupt gets called during a write process to th SD? Does it continue writing after finishing the interrupt? gets the data on the sd corrupted?

Is there a way to speed up the SD logging process? I am already basically just the things I have to.

What you need is to buffer the data, using an ISR to read data into the buffer only. Process the data and log it from the main loop.

However if you are using an Uno you may find there isn't enough RAM to make this really work. Also you
require an SD library that allows interrupts to run during updates of the SD card, and that involves a little
research (reading the code of the library) to figure out (it might just work, if you can think of a way to test
that the readings are happening as intended).

Note that SD libraries need their own (512 byte I think) buffer for writing to the SD card, and an Uno has 2k RAM, a Mega 8k RAM. A Mega (or better) microcontroller is obviously a much more able to buffer.

None of this can speed up the slow SPI access to the SDcard that microcontrollers use (the native interface is completely beyond small microcontroller hardware to handle).

If there is no way to accomplish this with programming tricks...

The fundamental problem is that I2C read and SD write both block: they sit in a while loop until the operation is complete. Interrupts are enabled during these operations, because, well, the libraries use interrupts to know when to read/write the next byte (among other things). But the Arduino can only do one of those operations at a time, from start to completion.

In your case, the pressure sensor I2C read time (10ms) is longer than the gyro cycle time (I2C? SPI? read time?). Should I mention the occasional 100ms delay from an SD write? (Depends on your specific card.)

This same problem just cropped up over here. The gist is that the IMU read time was ~6ms and the SD write time was occasionally ~10ms (it may be more). The cycle time is only 10ms. The only way to do both in 10ms is to make the operations concurrent, not sequential.

Part One of The Trick is to modify the I2C library so that it will do a read in several non-blocking steps:

  1. start the read (this actually writes 3 bytes to the device);
  2. check the read state (like Serial.available()); and
  3. finish the read (like Serial.read() this returns bytes that were saved during an I2C interrupt).

Part Two of The Trick is to modify the SD library so that it will call a helper function while it's waiting for the a write to complete (actually, the previous write must complete before the current write can begin). There were murmurings of calling the yield function, but all the versions I have seen do not.

With both Parts of The Trick, you can start an I2C transfer, go do other work, watch for the I2C transfer to complete, and start writing data to the SD card (described here, code here). Provide a yield function that does all of those things except writing to the SD card (recursion!). As long as you have sufficient buffers to span the SD write time, it will work because yield is called from the SD write method while it's blocked.

You must also be careful to keep the write data rate below the SD card speed. 200KB/s to 300KB/s is easily doable.

Oh, and don't try to write all this to Serial @ 9600 baud, because the prints will block while those character trickle out, 1 character per ms.

Cheers,
/dev

Thanks a ton for your great help! This forum is truly amazing!

MarkT:
What you need is to buffer the data, using an ISR to read data into the buffer only. Process the data and log it from the main loop.

I don't quite understand how a buffer is supposed to help me in this situation - I need to write the data to the SD sooner or later. If I am buffering it it just accumulates. It is basically prograstinating for data xD. But maybe I am missing something here. And could you please tell me what ISR is?

/dev:
The fundamental problem is that I2C read and SD write both block: they sit in a while loop until the operation is complete. Interrupts are enabled during these operations, because, well, the libraries use interrupts to know when to read/write the next byte (among other things). But the Arduino can only do one of those operations at a time, from start to completion.

In your case, the pressure sensor I2C read time (10ms) is longer than the gyro cycle time (I2C? SPI? read time?). Should I mention the occasional 100ms delay from an SD write? (Depends on your specific card.)

So does this mean I can't use interrupts during neither i2c communication nor sd writing no matter what? What about the code provided by jremington?

The gyro is hooked up via i2c and the read time ~1.4ms.

/dev:
Part One of The Trick is to modify the I2C library so that it will do a read in several non-blocking steps:

Frankly I don't consider myself experienced enough to start tinkering with libraries yet...

would it be feasible to have one arduino… sending to a second arduino?

I don’t quite understand how a buffer is supposed to help me in this situation…

So does this mean I can’t use interrupts?

Frankly I don’t consider myself experienced enough to start tinkering with libraries yet…

Could you please tell me what ISR is?

I’m usually the last person to discourage a new user, but if you have these questions, you’re not ready to design and debug a program that logs data @ 384.6Hz, possibly with 2 Arduinos.

Please let me encourage you to get some other projects under your belt:

  • Start with a SD Write Speed test to understand your card’s capabilities. What are the min and max write times?

  • Configure a PWM pin (output) and connect it to another Arduino pin (input). Attach an interrupt function to the input pin and see how fast it can count, or save the current micros() in an array. Then calculate and print something in loop(), once per second, WITHOUT USING delay.

  • Learn how Serial print affects the above sketches.

  • Follow the code from a file.print all the way down into the SD library, to where it performs an SPI operation (another library).

  • Follow the code from gyro read call all the way down to twi.c, which does the I2C write (and wait and read).

  • Think about how this guy does his trick:

balancing plates.jpg

Then you’ll know why you probably shouldn’t read the gyro every 4ms from an interrupt routine, and why buffering can help relieve some of the time-critical sections, even when some part of the system blocks.

Cheers,
/dev