Go Down

Topic: 128kSPS fast ADC data logging and processing (Read 7068 times) previous topic - next topic

hgpt

Oct 16, 2017, 03:06 pm Last Edit: Oct 16, 2017, 03:18 pm by hgpt
Hi all,

I recently got a task to do using the AD7767 128 kSPS 24-bit ADC. The task states that I need to plot the output of the ADC on a time scale. (the ADC is hooked up to a load cell).

I've worked with slower ADCs in the past (@4.8kSPS) and what I used to do then is print the values along with millis() to serial console and use Processing IDE to capture the data and save it to a CSV file. By doing so I was then able to copy the data from CSV straight into Matlab and plot the graph. The problem with AD7767 is that it is much faster than any previous ADC I've used and the above method will not work due to the increased sampling speed.

I've managed to get the AD7767 ADC working using an external oscillator and some other components. The problem that I face is that the ADC is way to fast for the serial console monitor (even after setting the baud rate up to 2000000). I'm able to see some values, but not all of them.
128kSPS = 3,0720,00 bps which is much faster than 2,000,000 bps (maximum speed for Serial Monitor).

I was thinking about logging the data to an external memory (flash, EEPROM etc) and read it afterwards. Do you have any idea if that would work?


Also any suggestions would be greatly appreciated. Thanks!

ard_newbie

#1
Oct 16, 2017, 03:53 pm Last Edit: Oct 16, 2017, 04:06 pm by ard_newbie
A few thoughts:

With an AD7767, you will receive a 24_bit sample every 7 us. I don't know how you will receive your samples (SPI, I2C,on 24 parrallel pins,...) but I guess this sample will be stored into a 32_bit integer.

An Arduino DUE has 96KBytes of SRAM, so let's say you will use 90KBytes to log your samples plus a time scale like micros() into another 32_bit integer.

Once you have logged 11520000 samples (plus the associated time scale), you send the entire buffer to your PC if this amount of samples is enough, if it's not, send periodically smaller buffer (e.g. 1000 samples) to your PC.

hgpt

#2
Oct 16, 2017, 04:54 pm Last Edit: Oct 16, 2017, 05:00 pm by hgpt
You've guessed it right, I'm storing the values using an unsigned long variable; so, yes, 32 bits.

I'm not quite sure that 96KBytes is enough. AD7767 outputs 3072000 bps or 384 KBps (384000 bytes/s) (correct me if I'm wrong). Given this amount  of data 96KBytes will do no more than a maximum of 250ms of recorded data, which isn't that much at all.

I'm looking for about 10-15s of data to be registered before it gets transferred to the PC. 10s are about 1,280,000 samples, that's about 4MB worth of data. Also, it would be nice is to be able to record new data as old one gets transferred to the PC.

I was thinking about using a dual core ESP32 board (I do have a bit of experience with this board) to register and transfer data at the same time, using 2 buffers and ping-pong technique to be able to write new values to the one buffer while reading old ones from the other one.

Also, is it possible to do the same thing using an Arduino, 2 external flash storage devices and an external intererupt from the ADC (DRDY pin)?

Any other ideas?

ard_newbie


Assuming you are using an Arduino DUE:

Send data to a PC via UART at 1 Mbps ( clock your DUE at 96 MHz, configure UART _BRGR for 1Mbps baude rate, select PDC DMA for UART transfers), whilst using SPI to read 24_bit samples from AD7767 at 128 KHz sampling frequency. Note that you have SPI via SPI header plus USART0 and USART1.

However, I doubt that you will really get a 24_bit precision ADC conversion because of the noise  :o 

hgpt

Unfortunately the only boards that I have available right now are:
- Sparkfun ESP32 Thing
- NodeMCU ESP8266
- Arduino Uno R3
- Arduino Pro Mini
- Arduino Nano

I guess out of them all ESP32 is the one to use regarding this task.

I still don't see how a 1Mbps connection to PC via UART will work, if you take into consideration the fact that the ADC outputs data at more than 3Mbps. The objective is to plot a graph of ADC value/time, while taking into consideration all of the samples provided by the ADC, not just some of them.

hgpt

#5
Oct 17, 2017, 12:12 pm Last Edit: Oct 17, 2017, 12:19 pm by hgpt
Further update using an ESP32 board.
I've tried doing the most basic thing, which is printing millis() as soon as the Data Ready Pin (DRDY) signals the end of an ADC sample conversion. To my surprise, even at 2,000,000 baud rate I get at most 33k samples. (code bellow).
Code: [Select]

#define DRDY  12

void setup() {

  Serial.begin(2000000);

}

void loop() {
  if (!digitalRead(DRDY))
    Serial.println(millis());
}


Outputs:
Code: [Select]

....
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
52
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
53
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
54
55
.....


As you can see from the above output there are around 35 values per millisecond or about 35k samples a second. If I try printing millis() along with ADC value I get an even lower sample rate, about 22k sample a second. I've also tried hooking the DRDY pin to an interrupt pin and use an external interrupt to log the event, but the results are the same.

I've verified the frequency at which the DRDY pin changes it state (using a scope) and it's around 129KHz (image bellow) which corresponds to the AD7767 datasheet. Does this have to do anything with the microcontroler being used? I doubt so given the fact that the ESP32 runs at 240MHz.


Any idea on what seems to be the root of the problem?

MarkT

A standard asynchronous serial interface isn't normally chosen for a high speed data acquisition system.
This goes for any external memory chip as well as uploading the results.

A microcontroller with lots of RAM, or with the ability to talk to an external parallel RAM chip would
be my choice.  Some can directly interface DDR memory (but that's all high speed surface mount
stuff).

This might be better suited to a full blown compute engine like a RaspPi.
[ I will NOT respond to personal messages, I WILL delete them, use the forum please ]

dc42

Two suggestions:

1. Unless you really need 24 bits of data in each reading, send fewer bits to the PC e.g. 16.

2. Use an Arduino Due, and use the native USB port to send data to the PC.
Formal verification of safety-critical software, software development, and electronic design and prototyping. See http://www.eschertech.com. Please do not ask for unpaid help via PM, use the forum.

hgpt

#8
Nov 03, 2017, 12:55 pm Last Edit: Nov 03, 2017, 01:45 pm by hgpt
A standard asynchronous serial interface isn't normally chosen for a high speed data acquisition system.
This goes for any external memory chip as well as uploading the results.

A microcontroller with lots of RAM, or with the ability to talk to an external parallel RAM chip would
be my choice.  Some can directly interface DDR memory (but that's all high speed surface mount
stuff).

This might be better suited to a full blown compute engine like a RaspPi.
Thank you for the info. I was thinking that I might do something wrong, but from what you're saying it appears that the problem has to do with hardware limitations.

The CP2102N USB to UART bridge that I use supports up to 3Mbaud. If you take into account that I'm sending 24 bits packed in a 32bit long type variable, along with another 32bits from millis(), that's about 64 bits for each serial print.
3,000,000 / 64 = 46,8 ksps ≃ 46 ksps.
So, I should be getting around 46 ksps. However, practically I only get around 22ksps. Where are the remaining 24ksps?

Meanwhile, using an ESP32 board that I have, I've tried using the internal 4MB flash memory (GD25Q32C) that's hooked up on the SPI interface to store ADC data @32ksps (I've used a slower ADC) and.... success.  :)
Down bellow you'll find the graph. It does correspond to the action that I was performing on the load cell (3 compressions in a row). It works with 128ksps too.
So, using a buffer in RAM and dumping the data in Flash works, but a cost: I only get a few second of recording, till the flash is full.

I still don't understand why the serial interface doesn't work at it's full potential of 3Mbaud as stated by the datasheet. Any ideas on that? Is the Raspberry Pi the only solution regarding this problem?

Two suggestions:

1. Unless you really need 24 bits of data in each reading, send fewer bits to the PC e.g. 16.

2. Use an Arduino Due, and use the native USB port to send data to the PC.
1. I really need the full 24bits of data along with the fast sampling rate.
2. I don't see how that would help.

If it helps anyone here's the code that I've used to store data to flash.
Code: [Select]
#include <SPI.h>
#include "esp_spi_flash.h";

//board chip GD25Q32C

#define SCK   27
#define MOSI  26
#define MISO  25
#define CS    5
#define DRDY  12

#define SERIAL_BUFFER_SIZE 256

unsigned long fsize, var;

//1 block =  16 sectors
byte sectors_per_block = 16;

//1 sector = 4096 bytes
//1 sector = start_address ---- finish_address
//size of 1 sector = 0x001000 (4096 bytes)
int size_of_one_sector = 0x001000;

//5 - 60

byte start_block = 5;
byte blocks_to_write = 20;
byte finish_bock = start_block + blocks_to_write;

byte ok = 0;

struct adcStruct {
  unsigned long value;
  unsigned long time_ms;
};
adcStruct samples[512];

void setup() {

  Serial.begin(2000000);

  // HG ESP32 SPI pins
  pinMode(MISO, INPUT);
  pinMode(MOSI, OUTPUT);
  pinMode(SCK, OUTPUT);
  pinMode(CS, OUTPUT);
  pinMode(DRDY, INPUT_PULLUP);

  SPI.begin(SCK, MISO, MOSI, CS);

  spi_flash_init();
  fsize = spi_flash_get_chip_size();

  Serial.print("Flash size...");
  Serial.println(fsize);

  erase_targhet_blocks(start_block, blocks_to_write);
}

void loop() {
  if (!ok) {
    Serial.println("Started...");
    Serial.println(millis());

    for (int block = start_block; block < finish_bock; block++)
      adcRecordBlock(block);


    Serial.println("Finished...");

    adcDisplay();

    ok = 1;
    Serial.println("Finished printing values...");
  }
}

void adcDisplay() {
  int start_sector, finish_sector;
  unsigned long current_addr;

  for (int block = start_block; block < finish_bock; block++) {

    start_sector = block * sectors_per_block;
    finish_sector = start_sector + sectors_per_block;

    for (int sector = start_sector; sector < finish_sector; sector++) {
      current_addr = sector * size_of_one_sector;

      spi_flash_read(current_addr, samples, sizeof(samples));

      for (int i = 0; i < 512; i++) {
        Serial.print(samples[i].value);
        Serial.print("\t");
        Serial.println(samples[i].time_ms);
      }
    }
  }
}

void adcRecordBlock(int block) {
  int start_sector = block * sectors_per_block;
  int finish_sector = start_sector + sectors_per_block;

  for (int sector = start_sector; sector < finish_sector; sector++)
    adcRecordSector(sector);
}

void adcRecordSector(int sector) {
  unsigned long current_addr = sector * size_of_one_sector;
  unsigned long previous_ms;
 
  for (int i = 0; i < 512;) {
    if (!digitalRead(DRDY)) {
      SPI.beginTransaction(SPISettings(35000000, MSBFIRST, SPI_MODE0));
      var = B00000000;
      var = var * 256 + SPI.transfer(0xFF);
      var = var * 256 + SPI.transfer(0xFF);
      var = var * 256 + SPI.transfer(0xFF);
      SPI.endTransaction();

      if(var >= 16000000 && i>=1) var = samples[i-1].value;
      samples[i].value = var;
      samples[i].time_ms = millis();

      i++;
      //wait for DRDY to go high again
      //for is much faster than DRDY --> DRDY stays low for 1 sample,
      //but due to for speed that sample is at risk of beeing aquired multiple times
      //Solution: wait for DRDY to go high again before storing another sample
      while (digitalRead(DRDY) != 1) {}
    }
  }

  spi_flash_write(current_addr, samples, sizeof(samples));
}


void erase_targhet_blocks(byte start_block, byte blocks_to_write) {
  byte finish_bock = start_block + blocks_to_write;
  int start_sector, finish_sector;
  unsigned long current_addr;

  for (int block = start_block; block < finish_bock; block++) {
    start_sector = block * sectors_per_block;
    finish_sector = start_sector + sectors_per_block;

    for (int sector = start_sector; sector < finish_sector; sector++)
      spi_flash_erase_sector(sector);
  }
}

MarkT

I still don't understand why the serial interface doesn't work at it's full potential of 3Mbaud as stated by the datasheet. Any ideas on that?
You have to allow for latency, not just throughput.  At 3Mbaud the time to format values will start to
become significant - that's partly due to C++ method call overhead and partly because you have
to actually do the conversion before sending the characters.  You also have to start worrying about
detailed timing of ISR handling and interrupt latencies.  Handling an interrupt often has to dump
lots of registers onto the stack and restore them later.  If the serial hardware doesn't have a FIFO then
interrupt handling is a big limiting factor - if it does have a FIFO then in theory its not too hard to
keep it from emptying and get full rate.  But that means the software library has to be written to
handle the FIFO properly (which is more complex than just feeding a character at a time).

In short there can be many reasons full potential of hardware is not met - its normal not achieve it
in fact.
[ I will NOT respond to personal messages, I WILL delete them, use the forum please ]

Go Up