Show Posts
Pages: 1 2 [3] 4 5 ... 109
31  Using Arduino / Storage / Re: Reading from CSV file and storing into multiple arrays with SdFat library on: July 24, 2014, 06:36:56 pm
First, you should check for errors when you read from a file like this:
  if (!(sdin >> x[xyzpoints] >> y[xyzpoints] >> z[xyzpoints])) {
    // Handle read error.

Second, you must provide a char variable to read the comma separators:

  if (!(sdin >> x[xyzpoints] >> sep1 >> y[xyzpoints] >> sep >> z[xyzpoints]) || sep1 != ',' || sep2 != ',') {
    // Handle read error.

Where sep1 and sep2 are type char:
  char sep1, sep2;
32  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 24, 2014, 09:51:15 am
This feels like a search for the Holly Grail.
He's not included any of the ARM core MCUs. If he expanded it to Cortex-M3.. for DUE.. .Then we'd want him to expand it to Cortex-M0 for ZERO.... then there would be others of us who'd like to see it expanded to Cortex-M4 for TI Tiva C series.
I am mainly interested in Cortex-M processors.   There is no hope that one person or even a few people can provide an open source firmware library for these devices.

Here is a project with the goal for providing such a library
The libopencm3 project (previously known as libopenstm32) aims to create a free/libre/open-source (LGPL v3, or later) firmware library for various ARM Cortex-M0(+)/M3/M4 microcontrollers, including ST STM32, Ti Tiva and Stellaris, NXP LPC 11xx, 13xx, 15xx, 17xx parts, Atmel SAM3, Energy Micro EFM32 and others.
They barely have software to configure the pins on devices. There is no support for SDIO on any device

Kowalski's work with Cosa is a really interesting development for a C++ OO Framework. It's a bare metal approach that will give us some task scheduling that might allow us to do some things that we expect an RTOS to accomplish.

Embedded systems are about time and Costa almost ignores problems of concurrency.  The importance of preemptive systems has been understood for over 40 years.
Liu, C. L.; Layland, J. (1973), "Scheduling algorithms for multiprogramming in a hard real-time environment", Journal of the ACM 20 (1): 46–61,

Here is a summary of this important theoretical result

There are lots of resources for Cortex M  so I can't complain too much.
33  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 23, 2014, 08:28:25 am
Small simple projects will continue to use the bare metal approach.  An OS makes little sense for under 32 KB of program space.  A simple development environment is adequate.

For larger DIY/hobby projects using 32-bit processors there are now many hardware options and the future will bring many more.  It would be a shame to be limited to choices made by the Arduino company.

About 30 years ago big science faced a similar situation.  From the late 1960s to mid 1980s DEC (Digital Equipment Corp)  minicomputers dominates control and data acquisition in large science experiments.  DEC supplied the software environment with systems like RSX/M, a real time system, and VMS for larger VAX machines.

Around 1985 single board computers based on the MC68000 and other microprocessors started to be used There was no standard OS for these boards.

Fortunately two of my friends, Jerry Fiddler, and David Wilner, had left the Lab in 1982 and founded Wind River Systems.  Wind River developed VxWorks which is a flexible RTOS that can be targeted to a wide variety of processors.  VxWorks became very popular in large physics experiments, space experiments like the NASA Clementine moon probe, and the Mars landers.  This defacto standardization was really important for big science at that time.

UNIX on SUN and other work stations became the standard OS for operator consoles and other non-real time applications . Linux is now  the best choice for boards like Rpi and BB that have Cortex Application processors.

The problem is that there are too many RTOS choices for micro-controllers like Cortex M and PIC32.  Many companies are supporting FreeRTOS but it is not technically outstanding.

I like ChibiOS/RT but it only has great HAL support for STM32.

I think Arduino is like Digital Equipment Corp and will fade in the same way.  There will are too many other outstanding hardware options and there will be even more in the future.

There is not an obvious software solution for DIY/hobby users.  FreeRTOS supports a wide variety of hardware, just look at this list  FreeRTOS is just not very exciting.
Too bad it is so easy to build a little RTOS kernel, this means there are many half baked RTOS systems instead of a project like UNIX and it's follow on, Linux.

westfw is right, the problem is the proper OS for DIY/hobby users.
Or an OS for  Teensy/NXP/ST/TI that is better than the current bare metal, without sacrificing the "ease of use" of the Arduino libraries.  But I don't think that either one exists yet.
34  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 22, 2014, 03:34:07 pm
I've just received a Tiva C-Series TM4C1294XL evaluation board from Texas Instruments.
Looks like a nice board.

This is great, the major companies are all building boards that can compete with Adruino and are talking to users.
We are looking forward meeting Makers and showcasing the MCU LaunchPad development ecosystem at Maker Faire Bay Area in San Mateo, Calif.

These companies are also working with universities to provide better tools and products for labs.
35  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 22, 2014, 10:05:06 am
The example shows how to use a per-allocated contiguous file.  Study it and use a similar method on Teensy.

Are you a lazy student trying to get others to think for you?

I won't be replying to any more of your questions since the examples I pointed out have the basic solution.

36  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 22, 2014, 07:08:01 am
So basically it would be sufficient to check if the card is busy in checking if DO is low, and only send commands and / or data packets to the card when DO is high

Won't work.  Please read the following carefully.

When you write to a file, a large number of SD commands and operations operations are required.  There is no way you could check for busy before executing this statement.
logfile << charBuf << flush;
This statement may require a cluster to be allocated.  That requires reading FAT blocks until a free cluster is found.  Updating the FAT block in memory, writing it back to the SD, then updating the backup FAT.  Even if you write 512 bytes, the data may cross a block so there could be read of a block, copy part of the data to that block, write the block back, then write a second block.  Then the flush requires reading the directory entry for the file, updating the directory entry and writing it back.

The SdFat library does check for busy before every write since the write would fail otherwise.  The problem here is that the card won't indicate busy until you send the write command.  Then you must wait before sending the data packet. If you write a RU which may be 16 KB, there may be 32 busy periods.

SdFat can't check for busy before reads.  The card sends a stream of busy tokens for read and finally sends a start data token.   

No reasonable SD library can do what you want.  Chan's FatFS does not have a non-blocking mode.  FatFS is widely use and the latency problem is handled by using FatFS in an RTOS.

I also have a new replacement for SdFat that I use in RTOSs like ChibiOS and FreeRTOS.  The problem is simpler on ARM processors with more features than Teensy.  You can use SDIO and write a driver that sleeps while the SDIO controller does all the busy checking for a large read or write.  Other threads run and no CPU time is lost in busy checks.

There is another possibility.  You can create a large continuous file and write it with low level SD operations.  You must still buffer data but the check of MISO should work. 

This program used one large multiple block write and avoids most busy periods. It can log data much faster than your requirement on an Uno .
I did a reliability test logging five analog pins at 5,000 samples per second.  This is an ADC rate of 25,000 values per second.  I logged 512 MB of data without dropping any values.

37  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 21, 2014, 11:02:13 am
This won't work.
What about the idea to first open a file on the SD card (T). Then pre-erase a block of 8 KB on the SD card in the next loop (T+1). Send 8 KB from the buffer to the SD card in the following loop  (T+2) , but in a way that the code is not blocked. Then a few loops later (T + 10) close the SD file.

Use an RTOS, that will work.
There are several ways to deal with this problem.  You can use an RTOS to buffer sensor data in a high priority thread and write to the SD in a low priority thread.  See the SD logger example in NilRTOS
38  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 20, 2014, 09:54:58 pm
This would also be true of any smaller buffer size?

SPI mode performance of the new generation of SD cards varies greatly and is not very predictable.  Manufactures only include SPI mode since the SD spec requires it.  Sometimes larger buffers reduce the frequency of long latency write but not the time.  In other cases the time is reduced.

New cards are optimized for phones and tablets in 4-bit SDIO mode.  There is little correlation between SPI performance with small writes and large writes in SDIO mode. 

I don't think there is a future in optimizing SPI mode performance for fast data logging. 

I am buying  development boards with SDIO mode SD sockets.  The SDIO controller on these boards have CRC and a fast DMA mode.
39  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 20, 2014, 06:40:43 pm
logfile << charBuf << flush;      //Write to SD Card  
Writing single 512 byte blocks to files on modern SD cards will always have long unpredictable write delays.  The basic flash write size is not 512 bytes.

Modern SD cards have a Recording Unit (RU) that is a multiple of 16 KB.  High performance cards may have 32 KB or larger RUs.

When a 512 byte block is forced to the card, a new RU must be written.  Existing data must be copied to the new RU by the card controller.  This could mean 32 KB of data is written to flash each time you do a flush.  This soon exhausts the free erased RUs and the card will block while it erased an AU of flash which may be several MB.

Check if the SD card is ready to accept data.
There is no command to do this.  Any new features that could help are not available in SPI mode.  SPI mode is very limiting and only marginally supported in modern cards.

There are several ways to deal with this problem.  You can use an RTOS to buffer sensor data in a high priority thread and write to the SD in a low priority thread.  See the SD logger example in NilRTOS

I wrote SdFat which is also the base for SD.h.  I have no plans to add new features to avoid this problem.  Writing to a file system using SPI mode with a 512 byte cache buffer will have long delays with modern SD cards.

I am experimenting with SDIO using 32 KB buffers on an STM32F407.  I can write at over 5 MB/sec with no busy delays.  I can't port this code to Arduino.

Edit: I can write at over 20 MB/sec with only small, less than 1 ms, occasional busy delays using 4-bit SDIO.
40  Using Arduino / Storage / Re: file creation dates with on: July 14, 2014, 02:00:06 pm
Can somebody explain why the actual creation (last modification?) dates aren't shown correctly? Thanks.

Files will only be time/date stamped if you have a RTC, RTC library, and time/date callback functions for SdFat, the base library for SD.h.
41  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 14, 2014, 11:48:59 am
While I agree that micontrollers will have a place for quite some time, I'm not as
sure about the 8 bit AVRs.
I started to use Arduino because I wanted an easy way to work with microcontrollers. 

I started using Arduino in 2008 and wrote the Fat16 library for the 168 Arduino with 16 KB flash and 1 KB SRAM.  It was just a toy to play with.

Now I want a microcontroller like Cortex M4 with more power than AVR but not an application processor designed for "hosting a rich OS platform" like the Cortex A series.

I have been developing a new FAT file system using STM32F4 processors.  The difference in capability over AVR is overwhelming.  I tried to love the Arduino Due when it came out but couldn't.

Here are two examples. 

The ADC on AVR Arduinos run at a max of about 10 ksps with analogRead.  You can increase the clock speed and get fair, not full 10-bit, accuracy at 40 ksps.  Low end STM32F4 ADCs can do 2.4 msps, about 60 times faster than AVR with 12-bit accuracy and very flexible DMA.  High end STM32F4 chips can run three ADCs interleaved for 7.2 msps.

I am developing a new file system for high end embedded processors.  The STM32F4 has a good, not great, SDIO controller which can run the 4-bit bus at over 48 MHz.  I have achieved sustained read speeds over 20 MB/sec.  I have sustained write speeds of 12 MB/sec but this is a limited by the SD card I am using.  I have ordered a new Samsung Pro card and hope to write at close to 20 MB/sec.

The AVR can never read or write faster than about 500 KB/sec that's about the best you can do with program I/O on the 8 MHz SPI bus.  In addition SD cards don't present their fast commands on the SPI bus, only on the SDIO bus.  This means you will have very long occasional write latencies with SD cards on the SPI bus.

The Arduino system software is not adequate for these new chips. You really need priority based pre-emptive scheduling to take full advantage of the features of these chips.  Most chip manufactures are now providing a free RTOS solution.  ST and Atmel provide FreeRTOS ports, Freescale has MQX, and there are other great systems like ChibiOS.

I really enjoy development with boards like the  STM32F4 Nucleo boards.  ChibiOS can do about 1.2 million context switches per second on the NUCLEO-F411RE.  I am using a Olimex stm32-H407 and it can do a context switch in 400 ns.  The 407 has 196 KB SRAM and 1 MB flash.
42  Using Arduino / Storage / Re: 0X6 and 0X4 errors in ReadWriteSdFat on: July 13, 2014, 01:17:57 pm
The error probably happened in the close before the re-open for read.  The error is a write error and likely happened when buffers were flushed in the close.
// close the file:
  myFile.close(); // <--------------------Error here

  // re-open the file for reading:
  if (!"test.txt", O_READ)) {
    sd.errorHalt("opening test.txt for read failed");

I didn't add a check for close failure when I modified the Arduino SD.h example.

I use an LC Studio SD shield
This module is very unreliable.  Search the forum for articles about this module.  It can be fixed but it is not worth the time and money.
43  Using Arduino / Storage / Re: fastest data logging option ? on: July 12, 2014, 06:51:32 pm
The 23LCV1024 has a battery backup pin so it can retain data

Another option is FRAM
44  Using Arduino / Storage / Re: Reading 250,000,000 values from sensor and storing it on SD card on: July 09, 2014, 07:05:36 am
I ve also got Raspberry pi if it is easier on that..though i ve never used it till now .. !!

The Pi does not include a hardware analog to digital converter, but a external ADC (such as the MCP3008) can be used along with some bit banged SPI code in python to read external analog devies. Here is a short list of some analog inputs that could be used with this setup.

Raspberry Pi won't come close for your needs.
45  Using Arduino / Storage / Re: Reading 250,000,000 values from sensor and storing it on SD card on: July 08, 2014, 02:20:35 pm
Here is the ($24 from Future Electronics)  STM32F429 board I am using

It's almost trivial to solve your problem with this board if you are a skilled STM32 programmer.

The board has an 8MB RAM device that is accessible by the DMA controller.  You just setup a timer, the ADC, and a DMA channel and acquiring up to 4 million samples is almost automatic.  The board has three ADCs and each ADC can digitize samples at 2.4 million/sec.

The three ADCs can run in interleaved mode to digitize one channel at 7.2 million samples/sec.

Too bad you don't have the skills to use this board.  It kills Due at 180 MHz,  2 MB flash, 256 KB fast RAM, and 8 MB SDRAM.
Pages: 1 2 [3] 4 5 ... 109