Show Posts
Pages: [1] 2 3 ... 107
1  Using Arduino / Storage / Counterfeit SD Cards. on: Today at 05:05:42 pm
I recently bought a number of SanDisk cards from Amazon and some from Amazon sellers.   Several were counterfeit.

Here is an example.  I bought Sandisk Ultra 8GB, 16GB, and 32GB.  The 8GB is clearly counterfeit  Here is it's CID and a performance test:
Manufacturer ID: 0X12
Product: MS
Version: 0.0
Manufacturing date: 5/2014

Type is FAT32
Card size: 7.99 GB
File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
Starting read test, please wait.

read speed and latency

The write speed, 71 KB/se,c and write latency,119 ms, are really poor.  The CID is totally wrong.

Here is the 16 GB version of the card.  I think it is genuine.
Manufacturer ID: 0X3
Product: SL16G
Version: 8.0
Manufacturing date: 4/2014

Type is FAT32
Card size: 15.93 GB
File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
Starting read test, please wait.

read speed and latency
Seems OK.  New cards don't do too well on SPI but 243 KB/sec is way better than 71 KB/sec and 20 ms compared to 119 ms for write latency.

Here is a 32 GB that seems to have a good CID but poor latency.
Manufacturer ID: 0X3
Product: SL32G
Version: 8.0
Manufacturing date: 6/2014

Type is FAT32
Card size: 31.91 GB
File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
Starting read test, please wait.

read speed and latency
The long unstable write latency might mean it's not genuine.  I don't have another 32 GB card.  I have another 16 GB card and it has short write latency.

Kind of disgusting.
2  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 28, 2014, 08:51:44 am
Unfortunately, there is no way to create OS-like functionality without incurring some overhead.
Modern embedded OSes are often more efficient than ad hoc solutions.  The algorithms in these systems have been under development for over 40 years.  It's like the difference between a bubble sort and quick sort.

A lot of intermediate-sized OS development gets distracted by "real time", which complicates everything and may not be necessary (arguably, Arduino is already NOT "real-time.")
Real-time and RTOS are unfortunate terms for an OS designed for embedded systems.  It's not about fast, it's about appropriate features.

Here are some quotes from "Introduction to Embedded Systems" by Lee & Seshia.  Download link
The computers in embedded systems often do not interact directly with humans in the same way that desktop or handheld computers do. As a consequence, the collection of services that they need from an operating system (OS) may be very different. The dominant general-purpose OSs for desktops today, Microsoft Windows, Mac OS X, and Linux, provide services that may or may not be required in an embedded processor.

An OS for embedded systems should have a few additional featues.
These OSs share many features with general-purpose OSs, but typically have specialized the kernel to become a real-time operating system (RTOS). An RTOS provides bounded latency on interrupt servicing as well as a scheduler for processes that takes into account real-time constraints.

A microkernel and modularity is desirable.
The core of any operating system is the kernel, which controls the order in which processes are executed, how memory is used, and how information is communicated to peripheral devices and networks (via device drivers). A microkernel is very small operating system that provides only these services (or even a subset of these services).

I also believe features to support the "Internet of Things" are also important.  The OS should make it easy to inter-operate with phones and other modern smart devices.

The problem is that there are thousands of RTOS to choose from, but drivers and middleware for them is lacking. I think this is one area where diversity doesn't help, and a single standard would allow people to develop drivers and middleware instead of re-inventing the RTOS.
I have evaluated RTOSes for many years.  If you screen the long list of OSes with a few key requirements such as HAL support or driver support for your chip and a modular micokernel, the list will soon be reduced to a manageable size.

The major chip companies all support an RTOS with FreeRTOS being most popular.

RTOSes are now used in the majority of 32-bit embedded systems.  Billions of copies of OSes like VxWorks and FreeRTOS are now in every day products.
3  Using Arduino / Storage / Re: SD card adapter bought on ebay on: July 27, 2014, 04:29:57 pm
I have one of those,  and it works fine with 5V supply and 5V on the signal lines.

The LC Studio modules do work with some SD cards.  I have checked several cases where cards work.  The card draws a lot of current on the 5V signal lines which may not good for the card or the Arduino  The Arduino can only supply enough current to reach about 4 V on the signal lines.

The card input lines should not draw current so I think the excess current is due to input protection devices in the card.

Not a desirable condition.
4  Using Arduino / Storage / Re: Reading from CSV file and storing into multiple arrays with SdFat library on: July 25, 2014, 05:10:27 pm
Sorry, I didn't notice that you have an extra comma at the end of input lines.  Either remove the extra comma in the input or add a third char variable to read this comma.
1120,538,0, <-- extra comma
1556,695,0, <-- extra comma

It appears to be copying the previous line's z value into each line's x value. I've been trying to track down the source of this issue without much success yet.

Trying to read a comma as a number causes an error and leaves the file's position to be incorrect.

You must provide a variable for every item in the input line.  White space is skipped so you don't need to read the CR/LF at the end of lines.
5  Using Arduino / Storage / Re: Reading from CSV file and storing into multiple arrays with SdFat library on: July 24, 2014, 06:36:56 pm
First, you should check for errors when you read from a file like this:
  if (!(sdin >> x[xyzpoints] >> y[xyzpoints] >> z[xyzpoints])) {
    // Handle read error.

Second, you must provide a char variable to read the comma separators:

  if (!(sdin >> x[xyzpoints] >> sep1 >> y[xyzpoints] >> sep >> z[xyzpoints]) || sep1 != ',' || sep2 != ',') {
    // Handle read error.

Where sep1 and sep2 are type char:
  char sep1, sep2;
6  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 24, 2014, 09:51:15 am
This feels like a search for the Holly Grail.
He's not included any of the ARM core MCUs. If he expanded it to Cortex-M3.. for DUE.. .Then we'd want him to expand it to Cortex-M0 for ZERO.... then there would be others of us who'd like to see it expanded to Cortex-M4 for TI Tiva C series.
I am mainly interested in Cortex-M processors.   There is no hope that one person or even a few people can provide an open source firmware library for these devices.

Here is a project with the goal for providing such a library
The libopencm3 project (previously known as libopenstm32) aims to create a free/libre/open-source (LGPL v3, or later) firmware library for various ARM Cortex-M0(+)/M3/M4 microcontrollers, including ST STM32, Ti Tiva and Stellaris, NXP LPC 11xx, 13xx, 15xx, 17xx parts, Atmel SAM3, Energy Micro EFM32 and others.
They barely have software to configure the pins on devices. There is no support for SDIO on any device

Kowalski's work with Cosa is a really interesting development for a C++ OO Framework. It's a bare metal approach that will give us some task scheduling that might allow us to do some things that we expect an RTOS to accomplish.

Embedded systems are about time and Costa almost ignores problems of concurrency.  The importance of preemptive systems has been understood for over 40 years.
Liu, C. L.; Layland, J. (1973), "Scheduling algorithms for multiprogramming in a hard real-time environment", Journal of the ACM 20 (1): 46–61,

Here is a summary of this important theoretical result

There are lots of resources for Cortex M  so I can't complain too much.
7  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 23, 2014, 08:28:25 am
Small simple projects will continue to use the bare metal approach.  An OS makes little sense for under 32 KB of program space.  A simple development environment is adequate.

For larger DIY/hobby projects using 32-bit processors there are now many hardware options and the future will bring many more.  It would be a shame to be limited to choices made by the Arduino company.

About 30 years ago big science faced a similar situation.  From the late 1960s to mid 1980s DEC (Digital Equipment Corp)  minicomputers dominates control and data acquisition in large science experiments.  DEC supplied the software environment with systems like RSX/M, a real time system, and VMS for larger VAX machines.

Around 1985 single board computers based on the MC68000 and other microprocessors started to be used There was no standard OS for these boards.

Fortunately two of my friends, Jerry Fiddler, and David Wilner, had left the Lab in 1982 and founded Wind River Systems.  Wind River developed VxWorks which is a flexible RTOS that can be targeted to a wide variety of processors.  VxWorks became very popular in large physics experiments, space experiments like the NASA Clementine moon probe, and the Mars landers.  This defacto standardization was really important for big science at that time.

UNIX on SUN and other work stations became the standard OS for operator consoles and other non-real time applications . Linux is now  the best choice for boards like Rpi and BB that have Cortex Application processors.

The problem is that there are too many RTOS choices for micro-controllers like Cortex M and PIC32.  Many companies are supporting FreeRTOS but it is not technically outstanding.

I like ChibiOS/RT but it only has great HAL support for STM32.

I think Arduino is like Digital Equipment Corp and will fade in the same way.  There will are too many other outstanding hardware options and there will be even more in the future.

There is not an obvious software solution for DIY/hobby users.  FreeRTOS supports a wide variety of hardware, just look at this list  FreeRTOS is just not very exciting.
Too bad it is so easy to build a little RTOS kernel, this means there are many half baked RTOS systems instead of a project like UNIX and it's follow on, Linux.

westfw is right, the problem is the proper OS for DIY/hobby users.
Or an OS for  Teensy/NXP/ST/TI that is better than the current bare metal, without sacrificing the "ease of use" of the Arduino libraries.  But I don't think that either one exists yet.
8  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 22, 2014, 03:34:07 pm
I've just received a Tiva C-Series TM4C1294XL evaluation board from Texas Instruments.
Looks like a nice board.

This is great, the major companies are all building boards that can compete with Adruino and are talking to users.
We are looking forward meeting Makers and showcasing the MCU LaunchPad development ecosystem at Maker Faire Bay Area in San Mateo, Calif.

These companies are also working with universities to provide better tools and products for labs.
9  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 22, 2014, 10:05:06 am
The example shows how to use a per-allocated contiguous file.  Study it and use a similar method on Teensy.

Are you a lazy student trying to get others to think for you?

I won't be replying to any more of your questions since the examples I pointed out have the basic solution.

10  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 22, 2014, 07:08:01 am
So basically it would be sufficient to check if the card is busy in checking if DO is low, and only send commands and / or data packets to the card when DO is high

Won't work.  Please read the following carefully.

When you write to a file, a large number of SD commands and operations operations are required.  There is no way you could check for busy before executing this statement.
logfile << charBuf << flush;
This statement may require a cluster to be allocated.  That requires reading FAT blocks until a free cluster is found.  Updating the FAT block in memory, writing it back to the SD, then updating the backup FAT.  Even if you write 512 bytes, the data may cross a block so there could be read of a block, copy part of the data to that block, write the block back, then write a second block.  Then the flush requires reading the directory entry for the file, updating the directory entry and writing it back.

The SdFat library does check for busy before every write since the write would fail otherwise.  The problem here is that the card won't indicate busy until you send the write command.  Then you must wait before sending the data packet. If you write a RU which may be 16 KB, there may be 32 busy periods.

SdFat can't check for busy before reads.  The card sends a stream of busy tokens for read and finally sends a start data token.   

No reasonable SD library can do what you want.  Chan's FatFS does not have a non-blocking mode.  FatFS is widely use and the latency problem is handled by using FatFS in an RTOS.

I also have a new replacement for SdFat that I use in RTOSs like ChibiOS and FreeRTOS.  The problem is simpler on ARM processors with more features than Teensy.  You can use SDIO and write a driver that sleeps while the SDIO controller does all the busy checking for a large read or write.  Other threads run and no CPU time is lost in busy checks.

There is another possibility.  You can create a large continuous file and write it with low level SD operations.  You must still buffer data but the check of MISO should work. 

This program used one large multiple block write and avoids most busy periods. It can log data much faster than your requirement on an Uno .
I did a reliability test logging five analog pins at 5,000 samples per second.  This is an ADC rate of 25,000 values per second.  I logged 512 MB of data without dropping any values.

11  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 21, 2014, 11:02:13 am
This won't work.
What about the idea to first open a file on the SD card (T). Then pre-erase a block of 8 KB on the SD card in the next loop (T+1). Send 8 KB from the buffer to the SD card in the following loop  (T+2) , but in a way that the code is not blocked. Then a few loops later (T + 10) close the SD file.

Use an RTOS, that will work.
There are several ways to deal with this problem.  You can use an RTOS to buffer sensor data in a high priority thread and write to the SD in a low priority thread.  See the SD logger example in NilRTOS
12  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 20, 2014, 09:54:58 pm
This would also be true of any smaller buffer size?

SPI mode performance of the new generation of SD cards varies greatly and is not very predictable.  Manufactures only include SPI mode since the SD spec requires it.  Sometimes larger buffers reduce the frequency of long latency write but not the time.  In other cases the time is reduced.

New cards are optimized for phones and tablets in 4-bit SDIO mode.  There is little correlation between SPI performance with small writes and large writes in SDIO mode. 

I don't think there is a future in optimizing SPI mode performance for fast data logging. 

I am buying  development boards with SDIO mode SD sockets.  The SDIO controller on these boards have CRC and a fast DMA mode.
13  Using Arduino / Storage / Re: Teensy 3.1 / Data Logging / SD Card / Latency / Non-Blocking SDFAT Library on: July 20, 2014, 06:40:43 pm
logfile << charBuf << flush;      //Write to SD Card  
Writing single 512 byte blocks to files on modern SD cards will always have long unpredictable write delays.  The basic flash write size is not 512 bytes.

Modern SD cards have a Recording Unit (RU) that is a multiple of 16 KB.  High performance cards may have 32 KB or larger RUs.

When a 512 byte block is forced to the card, a new RU must be written.  Existing data must be copied to the new RU by the card controller.  This could mean 32 KB of data is written to flash each time you do a flush.  This soon exhausts the free erased RUs and the card will block while it erased an AU of flash which may be several MB.

Check if the SD card is ready to accept data.
There is no command to do this.  Any new features that could help are not available in SPI mode.  SPI mode is very limiting and only marginally supported in modern cards.

There are several ways to deal with this problem.  You can use an RTOS to buffer sensor data in a high priority thread and write to the SD in a low priority thread.  See the SD logger example in NilRTOS

I wrote SdFat which is also the base for SD.h.  I have no plans to add new features to avoid this problem.  Writing to a file system using SPI mode with a 512 byte cache buffer will have long delays with modern SD cards.

I am experimenting with SDIO using 32 KB buffers on an STM32F407.  I can write at over 5 MB/sec with no busy delays.  I can't port this code to Arduino.

Edit: I can write at over 20 MB/sec with only small, less than 1 ms, occasional busy delays using 4-bit SDIO.
14  Using Arduino / Storage / Re: file creation dates with on: July 14, 2014, 02:00:06 pm
Can somebody explain why the actual creation (last modification?) dates aren't shown correctly? Thanks.

Files will only be time/date stamped if you have a RTC, RTC library, and time/date callback functions for SdFat, the base library for SD.h.
15  Development / Suggestions for the Arduino Project / Re: Does Arduino have a future? on: July 14, 2014, 11:48:59 am
While I agree that micontrollers will have a place for quite some time, I'm not as
sure about the 8 bit AVRs.
I started to use Arduino because I wanted an easy way to work with microcontrollers. 

I started using Arduino in 2008 and wrote the Fat16 library for the 168 Arduino with 16 KB flash and 1 KB SRAM.  It was just a toy to play with.

Now I want a microcontroller like Cortex M4 with more power than AVR but not an application processor designed for "hosting a rich OS platform" like the Cortex A series.

I have been developing a new FAT file system using STM32F4 processors.  The difference in capability over AVR is overwhelming.  I tried to love the Arduino Due when it came out but couldn't.

Here are two examples. 

The ADC on AVR Arduinos run at a max of about 10 ksps with analogRead.  You can increase the clock speed and get fair, not full 10-bit, accuracy at 40 ksps.  Low end STM32F4 ADCs can do 2.4 msps, about 60 times faster than AVR with 12-bit accuracy and very flexible DMA.  High end STM32F4 chips can run three ADCs interleaved for 7.2 msps.

I am developing a new file system for high end embedded processors.  The STM32F4 has a good, not great, SDIO controller which can run the 4-bit bus at over 48 MHz.  I have achieved sustained read speeds over 20 MB/sec.  I have sustained write speeds of 12 MB/sec but this is a limited by the SD card I am using.  I have ordered a new Samsung Pro card and hope to write at close to 20 MB/sec.

The AVR can never read or write faster than about 500 KB/sec that's about the best you can do with program I/O on the 8 MHz SPI bus.  In addition SD cards don't present their fast commands on the SPI bus, only on the SDIO bus.  This means you will have very long occasional write latencies with SD cards on the SPI bus.

The Arduino system software is not adequate for these new chips. You really need priority based pre-emptive scheduling to take full advantage of the features of these chips.  Most chip manufactures are now providing a free RTOS solution.  ST and Atmel provide FreeRTOS ports, Freescale has MQX, and there are other great systems like ChibiOS.

I really enjoy development with boards like the  STM32F4 Nucleo boards.  ChibiOS can do about 1.2 million context switches per second on the NUCLEO-F411RE.  I am using a Olimex stm32-H407 and it can do a context switch in 400 ns.  The 407 has 196 KB SRAM and 1 MB flash.
Pages: [1] 2 3 ... 107