next topic »

[on:]

On the Arduino right now, there are many libraries to access different kinds of external or internal EEPROM or Flash memories but there's no single API to do that. It would be very nice to have a standard API with which to open, read and write to streams or files, similar to the fstream C++ library or the stdio (fopen, fprintf, etc).

The idea is to be able to use similar code wether you are writing to the uart, the EEPROM, a dataflash or an SD card.

Does such a library already exist? If not, I am ready to code one and I would like to use this post to discuss with the community about what is the best option.

I am considering two options.

1. Code a sub-class of Print.h that would implement additional functions for streams and files (such as open(), close(), rewind(), seek(), read(), etc.) -- which would permit, among other things, to use the Streaming library for easiness.

Code example:

Code:

Dataflash f;
int x;
f.scanf("%d\n", &x);
f.rewind();
f.printf(f, "Hello word\n");
f.close();

2. Use the sdtio library (fopen, fprintf, etc.) The nice thing with this is that it's pretty standard for C users. It would be "the way to go" if Arduino had been designed for C users, but it's not -- and that's a good thing most of the time -- so it means that it's not very standard for Arduino users.

Code example (I don't even know if we can do that, I'm not super familiar with the stdio library):

Code:

FILE* f = open_dataflash();
int x;
fscanf(f, "%d\n", &x);
rewind(f);
fprintf(f, "Hello word\n");
fclose(f);

Any ideas, suggestions, questions?

[Reply #1 on:]

I would think that a standard API that can write serial data to either serial devices or memory devices would be very difficult, if even possible. Memory devices need to be told what address to write new data too, and somewhere something has to keep track of what address already have valid data stored in them, next free address, how to erase previous used memory location, etc.

I'm not a software guru at all, but what you are asking for seems darn near impossible, if not illogical?

Lefty

[Reply #2 on:]

The idea is to be able to use similar code wether you are writing to the uart, the EEPROM, a dataflash or an SD card.

It can be done but would it be worth the footprint it takes?? serious doubts.
It means for external EEPROM another layer e.g. around the I2C library costing extra scarce memory. For some applications it makes sense to do so, but not for all. Would it make (coding)  life easier? doubt it seriously

imho an EEPROM is more like one big array!  so why no array interface iso file interface? An SD card can have both an array interface (raw device) and a file interface.

my serial out cannot be rewind() so the 'file pattern' breaks quite fast.

I prefer every type of device its own interface, then I recognize them at least in the code

[Reply #3 on:]

First, I agree that there is a difference between streams and files. Of course, it's impossible to rewind or fseek the uart for example. It doesn't make the thing impossible though: it could easily be solved by using a class hierarchy comparable to that of the C++ iostream lib; or, if you just want one top class, you can just make those methods (rewind / fseek) do nothing or been unspecified in the case of streams.

Second, I don't really see why there would be a huge memory footprint, except maybe if we use solution #2 which uses <stdio.h> which has in itself a huge footprint from what I have read. The top (abstract) class would take no more space than the Print.h class, the bridge classes probably would take some space, but not disproportionately.

I personnally don't want to have to reimplement a big bunch of my code at the risk of making mistakes just because I switched from saving my data to the EEPROM to saving it to a SD card or another device. Of course, there's a cost for abstraction in terms of memory usage; the advantage is faster development and less risks of errors. And if you really end up feeling short on space, you can always switch back to a lighter, low-level library.

[Reply #4 on:]

still not convinced, sensors could also get a file interface, oven.scanf("%f", &temperature);  .... and IO pins.. Why not, reading from an IO pin just happens to be one bit at the time, analog pins read 2 bytes at a time. I can do digitalPin[12].print("1"); of analogPin[2].scanf("%d", &x);

but seriously build a prototype and suprise me .... The proof is allways in the pudding test !

[Reply #5 on:]

I didn't know, but the last version of Arduino actually includes a Stream abstract superclass from which Serial derives.
http://code.google.com/p/arduino/source/browse/trunk/hardware/arduino/cores/arduino/Stream.h

(although, I think they should have included the peek() function to it).

So, the API is basically there for streams. What is missing is a subclass of Stream for files (or file-like mediums) that give extra methods (seek, tell, eof). While we're at it, I would also add functions to read/write blocks, and possibly to read/write directly at a specific address.

[Reply #6 on:]

Yeah you're right I'll put something together and we'll see.

[Reply #7 on:]

2. Use the sdtio library (fopen, fprintf, etc.) The nice thing with this is that it's pretty standard for C users. It would be "the way to go" if Arduino had been designed for C users, but it's not -- and that's a good thing most of the time -- so it means that it's not very standard for Arduino users.

Code example (I don't even know if we can do that, I'm not super familiar with the stdio library):

Code:

FILE* f = open_dataflash();
int x;
fscanf(f, "%d\n", &x);
rewind(f);
fprintf(f, "Hello word\n");
fclose(f);

Any ideas, suggestions, questions?

There is an stdio.h that you can use, here's the documentation http://www.nongnu.org/avr-libc/user-manual/group__avr__stdio.html. If you look up examples of it around the internet you'll see that as long as you define a function to "put char" and "get char", you can use printf to wherever you want. Although, it might only be available for one stream (file if you wish).
Although, I do remember thinking stdio.h was the sliced bread of microcontrollers until I saw the memory footprint it left. But then again, at the time I didn't bother looking up the definitions of the compiler regarding floating point conversions and such... so that may also have an impact.

But, give it a try. Even if it's not feasible, or limited, it isn't a waste of time, since I'm quite sure that the 2560 would benefit form such implementation, or maybe a 512kb memory device in the future.

[Reply #8 on:]

I put something together in the form of the following "SeekableStream" library:
http://code.google.com/p/libinteract/source/browse/trunk/trunk/arduino/libraries/SeekableStream/SeekableStream.h

It's implemented as a subclass of Stream that adds seek/tell functionalities.

There's an issue with the flush() method that is unclear in Stream, in response I've included a flushInput() + flushOutput() (I believe it should just be included at some point in Stream).

I implemented such a SeekableStream for the ATD45DB161D library, here is the result:
http://code.google.com/p/libinteract/source/browse/trunk/trunk/arduino/libraries/ATD45DB161DStream

For the printf issue, I suggest using the Streaming library.

In the hope that it will be useful to others.