I'm searching for an method to transmit my temperature logging files stored on the SD Card through serial.
The challange is to transmit all the files (SD-Card fatfs) and check the packets/datablocks while transmitting and of course with low use of RAM (variable buffer / variable blocksize).
Every file (file1.log file2.log,file3.log..) is around 50kbyte big. and in total there are 370 files ~18 - 20Mbyte.
I searched for UART Transmission protocols and found X,Y, Z-Modem and Kermit but I can't find any portation to the embedded world. Maybe there is already something common which I don't know?
Why UART? Because I use RS485 to transmit the data to a PC that is 200meters away. The connection works with string in & ouput to control the Datalogger.
You haven't said what the SD card is connected to.
If you are trying to send 18 MBytes at about 11,000 bytes per second (115,200 baud) you are in for a long wait. And much longer still if you add data verification.
Why not just plug the SD card into your PC?
If the SD card is on an Arduino the first question is "Can the Arduino read the data from the files?"
If the answer to that is yes then it would be simple to write code to send a chunk of data to the PC over the Serial connection. It would also be simple to add, say, a CRC byte (or something more sophisticated) that allows the program on the PC to verify that it has received the data correctly and send a confirmation message to the Arduino before the Arduino sends the next chunk of data.
If you connect the Arduino to the PC over the USB cable I would expect you could work at 500,000 baud or even 1,000,000 baud but I doubt if those speeds are practical over long distances on RS485.
This seems to be something that should be easy to implement but not very practical.
A hardware option could be to buy a usb sd card reader for the pc and build some tri state buffers between the arduino and the SD card, e.g. ,make the card like dual port ram.
If you are using a Leonardo or Micro its possible to reprogram the board to also present as a disk drive to the PC.
I recall that Elektor magazine uses this system I.e arduino being a disk drive (memory stick)
To transfer data to their FPGA board
rogerClark:
A hardware option could be to buy a usb sd card reader for the pc and build some tri state buffers between the arduino and the SD card, e.g. ,make the card like dual port ram.
Without wishing to challenge your technical suggestion I still don't see the point of the OP's requirement - is the Arduino going to generate 18MB of data?
I presumed from what the OP said, was that the arduino was logging a lot of data to the SD card, probably over an extended period, he mentions 370 files each of 50k. Hence the total size on the SD card wold be tens of megabytes
I'm not sure how practical building a dual porting system for the card would be.
I envisaged using tristate buffers, one lot on the pc side of the cad and another set of buffers for the spi comms from the arduino
to disconnect from a PC SD card reader, when the arduino was accessing the card, and the when the PC needs to read the card, the other buffers would disconnect the arduino spi from the card.
Guess the hard thing would be for determining how to automatically switch the buffers, I.e who is bus master, the PC or the arduino, or perhaps there would need to be a physical switch or button on the arduino to do it
I guess its easier to manually take tha card out, but not if you do it often or need to do it unattended.
The LUFA USB disk solution is probably the nicest technical solution, but the most work
the problem is the logger is inaccessible when it's mounted and the SD-Card is connected via SPI to the arduino. I don't need to access the whole log's at once, it's more like downloading one log-file from the day before (50kbyte max).
Removing the Card isn't an option and using a tristate setup is not possible because I have a 200 meters distance to the PCm USB will not work (without any kind of extender).
I already installed the setup over the weekend with the RS485 and it works fine along the distance. I can control the logger and save data to the SD-Card.
Now I started to make a concept of a transmission function - maybe you guys have a better Idea:
Get the Filesize (Bytecount)
generate the crc for my blocksize of 1KByte
send the 1Kbyte of data + crc
wait for the response of host that the data is complete and the crc is correct
if false: repeat
if true: continue with the next packet
transmission ended when all bytes sended.
my father said that an alternative is to send the data bytewise and the host has to loop back the byte, so I can check that the transmitted data was correct. He mentioned that USB is working in a smiliar way.
superkato:
2) generate the crc for my blocksize of 1KByte
3) send the 1Kbyte of data + crc
Fitting a 1k block into Arduino Ram will be a challenge. As far as I know the SD Card library uses a 512k buffer so why not use that directly as your transfer block.
Personally I would experiment with much smaller transfers first to find the optimum size - especially if you need the PC to confirm successful reception or order a re-send.
you are right! I can use the SD Card directly without a buffer.
ok I will take a look on the easytransfer library, porting the other site to my visual basic program isn't that complicated. Thank you very much!
I'm wondering how people transfer large BMP files from the pc to the SD card. My dad (he is an 80C51 guy) he talked about Z-Modem but I think thats quite overboomed for my tiny application.
