So I have a pretty large stream coming in, about 92K and I wanted to know, how can I just have it dump the whole thing to an SD card.
First send the file size as an unsigned 32-bit integer then just send the data without doing any conversion. Write the data directly to the card.
First send the file size as an unsigned 32-bit integer
Why? How do you know how big the file will be?
If the data rate is slow enough you can just read from Serial and write to the file. The Serial stream has 64 byte input and output buffers. This will work if the the rate is under about 640 bytes per second.
If the rate is higher, you may overrun the buffer since SD write latencies can be 100 ms or more.
I wrote a special Serial library and logger for higher speeds SerialPort/SerialPortLogger at master · greiman/SerialPort · GitHub.
Yes I do know how large the file will be as the topic poster said the filesize will be around 92kb so a 16-bit integer could not be used. He could use a 24-bit variable which avr-gcc supports see avr-gcc - GCC Wiki I do agree the variable size was a bit excessive. I just forgot avr-gcc supported 24 bit integers.
I do agree the variable size was a bit excessive.
It's not the size but why write the size you expect a file to be at the start of a file? I didn't see a requirement for this and was just curious why it is needed. If the file is always the same size, you don't need to write the size. If the size varies, when do you know the size?
You can always get the real size of a file on any OS.
For Arduino SD.h
file.size()
With SdFat
file.fileSize()
Linux, OS X, and Windows have similar functions.
No I am saying the host should send how many bytes it is about to send to the Arduino that way the receiver program which is running on the Arduino knows when to stop and close the file. Yes your method if no bytes have been sent after x amount of time call sync() works too I just think that my method was easier to implement. The filesize is not written to the file. I am aware of how to get the filesize without storing it in the file already.
that way the receiver program which is running on the Arduino knows when to stop and close the file.
Most Serial logging applications don't know the file size at the beginning. You generally wrap data packets with a simple protocol to indicate EOF and other info.
There are lots of ways to solve the EOF problem.
If there is just one file, you can just call flush or sync every time there is a gap in the data. This is the same as closing the file so you can stop the receiver when ever the sender is done and not lose data.