Announcing - Arduino SDCard file manager sketch with ZModem

Projects Interfacing w/ Software on the Computer
1

I recently put together a sketch that creates an SDCard file manager with ZModem file transfers via the Serial interface that the Arduino USB port provides. I thought I would share it with the community; perhaps others will find it useful as well.

Video demonstration here:

Arduino SDCard file manager sketch with ZModem - Demonstration Video

Sketch available for download at GitHub here:

Arduino SDCard file manager sketch with ZModem - GitHub repository

First - much credit goes to Pete (El Supremo) who got this started in this thread a few years back:

Zmodem transfer from Teensy to PC

I managed to get the dynamic memory usage down to just under 4K, but unfortunately that means if you want to use this sketch, the Uno isn't going to cut it, you'll need a board like a Mega with a little more memory (Even Chuck Forsberg who wrote the original ZModem code back in 1987 probably had more than 2K of memory available!)

I realize that there are more modern options available than Serial, like WiFi for example, but my project involves an SDCard on a SparkFun MP3 shield and no WiFi. For those in a similar situation, this sketch may form a simpler way to get files to/from an Arduino SDCard without getting into wherever your Arduino is hidden away and unplugging the SDCard to move it to a computer.

On my Arduino Mega 2560 R3, I can receive files on the Arduino at up to 57600 baud reliably; maybe those using much faster boards can crank out more bps on receive, it's easy to configure and try.

I tried HyperTerminal, SyncTERM, and TeraTerm all successfully, send and receive. While SyncTERM is available for any platform you're likely to be using, I'm hopeful that this sketch will operate with other popular terminal applications on Mac and Linux platforms given that I got a perfect run on anything that still reasonably runs on Windows.

More information is available on the Wiki page of the GitHub repository.

I hope someone finds this sketch useful! Thanks again to Pete (El Supremo) for seeding this idea!

1 Like
2

I'm hoping someone with a smaller (Uno or Nano) board might find this functionality useful and test out my new sketch version on a small Arduino board.

I did a large scale overhaul to my ZModem sketch and I actually got it down (with a lot of very creative and unorthodox wrangling) to just under 30K of Flash and just under 2K of RAM needed. I even managed to do that and not have to choose to enable only one of send or receive. I didn't think it could be done at first, but it appears I proved myself wrong. I put some macros into zmodem_config.h that allow you to disable send, receive, or the file manager commands just incase, but I'm hopeful it's not necessary to do so.

It would be fabulous if someone with an SDCard on an Uno, or better yet a Nano because it has little less flash available, would take the latest version for a test drive and let me know how it runs! Over here at GitHub:

If you watch my little video on how to use the sketch (linked above), pretty much everything remains the same except the bits where I talk about how large the sketch is and the need for a large board.

3

What is ZModem used for?

4

Back in the early '90s and prior, in the days before the internet, ZModem was widely used for file transfers when dialed up to "Bulletin Board Systems".

In the context of the Arduino, it's useful if your Arduino has an SDCard wired up, but the Arduino has no connectivity mechanism (Ethernet, WiFi, etc.) outside of the Serial interface presented by the USB connection.

Watch the first minute or so of my Demonstration video and it should give you a better idea of the purpose in the Arduino setting.

5

Hi Monte Carlo. Thanks! This is exactly what I was looking for. I hadn't heard the term "zmodem" used in over 20 years, but instantly remembered using it on BBSs and realized it was the perfect solution for file transfer over serial.

I'm using an Arduino UNO Rev3. I commented out #define ARDUINO_SMALL_MEMORY_INCLUDE_SZ so that I could fit the sketch in memory. It compiles fine. I installed and configured syncterm. It communicates with the arduino and SD card -- I can use the file commands such as "DIR" and "CD".

I tried to upload a file. A small text file works fine, but a bitmap file (about 226kb) won't send. Here's a link to the test bmp file:

And here is the log of the upload attempt:

Mar 07 10:24:52 Log opened
Xfer Info: Sending f:\temp\upload\purple.bmp (225 KB) via ZMODEM
Xfer Info: Sending ZRQINIT
Xfer Info: Receiver requested mode (0x23):

Full-duplex, Can overlap I/O, CRC-32, Escape: Normal
Xfer Info: Receiver specified buffer size of: 1024
Xfer Info: Receiver requested mode (0x23):

Full-duplex, Can overlap I/O, CRC-32, Escape: Normal
Xfer Info: Receiver specified buffer size of: 1024
Xfer Error: Received ZRPOS at offset: 1024
Xfer Error: Received ZRPOS at offset: 512
Xfer Warning: ZACK for incorrect offset (1024 vs 256)
Xfer Error: Received ZRPOS at offset: 256
Xfer Info: Resuming transfer from offset: 1024
Xfer Error: Received ZRPOS at offset: 1152
Xfer Error: Received ZRPOS at offset: 1152
Xfer Error: Received ZRPOS at offset: 1152
Xfer Error: Received ZRPOS at offset: 1152
Xfer Info: Resuming transfer from offset: 1152
Xfer Error: Received ZRPOS at offset: 1280
Xfer Error: Received ZRPOS at offset: 1280
Xfer Error: Received ZRPOS at offset: 1280
Xfer Warning: Transfer failed on receipt of: ZRPOS
Xfer Notice: Hit any key or wait 60 seconds to continue...
Mar 07 10:25:25 Log closed

I tried multiple attempts. I even tried a lower baud setting. Any ideas what might be causing the failure?

Thanks again,
Dan

6

dperrigan:
Xfer Error: Received ZRPOS at offset: 1280
Xfer Warning: Transfer failed on receipt of: ZRPOS
Xfer Notice: Hit any key or wait 60 seconds to continue...
Mar 07 10:25:25 Log closed

I tried multiple attempts. I even tried a lower baud setting. Any ideas what might be causing the failure?

Thanks again,
Dan

Thanks for trying the sketch on an Uno, Dan!

I was hoping that the Uno and Mega would behave the same once the sketch was small enough to operate in the limited memory space of the Uno. That may have been wishful thinking.

I did have some problems with SyncTerm while I was developing the sketch, but was hopeful that I had ironed those out with the various optimizations I made where it ended up working on a Mega (you can see in my video I upload a ~100K file at 57600 in SyncTerm).

Would you be willing to run a test with HyperTerminal? I found that HyperTerminal was more tolerant while I was developing the sketch. You're almost certainly not running a version of Windows that is old enough to have HyperTerminal built in, but you can download it over here:

HyperTerminal

Just get the Hyperterminal_WindowsXP.zip, unzip it and run hypertrm.exe. It runs in Windows 10 okay, so it'll be fine on 7, 8, 10, whatever you are using.

Use Protocol "Zmodem with Crash Recovery" when uploading.

If that still craps out, maybe you can try 9600 Baud? It will be very slow, the idea is just to try to prove whether or not there is a buffer overrun or some other issue going on.

Crossing my fingers this can be made to work!

7

Success!

Hyperterm was the solution. It uploaded the 226k bmp file at 57600 with no problems at all. This is great!

I wonder if this can be adapted to use the wifi shield instead of the USB cable...

Thanks again,
Dan

8

dperrigan:
Success!

Hyperterm was the solution. It uploaded the 226k bmp file at 57600 with no problems at all. This is great!

I wonder if this can be adapted to use the wifi shield instead of the USB cable...

Excellent, thanks for the update Dan! I guess SyncTerm's ZModem implementation is more fragile than HyperTerminal. I'm not sure how it works for me with the Mega, but it could just be luck (eg. specific contents of the file perhaps). I wonder if I should redo my demonstration video to just use HyperTerminal and be done with the flakier SyncTerm.

I wrote this sketch for folks that have no other connectivity to their Arduino other than Serial over USB. With WiFi or Ethernet you'd probably want to take an entirely different approach; there's a reason you haven't heard about ZModem since the end of the BBS era. :slight_smile:

Enjoy the sketch in the mean time in any case!

9

Just what im looking for will it work on the DUE?

Cheers

10

I feel kind of bad that I didn't see this post a year and a half ago. There wasn't a lot of interest a few years back so I kind of just let this sketch go. Someone recently filed a Problem report on GitHub because they couldn't get it to compile with the newer IDE and Teensy libraries so I poked around this topic again wondering if anything else ever happened.

Anyway, I don't see why the sketch shouldn't work on any Arduino based device that presents a Serial interface on the USB interface (don't they all?) DUE should be just fine. The guy that wanted the bug fix recently got it working on a Teensy 3.0.