So it turns out that I need a little more help ...
When I removed the cap that I thought was bad I still get continuity across it's connections on the board (I also discovered that the capactor settings on my multimeter don't work so that was fun.)
After staring at the board for a good long time I also took out the regulator - no change
Then I found a tiny zener diode and when I took that out the erroneous continuity went away.
I don't know much about such diodes but I was able to get continuity in both directions but maybe my multimeter was giving it enough voltage to let current flow backwards.
I put the multimeter on the diode checker setting and it gives me about 0.8 in both directions. According to the internets that means it's bad but how can I know what value zener to replace it with? It's far too tiny to have any markings on it that I can read.