Dear Sam...
You are asking the right questions! I wish I had the answers for you... and me!
I so often find myself just wanting to USE something... without spending hours figuring out what is going on inside it. But, absent the documentation of usage details, we have to figure it out to know HOW something can (successfully!) be used. Sigh.
I'll have a stab, expecting... hoping!... that people who know more than me will jump in and correct the bits of my partial grasp which are wrong....
(All of the following based on the Arduino Ethernet Shield V5 schematic. In another thread...
http://arduino.cc/forum/index.php/topic,76334.0.html
...you will find that another reseacher (westfw... lots of Good Stuff in his post) didn't find all of the following in the V6 schematic... but at least D10-13 were still in use.)
The "trick" is that certain pins... at least D10, D11, D12 and D13... are shared between the ethernet interface and the SD card reader.
Bear with me... I need to "name some parts"... The following is not comprehensive. For instance, D12 not only goes to the "X1" chip, but also to the W5100. ("X1": the chip associated with the "FPS009-3001", which is the physical connector for the SD card.)
From the Arduino's point of view ("Apov"), pin D13 is being used as an output. It drives the SCK signal. (So it is an INPUT, as far as the shield is concerned. "Gotcha" number one: Getting mixed up along these lines.)
Still from Apov, D11 is used as an output called "MOSI". (Gotch-2: Keeping "MOSI" and MISO"... we'll meet it... separate in your head.)
MOSI and SCK (5v when "high", from Arduino) are conditioned via voltage dividers before reaching the X1 chip... the main reason I assume they are outputs! (That chip, however, really has me confused. First of all, with no part specification, I can't look up the datasheet. Secondly, while MOSI and SCK go in via the voltage dividers, elsewherethere are things which connect(?)... un-numbered(!) connections... to Arduino lines and to 5v through 10k pull-up resistors. There's also what looks like a 3v3 supply going into X1. My guess?....
X1: Basically a 3v3 chip, with SOME inputs tolerant of 5v, but a few, which sometimes get passed through to the SD memory card, are expected to be kept at 3v3 or below, as the other things a designer will expect to have on THOSE pins will NOT be 3v3 tolerant, anyway. Ah! Or maybe X1 COULD tolerate 5v on the input, but the SD card, on the same signal, could not. I'll come back to the "SOME inputs..." later.
D12, driving MISO, has a direct connection from X1, a 3v3 chip, to 5v I/O lines of the Arduino... two hours ago, I thought that this meant that the ethernet shield designers expect D11 to be configured for input to Arduino, never output. However, I've since learned that for SOME chips... quite possibly X1.. this would not necessarily be the case. It is certainly NOT SO the 5100, which we will come to in a minute. That chip runs on (Vdd) 3.3 volts, but can accept 5v on an input! (Sorry to be ignorant... some of you knew that. I'd never met such a chip before. I'll still be very, very careful not to apply an input that is higher than the chip's Vdd / Vcc (is there a difference?) as a general rule!
From other... and more reliable(?) indications, I now believe that MISO is an output, from the Arduino's point of view.
Onward!...
Pins D11-D13 also connect to the ICSP connector (2 rows of 3 pins), by the way, in case you ever want to attach something there, you'll know the implications. But that interconnect is being "written out" of newer shields, as it causes problems for Mega users. (Megas drive MOSI, MISO and SCK from different pins.)
Pin D10, signal "SS", goes into the W5100 at pin 29, called (there) /SCS, "Not SPI Slave Select". ("CS" from "Chip Select", my guess.) It also drives, through an inverter, W5100's pin 31 "SEN", "SPI Enable".
Now I want to talk about X1's "write_protect" and "detect". X1, remember, was thought to be the interface chip for the SD card. Those two (un-numbered) connections go to the Arduino's Analog I/O lines A0 and A1, respectively. Both lines have 10k pull up resistors to 5v. Initially, that made me conclude that they were outputs from the Arduino pov. Not a valid conclusion! The names saved me. I now suspect that both are being used as, Apov, inputs. There's a rather neat little jumper on the board which, if closed, affects the "write protect" status of the card. I'm not sure if it makes every inserted card protected, or if it over-rides "write protect" requests elsewhere... but I think it does one or the other.
I also see connections from the Arduino's D2 and D4 lines. "PD2" and "SD_CS", respectively. I couldn't find where PD2 went to, have no idea what it is for. "Power down"? Don't forget that we are only looking at the hardware. Maybe the engineer was noting that the software would send a signal on D2 for some feature that this board doesn't have? SD_CS goes, through a voltage divider, to X1, so I conclude that, Apov, it is an output, to select X1.
Ok! Parts named. So what?
Roughly spoeaking, The shield is using a multiplexed bus. Let's say you want to send the byte "57" to the ethernet, and a moment later you want to send "58" to the SD card.
Those numbers will BOTH appear (at different times!) in the SAME wire, the "data line".... a wire connected to BOTH the ethernet interface, AND the SD card interface. So how come both devices don't process both numbers? OTHER wires, the "control signals", are connected thus: some only to the ethernet interface, and some only to the SD card interface. To do what we wanted, we started in a state where the control signals were saying to their devices "ignore what is on the data line." Then we changed the control signal to the ethernet, said "watch this". Sent the 57. (It appeared at the SD interface, too, but that was ignoring data at the time.) After the 57 was away, we put the ethernet control signal back to "ignore stuff". Then we made the SD control signal say "watch this". Sent the 58. Put the SD interface off line again.
My word. It is amazing it EVER works... No wonder people have to ask questions at the forum!!!
===
SO WHAT DOES ALL THIS MEAN????
From the above, I hope you now see that when you plug in a shield, you have to know what pins from the Arduino are being tapped into. What pins are being connected to "stuff" on the shield. (Some pins will be physically connected to the shield... but not connected to any electronics on the shield.
The safest, easiest course is then to Just Not Use the pins that the shield is using.
If you "need" (want!) to use a pin the shield is using, it is critcal to know if the software accessing the shield is setting that pin as an input or as an output... or maybe changing it from one to the other from time to time! (And know something about the electronics on the shield which, though the pin, will connect to your external circuits.
If you create a mismatch, not only will things "not work", but you can end up damaging the electronics on the shield and/or the Arduino itself....
Sigh.
Sorry that went on a bit... I hope it was not too far wrong, and even where it was wrong, I hope it illustrated issues that you need answers to if you are going to have happy times with shields.
Why is your project not working? Quite possibly because part "A" of your project is doing things which clash... in hardware or software... with things part "B" of your project is doing. I hope the above helps you understand the concepts which will help you find the places those clashes might be happening.
===
While the 70 pages of the W5100 daasheet will never be bedtime reading, and there will be LOTS that make little sense to lots of us... There IS a nice discussion of SPI mode operation starting near the bottom of page 60 of the version 1.1.8 edition. I downloaded my copy from Sparkfun. It will tell you a lot about SPI in general, not just about how the W5100 is using it.
Thank you for struggling through all of the above! I don't expect it was useful to everyone, but I hope you feel your time was suitably rewarded.