Since it is a sensor, it probably has a fairly low impedance, so unless there is a lot of noise, you can use a fairly big resistor, 10k to 100k.
The other purpose of the series resistor is to prevent your clamping from affecting the signal - if the signal is 12 volt, a clamp to 5 will mean it will be 5.5 volts (diode drop) at the end of the resistor, so roughly 7 volts / 10k = .7 mA which is good for most auto sensors. Do watch ignition noise right near the arduino, but the point is .7ma won't push or pull the other end. The reason for the small resistor between the clamp and input is similar - if it is at 5.5v, and the internal protection wants 5.4v, you want to limit the current there too. There will always be some variation, but there isn't much current going anyway.
The internal protection will actually power the Arduino (there was a project creating an RFID using the excess voltage from the coil which would be shunted to Vcc and power the chip). You really don't want to do this, and you won't with a large enough resistor.
The internal protection is also basically a set of diodes, and with a large enough series resistor you really don't need the external diodes, depending on the voltage and current that needs to go through.
If the sensor is high-impediance (low current), you can't use a big resistor since .7 mA might be a large amount of current for the sensor and it will pull down the signal to where the system won't work. One common automotive example is an oxygen sensor. For those, you need to use something with higher impedance like an op-amp.
The inputs on the Arduino are high impedance (if you turn the pull-ups off) but are still clamped and the clamping can act more like a dead short when you get out of the voltage range. Often just the series resistor alone works, but it depends on the signal coming in and what it can drive and the voltage range.