I have a few major concerns...
4 inputs; pc, blu-ray, tv, spare
4 outputs; dac, speakers, hifi system, spare
1. The signal over S/PDIF/TOSLINK is sometimes encoded, such as Dolby AC3 from a DVD or Blu-Ray. These encoder/decoder formats are covered by patents, and (I'm not 100% sure about this) I don't think you can buy the decoder chips without a license. If you simply want to switch & pass-through the digital signal to your home theater receiver this is not an issue, but if you want to decode & convert to analog, it's a concern,
2. S/PDIF/TOSLINK is already outdated! Some of the high-resolution Blu-Ray formats only work over HDMI. If your Blu-Ray player (or computer) has an S/PDIF connection, it should "work" with any DVD or BLu-Ray disc, but you won't have all of the formats and depending on the player's features, high-resolution surround formats may be down-mixed to stereo.
3. Surround sound is now standard with DVDs & Blu-Ray, and you are talking about 4 outputs (two stereo outputs?). You are putting-in a TON of effort on something that can't handle surround sound.
A few other thoughts...
Anything related to TOSLINK is likely to be surface mount. So far, I've been able to avoid surface mount on my home projects but most "complex" chips are surface-mount only. You may be able to find one that has fairly reasonable lead spacing. Where I work, we have lots of fine-pitch surface mount parts (CPUs & memory, etc.) and I wouldn't even attempt hand-soldering these.
600ohm headphones by itself. as my headphones a DT770 pro 80ohms my tv can hardly power them
It's a line-level output. You need to add a headphone-amplifier stage.
You can buy low-power amplifier chips, and the circuit is easy to build. The LM386 is very popular. I've used the LM380 and some other stereo chip that I can't remember maybe (LM1887?).
If your TV has RCA connectors it's also a line-output, not designed to drive headphones.
...like a normal audio mixer.
Do you really need to mix signals? Doing that in analog is simple with a summing amplifier made from an op-amp. It's also simple in digital (since it's just summing), but for that you need a CPU (or microcontroller).