But something must be wrong with my (ugly style) circuit. Because I get a way to high voltage reading (over 5 Volts) on the reflected side. I connected a Yaesu FT891 with 40W Power. On the antenna side there's a 50 Ohm dummy load. The torroids used are as described in the schematic. Are my torroid windings/connections correct?
When we wind toroid coils, we have to make sure the wire is wound close to the toroid all the way around. Your wire turns are in too much of a loop, not wound tight enough.
It looks like you have grounded both the ends of each pot. Disconnect one end of each pot so you can have full range of resistance control.
I found out that one of the coaxial cables was faulty. What happened is that the RG58 going to the dummy accidentally made a short sometimes. This led to improper readings on the SWR bridge. Now that I changed the coax, all the readings are fine after tuning the variable caps.