Southpark:
Whatever you're doing..... .

Im trying to display the angle of a resolver, based on sine and cosine signals

Southpark:
..... and phase difference between the two square-wave waveforms.

this is achieved by hardare, ie, ref signal is fed to optocoupler and output transistor is digitalread. so if Vp happens when digitalread is 1 signal is in phase with ref. Same with other signal

Southpark:
You should take several readings, and average the reading.

I take an average of 5 last full cycles, thats why i have said that measurements are not flikkering

Southpark:
Also..... specify what kind of absolute error you're after .... +/- 0.1 degree? +/- 0.05 degree ?

Since Im working with sine ang cos and their variation is in 1 quadrant (other quadrants come out as about you suggested, by phase timing, and since I have 10 bits, I expect ideally about 0,1 deg (900 bitsteps). BUT :
ADC has an accuracy of +-2 lsb max. From this somebody could say that worst case is 4lsb or 0.4 deg. This is correct for start. If use some trigonometry its much better. Anyway, I expect 0,12 deg
In practice it is achieved for the full 1st and 3rd quadrant.

And my problem keeps going : Why it is the much greater error when one signal is opposite phase from the other (about +0.4 and -0.25 at midquadrants). The error is less as angle goes nearer the full quadrant region.

Southpark:
Also, if you want to get peak value of a sine-wave.... I guess you could measure a bunch of points over 1 period.... and repeat....and repeat... get a few sets of data. Then run it through a curve-fit process to get a least-squares fit - applied to a sinusoid.

I see is a good approach and thank you but dont have time to analyse it. Perhaps I will stay with my errors and correct them with a lookup table (they are steady).