calculating error, quickest way

I've got a project wher the frequncy of a timer is changed from 1hz to 1mhz, using the 16 bit timer is fine for frequency below a few khz, however above that the frequency doesn't. Transistion smoothly and starts jumping a few values, one its above 300khz its jumping by tens, 400khz by hundreds, is there a way to instead of scrolling from say 400khz to 430khz manually by changing the number until the value is enough to actually cause a change, what would be an easier way to start jumping by the error so it goes from one correct value to the next, I know I can precalculate the error and use a lookup table but I imagine there has to be a simple formula to do the same since the error is the same every time and can be calculated easily
Any ideas?

I'm using this to change the OCR1A value to get the desired freq
OCR1A = (8000000/freq)-1
So say I want 247khz the ocra would be 31 and the actual freq is 258064, so quite an errror, I could just jump by ocra values at a certain point where the error gets large but idk if there's an easier way

Say OCR1A was zero (so it counts to 1), then you would get a pulse every 8 MHz.
Then you change it to 1 (so it counts to 2) and you get a pulse every 4 MHz.

There is no in-between value. So if you want a 6 MHz frequency you are out of luck.

I know lol, but my gui doesn't know that, I would like it to automatically know
Like where normally I change by one instead I change to the next valid number

you should just mimic the (integer) math of the timer and show the calculated values in the UI

you type 247000,
the Arduino does something like:
long d = (8000000/247000) - 1 = 31;
long freq = 8000000/d = 258064;
Display.print(freq);

Well the problem is its not typed in but in/decremented with a rotary encoder, so for lower frequencies it just goes one at a time, at higher it would need to in/decre,ent by larger values, problem is that value changes, that's why I'm looking for a formula basically that mimics the curve of the error, I'm sure its simple I'm just not too well versed in math lol

OK make stepsize depending on the count

some partial code to get the idea

#include <math.h>   // for log10 

long value = 0;
if (turnleft) value = value + stepsize(value);
if (turnright) value = value - stepsize(value);
....

long stepsize(long val)
{
  long st = 1;
  if (val > 0)
  {
    int x = max(0, log10(val)-2); // base 10 log ~ length of number 
    for (int i=0; i< x; i++;) st *= 10;
  }
  return st;
}

This function step takes the value (e.g. 4567)
calculates log10(4567) = 3.6596 -2 = 1.6596
floors this to an integer => 1.
Then the for loop multiplies the stepsize by that amount of 10, so the final stepsize = 10

step(123) => x = 0; ==> step = 1
step(10) => x =0 (due to max) ; ==> step = 1
step(100100) => x= 3 ==> step = 1000

easier variation

long stepsize(long val)
{
  if (val >= 1000000000) return 10000000;
  if (val >= 100000000) return 1000000;
  if (val >= 10000000) return 100000;
  if (val >= 10000000) return 10000;
  if (val >= 1000000) return 1000;
  if (val >= 100000) return 100;
  if (val >= 10000) return 10;
  return 1;
}

hope this helps

That's an interesting idea however it will not match the actual output because the error appears to change logarithmically(as in a graph of error vs frequency) tho I'm sure there is a linear equation because I believe when I graphed the difference between errors its linear, I just can't figure it out

The best approach really is to use a dds.

Short of that, google "dithering".

Yeah I wish I thought of tnat before I finished the pcb, also my budget is kinda small but anyway, how could I apply dithering ? I can't find a decent source of the actual process and how to apply it to my thing

Well, Yeah But you can buy an AD9850 board on Ebay for 5 dollars or less and get a square wave of variable duty cycle with the DDS as well as triangular.
There is also a rather pricey but very nice function generator with a rotary encoder it's a small board perhaps the size of a 20 X 4 display in outline with a 16 X 2 display and some pots and switches and a big knob , runs on 9V dc and .... I bought one as I thought a continuous FG from DC to 5 MHz was a really nice bench tool. I also have a bunch of unity gain 50 Mhz 5V 100 mA output (Load) curent Op-Amps (EL2157) and those will make it reasonably idiot proof.

Bob

How much did that board cost? Lol I already spent 70$ on this, and I'm finally just in the final proggramming phase working on the gui,
but unless someone can figure out that formula perhaps after a certain frequency I will just have it directly change the OCR1A number and display the frequency

winner10920:
Well the problem is its not typed in but in/decremented with a rotary encoder, so for lower frequencies it just goes one at a time, at higher it would need to in/decre,ent by larger values, problem is that value changes, that's why I'm looking for a formula basically that mimics the curve of the error, I'm sure its simple I'm just not too well versed in math lol

You know (or can work out) what discrete frequencies are achievable. You know (or can find out) what the granularity of your rotary encoder is and how many increments/decrements you're prepared to use to adjust the frequency to an arbitrary position.

Now you need to decide what frequencies you want to have available, and how you want to use the encoder to select them.

For example you might decide that one step in the 'increase' direction will change the frequency to the next higher frequency that is more than 10% higher than the current frequency, or something like that. Or you might prefer to step between frequencies which are nice round numbers. This isn't a software problem (pretty much any algorithm you can imagine can be implemented in software) but you need to decide what algorithm you want to use. The 'how do I implement it in software?' question comes later.

how could I apply dithering ?

If you generate 1Mhz 50% of the time, and 2Mhz 50% of the time, you get a 1.5Mhz output.

PWM is a form of dithering.

Now that is an interesting idea, also would allow me to get those higher frequencies with a litle more accuracy

To keep it simple for now what I'm gonna try and do is that for when the frequency is <2700 scroll by frequency since there will be no error, >2700 start scrolling by OCR1A then update the frequency so it appears the frequency is being scrolled thru yet its still accurate although jumping around