How much can 328p handle?

Oh yeah - way faster
add this
#include <SPI.h>

Get rid of all these
 #define W_CLK 8       // Pin 8 - connect to AD9850 module word load clock pin (CLK)
 #define FQ_UD 9       // Pin 9 - connect to freq update pin (FQ)
 #define DATA 10       // Pin 10 - connect to serial data load pin (DATA)
 #define RESET 11      // Pin 11 - connect to reset pin (RST).

use this:
// pin 13 to clock
// pin 12 to devices data out pin if you will read anything back
// pin 11 to devices data in pin that you are sending data to
 #define FQ_UD 10       // Pin 10 - connect to freq update pin (FQ)
 #define RESET 9      // Pin 9 - connect to reset pin (RST).

Put your 5 bytes in an array, and add this in place of the bit banging you have now"

PORTB = PORTB & B11111011; // clear pin 10
SPI.transfer(dataArray[0]);
SPI.transfer(dataArray[1]);
SPI.transfer(dataArray[2]);
SPI.transfer(dataArray[3]);
SPI.transfer(dataArray[4]);
PORTB = PORTB | B00000100; // set pin 10