What type of Arduino do you have? What Arduino uses pins 10, 11, and 12 for SPI? How do you have things connected? Why are you using SPI.h but appear to be bit-banging the SPI protocol? Is SPI.h from the SPI library or something of your own?
Where is your "... SPI.beginTransaction() to begin using the SPI port." ?
I could be wrong, but I don't think anyone can help you until you provide a lot more information.
I admit that I have not used SPI, but this doesn't look right.
According to the data sheet at http://cds.linear.com/docs/en/datasheet/1257fc.pdf (page 6):
"VOUT (Pin 7): The buffered DAC output is capable of
sourcing 2mA over temperature while pulling within 2.7V
of VCC. ... "
You are pulling within 1.3 volts (5 - 3.7) of Vcc. You are getting better than specification value. [Perhaps. You are not pulling 2mA.] What did you expect?
You cannot get an output of 12 volts using a Vcc of 5 volts.
Notice also (same URL, page 1): "... An external reference can be used to override
the internal reference and extend the output voltage range
to 12V. ..."
You also appear to have violated (same URL, page 8 ): "... The external reference must be greater
than 2.475V and less than VCC – 2.7V, ..." because your reference is 12 volts but your Vcc is 5 volts.
You may get close to 12 volts out with 12 volts input IF YOU USE A DIFFERENT PART. The LTC1257 would need a Vcc of almost 15 volts to get 12 volts out.
I wanted to help, but the time to help would have been prior to selecting this part. It's a nice part, but it can't accomplish this task. This could be fixed with an op amp, some resistors and some capacitors, but I suspect you don't want more parts.
Tread carefully, read the data sheet, and perhaps get more education as an electrical engineer. So far, you have gone about this incorrectly. Fortunately, you do not appear to have destroyed anything - yet.
This was my first reading of this data sheet. I do not plan to delve any deeper. I don't intend to sound harsh, I am just frustrated.
You are absolutely right, I am not an electric guy. i am more into web developing, but I have a project that i need to solve.
I wanted to hire someone, but I could not find anybody...
I am not sure that LTC1257 is good for me after your explanation.
Can any recommend a DAC that does not need any extra component and works with arduino without any extra component, that can be output 0-Vref. 0 Volt on code 0, Vref/2 Volt on code 2048, Vref Volt on code 4095, etc.
If it is not 12 bit but 10 bit, that is also good.
If this is still relevant, I will use a +15V source for Vcc, and a linear regulator to make a +10V for the Vref. The digital interface will still be +5V, as the LTC1257 has an internal reference for it. It will have only 1 external component: the linear regulator.