"The problem is now to find the suitable speed. When I change speeds, the reply from slave changes. It seems like 500 kbps gives the most reliable data but how do I know the right speed?"
if you have the manual for the device if should be able to tell you that! 
otherwise using a scope might be one way to determing the correct baudrate. 500kbps is probably right though! 
"My device ID is 0x11. I send RTR = 1, IDE = 0, DLC = 8. This is the question to ask for "Read A/D regs". I will get an answer: ID 464, data bytes = 0 when I should get ID 0x11, data bytes = 8."
message.id.std = 0b00100010000;//get A/D regs
0b00100010000 = 0x110 NOT 0x11!!! :o
why not just use the HEX notation to input the device ID....