convert serial read to int

You probably want Serial.parseInt(). Check the reference for details.

ASCII is ASCII, there is no decimal ASCII or hexadecimal ASCII.