Reading the second file you uploaded, I don't think the sensor is using Modbus. The text suggests that the sensor sends the information continuously, rather than wait to be asked for the information.
There is a basic serial handling tutorial here in the forums:
You don't appear to have start and end markers as such. However, it seems that your returned data message will always begin with 0xAA 0xBB and will be 12 bytes long. If you had a buffer 12 bytes long, and stored the received bytes, shifting out the oldest byte when the buffer fills, you can check for the first 2 bytes matching 0xAA & 0xBB. If you get a match, then calculate the checksum and compare it with the 12th byte. If you get a match, then you have a complete message and can extract the data bytes as needed.
Additionally, the text file suggests that the data is transferred as ASCII hex and that possibly each message may be terminated by a CR LF - as it seems to say that you can view this data using a serial port monitor. You should be able to easily determine this for yourself.