Decoding Datastrings with unknown encoding

Community General Discussion
1

Hello everyone,

I’m working on a project that involves decoding a series of complex datastrings. I’ve identified some patterns and meanings for the characters, but I’m struggling to develop a reliable method to decode these strings consistently.

My issues appear because in this case not a character stands for only one other characters. Some characters appear to have more than one meaning and I dont manage to see a pattern yet. I know the meaning of each answer.

The device is an indicator that measures deflections in mm. Currently I am communicating with it via a serial port. The Buadrate is 9600, Data bits 8, stop bits 1, Parity even and no flow control. That’s how it’s set in the device and the receiver. Those settings are changeable (but please don’t go through settings with me, I have tested them all and went through it with support). Trust me when I tell you that’s it encoded. The terminator adds CR and LF. If I forgot anything please tell me.

The message is typically build up as following:

\!>|R||||||||

this one more specifically stands for GN00,+0000.0000CRLF

What Ive identified so far, is that only the last nine are interesting for me. R/P stand for a + and H/J for a - the rest are numbers 0000.0000.

Here are some examples:

|Nr.|Command|Measurement|Output|
|---|---|---|---|
|1.|GA00|0.0000mm|\!>xR|||||||||
|2.|GA00|0.0000mm|\!>|R|||||||||
|3.|GA00|0.0000mm|\!>|R|||||||||
|4.|GA00|0.0000mm|\!>|R|||||||||
|5.|GA00|0.0000mm|\!>|R|||||||||
|6.|GA00|0.0000mm|\!>|R|||x|||||
|7.|GA00|0.0000mm|\!>|R|||||||||
|8.|GA00|0.0000mm|X!>xR|||||||||
|9.|GA00|0.0000mm|\!>xR|||||||||
|10.|GA00|0.0000mm|X!>|R|||||||||
|11.|GA00|0.0005mm|\!>|R|||||||T|
|12.|GA00|0.0005mm|\!>|R|||||||T|
|13.|GA00|0.0005mm|\!>|R|||||||T|
|14.|GA00|0.0005mm|\!>|R|||||||T|
|15.|GA00|0.0005mm|\!>|Rx||||||T|
|16.|GA00|0.0005mm|\!>|R|||||||T|
|17.|GA00|0.0005mm|\!>|R|||||||T|
|18.|GA00|0.0005mm|\!>|R||x||||T|
|19.|GA00|0.0005mm|\!>|P|||||||T|
|20.|GA00|0.0005mm|\!>|R|||||||T|
|21.|GA00|0.0010mm|\!>|R||||||t>|
|22.|GA00|0.0010mm|\!>|R||||||t<|
|23.|GA00|0.0010mm|\!>|R|||xx|t>|
|24.|GA00|0.0010mm|X!>|R||||x|t<|
|25.|GA00|0.0010mm|\!>|R||||||t<|
|26.|GA00|0.0010mm|\!>|R|||||xt>|
|27.|GA00|0.0010mm|\!>|R||||||t>|
|28.|GA00|0.0010mm|\!>xR|||||xt<|
|29.|GA00|0.0010mm|\!>|R||||||t>|
|30.|GA00|0.0010mm|\!>|R||||||t>|
|31.|GA00|0.0020mm|\!>|R||||||l>|
|32.|GA00|0.0020mm|\!>|R||||||l>|
|33.|GA00|0.0020mm|\!>|R||||||l>|
|34.|GA00|0.0020mm|\!>|R||||||l>|
|35.|GA00|0.0020mm|\!>|R|||||xl>|
|36.|GA00|0.0020mm|\!>|P||||||l>|
|37.|GA00|0.0020mm|X!>|R||||||l>|
|38.|GA00|0.0020mm|\!>|R||||||l>|
|39.|GA00|0.0020mm|\!>|Rx|||||l>|
|40.|GA00|0.0020mm|\!>|R|||x||l>|
|41.|GA00|0.0030mm|\!>|R||||||d||
|42.|GA00|0.0030mm|X!>|R||||||d||
|43.|GA00|0.0030mm|X!>|R||||||d||
|44.|GA00|0.0030mm|\!>|R||||||d||
|45.|GA00|0.0030mm|\!>|R||||||d||
|46.|GA00|0.0030mm|\!>|R||||||d||
|47.|GA00|0.0030mm|\!>|Rx|||||d||
|48.|GA00|0.0030mm|\!>|R||||||d||
|49.|GA00|0.0030mm|\!>|P||||||`x|
|50.|GA00|0.0030mm|\!>|P||||||d||
|51.|GA00|0.0040mm|\!>|R||||||\>|
|52.|GA00|0.0040mm|\!>|R||||||\<|
|53.|GA00|0.0040mm|\!>|R||||||X>|
|54.|GA00|0.0040mm|\!>|R||||||\>|
|55.|GA00|0.0040mm|\!>|R||||x|\>|
|56.|GA00|0.0040mm|\!>|R||||||X>|
|57.|GA00|0.0040mm|\!>|R||||||X<|
|58.|GA00|0.0040mm|\!>|R|||||x\>|
|59.|GA00|0.0040mm|\!>|R||||||\>|
|60.|GA00|0.0040mm|\!>|R||||||\>|
|61.|GA00|0.0050mm|\!>|P|||||xT||
|62.|GA00|0.0050mm|\!>|R||||||T||
|63.|GA00|0.0050mm|\!>|R||x|||T||
|64.|GA00|0.0050mm|\!>|R||||||P||
|65.|GA00|0.0050mm|\!>|R|||||| ||
|66.|GA00|0.0050mm|\!>|R||||||T||
|67.|GA00|0.0050mm|\!>|R||||x|T||
|68.|GA00|0.0050mm|\!>|R||x|||T||
|69.|GA00|0.0050mm|\!>|Px|||||T||
|70.|GA00|0.0050mm|\!>|R||||||T||
|71.|GA00|0.0060mm|\!>|R||||||L||
|72.|GA00|0.0060mm|X!>|R||||||L||
|73.|GA00|0.0060mm|\!>|R|||x||L||
|74.|GA00|0.0060mm|\!>|R||||||L||
|75.|GA00|0.0060mm|\!>|P||||||L||
|76.|GA00|0.0060mm|\!>|R||||||Lx|
|77.|GA00|0.0060mm|\!>|P||||||H||
|78.|GA00|0.0060mm|\!>|R|||x||H||
|79.|GA00|0.0060mm|\!>|R||||||L||
|80.|GA00|0.0060mm|\!>|R||||||L||
|81.|GA00|0.0070mm|\!>|R||||||D>|
|82.|GA00|0.0070mm|\!>|Rx|||||D>|
|83.|GA00|0.0070mm|\!>|R||||||D<|
|84.|GA00|0.0070mm|\!>|P||||||D>|
|85.|GA00|0.0070mm|\!>|R||||||D>|
|86.|GA00|0.0070mm|\!>|R||||||D>|
|87.|GA00|0.0070mm|\!>|R||||||D>|
|88.|GA00|0.0070mm|\!>|R||||||D<|
|89.|GA00|0.0070mm|\!>|R||||||D>|
|90.|GA00|0.0070mm|\!>|R||||||D>|
|91.|GA00|0.0080mm|\!>|P||||||<>|
|92.|GA00|0.0080mm|X!>|R|||x||<<|
|93.|GA00|0.0080mm|\!>|R||||||<<|
|94.|GA00|0.0080mm|\!>|R||||x|<>|
|95.|GA00|0.0080mm|\!>|P||x|||<>|
|96.|GA00|0.0080mm|\!>|R||||||<>|
|97.|GA00|0.0080mm|\!>|R||||||<>|
|98.|GA00|0.0080mm|\!>|R||||||<>|
|99.|GA00|0.0080mm|\!>|Rx|||x|<>|
|100.|GA00|0.0080mm|\!>|R||||||<>|
|101.|GA00|0.0090mm|\!>|R||||||4||
|102.|GA00|0.0090mm|X!>|R||||||4||
|103.|GA00|0.0090mm|\!>|R||||||4||
|104.|GA00|0.0090mm|\!>|R||||||4||
|105.|GA00|0.0090mm|\!>|R||x|||4||
|106.|GA00|0.0090mm|\!>|R||||||4||
|107.|GA00|0.0090mm|\!>|R||||||4||
|108.|GA00|0.0090mm|X!>|R||||||4||
|109.|GA00|0.0090mm|\!>|P||||||4||
|110.|GA00|0.0090mm|\!>|P||||x|4||

I could surely provide more examples.

That are all the possible meanings I have found so far:

‘-’ = H, J
‘+’ = R, P
9 = 4, &
8 = <, 8
7 = D
6 = L,
5 = T, *, (
4 = , F
3 = d
2 = l
1 = t, :
0 = |, <, >

x=|
X=\

Thank you!

2

does the device have a name ?

3

Yes it’s a Digimatic Indicator ID-H 0530/0560

4

Print the values ​​in Hexadecimal.
It's easier to understand and analyze.

5

I will as soon as I have access to it again!

6

have you tried contacting Mitutoyo to see if they can provide the documentation ?

there seems to be some doc about the RS232 communication in the user manual

7

image
image987×448 27.6 KB

Ref User manual pag 21.
UserManual.pdf (1.9 MB)

2 Likes
8

Yes, I have. All they could provide, were the pin specifications though. If possible I would prefer to keep the focus of this post on the decoding of the strings.

9

Yes I went through the manual. What you have sent in here is the data format for the SPC output. I cannot send commands via SPC though and can only read information out by manually sending them on the device. That is not an option. The communication used is referred to as RS232 in the manual.

10

That’s my point above

There is some documentation about the commands and the output, is what they described matching what you see?

11

But are they using ASCII, American character set or Japanese character set?

12

It is not a strings, it is a byte array. So decoding it as a strings seems pointless for me.

13

@foxytio there seems to be confusion over the format of the data you are trying to decode.

Can you provide a sample of the raw data that you are trying to decode.

14

Digimatic is the standard protocol used by Mitutoyo with it's own devices.

Try this library:

Check also this project:

15

Yes and no the commands described in this part of the Manual work perfectly. I can change settings on the device and everything else. The output that I get from the device though, does not fit what’s described in the manual. I gave an example for both - how it should look and how it does - in the topic post above. Does that answer your question?

16

Maybe I am using some words wrongly. I apologies for that. The output I protocoled above is what I get and what I would like to to find a using method to encode for.

17

Maybe I am wrong about this but the data format explained in the pdf under the first link does not fit the outputs I receive neither in length nor in position.

And for the second link. It thematizes SPC to USB which isn’t interesting for me either, since I am communicating with the device over its RS232 setting

18

it does. I'm confused though with your

they describe

image
image1046×1498 98 KB

So I don't see any pipes '|' and I'm also confused about

➜ is this the real raw output or what your code does to the output?

what would help is seeing the HEX codes for each payload answer without any modification.

2 Likes
19

That is the raw output. I didn’t change anything. What you see below is just what stands above it (the column titles) put together in one document by a Python script.

I could provide everything in hex code, would have to convert it then though. But as I said earlier, I will anyways when I have access to the device again. I hope this helps

20

Hex code would be appreciated - don’t transform what you have already but run a real extraction and print each byte in. HEX

The reason we ask is to see if the communication would be in Unicode or utf8 or something else and what you see as ASCII might not tell the whole story.