Go Down

Topic: Serial Monitor - What characters am I actually seeing? (Read 193 times) previous topic - next topic

Thee_Captain

The serial monitor is printing out:

A number in hexidecimal
String of characters
A number in hexidecimal

Code: [Select]

138
{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}
0


When I copy and paste this into my text editor I get

Code: [Select]

138

{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}

0


In my text editor I can "show all characters" I get

Code: [Select]

138CRLF
CRLF
{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}CRLF
CRLF
0CRLF
CRLF


I feel like I'm getting an extra CRLF between each line. Are these characters present or are they being added in by my Windows copy paste clipboard?

It is important because I am dealing with chunked HTTP responses and the hex numbers are the size of the chunks, but the it seems like there are characters unaccounted for.

GolamMostafa

#1
Mar 23, 2019, 01:36 am Last Edit: Mar 23, 2019, 03:33 am by GolamMostafa
It is good time to be familiar with various sub-windows/fields of the Serial Monitor (SM) of the Arduino IDE.

Figure-1: Serial Monitor of IDE

When we pop-down the 'Line ending tab', we see the following options:
1.  Newline
When this option is selected, the SM sends 'non printable' newline character (NL) sometimes called linefeed (LF) with ASCII code 0x0A. This character/code brings the cursor below the current line.

2.  Carriage return
When this option is chosen, the SM sends 'non printable' carriage return character (CR) with ASCII code 0x0D. This character/code brings the cursor below the current line.

3.  Both NL & CR
When this option is active, the SM sends 'non printable' carriage return character (CR) with ASCII code 0x0D and 'non printable' newline character (NL/LF) with ASCII code 0x0A. This character/code brings the cursor below the current line.

4.  No line ending
The choice of this option does not send any character at all.

Hope, the above discussion will help to interpret the message of your of your text editor/serial monitor.

Thee_Captain

Awesome, thank you. When you say "Serial Monitor sends", does this mean filling in the form at the top of the monitor and pressing the Send button?

Does this also happen when the Serial Monitor is receiving the characters?

I now realize I should have provided more information in my question. I am reading those characters from an HTTP response and Serial.print()-ing them to the Serial Monitor. I am not using the Serial Monitor to send this data to the micro controller. Then I am copy and pasting them into a text file and in the text file they are showing extra characters.

I will also go back through my code and update a counter to count how many bytes I'm reading in from the response and how many I'm printing to the Serial Monitor. This will give me some more insight into what's being printed.

If I am still running into the problem, I will try to write a small program to demonstrate my problem that I can inline with code tags.

Thank you for your help.

cattledog

Quote
It is important because I am dealing with chunked HTTP responses and the hex numbers are the size of the chunks, but the it seems like there are characters unaccounted for.
I'm not so sure about that. 0x138 = 312 decimal. I actually can count 312 characters in the message you posted when I leave out the last CRLF . That seems consistent with the protocol
https://en.wikipedia.org/wiki/Chunked_transfer_encoding

Code: [Select]
{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}CRLF

It's easier to see what you have if you break the message into groups of 20 characters. I count 15 lines of 20 plus 12 characters to the last CRLF.

Code: [Select]

{"id":"3","macAddres
s":"D8:A0:1D:41:C1:2
","userId":"0","firm
Id":"4","registratio
nNumber":"1","title"
:"ESP32 Pico","descr
iption":"Testing rob
ot state using the E
SP32 Pico.","hardwar
eLayout":"","current
State":"0","red_led"
:"0","blue_led":"1",
"_links":{"self":{"h
ref":"https:\/\/****
**************.com\/
robot\/3"}}}CRLF

Thee_Captain

Thank you cattledog. Right, so you're counting from brace to brace? {...} I count that to be correct  as you do at the 312 chars. I'm wondering about the extra CRLF's; 2 after 138 and the 2 after the body. I thought I'd be expecting 1 after the 138 and 1 after the body.

I also suppose, if it is consistent, it doesn't really matter. I guess I'm just not completely confident with how I read the characters to begin with.

After reading the Wiki you provided me, it does appear that there are two CRLF's after the 0, but I'm not seeing 2 after the hex numbers before the chunks.

From HTTP Wiki
Code: [Select]
4\r\n
Wiki\r\n
5\r\n
pedia\r\n
E\r\n
 in\r\n
\r\n
chunks.\r\n
0\r\n
\r\n


I do think I'm getting closer to seeing where all the characters are coming from. Thank you everyone for talking me through this. I think after I count what I'm actually reading in I'll be satisfied with how many chars I'll need to read for each chunk.

cattledog

Quote
I guess I'm just not completely confident with how I read the characters to begin with.
If you want that reviewed, you'll need to post your reading method. How is the message composed by the sender?

Thee_Captain

Code: [Select]
void recvWithEndMarker() {
  static int ndx = 0;
  char lastReadChar = '\0';
//  char endMarker[] = {'>', '>'};
//  char endMarker[] = {'Q', 'Q'};
  char endMarker[] = {'\r', '\n'};
  char rc;

   
  while (client.available() > 0 && newData == false) {
    rc = client.read();

    if ( !(rc == endMarker[1] && lastReadChar == endMarker[0]) ) {
      receivedChars[ndx] = rc;
     
      ndx++;
      if (ndx >= numChars) {
        ndx = numChars - 1;
      }
    }
    else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
    }
    lastReadChar = rc;
  }
}


The message is being sent from an Apache Web Server running PHP. I'll have to dig through the php.ini and some .htaccess files to know what it's doing for sure.

When I hit the endpoint with anything other than my Arduino (Chrome developer tools / ARC Advanced REST Client) I'm not getting the hex chunk sizes showing up to analyze unfortunately.

That's why I resorted to copy and pasting from the Serial Monitor in a text editor. Then I was surprised by what I'm still not completely convinced aren't misplaced CRLF's. At least the one after 138.

I really appreciate the extra sets of eyes.

GolamMostafa

When you say "Serial Monitor sends", does this mean filling in the form at the top of the monitor and pressing the Send button?

Does this also happen when the Serial Monitor is receiving the characters?
I hope that the following examples will clear up the matter to you.

Select 'Newline' option in the 'Line ending tab'. Place the cursor in the InputBox of the Serial Monitor (SM). Click on the Send Button of the SM. In response, the SM/PC will send only the code 0x0A.

Select 'Newline' option in the 'Line ending tab'. Place the cursor in the InputBox of the Serial Monitor (SM) and enter the character A from the Keyborad of PC.  Click on the Send Button of the SM. In response, the SM/PC will send the code 0x41 (ASCII code of A) first and then the code 0x0A.


cattledog

I do not think that the reported chunk size will match the received message.

The reading method with the test for the two end markers will not include the final \n but will include the \r as the last character of receivedChars.

I think that strlen receivedChars will show one more than the chunk size calculated by the sender which should not include the final \r\n according to the protocol shown in the Wiki.

I do think there is something strange going on with the copy and paste into the text editor.

I would trust the Arduino and the tools available to you for working with the null terminated character arrays you generate from the reading code.

Thee_Captain

I do not think that the reported chunk size will match the received message.

The reading method with the test for the two end markers will not include the final \n but will include the \r as the last character of receivedChars.

I think that strlen receivedChars will show one more than the chunk size calculated by the sender which should not include the final \r\n according to the protocol shown in the Wiki.

I do think there is something strange going on with the copy and paste into the text editor.

I would trust the Arduino and the tools available to you for working with the null terminated character arrays you generate from the reading code.

Thank you so much. This is what the problem had "felt" like to me. I'll go through my reading method and make the same calculations today. Then I'll understand exactly what I should modify to make it work the way I need it to. Then I'll do some better unit testing next time.

Go Up