Serial Monitor - What characters am I actually seeing?

[Thee_Captain]

The serial monitor is printing out:

A number in hexidecimal
String of characters
A number in hexidecimal

138
{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}
0

When I copy and paste this into my text editor I get

138

{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}

0

In my text editor I can "show all characters" I get

138CRLF
CRLF
{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}CRLF
CRLF
0CRLF
CRLF

I feel like I'm getting an extra CRLF between each line. Are these characters present or are they being added in by my Windows copy paste clipboard?

It is important because I am dealing with chunked HTTP responses and the hex numbers are the size of the chunks, but the it seems like there are characters unaccounted for.

[GolamMostafa]

It is good time to be familiar with various sub-windows/fields of the Serial Monitor (SM) of the Arduino IDE.

Figure-1: Serial Monitor of IDE

When we pop-down the 'Line ending tab', we see the following options:
1. Newline
When this option is selected, the SM sends 'non printable' newline character (NL) sometimes called linefeed (LF) with ASCII code 0x0A. This character/code brings the cursor below the current line.

2. Carriage return
When this option is chosen, the SM sends 'non printable' carriage return character (CR) with ASCII code 0x0D. This character/code brings the cursor below the current line.

3. Both NL & CR
When this option is active, the SM sends 'non printable' carriage return character (CR) with ASCII code 0x0D and 'non printable' newline character (NL/LF) with ASCII code 0x0A. This character/code brings the cursor below the current line.

4. No line ending
The choice of this option does not send any character at all.

Hope, the above discussion will help to interpret the message of your of your text editor/serial monitor.

[Thee_Captain]

Awesome, thank you. When you say "Serial Monitor sends", does this mean filling in the form at the top of the monitor and pressing the Send button?

Does this also happen when the Serial Monitor is receiving the characters?

I now realize I should have provided more information in my question. I am reading those characters from an HTTP response and Serial.print()-ing them to the Serial Monitor. I am not using the Serial Monitor to send this data to the micro controller. Then I am copy and pasting them into a text file and in the text file they are showing extra characters.

I will also go back through my code and update a counter to count how many bytes I'm reading in from the response and how many I'm printing to the Serial Monitor. This will give me some more insight into what's being printed.

If I am still running into the problem, I will try to write a small program to demonstrate my problem that I can inline with code tags.

Thank you for your help.

[cattledog]

It is important because I am dealing with chunked HTTP responses and the hex numbers are the size of the chunks, but the it seems like there are characters unaccounted for.

I'm not so sure about that. 0x138 = 312 decimal. I actually can count 312 characters in the message you posted when I leave out the last CRLF . That seems consistent with the protocol

Chunked transfer encoding

{"id":"3","macAddress":"D8:A0:1D:41:C1:2","userId":"0","firmId":"4","registrationNumber":"1","title":"ESP32 Pico","description":"Testing robot state using the ESP32 Pico.","hardwareLayout":"","currentState":"0","red_led":"0","blue_led":"1","_links":{"self":{"href":"https:\/\/******************.com\/robot\/3"}}}CRLF

It's easier to see what you have if you break the message into groups of 20 characters. I count 15 lines of 20 plus 12 characters to the last CRLF.

{"id":"3","macAddres 
s":"D8:A0:1D:41:C1:2 
","userId":"0","firm 
Id":"4","registratio 
nNumber":"1","title" 
:"ESP32 Pico","descr 
iption":"Testing rob 
ot state using the E 
SP32 Pico.","hardwar 
eLayout":"","current 
State":"0","red_led" 
:"0","blue_led":"1", 
"_links":{"self":{"h 
ref":"https:\/\/**** 
**************.com\/ 
robot\/3"}}}CRLF

[Thee_Captain]

Thank you cattledog. Right, so you're counting from brace to brace? {...} I count that to be correct as you do at the 312 chars. I'm wondering about the extra CRLF's; 2 after 138 and the 2 after the body. I thought I'd be expecting 1 after the 138 and 1 after the body.

I also suppose, if it is consistent, it doesn't really matter. I guess I'm just not completely confident with how I read the characters to begin with.

After reading the Wiki you provided me, it does appear that there are two CRLF's after the 0, but I'm not seeing 2 after the hex numbers before the chunks.

From HTTP Wiki

4\r\n
Wiki\r\n
5\r\n
pedia\r\n
E\r\n
 in\r\n
\r\n
chunks.\r\n
0\r\n
\r\n

I do think I'm getting closer to seeing where all the characters are coming from. Thank you everyone for talking me through this. I think after I count what I'm actually reading in I'll be satisfied with how many chars I'll need to read for each chunk.

[cattledog]

I guess I'm just not completely confident with how I read the characters to begin with.

If you want that reviewed, you'll need to post your reading method. How is the message composed by the sender?

[Thee_Captain]

void recvWithEndMarker() {
  static int ndx = 0;
  char lastReadChar = '\0';
//  char endMarker[] = {'>', '>'};
//  char endMarker[] = {'Q', 'Q'};
  char endMarker[] = {'\r', '\n'};
  char rc;

    
  while (client.available() > 0 && newData == false) {
    rc = client.read();

    if ( !(rc == endMarker[1] && lastReadChar == endMarker[0]) ) {
      receivedChars[ndx] = rc;
      
      ndx++;
      if (ndx >= numChars) {
        ndx = numChars - 1;
      }
    }
    else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
    }
    lastReadChar = rc;
  }
}

The message is being sent from an Apache Web Server running PHP. I'll have to dig through the php.ini and some .htaccess files to know what it's doing for sure.

When I hit the endpoint with anything other than my Arduino (Chrome developer tools / ARC Advanced REST Client) I'm not getting the hex chunk sizes showing up to analyze unfortunately.

That's why I resorted to copy and pasting from the Serial Monitor in a text editor. Then I was surprised by what I'm still not completely convinced aren't misplaced CRLF's. At least the one after 138.

I really appreciate the extra sets of eyes.

[GolamMostafa]

Thee_Captain:
When you say "Serial Monitor sends", does this mean filling in the form at the top of the monitor and pressing the Send button?

Does this also happen when the Serial Monitor is receiving the characters?

I hope that the following examples will clear up the matter to you.

Select 'Newline' option in the 'Line ending tab'. Place the cursor in the InputBox of the Serial Monitor (SM). Click on the Send Button of the SM. In response, the SM/PC will send only the code 0x0A.

Select 'Newline' option in the 'Line ending tab'. Place the cursor in the InputBox of the Serial Monitor (SM) and enter the character A from the Keyborad of PC. Click on the Send Button of the SM. In response, the SM/PC will send the code 0x41 (ASCII code of A) first and then the code 0x0A.

[cattledog]

I do not think that the reported chunk size will match the received message.

The reading method with the test for the two end markers will not include the final \n but will include the \r as the last character of receivedChars.

I think that strlen receivedChars will show one more than the chunk size calculated by the sender which should not include the final \r\n according to the protocol shown in the Wiki.

I do think there is something strange going on with the copy and paste into the text editor.

I would trust the Arduino and the tools available to you for working with the null terminated character arrays you generate from the reading code.

[Thee_Captain]

cattledog:
I do not think that the reported chunk size will match the received message.

The reading method with the test for the two end markers will not include the final \n but will include the \r as the last character of receivedChars.

I think that strlen receivedChars will show one more than the chunk size calculated by the sender which should not include the final \r\n according to the protocol shown in the Wiki.

I do think there is something strange going on with the copy and paste into the text editor.

I would trust the Arduino and the tools available to you for working with the null terminated character arrays you generate from the reading code.

Thank you so much. This is what the problem had "felt" like to me. I'll go through my reading method and make the same calculations today. Then I'll understand exactly what I should modify to make it work the way I need it to. Then I'll do some better unit testing next time.