Why am I getting \u000d\u000a appended to only SOME of my String variables

I’m using the Yun to act on commands from an HTML webpage with javascript. The webpage has a simple script for sending commands when either the “auto” or “on” radio button is selected:

HTML:

<div id="fan_controls">
  <div class="fanLabel"><label>Fan:</label></div>
  <div class="firstRadioButton"><input type="radio" name="fanState" value="auto" checked onClick="setFanAuto()"></div>
  <div class="firstRadioButtonLabel"><label>Auto</label></div>
  <div class="lastRadioButton"><input type="radio" name="fanState" value="on" onClick="setFanOn()"></div>
  <div class="lastRadioButtonLabel"><label>On</label></div>
</div>

javascript:

  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + '/');
  }

  function setFanOn() {
    $.get('/arduino/' + 'setFanState/' + 'on' + '/');
  }

The HTML and javascript codes seem to be working fine, but my problem is with the data being read in (and written to the bridge) from the Arduino sketch (using Arduino IDE 1.6.1). Here’s the relevant part of the sketch:

Arduino:

void process(YunClient client) {
  // read the command
  String command = client.readStringUntil('/');
  Console.print("[command = ");
  Console.print(command);
  Console.print("] ");
  
  // is "setFanState" command?
  if (command == "setFanState" ) {
    // Read fanState
    String fanState = client.readStringUntil('/');
    Console.print("fanState = ");
    Console.print(fanState);
    Console.println("----------------------");
    Bridge.put("command", command);
    Bridge.put("fanState", fanState);
  }

The SerialMonitor output is:

[command = setFanState] fanState = auto
----------------------
[command = setFanState] fanState = on
----------------------

and /data/get yields

{"value":{"D13":"0","intT":"70","intH":"40","D9":"50","D3":"50","fanState":"auto\u000d\u000a","command":"setFanState","humiditySetPointRequested":"20","D6":"50","outT":"60"},"response":"get"}

when the “auto” radio button is clicked, and

{"value":{"D13":"0","intT":"70","intH":"40","D9":"50","D3":"50","fanState":"on\u000d\u000a","command":"setFanState","humiditySetPointRequested":"20","D6":"50","outT":"60"},"response":"get"}

when the “on” radio button is clicked. Why am I getting \u000d\u000a appended to “auto” and “on”, while nothing is being appended to the “fanState” value for the command? Maybe more importantly: how do I get rid of these extraneous characters? I have “googled” the characters and understand them to be Unicode codes for carriage return and newline, but why are they being stored as part of the String variable “fanState”? I don’t see any difference between how my sketch is reading and storing the values for “command” and “fanState”, but it’s clear from the SerialMonitor output that these codes really are a part of the data being stored for the variable “fanState”. What am I missing??

My first thought is the the CR/LF are appended since you're reading to the end of the line. But while I'm not fluent in javascript, it looks like you are sending the command with the trailing slash, so it should only be reading up to that point and not actually reading the final CR/LF.

I can't say for sure why you're getting it, but it's easy to remove:

  fanstate.trim();

The trim() function will remove all leading and trailing whitespace characters from the string (spaces, tabs, CR, LF, etc.) Once you add that after the client.readStringUntil() call, you will have to change the print() statement that prints it to a println() in order to maintain the same output since the CR/LF will no longer be part of the printed string.

@jverive,

in your code you have:

  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + '/');
  }

you want:

  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/');
  }

The extra slash is creating an empty object. I have not tested this, because your code is incomplete. Perhaps someone else can test it.

Jesse

Thanks, ShapeShifter. Using fanState.trim() works great. I knew there had to be a simple solution, even if I don't quite understand why the control characters are getting appended.

I also tried Jesse's suggestion, but it made no difference. But then I found that if I add a space between 'auto/' and '/' either by

  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + ' ' + '/');
  }

or

  function setFanOn() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + ' /');
  }

I also get fanState without the funky appended characters. I have a more complex set of radio buttons further down in the webpage, so I'll get more opportunities to investigate what is happening - but at least I have a couple of simple solutions.

jverive:
::::SNIP::::
I also get fanState without the funky appended characters. I have a more complex set of radio buttons further down in the webpage, so I'll get more opportunities to investigate what is happening - but at least I have a couple of simple solutions.

Okay, post your final code. There is a good chance we can fix it.

Jesse

jessemonroy650:
The extra slash is creating an empty object.

That's what's confusing. Yes, he has two trailing slashes after the 'auto' or 'on' but it shouldn't affect anything the way he has the reads configured.

He first reads the command with readStringUntil('/') which should copy out characters until it huts the slash after the command string. The slash is consumed and not stored in the command. Then he reads the value using the same call. Thus should copy the "auto" or "on" and stop at the slash, which is consumed but not stored. The input string should still have that last slash in it which would be ignored. It's an extra character of overhead but shouldn't make any difference to the values he gets.

But what has me confused is why he gets the CR/LF at the end of the value? That usually happens when the trailing slash is missing and readStringUntil() hits the end of the string. In that case, everything that remains up to the end of string is returned, including the CR/LF.

Yes, we need to see the complete code, but I'm hard pressed to think of what might be in the hidden portion that would explain this behavior.

A sudden thought... jverive: might you have a revision control problem where you have mismatched code? Did you recently add the trailing slashes to the JavaScript code? Might you be running an older version of the code than you think you are? It really sounds like the string you're receiving is not what you think you're sending.

I’ve trimmed the various codes down to just the functionality needed to demonstrate the problem.

HTML (javascript embedded):

<!doctype html>
<html lang="us">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Home Control v2</title>
<link rel="stylesheet" type="text/css" href="styles.css" media="screen">
<script src="jquery.js"></script>
<script>
  $(window).load(function() {
    setTimeout(function() {
    },50);
  });
</script>
<script>
  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + ' /');
  }

  function setFanOn() {
    $.get('/arduino/' + 'setFanState/' + 'on/' + '/');
  }
</script>
</head>
<body>

<form action="">

<div class="climate">
<div id="fan_controls">
  <div class="fanLabel"><label>Fan:</label></div>
  <div class="firstRadioButton"><input type="radio" name="fanState" value="auto" checked onClick="setFanAuto()"></div>
  <div class="firstRadioButtonLabel"><label>Auto</label></div>
  <div class="lastRadioButton"><input type="radio" name="fanState" value="on" onClick="setFanOn()"></div>
  <div class="lastRadioButtonLabel"><label>On</label></div>
</div>


</form>

</body>
</html>

Arduino Yun sketch:

#include <Bridge.h>
#include <YunServer.h>
#include <YunClient.h>
#include <Console.h>

YunServer server;

void setup() {

  // Bridge startup
  pinMode(13,OUTPUT); //LED13 is used for confirming Bridge initialization
  digitalWrite(13, HIGH);
  Bridge.begin();
  digitalWrite(13, LOW);
  
  //Console is used for debugging
  Console.begin();
  
  server.listenOnLocalhost();
  server.begin();
}

void loop() {
  // Get clients coming from server
  YunClient client = server.accept();

  // There is a new client?
  if (client) {
    // Process request
    process(client);

    // Close connection and free resources.
    client.stop();
  }

  delay(250); // Poll every 250ms
}

void process(YunClient client) {
  // read the command
  String command = client.readStringUntil('/');
  Console.print("[command = ");
  Console.print(command);
  Console.println("] ");
  
  // is "setFanState" command?
  if (command == "setFanState" ) {
    // Read fanState
    String fanState = client.readStringUntil('/');
    Console.print("fanState = ");
    Console.println(fanState);
    Console.println("----------------------");
    Bridge.put("command", command);
    Bridge.put("fanState", fanState);
  }
}

CSS:

body {
  font-family:Arial,Helvettica,sans-serif;
  background-color:#ffffff;
  font-size: 0.7em;
}


.climate #fan_controls {
  display: inline-block;
  width: 100%;  
}

.climate .fanLabel label {
  display: inline-block;
  position: absolute;
  font-size: 1.2em;
  font-weight: bold;
  margin: 5px 5px 0px 0px;  
}

.climate .firstRadioButton input {
  float: left;
  position: relative;
  margin: 3px 0px 5px 65px;
}

.climate .middleRadioButton input {
  position: relative;
  float: left;
  margin: 3px 0px 5px 0px;
}

.climate .lastRadioButton input {
  position: relative;
  float: left;
  margin: 3px 0px 5px 0px;
}

.climate .firstRadioButtonLabel label {
  float: left;
  position: relative;
  display: inline-block;
  text-align: left;
  font-size: 1.1em;
  margin: 6px 12px 5px 1px;
}

.climate .middleRadioButtonLabel label {
  float: left;
  position: relative;
  display: inline-block;
  text-align: left;
  font-size: 1.1em;
  margin: 6px 12px 5px 0px;
 }

.climate .lastRadioButtonLabel label {
  float: left;
  position: relative;
  display: inline-block;
  text-align: left;
  font-size: 1.1em;
  margin: 6px 0px 5px 0px;
  clear:right;
 }

You’ll notice that in my javascript, I’ve added a space between the ‘auto/’ and the trailing ‘/’, but have not done so with the ‘on/’ function; this was done just to demonstrate the difference between the two formats. The SerialMonitor output clearly shows the effects (added line break after ‘on’ is received):

[command = setFanState] 
fanState = on

----------------------
[command = setFanState] 
fanState = auto
----------------------

and the /data/get responses are also clearly different between ‘auto/’ and ‘on/’:

{"value":{"command":"setFanState","fanState":"auto"},"response":"get"}

{"value":{"command":"setFanState","fanState":"on\u000d\u000a"},"response":"get"}

Arduino IDE version is 1.6.1, and jquery.js library version is v1.10.2 (downloaded from jquery.com). I’m saving all code (HTML and CSS) in the www directory of the sketch folder before compiling and uploading the sketch over the network, so I’m pretty sure there are no discrepancies between what I’m editing and what I’m running. And I reload the webpage prior to testing after each compile, to ensure that I’m actually running the most recent HTML code.

It looks to me like the client.readStringUntil(’/’) method is actually reading beyond the ‘/’, resulting in end of line detection (and appending), but haven’t looked into the code for the method. I appreciate the effort you folks are putting into this!

And just for completeness, I’ve modified the code to demonstrate the effects of Jesse’s recommendation to completely remove the last ‘/’. In the following code (in javascript functions) you can see that a space is added between ‘auto/’ and the trailing ‘/’, the ‘on/’ with trailing ‘/’ are as intitially coded, and the ‘off/’ command has no trailing ‘/’:

<!doctype html>
<html lang="us">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Home Control v2</title>
<link rel="stylesheet" type="text/css" href="styles.css" media="screen">
<script src="jquery.js"></script>
<script>
  $(window).load(function() {
    setTimeout(function() {
    },50);
  });
</script>
<script>
  function setFanAuto() {
    $.get('/arduino/' + 'setFanState/' + 'auto/' + ' /');
  }

  function setFanOn() {
    $.get('/arduino/' + 'setFanState/' + 'on/' + '/');
  }

  function setFanOff() {
    $.get('/arduino/' + 'setFanState/' + 'off/');
  }
</script>
</head>
<body>

<form action="">

<div class="climate">
<div id="fan_controls">
  <div class="fanLabel"><label>Fan:</label></div>
  <div class="firstRadioButton"><input type="radio" name="fanState" value="auto" checked onClick="setFanAuto()"></div>
  <div class="firstRadioButtonLabel"><label>Auto</label></div>
  <div class="middleRadioButton"><input type="radio" name="fanState" value="on" onClick="setFanOn()"></div>
  <div class="middleRadioButtonLabel"><label>On</label></div>
  <div class="lastRadioButton"><input type="radio" name="fanState" value="off" onClick="setFanOff()"></div>
  <div class="lastRadioButtonLabel"><label>Off</label></div>
</div>


</form>

</body>
</html>

The SerialMonitor output shows extra line break only for ‘on’ and ‘off’ fanState values:

[command = setFanState] 
fanState = auto
----------------------
[command = setFanState] 
fanState = on

----------------------
[command = setFanState] 
fanState = off

----------------------

and the /data/get results confirm the extra characters only for ‘on’ and ‘off’:

{"value":{"command":"setFanState","fanState":"auto"},"response":"get"}

{"value":{"command":"setFanState","fanState":"on\u000d\u000a"},"response":"get"}

{"value":{"command":"setFanState","fanState":"off\u000d\u000a"},"response":"get"}

HTTP headers for each /data/get do not reveal any addition of funky characters (outside of the added space in the ‘auto’ command - encoded normally as %20):

Request URL:http://99.181.164.173/arduino/setFanState/auto/%20/

Request URL:http://99.181.164.173/arduino/setFanState/on//

Request URL:http://99.181.164.173/arduino/setFanState/off/

So to me, the appending seems indeed to be a manifestation of the client.readStringUntil(’/’) method.

Any further ideas?

Somebody else has noticed this problem and has concluded that the last segment of the URL should be read with client.readStringUntil('\r') instead of client.readStringUntil('/'):

I've tried it, and it works. Must be a peculiarity of the stream class?

readStringUntil(’/’) should be able to properly read the value for each of these strings:

jverive:

Request URL:http://99.181.164.173/arduino/setFanState/auto/%20/

Request URL:http://99.181.164.173/arduino/setFanState/on//

Request URL:http://99.181.164.173/arduino/setFanState/off/

This is the code:

String Stream::readStringUntil(char terminator)
{
  String ret;
  int c = timedRead();
  while (c >= 0 && c != terminator)
  {
    ret += (char)c;
    c = timedRead();
  }
  return ret;
}

timedRead() is simply a function that attempts to read from a stream with a timeout. It either returns the character that was read if one is available, or returns 0 if no characters are available within the timeout period.

So, the code will keep copying files until the terminator is found, or until no character is available (or a NULL character was read.) In this case, there is a proper ‘/’ terminator after each value, so it should work! It should not be necessary to use ‘\r’ for the last call.

At this point, I suspect that the YunClient object is stripping off trailing slashes before the sketch even gets the string, just as it strips off the leading address and /arduino/. In that case:

  • http://99.181.164.173/arduino/setFanState/auto/%20/ becomes setFanState/auto/%20\r\n
  • http://99.181.164.173/arduino/setFanState/on// becomes setFanState/on\r\n
  • http://99.181.164.173/arduino/setFanState/off/ becomes setFanState/off\r\n

Since the first case still has a trialing slash after the value, the readStringUntil(’/’) stops at the slash and consumes it. However, the second and third cases no longer have a trailing slash, so readStringUntil(’/’) never finds it, keeps copying characters including the CR/LF, and finally stops when it hits the end of the string.

That’s my best guess based on the evidence so far. I don’t have time to trace through all of the YunClient code and figure out what it does, so at this point I can only suggest the two known work-arounds:

  • Use readStringUntil(’\r’) for the last token so the CR never gets copied
  • Use readStringUntil(’/’) which will copy the CR/LF for the last token, then use trim() to remove them.

Personally, whenever reading in a string and then comparing to fixed string values, I think it’s a good idea to always call trim() on the input value to get rid of any possible extraneous whitespace. For example, if the value were read in from a serial port, it would still allow the comparison to work even if the user typed some space or tab characters before the command string.

I agree that the client.readStringUntil('/') SHOULD be working, and I'm also quite inclined to believe that the client is stripping off the last '/'. I've examined the source code for the Stream class and associated functions - including timedRead() - and I don't see any problems.

I'll have to look into the client object to see what's going on, but right now I have "real" work to do.