Losing HTTP connection after 15 post messages

Hi,

I have a Arduino Mega2560 with ethernetshield2 (client) that connects to a PTZ (pan tilt zoom) JVC KY100 camera (server). The connection between Arduino and camera is via HTTP. I try to control the basic camera function via the Arduino.
I am able to login to the camera (digest authentification) after some research and do some post requests in a loop (Pan, stop, Go to home positon) with 1 second delay between the steps.
The result of the response of the camera is succesfull and the camera moves to the programmed positons and speed, using the API v1.13 of JVC.
However after 15 commands (5 times through te loop) the connection is lost.
When a lost connection is detected I do a reconnect (client connected again and do a Login). Again after 15 commands the connection is lost again and goes on and on every time after reconnecting and 15 commands.

  • The Arduino mega 2560 with ethernetshield2 works with other sketches (TCP/IP connection, not HTTP) perfectly.
  • I have the latest Ethernet2 library.
  • I have the latest firmware of the camera.
  • I use external power supply or POE makes no difference
    What could be wrong, I'm looking now for several days to solve this issue, but I am not an experienced c++ programmer.
    Thanks in advance.

Not sure it has to do with your C++ skills, seems your sessions times out.

What does the camera expect to maintain the session? is there a cookie, something in the URL or client side storage or a javascript heartbeat ?

what do you see being returned if you curl to the camera?

"I am able to login to the camera (digest authentification) after some research and do some post requests in a loop (Pan, stop, Go to home positon) with 1 second delay between the steps."

So you are posting commands to the camera server once per second? Does your camera server respond to GET request or request from a browser web page?

Hi,

The camera works well when connected to Firefox or Chrome?

When logging in to the camera I have to send Get requests, so this seems oké.
The first command (Post Pan command) is after 1800 msec and the other 2 (Stop and Go home) is send after 1000msec.
Making the values shorter i.e. 500msec also gives a lost of connection after 15 times.
The response of the camera after a post says it is succesfull.

Changing the commands to only post a get system information does not change the behaviour, so also lose connection after 15 times. When making the intervals from 1800 to 20000 msec gives a timeout and no commands can be send.

I also tried with a Keep alive time out of 30 or 60 seconds but without succes.
It goes wrong after the 15th post command is send.
The camera respons says it is oké, but in the client.available loop, which waits for the HTTP message no characers are received anymore and also the client.connected becomes false.

So I'm lost where to search for this behaviour.
What could i do else?
Thanks in advance

Trace the connexion from ssh and simulate your GET commands
There must be a way for the browser to stay identified with the server as http by itself is stateless and does not maintain an open socket to the server
You need to find out how this works in your browser

Can you post a link to this specification: API v1.13 of JVC
Also post some sample get/post requests which you are issuing.
It does seem , as has been suggested, like a keep-alive method is missing .

haeleng:
@6v6g6
The link for the JVC api is here http://pro.jvc.com/pro/attributes/ip/manual/JvcCamcorderApiReferenceV113_public.pdf
When opening in a browser the url is http://192.168.1.10/main.php

below the C++ Code for sending a HTTP Post request
void Send_PTZ_Command(String cmdTxt, String Command) {

Serial.println("Send Post " + cmdTxt + " Request");
Serial.print(cmdTxt + " message lengte= ");
Serial.println(Command.length());
SendDefaultPostRequestCommand(String(Command.length()));// length of content

client.println(CalcDigestAuthCommand());
client.println(F("Connection: keep-alive"));
client.println(F("Pragma: no-cache"));
client.println(F("Cache-Control: no-cache"));
client.println();// send end of Header and body
//Send JSON message
client.println(Command);
//Serial.println(Command);
}

void SendDefaultPostRequestCommand(String Lengte)
{

client.println(F("POST /cgi-bin/cmd.cgi HTTP/1.1"));// achter post /cgi-bin/cmd.cgi
client.println(F("Host: 192.168.1.10"));
client.println(F("User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0"));
client.println(F("Accept: application/json, text/javascript, /; q=0.01"));
client.println(F("Accept-Language: ja,en-us;q=0.7,en;q=0.3"));
client.println(F("Accept-Encoding: gzip, deflate"));
client.println(F("Content-Type: application/x-www-form-urlencoded; charset=UTF-8"));
client.println(F("X-Requested-With: XMLHttpRequest"));
//client.println(F("Referer: http://192.168.1.10/main.php"));// main.php
// referer info not needed
client.println("Content-Length: " + Lengte);
//client.println(F("Origin: http://192.168.1.10"));
}

I just looked quickly and saw this:

3.2.2. Session renewal
Renew a sessionUpdating session is needed within 25 seconds as always.This command has become obsolete and session renewal is no longer required.Please don't use this command because it causes malfunction on Web access.

Does the 25 seconds about match the problem you described ?

Thanks for the suggestion. I do not use this renewal command in my code.
When I change my intervals to see code below (45 sec for 15 commands) it does work until 15 commands are sent and then again the connection is lost and a retry will be done.

Another issue I sometimes get after a connection is lost is that a request response says SessionError, which according the api doc should mean that Authentication error occured. I do not know if this is related the mentioned issue, or that this is a seperate problem.
I use a fixed supply or POE and see no difference. Also a direct connection between camera and Arduino is not different than via a switch.

void loop() {
	//int x = freeRam();
	//Serial.println(x);
	if (!client.connected()) {
		Serial.println(F("Not connected to PTZ, will try to connect"));
		if (ClientConnectedBefore) {
			client.stop();
			//SessionErrorNr = 0;
			delay(100);
		}
		ConnectToPTZ();
		delay(100);
		LogedIn = false;
	}
	// connection can get lost within send command therefore local connection test.
	if (LogedIn)
	{	
		switch (caseNr) {
		case 1:
			SendCommand(static_cast<byte>(ePan), "Right", 20, caseNr);
			NextCaseCommand(2, 1500);
			break;
		case 2:
			SendCommand(static_cast<byte>(ePan), "Stop", 0, caseNr);
			NextCaseCommand(3, 3500);
			break;
		case 3:
			SendCommand(static_cast<byte>(ePan), "Home", 0, caseNr);
			NextCaseCommand(4, 5000);
			break;

		case 4:
			caseNr = 1;
			break;
		}
		// has check for client connect
		WaitForCommandResponse();
	}
	else LoginToPTZ();// try to login
}// end loop

I can't really say too much more about it because I am not familiar with this interface.

However, the things I would check first are:

a) Because of the conflicting information about the session time outs in 2.2 and 3.2.2 in the interface specification, indicating that some changes have taken places since the previous version, check that the API version current on your camera is 1.13. say by looking at 3.2.1.

b) With the Mega you have 8kB of RAM and some of those JSON responses can get quite big. You are also using the String class which can lead to storage (heap) fragmentation. Maybe it is worth sending number of smaller commands just to see if the failure rate is proportional to the size of the data handled, indicating a memory problem.

c) Talk to the camera over a web browser in which java script is disabled to rule out some crude "keep-alive" mechanism has been implemented which would fail without java script.

Other things to look at maybe the calculation of the http message length. If it is too long, the camera may wait until a timeout for the rest of it.

I had a quick check on the Internet and found this page

In the JvcCamcorderApiReferenceV107_public.pdf you can download from there, they explain how to use the SessionID nonce you got upon login. that's something you need to have in each request.


Please correct your post above and add code tags around your code:
[code]`` [color=blue]// your code is here[/color] ``[/code].

It should look like this:// your code is here
(Also press ctrl-T (PC) or cmd-T (Mac) in the IDE before copying to indent your code properly)

I guess you missed that the OP quoted (albeit rather obscurely) the document corresponding with his installation in post #6 http://pro.jvc.com/pro/attributes/ip/manual/JvcCamcorderApiReferenceV113_public.pdf

I had missed it but indeed that is the same thing, you need to pass the session ID in all your requests

6v6gt:
I can't really say too much more about it because I am not familiar with this interface.

However, the things I would check first are:

a) Because of the conflicting information about the session time outs in 2.2 and 3.2.2 in the interface specification, indicating that some changes have taken places since the previous version, check that the API version current on your camera is 1.13. say by looking at 3.2.1.

b) With the Mega you have 8kB of RAM and some of those JSON responses can get quite big. You are also using the String class which can lead to storage (heap) fragmentation. Maybe it is worth sending number of smaller commands just to see if the failure rate is proportional to the size of the data handled, indicating a memory problem.

c) Talk to the camera over a web browser in which java script is disabled to rule out some crude "keep-alive" mechanism has been implemented which would fail without java script.

Other things to look at maybe the calculation of the http message length. If it is too long, the camera may wait until a timeout for the rest of it.

Answer
a) Firmware version of camera is 1.13 so equal to documentation version. Renewal from 3.2.2 can not be used. I tried it but It gives a param error message.

b) Using smaller strings does not help. I used the GetSystemInfo from 3.2.1 which is a much smaller message than the string length of a Pan command, and also the GetSystem info give again an exit after 15 times. I used global string variables with fixed parameters instead of passing it via the function header but this did also not help.

c) Do not know how switch of java script in a web browser. I use Firefox or Chrome, to check the crude keep alive mechanism.

d) Changing message length to other values gives errors.

Do not know how to check if keep alive is active. I tried the Timeout parameter of Keep-alive but with no succes. Al 15 times I get a response message with the result succes and then it exits connection, but in the web browser is mentioned it stays alive.

Also changed interval times between sending commands, but every time I do get a lost connection after 15 times.

Any suggestions?
Thanks

c) Do not know how switch of java script in a web browser. I use Firefox or Chrome, to check the crude keep alive mechanism.

That's easy enough. Google for firefox disable javascript

You've still got to address the point made by @J-M-L

However, if the behaviour is that it consistently fails after 15 attempts, irrespective of the interval between attempts and irrespective of the size of the returned data (assuming here that you have dimensioned the size of the receiving structure in your program to match the reduced data volume you are expecting), then it looks clear that the problem is a systematic rejection by the camera.

Post the full code that you have used for the last test.

just to be clear, that's what I understand from the documentation: your camera maintains the session information in a cookie that is passed at every request. You need to emulate that.

At the end of the login phase, the camera likely returns a "Session ID" (see page 8 and 9)

You need to capture that Session ID in the response you get from the camera and save it for later use. For exemple when requesting to set the Streaming resolution, you'll send this

@J-M-L: I
already use the SessionID in my code

@6v6gt
See full test code as (ino) attachment because the size exeeds the max of 9000 characters to post it here in this window.

When switching off java script in Chrome, the web interface is shown but does not function. There is no camara picture and commands for pan tilt etc do not work.

Thanks

TestSendCommandsPTZ3.ino (16.8 KB)

With so many Strings being (over)used everywhere (at least use passing by reference between your functions as they did in the code you got from DigestAuthorization.ino!), are you sure you are not running out of memory (as suggested by @6v6gt)?

you have this freeRam() thingy in the code, did you try to use it?

PS:

curious about why you start of with the string "200" in your readString?String readString = String(200); //string for fetching header data from IP address

--> if you wanted to reserve 200 bytes (which would be a good idea), you need to call reserve() in the setup()

 String readString;

void setup() {
  // initialize serial and wait for port to open:
  Serial.begin(115200);
  myString.reserve(200);
  ...
}

an even better idea would be to rewrite without the String class at all.... but that's just me thinking aloud here..

Reserving space for Strings, as already pointed out, is always a good idea to limit general heap fragmentation.

I've looked through your code and does not look like you have made the classic String error of defining a String as a local variable in a function, then assigning its handle to a global String variable, then letting the function in which it is defined go out of scope. This leads to all sorts of mysterious behaviour because the referenced String may no longer be there.

Having said that, if you are doing an exercise say sending 15+ times the Get Session Information command, always print the text of the sent command to the serial monitor so you can see if any data (e.g. Session ID)has been corrupted in the mean time.

The other thing is that you are following examples such as:

void SendDefaultPostRequestCommand(String Lengte)
{

	client.println(F("POST /cgi-bin/cmd.cgi HTTP/1.1"));// achter post /cgi-bin/cmd.cgi
	client.println(F("Host: 192.168.1.10"));
	client.println(F("User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0"));
	client.println(F("Accept: application/json, text/javascript, */*; q=0.01"));
	client.println(F("Accept-Language: ja,en-us;q=0.7,en;q=0.3"));
	client.println(F("Accept-Encoding: gzip, deflate"));
	client.println(F("Content-Type: application/x-www-form-urlencoded; charset=UTF-8"));
	client.println(F("X-Requested-With: XMLHttpRequest"));
	//client.println(F("Referer: http://192.168.1.10/main.php"));// main.php
	// referer info not needed
	client.println("Content-Length: " + Lengte);
	//client.println(F("Origin: http://192.168.1.10"));
}

Alot of it is probably ignored and harmless, but it would be cleaner to strip it down to the minimum necesary. The specification for http 1.1 is here: HTTP/1.1: Protocol Parameters

Also, here it looks like you have copied a standard example.

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address

I guess it would only matter if you had a second device on the network with the same mac address. Ideally, you would find this address from a label on your ethernet card so it would be correct and unique.

Clearly there are some things you can do remotely on the camera from a web browser which you can't do from the Arduino like streaming data. I guess though you have defined a subset of commands which you can use, so the exercise still makes sense. But the fact you can successfully issue a number of commands (up to the apparent limit of 15) is certainly a very good start.

I’ve not checked in details but are you always emptying the client incoming buffer before moving on?