Ethernet GET a file with user and password

Projects Networking, Protocols, and Devices
1

Hi,
I have an IP camera with a username and password and a special port.
Let's say the IP is 192.168.0.102 with username "pet" and password "123" at port 8080, and the file is ipcam.jpg.
To capture a picture I can use this:

wget --user=pet --password=123 -O $TIMESTAMP http://192.168.0.102:8080/ipcam.jpg
firefox http://pet:123@192.168.0.102:8080/ipcam.jpg

They both work.

Now I want my Arduino to get that file (and store it on SD card).
The string "pet:123" in base64 is "cGV0OjEyMw==".

  if (client.connect( "192.168.0.102", 8080)) 
  {
    client.println( "GET /ipcam.jpg HTTP/1.1");
    client.println("Authorization: Basic cGV0OjEyMw==");
    client.println("Host: 192.168.0.102");
    client.println("Connection: close");  //close 1.1 persistent connection  
    client.println();

But I can't get the file. The ip camera tells me : "401 Authorization Required ".
Should I read everything from the ip camera until no more is available and do a new GET with the Authorization ?

Then I read this: http://forum.arduino.cc/index.php?topic=179583.0
And I tried to do a GET with Authorization and the file in a single line:

    client.println("GET /Authorization: Basic cGV0OjEyMw==@192.168.0.102:8080/ipcam.jpg HTTP/1.1");
    client.println("Connection: close");
    client.println();

The ip camera tells me : "HTTP/1.1 200 OK ", but also "The requested page does not exist".

So I tried a number of things, and this gave binary data:

    client.println("GET /ipcam.jpg /Authorization: Basic cGV0OjEyMw== HTTP/1.1");
    client.println("Connection: close");
    client.println();

I don't know yet if the returned binary data is a jpg file, it's too small (1800 byte instead of 30k) and not recognized by a viewer.

Is that GET valid ? With both the "/ipcam.jpg" and "/Authorization" ? Or is that only for my IP camera ?

2

Here is an article that might help you.
https://docs.oracle.com/cd/E27515_01/common/tutorials/authn_http_basic.html
When the server (camera) returns the 401 code, you must send the header again with the user:password. I have not tried this.

3

I know, but I'm too lazy to implement the 401 thing.
That is how I found the GET with both the file and Authorization in a single line. And my ip camera seems to accept a single GET with both things. But I can't find anywhere that something like that is valid.

4

I can not make it work.
After the GET, a block of data is returned and the "Content-Length: " is mostly between 24k to 36k for the jpg file.
However, exactly 1800 data bytes are available, and no more. No matter how long I wait. The ip camera disconnects after some time.
Those 1800 bytes do contain the first 1800 bytes of the jpg file.

When I try the 401 thing and request it again, there are zero data bytes available.

I think there is a buffer or timing issue between the Arduino and my ip camera. Bad luck.

5

Post the code you are using. You must wait for the entire picture to be downloaded. That means parsing the Content-Length value and waiting for that amount of data, using a timeout to prevent a lockup if the connection breaks or the camera "server" stalls. It will obviously be downloaded in several packets if the max packet size is about 1400 bytes.

edit: Actually, the server should close the connection when the file is complete, so parsing the Content-Length value may not be necessary.

Also, the camera's MTU may be too big for the w5100 buffers. Maybe the packets are too big.

6

Uhm.... the code is a big mess with my passwords and so...
To test what came available, I did this:

      unsigned long previous = millis();
      while( millis() - previous < 30000UL)
      {
        int n = client.available();
        for( int i=0; i<n; i++)
        {
          byte c = client.read();
          jpgFile.write( c);
        }
      }

And I have a lot of Serial.println, they show that after the 1800 bytes, nothing else is available, and the ip camera disconnects after a while (that is normal). I also added delays in many places. Nothing helps.

My Arduino is connected to my router with a cable, but my ip camera is connected to my router via a wifi client. That is the only wifi client of the three I have that worked with this ip camera. So there is too much going on. That ip camera has probably more issues than the Arduino.

The ip camera says that the packet size is 0900, that could be 900 bytes or 9*256 bytes or 900 integers (1800 bytes).

7

Try this as a test. Does it do any better?

while(client.connected()) {
   while(client.available()) {
      byte c = client.read();
      jpgFile.write(c);
   }
}
Serial.println("disconnected");
client.stop();

There is no timeout feature, but you can add that later.

8

I did that test, but a file of 1800 bytes was created. Inclusive the server response it is 2030 bytes.
But I started with something like that, I tried so many things.
Retrieving a file from the internet and storing it on the SD card is no problem, also the jpg file itself is no problem.

This is getting weird: when I get a *.html file of 5k from the ip camera, it works !
That *.html file requires the password in the same way, but that is a real file (in flash I presume), the ipcam.jpg file is created on the fly.

9

It may be the Arduino ethernet shield is not fast enough to keep up with the camera creating the jpg file "on the fly".

10

Yes, something like that. It is definitely something in the ip camera that is not compatible with the W5100 or the Ethernet library.
When I set the picture resolution and quality of the ip camera lower, my Arduino can capture images, even when they are a larger than 1800 bytes.
So I continue with a resolution of 160x120.

Next step: using picojpeg or TJpgDec to detect a brightness change of the picture.

Thanks for your help SurferTim.