Connection Problem using EtherCard lib: Unsupported Media Type

Hello there!

I try to get data from a website, it's only plain text, with json format. Sadly I'm not able to receive the data, it seems like the server is refusing my request.

The code I'm using is based on the webclient example from EtherCard.
Here is my code:

// Present a "Will be back soon web page", as stand-in webserver.
// 2011-01-30 <jc@wippler.nl> http://opensource.org/licenses/mit-license.php
/*
* Pin VCC -> +3.3V (rightmost, when facing the display head-on)
* Pin GND -> GND
* Pin CS -> Arduino digital pin 8
* Pin SI -> Arduino digital pin 11
* Pin SO -> Arduino digital pin 12
* Pin SCK-> Arduino digital pin 13
*/

#include <EtherCard.h>
#include <PCD8544.h>

#define STATIC 1  // set to 1 to disable DHCP (adjust myip/gwip values below)

#if STATIC
// ethernet interface ip address
static byte myip[] = { 10,0,0,200 };
// gateway ip address
static byte gwip[] = { 10,0,0,138 };
#endif


//Display
static unsigned const int led = 10;
static PCD8544 lcd;

// ethernet mac address - must be unique on your network
static byte mymac[] = { 0x74,0x69,0x69,0x2D,0x30,0x31 };

byte Ethernet::buffer[500]; // tcp/ip send and receive buffer

static uint32_t timer;

char website[] PROGMEM = "www.wienerlinien.at";

// called when the client request is complete
static void my_callback (byte status, word off, word len) {
  Serial.println(">>>");
  Ethernet::buffer[off+300] = 0;
  Serial.print((const char*) Ethernet::buffer + off);
  Serial.println("..."); 
  
  lcd.setCursor(0, 0);
  lcd.print((const char*) Ethernet::buffer + off);
}

void setup(){
  // PCD8544-compatible displays may have a different resolution...
  lcd.begin(84, 48);
  pinMode(led, OUTPUT);
  digitalWrite(led, LOW);
  lcd.setCursor(0, 0);
  lcd.print("U4-OSV");
  
  Serial.begin(57600);
  Serial.println("\n[Starting program to connect to wienerlinien.at]");
  
  Serial.println( "Initialize ethernet controller");
  
  timer = millis();
  
  if (ether.begin(sizeof Ethernet::buffer, mymac, 8) == 0) {
    Serial.println( "Failed to access Ethernet controller");
  } 
  
  #if STATIC
    ether.staticSetup(myip, gwip);
  #else
    if (!ether.dhcpSetup()) {
      lcd.setCursor(0, 3);
      lcd.print( "DHCP failed");
    }
  #endif
  
  Serial.println( "DNS Lookup, might take a while");
  
  if (!ether.dnsLookup(website)) {
    Serial.println("DNS failed");
  }
  
  ether.printIp("IP:  ", ether.myip);
  ether.printIp("GW:  ", ether.gwip);  
  ether.printIp("DNS: ", ether.dnsip);   
  ether.printIp("SRV: ", ether.hisip);
  
  Serial.println( "Setup finished!");
}

void loop(){
  
  
  ether.packetLoop(ether.packetReceive());
  
  if (millis() > timer) {
    lcd.setCursor(0, 1);
    lcd.print( "checking...");
    timer = millis() + 150000;
    Serial.println();
    Serial.print("<<< REQ ");
    ether.browseUrl(PSTR("/ogd_realtime/monitor?"), "rbl=4403&sender=te0KFsznWK", website, my_callback);
  }
}

Side note: I needed to set the ip manually to make it work

I do get a correct ip and dns:

IP:  10.0.0.200
GW:  10.0.0.138
DNS: 8.8.8.8
SRV: 193.178.171.175

The error I get is this one:

HTTP/1.1 415 Unsupported Media Type
Date: Thu, 06 Feb 2014 16:31:29 GMT
Server: Apache
X-Powered-By: Servlet 2.4; JBoss-4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)/JBossWeb-2.0
Content-Length: 1048
Connection: close
Content-Type: text/html;charset=utf-8
Set-Cookie: NSC_MCwtsw-Q-...

The data I want to get can be found here: http://www.wienerlinien.at/ogd_realtime/monitor?rbl=4403&sender=te0KFsznWK

Any help is highly appreciated, I've got no clue, whats wrong.

The EtherCard library's browseURL method doesn't send an "Accept:" header to the server (usually not needed for web pages) but the server you're sending your request seems to need that for this JSON request.

Try with "Accept: /\r\n" or if that does not work with "Accept: application/json\r\n" (to be inserted around line 485, method "www_client_internal_datafill_cb", in the GET header construction, file "tcpip.cpp").

Hey there plyon
Thanks a lot, this was the problem! Your first code worked. For anybody who has the same problem, you can download the modified version of EtherCrad from the attachment or you add the line of code in the library (EtherCard) in the file "tcpip.ccp" yourself:

if (client_postval == 0) {
      bfill.emit_p(PSTR("GET $F$S HTTP/1.0\r\n"
                        "Host: $F\r\n"
                        "Accept: */*\r\n"
                        "$F\r\n"
                        "\r\n"), client_urlbuf,
                                 client_urlbuf_var,
                                 client_hoststr, client_additionalheaderline);
    }

(Can be found in line 483 - 490).

Now I get this response

<<< REQ >>>
HTTP/1.1 200 OK
Date: Sat, 08 Feb 2014 18:05:21 GMT
Server: Apache
X-Powered-By: Servlet 2.4; JBoss-4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)/JBossWeb-2.0
Connection: close
Content-Type: application/json;charset=UTF-8
Set-Cookie: NSC_MCwtsw-Q-Fyu-SAw=ffffffffc3a0377645525d5f4f58455e445a4a423660;expires=Sat, 08-Feb-2014 18:07:21 GMT;path=/;httponly

Sadly I don't know how to get the data from the site, so the json code. I thought it would be in the same message, but increasing the buffersize does not help. Can anybody tell me, where I find the actual data?

To try out the program, use the code above with the library, I attched to this post.

EtherCard.zip (84.5 KB)

It looks like your web server holds a session and is not prepared to answer single requests. You probably have to store the provided Cookie and send it back in your next request.
So if you received the answer in your previous post, you have to add this header in your request:

Cookie: NSC_MCwtsw-Q-Fyu-SAw=ffffffffc3a0377645525d5f4f58455e445a4a423660

Remember that sessions usually expire so you won't be able to hard code that into your application. You probably cannot use the browseURL() method anymore as you have to construct your requests yourself.

Thank a lot for your research pylon. I think this small project is harder than I thought. I will probably let it be, but thanks again for your help!