Ethernet shields client read buffer

Hi, I hope this is the right place to post this.

I am trying to make my arduino interface with the utorrent web API. I have had quite a lot of luck until I hit the point where I requested a list of all the torrents. This list is so long that it fills the 2048 char buffer so I can't read the whole list.

void GetList()
{
  Token = GetToken(); 
  String ListString = "GET /gui/?token=" + Token + "&list=1 HTTP/1.1";
  
  client.flush();

  client.println(ListString);
  client.print("Authorization: Basic ");
  client.println("XXXXXXXXXXXXXXXXXXX");
  client.println();
  
  //Wait for response
  delay(500);
  int i = 0;
  
  //Read the response
  while (client.available()) 
  {  
    char c = client.read();
    Serial.print(c);
    
    if (c == '\n')
    {
      Line[i] = ServerListreply;
      i++;
      //client.flush();
    }
    else
    {
      ServerListreply = ServerListreply + c;   
    }  
  }
}

As you can see I flush the buffer before making the request, but still only have enough room for 6 our of the 71 torrents I currently have.

The server response looks something like this (well this is as much as I can get before the buffer overflows):

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 12208
Content-Type: text/plain
Cache-Control: no-cache

{"build":23703,"label": [
["TV Show",7]
]
,"torrents": [

["2549155C57D750F36085BB42B42B7906524779B3",200,"Adulthood.LiMiTED.DVDRip.XviD",734773566,1000,734773566,230801408,314,0,0,0,"",0,88,0,131,65536,-1,0],
["8A9925FDFD14EC2D1EE7B7E37BD5CFBFC48E0881",200,"Aladdin[1992]DvDrip[Eng]-Stealthmaster.avi",734035968,1000,734035968,538467163,733,0,0,0,"",0,382,0,612,65536,-1,0],
["6FA17D5E10098C273D1E1C1EA755DC483270B859",200,"Archer.2009.S02E04.Pipeline.Fever.HDTV.XviD-FQM.avi",183776848,1000,183776848,194215940,1056,0,0,0,"TV Show",0,874,0,874,65536,-1,0],
["E9698F9D6556493C2263355F27D07573234CBEE9",200,"Bambi[1942]DvDrip.AC3[Eng]-aXXo",733075244,1000,733075244,572938028,781,0,0,0,"",0,304,0,491,65536,-1,0],
["7801FFCD5E25942ACA3C9641BECF4E91826A0B2B",201,"Buried {2010} DVDRIP. Jaybob",724286462,1000,724286462,1067874302,1474,949,67,19552,"",34,678,0,962,393121,-1,0],
["A8A4CEE44A6FD92A8711ABBAB291DFB76577A793",136,"Chase & Status - More Than Alot [320kbs] + COVERS - Drum & Bass music album   ~mahasonaz~",116937446,1000,118903526,178326424,1499,0,0,0,"",0,0,0,0,65536,-1,0],
["4090EBD0197AAC71834A8E86D2752D237366F429",201,"Easy A[2010]DvDrip[Eng]-FXG",733630353,1000,733630353,346754975,472,45,5,-1,"",3,686,0,294,65536,-1,0],

What I would like to do is clear out the buffer after each line break. That (in theory) would mean I never run out of buffer, but if I do a regular client.flush() the "while (client.available())" fails and I drop out of the loop.

Is there any way I can flush out the old data without losing any new data? (sorry for the wall of code/output)

What I would like to do is clear out the buffer after each line break.

You are clearing the buffer by reading from it. Each byte you read makes room for another character. You simply need to read faster.

All kidding aside, this isn't helping

      ServerListreply = ServerListreply + c;

Each time you add a character to the string, a new chunk of memory is allocated, the old string is copied character by character to the new string. Another character by character copy is made by the assignment statement.

For the fastest reading of data, you really must quit using the String object.

I think you will need to find a way to request less data at one time from the utorrent server.

PaulS:

What I would like to do is clear out the buffer after each line break.

You are clearing the buffer by reading from it. Each byte you read makes room for another character. You simply need to read faster.

That makes sense. I wasn’t really considering the processing speed or efficiency of my code.

I will try replacing my strings with char arrays and see if I have any more luck. Thanks.

Unfortunately, I don’t think I can request less data. Which may mean this isn’t possible.

Unfortunately, I don't think I can request less data. Which may mean this isn't possible.

Perhaps not from utorrent, but you might be able to create an application on a web server, somewhere, that gets the data from utorrent, and puts it out one specific record at a time. Then, your Arduino would make a request that tells your server app to bother utorrent for some data. That request would return the number of records that the server app got from utorrent.

Then, the Arduino would loop, requesting record #n, for n=0 to max-1, which your server app would return from it's store.

If you only want a certain part of the data being sent, then you will need to write a routine that evaluates the data as it is being received and only save the desired part.

The server idea is a nice solution.

I want to have a push buttons to stop/start all the torrents. But unfortunately there is no stop all command in the API, so you have to retrieve a list of all the torrents and then loop through each one stopping (or starting) them. So it is only the hash number (the alpha-numeric at the beginning of each line) that I am really interested in.

Thinking as I type: As utorrent is on the server, I could write a script (.sh, php or something) on the server to start/stop all the torrents and just use the arduino to call that script (through a HTTP GET request).