[SOLVED]Faster Uploading from ESP8266

I need to upload data from ESP8266 to an webserver created by ESP8266 itself

WiFiClient client = server.available();
    if (!client) { return; }
    while(!client.available()){  delay(1); }
    client.println("HTTP/1.1 200 OK");
    client.println("Content-Type: text/html");
    client.println("");
    client.println("<!DOCTYPE HTML>");
    client.println("<html>");
    for(int i=0;i<100000;i++)
    {
      client.println(analogRead(A0));
       
    }
   
    client.println("</html>");
    client.stop();
    delay(1);

But this code gives me a speed of 1 reading/6ms. Since I need to upload uint8_t, I need to upload data with packet size 8 bits than regular 32 bit. Can somebody explain me how to do it ?

This links describes a way to do it, but I don't understand what is MTU size ?

what is MTU size ?

I assume that you have tried Google ?

....int i=0;i<100000;....

    for(int i=0;i<100000;i++)
    {
      client.println(analogRead(A0));
       i=i+1;
       
    }

Why are you f**king with the loop index in the body of the for loop?

Writing data to a buffer, and then sending the buffer to the client when it is full will be faster than writing one value at a time to the client. You need to do some research on network packet size and how big the buffer should be.

PaulS:

    for(int i=0;i<100000;i++)

{
      client.println(analogRead(A0));
      i=i+1;
     
    }



Why are you f**king with the loop index in the body of the for loop?

Sorry about that. I did some editing before posting the Q and i left the loop index undeleted

i<100000

What is the max possible value of an int.
Leo..

Wawa:
i<100000

What is the max possible value of an int.
Leo..

That's fine I can use long int

PaulS:
Writing data to a buffer, and then sending the buffer to the client when it is full will be faster than writing one value at a time to the client. You need to do some research on network packet size and how big the buffer should be.

We can do that but sampling rate becomes inconsistent if you are taking a large no of data points , let's say a million data points at a rate of 1 reading/ms. The only way I could think of is reducing the packet size for data transfer which is 32 bit to 8 bit. But I couldn't locate this in library files without help from community.

let's say a million data points at a rate of 1 reading/ms.

You have a choice. You can send the client 1,000,000 packets with one data point each, as you are doing now, and complaining about how long it takes.

Or, you can send the client 100,000 packets with 10 data points each, in 1/10th of the time.

Now, there are trade-offs, of course. If you send fewer packets, they won't take as long to send, so you'll need to find something else to complain about.

Sending 100,000 packets, instead of 1,000,000 packets will still take a lot of time, so you may still not be happy.

You might think that you'll lose information about the timing between the data points if you send more than one data point per packet, but the timing that you see is the time between packets being sent/arriving, which is NOT the same as the timing between data points.

What do you want to do with the data?
You could send every measurement as an 8-bit binary number, instead of ASCII text, and save around three times the bandwidth.

Or you could use WebSockets: WebSockets for awesome on the ESP8266 - YouTube

Pieter

PieterP:
You could send every measurement as an 8-bit binary number, instead of ASCII text, and save around three times the bandwidth.

Is there a datatype for binary numbers ?. I can certainly convert a integer to a binary array(which is of datatype byte) but it will take 64 bits.
Could you be please be more elaborate ?

Right now, you are converting your analogRead values to text: for example, if you read a value of 512, it's converted to "512", or ASCII 0x35 0x31 0x32, so it's three bytes long. If you convert it from a 10-bit unsigned int (ranging from 0 - 1023) to an 8-bit unsigned int (0 - 255) - giving up 2 bits of accuracy - you can send the number directly (without converting it to text first) as a single byte. For example, 512 converted from 10-bit to 8-bit is 512 >> 2 = 128. (>> is right bit-shift) Now instead of "printing" it as text ("128" or 0x31 0x32 0x38), you write it as a single byte binary number: 128 = 0x80. For the same number, you now have only one byte instead of three, so it'll be three times faster.

I've used the hexadecimal format to indicate that those are the actual bytes that are sent over the TCP connection, and not a string of text.

You could send it as a 10-bit binary number as well, but this is less efficient: you'll need 2 bytes (16 bits) and you only use 10 bits of data. It's still more efficient than sending it as ASCII text though.

But of course, it all depends on what's happening with the data on the receiver side. Most HTTP servers probably won't like non-ASCII data ...

We can't really help you any further if you don't tell us what you are trying to do with the data.

Pieter

PieterP:
We can't really help you any further if you don't tell us what you are trying to do with the data.

Pieter

I think I am printing integers not text. I am analogread function (which gives a ouptput(0-256), since my voltage is low enough ) from a sensor and storing it in a byte. and using client.print() function to print the byte
STUFF You will want to know:
Collecting voltage reading( which is generally from 1-256 ADC reading) from a methane Gas sensor for 15 minutes @1 kHz. Then I would like to send the data to a computer by any possible means in any possible format

STUFF you don't want to know:
Certain Research group at IISc is building a gas sensor from a Graphene material (floating in methane gas chamber with ESP8266 on board) whose resistance changes in presence of methane gas. I had build a current source which feeds 1 uA current to graphene material and measure voltage across it to measure the change in resistance. These reading undergo certain statistical analysis to give a very damn accurate Methane gas sensor.

arpit1ug:
I think I am printing integers not text. I am analog reading value from a sensor and storing it in a byte. and using client.print() function to print the byte

No, you are printing text.

analogRead() returns an int (32 bits, 4 bytes, where the 22 first bits are always 0). You could write these bytes directly, meaning that you take the exact 4 bytes of this value from memory, and send them over the TCP connection, for example:

  int value = analogRead(A0);
  client.write((uint8_t *) &value, sizeof(value));

If 'value' equals 1023 (or 0b1111111111 in binary), it will write out 0b0000 0000 0000 0000 0000 0011 1111 1111 over the TCP connection.

This is pretty inefficient, because you're sending all those leading zeros as well. If you want 10 bits of accuracy, it's better to use 16 bits instead of 32:

  uint16_t value = analogRead(A0);
  client.write((uint8_t *) &value, sizeof(value));

If 'value' equals 1023, this will write out 0b0000 0011 1111 1111.
Which is twice as fast, but you're still sending 6 useless zeros every time.

You could do some fancy tricks to package 10-bit numbers in 8-bit bytes, but the easiest way is to just lower the resolution to 8 bits:

  uint8_t value = analogRead(A0)>>2;
  client.write(value);

If the analog reading was 1023 like before, 'value' now equals 255 (0b1111 1111 in binary).
It's only one byte long, so it will write out 0b1111 1111 over the TCP connection.

However, if you use client.print:

  int value = analogRead(A0);
  client.print(value);

It will convert the number to ASCII characters before writing it out:
1023 → "1023" → 0x31 0x30 0x32 0x33 (ASCII), so it will write out 0b0011 0001 0011 0000 0011 0010 0011 0011. By converting to text, you add a lot of overhead, because you're only using 10 (digits 0-9) of the 256 possible characters per byte, so you need more bytes to represent the same number.

If you use client.println, it gets even worse:

  int value = analogRead(A0);
  client.println(value);

println adds a carriage return ('\r' = 0x0D) and a line feed ('\n' = 0x0A) at the end of the text:
1023 → "1023\r\n" → 0x31 0x30 0x32 0x33 0x0D 0x0A, so it writes out 6 bytes, while you could represent the same number in only 2 bytes if you use the binary method.

arpit1ug:
STUFF You will want to know:
Collecting voltage reading( which is generally from 1-256 ADC reading) from a methane Gas sensor for 15 minutes @1 kHz. Then I would like to send the data to a computer by any possible means in any possible format

Does it have to be real-time? Or could you save it to a file, and then send the file to the computer after 15 minutes of measurements?
If you have at least 900,000 bytes of flash available, you can just save everything to a .hex file in the SPIFFS (using file.write(), not print()), and just use a web server to download the .hex file to the computer.
Then you can use a simple python script to convert it from a binary file to a CSV, or whatever format you want for processing.

If you need it real-time, you could try WebSockets (it supports binary data), but you won't be able to reach 1kHz, I'm afraid ... Or you could use a direct TCP connection, like you are doing now, but with or without HTTP on top of it, just a binary stream of data, if you change the Content-Type to some binary format, or an octet stream, that should work. Then just use client.write(value) as explained above.
If that's still not fast enough, you could try UDP. If you don't mind some dropped packets or some packets arriving out of order. You might want to use some kind of buffer to minimize overhead.

Pieter

Edit:

(which gives a ouptput(0-256), since my voltage is low enough )

If you are sure that you're never going to get a reading higher than 255, you can use:

  uint8_t value = min(analogRead(A0), 255);

instead of:

  uint8_t value = analogRead(A0)>>2;

PieterP:

  uint8_t value = analogRead(A0);

client.write(value);

I updated my code to this one, then I opened my browser and typed the local IP address of my module and I got

aa

Yes, that is the expected behavior: the browser expects text data, and you gave it binary data.
Try again with the correct HTTP headers:

Content-Type: application/octet-stream
Content-Disposition: attachment; filename="methane.hex"

It should download the file instead of trying to display it.
Then you can open it using a hex editor, like Bless.
Or use a simple python script:

with open("methane.hex", "rb") as f:
    byte = f.read(1)
    while byte:
        print(int(byte[0]))
        byte = f.read(1)

(Delete anything else like "" etc., just keep the HTTP headers and the client.write() data.)

I'd personally use:

uint8_t value = min(analogRead(A0), 255);

To prevent an overflow to 0 when reading a value higher than 255. Now you're basically performing a modulo operation on the value.

Pieter

PieterP:
I'd personally use:

uint8_t value = min(analogRead(A0), 255);

To prevent an overflow to 0 when reading a value higher than 255. Now you're basically performing a modulo operation on the value.

Pieter

This Works :slight_smile: but only for 1000 data points, for higher number it takes too much time.

void loop()
{
    WiFiClient client = server.available();
    client.setNoDelay(1);
    if (!client) { return; }
    while(!client.available()){  delay(1); }
    client.println("HTTP/1.1 200 OK");
    client.println("Content-Type: application/octet-stream");
    client.println("Content-Disposition: attachment; filename='methane.hex'");
    client.println("");
    for(int i=0;i<1000;i++)
    {
      uint8_t a=(analogRead(A0));
      client.write(a);
    }
    client.stop();
    delay(1);

}

Why does it takes so much time to download hex file ?

Anyway it takes time to download but it works, thanks a lot Pieter, U are GOD

It could have something to do with the MTU size being 1460 bytes.
The TCP buffer is only 1460 bytes as well: Arduino/WiFiClient.h at master · esp8266/Arduino · GitHub
I'm not sure, I've never actually tried something like this ...

Could be useful:

Probably best to ask it on GitHub, because I have no idea how the TCP connection is actually implemented.

Pieter

Problem is solved, mostly . Thanks to Pieter. Final code

#include <ESP8266WiFi.h>

WiFiServer server(80);


void setup()
{
  Serial.begin(250000);

    WiFi.disconnect();
  delay(1000);
  
  Serial.println("");
  Serial.println("Connecting");
   WiFi.begin("SSID","*****");
  while ((!(WiFi.status() == WL_CONNECTED))){
    delay(300);
    Serial.print(".");

  }
  
  Serial.println("Connected");
  Serial.println((WiFi.localIP()));
  server.begin();
 
}
void loop()
{
    WiFiClient client = server.available();
    client.setNoDelay(1);
    if (!client) { return; }
    while(!client.available()){  delay(1); }
    client.println("HTTP/1.1 200 OK");
    client.println("Content-Type: application/octet-stream");
    client.println("Content-Disposition: attachment; filename='methane.hex'");
    client.println("");
    int prev=millis();
    for(int i=0;i<1000000;i++)
    {
      if(millis()-prev<2)
      {
        uint8_t a=(analogRead(A0));
        client.write(a);
        prev=millis();
      }
     }
    client.stop();
    delay(1);

}

Although there is still the problem of slow download, I have tested for 500 readings/s for 100,000 data points. I hope it will work for more