Go Down

Topic: Transfer speed of the W5100 (Read 239 times) previous topic - next topic

Kane12

Hi!

I have already read some information about this question but I still don't have it clear.

I'm using an Arduino Mega 2560 + W5100 chip (I know that's not the best chip) to implement a webserver. I'm using the officials Ethernet and SPI libraries. For the moment, I'm sendind data to the clients with a transfer speed of +- 60 KB/s.

To send data I follow this strategy (I have the data in the SD card. I use the SdFat library):

Code: [Select]
char buffer[512] = {0};
unsigned int readCharacters = 0;
while(file.available()) {
    readCharacters = file.read(buffer, 512);
    client.write(buffer, readCharacters);
}



With the SD card I have tested that I can read files at the maximum speed of the SD card (+- 500 KB/s).
Because of this, the bottleneck that I have is in the W5100 chip.

I have no modified the SPI config for the W5100 chip (I use the default configuration).

Do you think I can increase the transfer speed in some way?

PaulS

Quote
Do you think I can increase the transfer speed in some way?
Yes. The TCP/IP protocol sends data in packets of 512 bytes. Some of that packet is payload, but not ALL of it. When you write data to the client 512 bytes at a time, two packets are sent, because the 512 bytes won't fit in the payload of a single packet.

If you drop the read size to something like 480 bytes, and write them to the client, all of them will more likely fit in a single packet.

Some experimenting, and perhaps packet sniffing using wireshark, will tell you the optimum read size for sending the fullest packets, without going over the payload size.
The art of getting good answers lies in asking good questions.

Juraj

#2
Jul 12, 2018, 10:55 am Last Edit: Jul 12, 2018, 11:52 am by Juraj
use byte, not unsigned int for the buffer.
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

PaulS

use byte, not unsigned int for the buffer.
OP is using char for the buffer, which is fine. The unsigned int is for the amount of data read/to be written.
The art of getting good answers lies in asking good questions.

Kane12

@PaulS Ok. I will try it. TY!

@Juraj In another post I said that I was using your StreamLib library to send data. I have modified the code because now I think that I can't use it (or yes, but I need two buffers; one for read the data and another for using your library. I think that's a waste of time fill two buffers with the same information).
Is there a way to fill a single buffer in this way and use your library to send the information?

Juraj

#5
Jul 12, 2018, 01:05 pm Last Edit: Jul 12, 2018, 01:07 pm by Juraj
I
@Juraj In another post I said that I was using your StreamLib library to send data. I have modified the code because now I think that I can't use it (or yes, but I need two buffers; one for read the data and another for using your library. I think that's a waste of time fill two buffers with the same information).
Is there a way to fill a single buffer in this way and use your library to send the information?
I have

Code: [Select]
while (dataFile.available()) {
   bp.write(dataFile.read());
}


The SD library loads the file or part of the file into own memory cache so my copying from file by one byte into BufferredPrint should be fast.
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

Kane12

#6
Jul 12, 2018, 03:12 pm Last Edit: Jul 12, 2018, 03:13 pm by Kane12
I have tested that is so much faster read with a buffer than read byte a byte.

A 400 KB file:

- With a 512 byte buffer -> +- 1s

- Byte a byte -> +- 10s

Juraj

#7
Jul 12, 2018, 04:13 pm Last Edit: Jul 12, 2018, 04:31 pm by Juraj
I have tested that is so much faster read with a buffer than read byte a byte.

A 400 KB file:

- With a 512 byte buffer -> +- 1s

- Byte a byte -> +- 10s
the SD library has extra handling of 512 B and lager buffer. it doesn't use then it's own cache
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

PaulS

the SD library has extra handling of 512 kB and lager buffer. it doesn't use then it's own cache
I don't think Arduinos and 512 kB buffers go together.
The art of getting good answers lies in asking good questions.

Juraj

I don't think Arduinos and 512 kB buffers go together.
:-) true. thank you for correcting me. (second time today)
You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

Kane12

I have obtained the maximum transfer speed with a 2048 B buffer -> 440 KB / 6.4 s = +- 70 KB/s

I can't use a bigger buffer because the data is not send correctly. I think that's because of the maximum size of the W5100's TX buffer.

Code: [Select]
#define TX_RX_MAX_BUF_SIZE 2048

This define is in the W5100.cpp file.

My unique question is if 70 KB/s is the maximum speed. The SPI is not the bottleneck because I can communicate with the SD card at +- 500KB/s.


Kane12

Using this Ethernet library PS Ethernet library I have reached a transmission speed of +- 86 KB/s. Why with the SD card I obtain a transmission speed of +- 500 KB/s (it uses SPI bus) and with the W5100 I can only obtain a transmission speed of +- 86 KB/s (it also uses SPI bus)?

I have spent so much hours searching information about this question but I don't find anything clear.

Can someone explain it to me?

Thank you!

westfw

IIRC, the W5100 has a particularly high-overhead data transfer process, where each byte read or written requires that the address of that byte in the W5100 memory also be written.

Code: [Select]
uint16_t W5100Class::write(uint16_t _addr, const uint8_t *_buf, uint16_t _len)
{
  for (uint16_t i=0; i<_len; i++)
  {
#if !defined(SPI_HAS_EXTENDED_CS_PIN_HANDLING)
    setSS();   
    SPI.transfer(0xF0);
    SPI.transfer(_addr >> 8);
    SPI.transfer(_addr & 0xFF);
    _addr++;
    SPI.transfer(_buf[i]);
    resetSS();
   :
  }
  return _len;
}


The W5100 was sort-of originally designed with a parallel interface in mind, so the SPI ability is essentially a kludge in front of the parallel interface.   It means that the max transfer rate (not including any protocol overhead or inefficiencies in the IP/TCP implementation) is about 1/4 the speed of a "good" SPI implementation that would read/write hundreds of bytes at a time.

The later chips like the W5500 have a much smarter SPI capabilities.

I don't know exactly how the W5100 uses its buffers; if they are really only 512 bytes and ALL the packet overhead has to fit in there, you have at least 14 bytes of ethernet overhead, 20 bytes of IP, and 20 bytes of TCP.   That means if you wanted one "full-ish" packet per "command", you'd need to have a data size of about 450 bytes.  But it gets more complicated when you add in TCP "thought process" - some implementations (the other side of your connection) will get pretty upset if they can't send "full-sized" packets (576-1500 bytes), and leave the connection in some sort of "throttled" state.  Windowing says it might do better to have a stream of 200 byte packets rather than "packets as big as you can send."
There would be a lot to experiment with...  (and alas, very little that you can change, since the protocol processing is all deeply embedded inside the chip.)



Go Up