Hey Guys,
when I use my ethernet shield (V2) with the Ethernet.h built in library, I never get a better transfer rate than 1.2 kbyte/sec. which should be equal to 9600 bauds.
I can initialitze the serial monitor with serial.begin(115200) or something like that but of course this wouldnt help.
Is there any way how I could easily improve the transfer rate? It is not really satisfying if you have a 10/100 mbps ethernet shield which can only work with 9600 bauds.
Big thanks
It is not really satisfying if you have a 10/100 mbps ethernet shield
Do you really? Which one? Not theW5100 based shield...
Post your code.
If yo0u transfer one byte per packet, the transfer rate will be very slow.
You might give this Ethernet library a try:
Right now I'm (again) working on trying to figure out why all these libraries are so very slow. On WebClient, Ethernet2 achieves about 10 kbytes/sec. My optimizations speed it up to 95 kbytes/sec. But that's still pretty terrible. In theory, about 1 Mbyte/sec ought to be possible with 12 or 14 MHz SPI clock and W5500's protocol overhead.
However, if you're only achieving 12% of the maximum of the standard Ethernet library then there's probably a bottleneck in your code so the faster library might not help at all until you fix that issue.
I'm having similar issues, this thread is rather interesting 
If one were to download and add the Ethernet library from Github, how does one tell the compiler which library to use:
Multiple libraries were found for "Ethernet.h"
Used: C:\Arduino\arduino-1.6.12\libraries\Ethernet
Not used: C:\Users[USERNAME]\Documents\Arduino\libraries\Ethernet-master
Also, I want to override some w5500 settings, such as:
w5500.setRetransmissionCount(1);
It isn't obvious to me how to do this, the Github page suggests the w5500 has been tested with the code, but I can't see any references.
Any help is greatly appreciated
weird_dave:
how does one tell the compiler which library to use:Multiple libraries were found for "Ethernet.h"
Used: C:\Arduino\arduino-1.6.12\libraries\Ethernet
Not used: C:\Users[USERNAME]\Documents\Arduino\libraries\Ethernet-master
Rename C:\Users[USERNAME]\Documents\Arduino\libraries\Ethernet-master to C:\Users[USERNAME]\Documents\Arduino\libraries\Ethernet. The library in the sketchbook folder will take include priority over the library in the Arduino IDE installation folder if the folder name matches the included file name.
Brilliant, that works a treat!
Assuming it has picked up all the correct includes, there is some improvement in speed. Looking at the time between SPI bytes, there's still a rather large time delay between them, it looks like the buffer transfers are byte at a time rather than SPI.transfer (buffer, size) which is about 3x faster in testing over here: Ethernet2 (UDP) SPI transfers have a lot of dead time - Arduino Due - Arduino Forum
I'm running this on a Due and with an increased SPI speed of 28MHz
Any idea how to change the retry count and timeout with this library?
The retransmission time and count should be modified the same way as the IDE library.
It took a while to notice that the class reference was W5100 not w5100
So I have replaced my:
w5500.setRetransmissionCount(1);
w5500.setRetransmissionTime(1);
with
W5100.setRetransmissionCount(1);
W5100.setRetransmissionTime(1);
but I am seeing 7 ARP broadcasts where I was previously seeing 2 (1 retry) and the time between broadcasts has reduced to from 200ms, but definitely not to 100us as it should.
Which library are you using? Which Wiznet IC are you using?
edit: I recommend using the Wiznet library for any Wiznet IC other than the w5100.
Ah yes, my apologies for leaving out that vital info.
I'm using the w5500. The previous library I was using was Ethernet2 from the library manager.
I'm away from my kit at the moment so I'll have to wait for playtime.
I assume I'll install it then rename the folder to Ethernet like I did with Pauls library?
I see they've set it up for 42MHz SPI, I think I'll slow it to 28MHz for comparison (and my eyes aren't that fast)
Hey guys,
sorry for my late response, I had a little incident here.
My code is unfortunately too long to be placed here.
To put it in a nutshell, I use
client.write(file.read());
and
file.write(client.read());
to read/write from a sd card. Anyways that shouldn't matter since the result is the same when I just read a byte from a client and put it in a variable which is overwritten with the next byte (I tested this).
these commands are put in loops and write 1 Byte each time.
Can I expect a faster transfer rate by using this function?
client.write(buf, len)
Are there any other ways like changing some parameters in the library?
Thank you pert for the recommendation of Pauls library. I would prefere a built-in library solution, but if there is none, your library will be my next challenge.
I think you're either going to have to paste the code or explain what you're doing, because this is the first mention of SD card transfers.
Generally, dealing with multibyte buffers is faster than byte by byte for most things. If you're transferring bytes to/from SD and Ethernet, it's going to be pretty slow with all the overheads between the bytes.
Can I expect a faster transfer rate by using this function?
client.write(buf, len)
Yes. i get about a 4x increase sending 64 bytes at a time rather than 1.
char tBuf[65];
while(myFile.available()) {
clientCount = myFile.read(tBuf,64);
client.write((byte*)tBuf,clientCount);
}
Can I expect a faster transfer rate by using this function?
client.write(buf, len)
I suspected that that was the bottleneck in your code, which is why I asked you to post it. Yes, as SurferTim points out, you can speed things up that way.
Are there any other ways like changing some parameters in the library?
You know what the term "low hanging fruit" means? Post your code. Let's fix all the issues with it first. Then, if that is not sufficient, we can discuss changes to code in libraries.
Thanks guys!
I will perform the changes SurferTim suggested and will tell you the results later.
I actually assumed a hardware related bottle neck, since the transfer rate of 1.2kb/s is equivalent to 9600bauds. So I thought I rather had to change this value somewhere, but well, you guys are obviously right. I am already looking forward for the improvements.
Here comes the complete code. I have to split it on 2 Posts, since it is too long.
#include <SD.h>
#include <Ethernet.h>
//Im Debug-Mode werden zusätzliche, Prozess-relevante Ausgaben auf dem Serial-Monitor getätigt.
//(Auskommentieren, falls dies nicht gewünscht ist.)
#define DEBUGMODE
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };
int port = 1000;
EthernetServer server(port);
File file;
String serialInputString = "";
String clientInputString = "";
String fileName = "";
int i = 0;
unsigned long long1 = 0, long2 = 0;
unsigned char c = 0;
unsigned long long fileSize = 0;
unsigned long long transferredBytes = 0;
void setup() {
Serial.begin(9600);
while (!Serial) {}
digitalWrite(10,HIGH);
if(SD.begin(4) == 0){
Serial.println(F("SD init fail"));
}
Ethernet.begin(mac);
server.begin();
Serial.print("server is at ");
Serial.println(Ethernet.localIP());
delay(200);
}
void loop() {
delay(400);
Serial.print("System: Waiting for incomming connection on IP: ");
Serial.print(Ethernet.localIP());
Serial.println(" Port:" + String(port));
EthernetClient client = server.available();
if(client.connected()){
Serial.println("System: Client connected.");
while(client.connected()){
#ifdef DEBUGMODE
Serial.println("Waiting for Serial Input or Client Input...");delay(700);
#endif
/////////////////////////////////// Serial Input Handler //////////////////////////////////////////////
//Prüft, ob Eingaben auf dem seriellen Monitor getätigt wurden.
if(Serial.available()){
delay(100); //Serial vollständig in Puffer einlesen lassen.
while(Serial.available()){
serialInputString += (char)Serial.read();
}
Serial.println("Server(Arduino): " + serialInputString);
Serial.flush();
/////////////////////////////////// File Sender //////////////////////////////////////////////
/*Sendet die Datei, im Falle einer #SENDFILE#-Eingabe. Die Datei wird im obersten Verzeichnis
gesucht und zwar mit dem Namen, der nach #SENDFILE# geschrieben wurde. Bsp: #SENDFILE#abc.txt
*/
if(serialInputString.startsWith("#SENDFILE#")){
serialInputString.remove(0, 10);
#ifdef DEBUGMODE
Serial.println("So file name is:" + serialInputString);
#endif
fileName = serialInputString;
file = SD.open(fileName,FILE_READ);
if(!file){Serial.println(F("SD open fail"));break;}
if(!file.seek(0)){Serial.println(F("Rewind fail")); file.close();break;}
Serial.print("Sending File: " + fileName + " with file size of: ");
Serial.print(file.size());Serial.print(" Bytes. That's ");Serial.print(file.size()/1024);
Serial.print(" KBytes or ");Serial.print(((file.size()/1024)/1024));Serial.println(" MBytes.");
client.print("#SENDFILE");
fileSize = ( fileSize | file.size() );
for(i=7; i>=0; i--){
client.write((fileSize >> i*8));
}
client.print(fileName);
client.write("\n");
if(file.size()>100){ Serial.println("Progress:"); }
for(transferredBytes = 0; transferredBytes < file.size(); transferredBytes++){
client.write(file.read());
if(( (transferredBytes%(fileSize/100)) == 0) && fileSize>100){
Serial.print((unsigned int)(transferredBytes/(fileSize/100)));
Serial.print("% ");
if((transferredBytes/(fileSize/100))%20 == 0 && (transferredBytes/(fileSize/100)) != 0){ Serial.print("\n"); }
}
}
Serial.println("100%\nTransfer complete.");
#ifdef DEBUGMODE
Serial.print("transferredBytes is: ");Serial.println((unsigned long)transferredBytes);
Serial.print("file.size() is: ");Serial.println(file.size());
#endif
file.close();
fileName = "";
transferredBytes = 0;
serialInputString = "";
fileSize = 0;
}
/////////////////////////////////// Text Sender //////////////////////////////////////////////
//Sendet die Eingabe als Textnachricht
else{
unsigned long bytesSent = 0;
bytesSent = client.println(serialInputString);
#ifdef DEBUGMODE
Serial.print(bytesSent);
Serial.println(" bytes sent by client.print().");
#endif
serialInputString = "";
}
}
/////////////////////////////////// Client Input Handler //////////////////////////////////////////////
/*Prüft, ob Eingangsdaten im Socket vorliegen. Liest in diesem Fall das erste Byte oder mehrere ein,
um die Art der Nachricht festzustellen*/
if(client.available()){
delay(50);
#ifdef DEBUGMODE
Serial.println("Oh, some incoming data is waiting to get retrieved!");
#endif
clientInputString.concat((char)client.read());
#ifdef DEBUGMODE
Serial.println("One byte of that data has been concatenated to the clientInputString.The whole string is now:" + clientInputString);
#endif
delay(200);
if(clientInputString == "#"){
#ifdef DEBUGMODE
Serial.println("Oh, the clientInputString is exactly #!");
#endif
while(1){
clientInputString.concat((char)client.read());
/////////////////////////////////// File Receiver //////////////////////////////////////////////
/*Erzeugt und schreibt die Datei im Fall einer Dateiübertragung*/
if(clientInputString.startsWith("#SENDFILE")){
#ifdef DEBUGMODE
Serial.println("Oh, string is now '#SENDFILE' !! getting long...");
#endif
for(i=0; i<8; i++){
c = client.read();
#ifdef DEBUGMODE
Serial.print(c, HEX); Serial.print(" "); Serial.println(c, BIN);
#endif
fileSize = ( fileSize << 8 );
fileSize = ( fileSize | c );
}
#ifdef DEBUGMODE
Serial.println("Reading the long value (8bytes) done.");
#endif
long1 = (unsigned long)(fileSize >> 32);
long2 = (unsigned long)(fileSize);
if(long1==0){
Serial.print("File size is:");Serial.print(long2);Serial.print("Bytes. That's ");Serial.print((long2/(unsigned long)1024));Serial.print("KBytes or ");Serial.print((long2/((unsigned long)1024)/1024));Serial.println("MBytes.");
}
else{Serial.print("File size is:");Serial.print((unsigned int)(fileSize/(unsigned long long)1024));Serial.print("KBytes. That's ");Serial.print((unsigned int)(fileSize/(unsigned long long)1048576));Serial.print("MBytes.");}
#ifdef DEBUGMODE
Serial.println("Longs are:");
Serial.print(long1,HEX);Serial.print(" ");Serial.println(long1, BIN);
Serial.print(long2,HEX);Serial.print(" ");Serial.println(long2, BIN);
Serial.println("Getting file name until new-line-command...");
#endif
while(!fileName.endsWith("\n")){ fileName.concat((char)client.read()); }
fileName.trim();
#ifdef DEBUGMODE
Serial.println("File name after trim is: " + fileName);
#endif
SD.remove(fileName);
file = SD.open(fileName,FILE_WRITE);
if(!file){Serial.println(F("SD open fail"));}
if(!file.seek(0)){Serial.println(F("Rewind fail")); file.close();}
Serial.println("Receiving File...");
if(fileSize>100){Serial.println("Progress:");}
for(transferredBytes = 0; transferredBytes < fileSize; transferredBytes++){
while(!client.available()){delay(1);}
file.write(client.read());
if(( (transferredBytes%(fileSize/100)) == 0) && fileSize>100){
Serial.print((unsigned int)(transferredBytes/(fileSize/100)));
Serial.print("% ");
if((transferredBytes/(fileSize/100))%20 == 0 && (transferredBytes/(fileSize/100)) != 0){Serial.print("\n");}
}
}
Serial.println("100%\nTransfer complete.");
#ifdef DEBUGMODE
Serial.print("fileSize is: ");Serial.println((unsigned long)fileSize);
Serial.print("transferredBytes is: ");Serial.println((unsigned long)transferredBytes);
Serial.print("file.size() is: ");Serial.println(file.size());
if(file.available()){Serial.print("ERROR... There is something more to read! Bytes available to read: "); Serial.println(file.available());}
if(!file.available()){Serial.println("Nothing to read!");}
#endif
file.close();
fileName = "";
fileSize = 0;
transferredBytes = 0;
clientInputString = "";
long1 = 0;
long2 = 0;
c = 0;
break;
}
}
/////////////////////////////////// Text Receiver //////////////////////////////////////////////
/*Empfängt den Text im Fall einer Textmitteilung und gibt diesen aus*/
}else{
#ifdef DEBUGMODE
Serial.println("So client inputstring is not exactly #. Going on concatenating incoming bytes until new-line-command");
#endif
while(!clientInputString.endsWith("\n")){
clientInputString.concat((char)client.read());
}
Serial.print("Client: " + clientInputString);
clientInputString="";
}
///////////////////////////////
//Teilt lediglich mit, dass der Client nicht mehr verbunden ist.
}
if(!client.connected()){
Serial.println("System: Client disconnected.");
Serial.flush();
}
}}}
String serialInputString = "";
String clientInputString = "";
String fileName = "";
String processing is far slower than string processing AND can fragment memory. Your program will be faster if you do NOT use Strings.
void loop() {
delay(400);
Now, I REALLY have to wonder if speed is all that important...
I gave up reading your code. It is just riddled with useless delay(), so I am forced to conclude that speed isn't really that important.
The delays are not placed inside the loops where incoming or outgoing data is read/written. The delays are outside where no intensive communication process take place.
Here are the loops where data is read/written:
Data receiving part (don't get confused by commands which observe the progress) Basically it's all about that one line with file.wirte(client.read()). The delay(1) will only happen if reading has been performed faster than the incoming of the data.
for(transferredBytes = 0; transferredBytes < fileSize; transferredBytes++){
while(!client.available()){delay(1);}
file.write(client.read());
if(( (transferredBytes%(fileSize/100)) == 0) && fileSize>100){
Serial.print((unsigned int)(transferredBytes/(fileSize/100)));
Serial.print("% ");
if((transferredBytes/(fileSize/100))%20 == 0 && (transferredBytes/(fileSize/100)) != 0){Serial.print("\n");}
}
}
And here is the sending part. It's basically exactly the same.
for(transferredBytes = 0; transferredBytes < file.size(); transferredBytes++){
client.write(file.read());
if(( (transferredBytes%(fileSize/100)) == 0) && fileSize>100){
Serial.print((unsigned int)(transferredBytes/(fileSize/100)));
Serial.print("% ");
if((transferredBytes/(fileSize/100))%20 == 0 && (transferredBytes/(fileSize/100)) != 0){ Serial.print("\n"); }
}
}
I can comment out the part with the progress calculation and serial output and the transfer rate is the same, namely exactly 1.2 KB/s.
However, I will now implement the function with the buffer as Tim suggested and will post news. Stay tuned!
I have 2 questions so far.
Why does the array have to have 65 and not 64 fields?
What will happen, if there are not 64 bytes ready to be read from the client, i.e. because incoming data rate is too slow or there are some lags or simply because the EOF has reached after 30 bytes or so? Will those funcions block or will myFile.read() leave the rest of the fields empty and client.write() stop writing after reaching the last written byte by myFile.read() ? Or will some empy bytes (0's) be sent?
SurferTim:
Yes. i get about a 4x increase sending 64 bytes at a time rather than 1.char tBuf[65];
while(myFile.available()) {
clientCount = myFile.read(tBuf,64);
client.write((byte*)tBuf,clientCount);
}
EDIT: Sorry I didn't look at the details. I just realized that clientCount will make sure that only those bytes are sent which have been read. So this should work perfectly!
BUT how would I write this part when it comes to data receiving?
char tBuf[65];
while(client.available()) {
clientCount = client.read(tBuf,64);
file.write((byte*)tBuf,clientCount);
}
I think a code like this would not work here, since the loop would be left it reading was faster than receiving. I guess I will have to make it somehow like this, right?
while(file.size() < fileSize){...}
where fileSize is the expected file size (which is known because it is transmitted in advance).