Library cannot be found after upgrading to lateset IDE 1.8.6

Hi all :slight_smile:

I am using the following libraries :

#include <utility\w5100.h>
#include <utility\socket.h>

However, after upgrading to the latest version of IDE, the compiler can no longer find the library files

I have even tried to modify the include statement with the whole path like this :

#include <C:\Program Files (x86)\Arduino\libraries\Ethernet\src\utility\w5100.h>
#include <C:\Program Files (x86)\Arduino\libraries\Ethernet\src\utility\socket.h>

but the compiler still complains that the files are not there!

Needless to say that I have actually checked the above folder and the files are present.

I am sure this is something dead simple... but I cant figure out whats wrong :frowning:
Any ideas?

Watcher:
the compiler can no longer find the library files

Files or just file? On my system the only one of those two it can't find is socket.h.

Watcher:
Needless to say that I have actually checked the above folder and the files are present.

Are you quite sure of that? Arduino IDE 1.8.6 uses Ethernet library 2.0.0. That version does not have a utility/socket.h so the error you're getting is quite expected.

Do you really need socket.h? Although the last Ethernet library release incremented the major version, it was not intended to have any breaking changes. The major version increment was decided on because to reflect the very significant (but non-breaking) changes made. Only the header files directly under the src folder are considered part of the public interface so it's always a bit risky to use files in a subfolder.

@pert:

You are right. socket.h is no longer there!

I do need it because i am using the following to detect and close unused sockets after a telnet connection:

void CheckSockStatus() {
  
  TimeForSocketStatus = millis();
  
   for (int i = 0; i < MAX_SOCK_NUM; i++) {
  
    uint8_t s = W5100.readSnSR(i);
    socketStat[i] = s;
   
    uint8_t dip[4];
    W5100.readSnDIPR(i, dip);
   
      if(s == 0x1C) {
       close(i);
       LogSDCard("Socket forced closed",0,20,2);
       closeConnection();
       PORT.print(F("Socket#"));
       PORT.print(i);
       PORT.print(F(" Closed"));
       
     }
     
   }
}

oh.... and socket.cpp is still there but socket.h is gone!

Short & easy answer: Use the library manager to downgrade Ethernet back to version 1.1.2.

Can you tell me which library you're using with these includes?

Removal of socket.h from Ethernet 2.0.0 was my doing. Looks like I'm now the de-factor maintainer of the Arduino Ethernet lib (it's a thankless job that doesn't pay anything, btw, but someone had to do it). Long term, having "socket.h" in a widely used Arduino library is going to be a liability for conflict with the standard C library, since POSIX standardizes this header and all PC-class standard C libraries implement it. We've already seen these sorts of conflicts with the Time library (which I also maintain) as the avr libc library has "improved" over the years, so with 2.0.0 I wanted to put us on a long-term path where Ethernet won't also suffer these problems.

Unlike other attempts to update Ethernet (one of which is still lingering on a branch as version 1.2) which renamed the w5100 class and header, I did try to keep as much backwards compatibility with sketches and libraries that include w5100.h and fiddle directly with the Wiznet registers. Whether that was a good idea might be debatable....

So the longer answer, which isn't really an answer at all but rather a question, is which libraries out there are trying to access Ethernet at this intermediate socket layer? I probably won't do anything more on Ethernet until mid-2019... but when I do work on the code again, would be nice to know more about these backwards compatibility issues.

Can you tell me which library you're using with these includes?

These are all the libraries I am using in this project :

#include <SD.h>   
#include <SPI.h>    
#include <EEPROM.h>
#include <avr/pgmspace.h>
#include <Wire.h>
#include <Ethernet.h>
#include <EthernetUdp.h> 
#include <avr/wdt.h>                                  
#include <OneWire.h>
#include <DallasTemperature.h>
#include <utility/w5100.h>
#include <utility/socket.h>
#include <TimeLib.h>
#include <pins_arduino.h>

Could I just manually add the files in the above folders or create a new library folder?

Regarding this code, perhaps with Ethernet 2.0.0 is may not really be needed anymore? I rewrote the socketBegin() function to reclaim effectively-closed sockets. Whether that fully solves the problem this old code was meant to address is a good question. If it doesn't, I'd really like to know (for Ethernet 2.1.x sometime next year).

If you really do need to get close(socknum), probably the simplest way would be to create an EthernetClient instance (as a local variable is fine) using the constructor that lets you force it to a particular socket number. Then call its stop() function.

Regarding this code, perhaps with Ethernet 2.0.0 is may not really be needed anymore? I rewrote the socketBegin() function to reclaim effectively-closed sockets. Whether that fully solves the problem this old code was meant to address is a good question. If it doesn't, I'd really like to know (for Ethernet 2.1.x sometime next year).

This code was meant to reclaim sockets left open due, to for example, a lost connection resulting from the remote party or otherwise. I am using all 4 available sockets ( telnet,ftp, web) so it was important to periodically check and close sockets that were left open.

Are you now saying that the latest version of ethernet.h does this on its own?

Looked just now, and sadly no. It does reclaim sockets stuck in LAST_ACK, TIME_WAIT, FIN_WAIT, & CLOSING. I did put in code for CLOSE_WAIT, but it's currently commented out. Now I honestly don't remember why I took that out before releasing 2.0.0.... maybe it was just overly cautious on my part?

Ok, thanks for looking!

So what would be your recommendation as to get around this?
Shall I downgrade to the previous Ethernet version (1.1.2 i think) ?
or would another approach be better?

On the other hand, surely lots of other people would be keen to reclaim unused sockets so I would expect this issue to come up again...

The tough thing about the CLOSE_WAIT state, from the perspective of designing the Ethernet library, is how to handle the case where some client object is still using the half-open connection.

In your program, you know (or should know) whether you're still making use of that socket. Just because the other side send a FIN packet to close the connection doesn't necessarily mean your code has managed to read the data the remote host sent before the FIN.

In fact, one of the many bugs I fixed in 2.0.0 is the case of trying to open a connection to a server which rapidly sends a message then closes the connection. The old open code didn't properly check for CLOSE_WAIT, so if the remote host was able to send the entire reply and close the connection, prior to 2.0.0 you couldn't even get open to succeed.

Perhaps in 2.1.x I'll add a bitmask that's updated by the client constructors and destructors, so we know if anything is actually using the socket....

As far as what to do, it's hard to make a really specific recommendation when I can only see small fragments of your code. But as a general suggestion, the lingering CLOSE_WAIT situation really should not occur if you call the client stop() function when you are done using the connection.

But as a general suggestion, the lingering CLOSE_WAIT situation really should not occur if you call the client stop() function when you are done using the connection.

True! but what if the other party has lost the connection.
example :
I telnet arduino from a tablet over wifi (Arduino is on wired ethernet) and the tablet's wifi connection is lost.
The arduino has no way of knowing this and the socket remains open.

PS: Another issue i was facing and unable to fix was the inability to maintain a connection in some cases. In other words the connection would open and quickly close. I had to either keep trying or reset the arduino.

Maybe we're talking about 2 different things here?

Lingering connections are a generic problem of TCP networking on all systems, not just Arduino. If neither side transmits data, the connection remains open even if there is no longer connectivity. Lost connection is only detected when packet transmission fails. There is a feature called "keepalive", which may or may not be supported by these chips. It's also questionable how well these Wiznet chips handle every case and manage to close the connection....

CLOSE_WAIT, as I understand it, is a completely different scenario. When the remote host closes the connection, it sends a FIN packet. If you haven't closed, the connection is said to be "half open". On your end, if you haven't received everything yet, you can still read whatever data arrived before the FIN packet.

Maybe I should dive deeper into ehternet...

My objective was to close sockets that remained open when a connection was unexpectedly lost and obviously before a FIN packet was sent/received.
Before the function above was included, there were cases when a new connection could not be established because no more sockets were available even though none was actually used anymore.

I know your goal is to get your application working.

But if you find a case where the Ethernet library really isn't handling something else, perhaps put a little extra time into creating a test program and procedure to recreate the problem.

I'll probably work on Ethernet again in mid-2019. The main goal will be non-blocking APIs which so many people have requested. I also want to fix these sorts of problems, but the reality is there's usually not much I can do until I find a way to recreate the problem....

Ok... thank you for your input, time and effort :slight_smile:

I will give a try and maybe get back with some questions / comments!

One more question:

Is there a way to report the library version number available at compile time so that different code oprions can be #ifdef-ed in and out?

Thanks