That was my first idea of how it should work. But I could not figure out how to send commands from my Linux box to the Arduino when the Arduino is a client. My understanding is that a client only asks from a server. When the server wants to push something to the client (in this case telling the Arduino to change an output port for example) it requires software on the client side that is not doable on an Arduino. So the Arduino also needs to be a server so it can listen to the Linux box.
Or at least that's how I understand it, but I'm not exactly an expert on these matters...