I do not need SD card reader or any other device. Only Ethernet.
Is it neccesarry to use H1012NL transformer (or similar) and connect RJ45 socket to W5100 through this transformer? Is it possible to buy RJ45 socket with integrated protection? Or how it works on original shield?
I can't use original shield because is to big and I need small and manufacturable board. If it is important I can put 16MHz external crystal to ATmega.
I found a lot of minimal schematics for ENC28J60 but not for W5100
Well, that's pretty much how it's done. Compare that to what's in the W5100 datasheet. I've also used the WIZ811MJ and WIZ820io (W5200) modules. Unless we're talking considerable volume, those modules may represent a better choice. If you are really bent on rolling your own, then it sounds like you will have a learning curve.
This is all deja-vu to me. I have the Ethernet shield, I have a few of the WIZ modules, and I'm sourcing parts to begin a home-brewed W5100 design. Yes, you'll need integrated magnetics. One of the Pulse MagJack connectors will work. Get one for 10/100 with auto-MDIX support. (This ensures the xformer etc. are all symmetric so the RX pins work as TX and vice versa.)