Books I read and do recommend to all security enthusiast:

• Internetworking with TCP/IP (vol. 1-3) by Douglas Comer
• Modern Operating System by Tanenbaum
• Hacking Exposed by Joel Scambray, Stuart McClure, and George Kurtz
• Hacking Exposed Web Application by Joel Scambray, Mike Shema, and Caleb Sima
• Linux Hacking Exposed by Brian Hatch, James B. Lee, and George Kurtz
• Hacking: The art of exploitation by Jon Erickson
• Web Hacking: Attacks & Defense by Stuart McClure, Saumil Shah, and Shreeraj Shah
• Hacknotes(tm) Linux and Unix security portable reference by Nitesh Dhanjani
• The Shellcoder's Handbook: Discovering and Exploiting Security Holes by Jack Koziol, David Litchfield, Dave Aitel, Neel Mehta
• Exploiting software: How to break code by Greg Hoglund and Gary McGraw
• Reversing: secrets of reverse engineering by Eldad Eilam
• Hacker Disassembling Uncovered by Kris Kaspersky
  

Note:
A very good friend of mine and a mentor of my early career told me once, "Learn and understand how networking is done, if you really want to get into hackers footstep".