There's an interesting article in The Register outlining what we've touched on here at ITBE a few times in the past-that PCI Express has the chops to form a robust server fabric.
A company called VirtenSys has come up with a plan to offer virtual I/O based on PCIe that would greatly simplify network infrastructures, largely by cutting down on the need for network adapters.
Essentially, the company is extending the PCIe bus outside the server directly to a network switch outfitted with virtual adapters for other servers on the network. If you have 10 servers hooked up to the switch, they would share only one physical NIC and HBA-cutting both capital and operation costs.
You would still need Ethernet or IB to link to storage and broader data center networking, but at least you can pare down your server infrastructure. And these days, every little bit helps.