I ran into an interesting problem at a client site today that I would like to share with others for your edification:

A client had an Astaro 3X box's interface going through a switch to a gateway; we could ping and transfer packets to the gateway fine. The client went to cut in a 4X box: pinging was no longer happening. Our first impression was that it was something being done 'wrong' on the 4X box.

arp's showed as "incomplete" on the interface. We then cut in a crossover cable: bang, pings and packet transfers worked. We replaced the switch with another brand of switch, and then that too worked.

So to recap, a switch that worked with Astaro 3X wouldn't work with Astaro 4X (!).

So it was either due to changes in the network stack of the different Linux kernels and how arps are composed, or changes in the drivers between the OS's (Ethernet packet fragmentation, framing, padding, etc.). The newer format arps could not be handled by the mediocre brand switch.

Something to look out for when you're upgrading: use a crossover cable in your testing if you hit the wall...

Where is the mystery? Flush the ARP cache on the switch (some seem to have incredible high default timeout values) - that's it.

Greetings
cyclops

But what if I told you the switch was repowered?
AND the arps were cleared from the interfaces??

Upon repinging after all that, no go; new arps showing again as "incomplete".

Put back the 3X box: works...

One other possibility: maybe the switch couldn't handle the value of the MAC address of the new box's NIC?
Unlikely, but listing EVERY possibility...

P.S. All cables checked...

P.P.S. Different ports on the switch were used; the behavior was consistent across multiple ports...

Just out of curoisty what type (brand) switch was it ?

Dlink...

(you asked; I didn't volunteer it...)