After like 4 passes at the configuration of our network (switches, firewall’s, load balancers, etc) Danny found an abonormality that we wanted to correct and see what happened. On our core switch side we had the port where our Pix went set to full duplex 100meg, but on our Pix (configured years earlier than our core) it was set to auto. Turns out the Pix does not auto sense the 100meg full, but does not error out in the situation. You don’t even get lost packets, but you do get some collisions. Well some is an understatement.
Later in the day we set the port on the pix to be full duplex 100meg and within a few hours our metrics back to normal. This little change took us weeks to find. This is not the first time I have been burned by a port mismatch. Knowing that we even took steps to prevent this, or we thought we did.
It is frustrating to find such a little issue that does not show with errors causes so much problems..