Let's begin with an example of typical switched network with multilayer distribution switches.
Assume that all the links between switches are trunk links, that both DL1 and DL2 are participating in HSRP groups where DL1 active for VLAN1 and VLAN3 and DL2 for VLAN2. STP is configured proportionally to block the rightmost links at access layer switches for even VLAN's and the leftmost links for odd VLAN's.
Now let's pretend that host C1 pings host C2.
Let's follow the path of the packet on its way to C2:
1. Since C1 resides in the different VLAN than host C2, then it should send the packets to its default gateway (DL1's VLAN1 interace). By means of ARP request C1 discovers MAC address of VLAN1 interface and sends the first ICMP packet.
2. DL1 routes the packet. VLAN2 is a directly connected network, so DL1 discovers with ARP MAC address of C2 and sends the packet to it (Packet traverse the link between DL1 and DL2 and then to AL2 since the leftmost link is blocked by STP for VLAN2 on AL2).
Important note: to this moment internal switch on DL1 have the following MAC-addresses: C1 in VLAN1 and C2 in VLAN2.
ICMP-reply from C2 follows the same path except that its default gateway will be DL2's VLAN2 interface. This means that in this network the routing between VLAN's is asymmetric.
What happens if the pings are coming for example for 10 minutes?
DL1 ages MAC-address entry for C2 in VLAN2 since it sees no new frames from it (remember, that reverse packets are first routed at DL2 and then sent from its VLAN1 interface?) and begins to flood the frames for C2 out of all its ports in VLAN2. Mirrored result is seen on DL2 for reverse packets.
The problem relies to the fact that router holds the arp entries for 14,400 seconds or 4 hours, but the switch ages CAM table in 5 minutes.
For this not to happen the timers on ARP and CAM table should be in SYNC!
The easiest way is to increase MAC aging time to 14,400 seconds.
For Catalyst 6500 it can be done with the following command:
mac-address-table aging-time seconds [vlan vlan_id]
Happy switching!