The Scenario
You have dual NICs and you disable NIC1 which uses 192.168.1.1 as its gateway. With NIC2 you enable it/connect it to another network which also has the gateway 192.168.1.1
Everything will work fine at this point.
When switching back to NIC1 even with NIC2 disabled and even unplugged, the OS basically can't pick up the new/updated ARP entry of the old device for 192.168.1.1 and perhaps thinks it is a security risk or spoof of some sorts and blocks i........
Here is the scenario you or a client have a remote machine that was installed as a standard/default minimal Centos 6.x machine on a single disk with LVM for whatever reason. Often many people do not know how to install it to a RAID array so it is common to have this problem and why reinstall if you don't need to? In some cases on a remote system you can't easily reinstall without physical or KVM access.
So in this case you add a second physical or disk or already ha........
In my case I could login with the initial install but I rsync'd everything over while preserving ownership and permissions to another RAID partition and booted from that.was fine before. The problem is that you are kicked out the second you login and the problem was SELINUX for some reason (perhaps it noticed something strange when it was moved to the new partition)
login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
login: ROOT LOG........