syst (192.168.1.122) at 21:96:43:3a:62:a2 [ether] on eth0
arp -d 192.168.1.122
If you get an error do:
arp -d 192.168.1.122 -i eth0
Sometimes the arp cache just won't clear either, some say it should automatically but there are cases where the above doesn't work. I believe it may be that some network applications are keeping the entry alive but this is just a guess.