After spending most of the day extensively testing the new resilient firewall setup with no problems, we went to move in into production just after 5pm thinking it would take 45 minutes or so. However thanks to some bizzare compatibilty problem between our HP Procurve 4000 switch, and the shonky BT supplied Cisco routers, it turned in to a packet loss nightmare. We ended up having to bypass the Procurve switch which has the unwelcome side effect of breaking the high availability cluster unless you put up with around 50% packet loss.
Tomorrow I need to shout at BT a lot, faff around with various duplexing settings, and try to find another way to connect to rooms at opposite ends of the site without using fibre as we seem to have run out of media converters.
I need beer.