Partial Network Outage

Posted by Blue Box Group on Tue Jun 30 18:10:00 UTC 2009

From 5:00pm to 5:20pm (Wed Jul 01 00:00:00 UTC 2009 to Wed Jul 01 00:20:00 UTC 2009), three distribution switches on our network experienced a VLAN issue with caused traffic to be incorrectly switched to hosts they serve for a maximum period of about 20 minutes. Our network technicians immediately identified the problem and began implementing the fix at 5:05pm. Network access to all hosts was restored by 5:20pm. A number of hosts saw a much briefer outage.

This event was isolated to about 50 machines on our network. We’ve taken preventative measures to ensure that any switches on our network are configured in such a way that this can not happen again, and are writing automated checks to ensure that any new switches brought onto our network are also configured in this protected manner.

THE LATEST

THE ARCHIVES

THE FEED