Network Maintenance: February 26, 2009

Posted by Blue Box Group on Thu Feb 26 02:57:00 UTC 2009


We will be continuing work on the network core to address issues discovered as a result of the February 12th outage. This network maintenance will start at 11:00pm on Thursday, February 26 and may continue until 3:00am. While no outage is expected as a result of this work, an increased risk of outage exists for the duration of the maintenance.

During this round of maintenance we will be performing the following:

  • Our other distribution switch will be upgraded (note that it is already acting as ‘secondary’ as a result of the network maintenance on Feb. 25)
  • Access-level switches will have the other side of their redundant uplinks moved to the new hardware.
  • Our two OpenBSD firewalls will be removed from the network core, and the border firewall functions will be moved over to our new core routing gear.

Update Feb 26, 2:57am PST (10:57 UTC): We’re done performing network maintenance for the night. All of the above goals were accomplished. Some customers (notably VPS customers on our shared VPS infrastructure and shared hosting customers) may have noticed around 30 seconds of down-time as gateway addresses were moved from the OpenBSD firewalls to the core routing gear. At this point we’ve addressed the lion’s share of the concerns brought up by February 12th’s network outage. We will be addressing the last of these in coming weeks, however none of this affects the core routing infrastrusture or network topology, so network maintenance events may not be posted for this.

If any customers are experiencing any problems on the network as a result of tonight’s work, we urge you to open urgent support tickets so we can address them ASAP.