Network Issues:

• [2008-08-27 13:26 ]
  On Wednesday, August 27th, 2008, at approximately 5AM, ISWest experienced a cascading event that disrupted services until 11:30AM. At 5AM, an entire AT&T fiber shelf failed at our Equinix El Segundo facility and had to be rebuilt by the fiber engineers. The loss of both our primary and secondary paths to our equipment exposed an operating system problem in our failover locations. When the failover locations were placed under a higher than usual load they experienced intermittent significant packet loss and latency. This operating system problem did not manifest itself in our manufacturer recommended upgrade 4 hours prior to the event under normal load. The intermittent packet loss made it extremely difficult to identify the root cause of the impact. After several attempts to route around the issues we were forced to revert to the previous operating system on all of our core routers. The coordination and execution of the rollback was very involved and took several hours. We have collected the information and log files required to escalate this issue to the manufacturer and we should have a definitive answer in the next few weeks as to what happened and why, so that we can avoid it in the future. However, we have no plans to make any changes to the core network at this time.
  
  
• [2008-08-27 10:10 ]
  ISWest is currently experiencing moderate packet loss to some providers on our edge routers. Our network Engineers are working on the issue and hope to have an ETR at 11AM.
  
  [2008-08-26 9:28 ]
  At 8:04AM Tuesday, August 26, ISWest experienced a physical loss of connectivity on one of our core internet connections for 22 minutes. The temporary changes we initiated to address the problems encountered on August 14th prevented the automatic failover of our redundant network routing. We have implemented an interim fix to re-establish connectivity. The scheduled maintenance for this friday at 11:30PM was designed to rectify all of these outstanding issues. We feel we need to accelerate the maintenance schedule and will be executing an emergency maintenance window tonight at 11:30PM. If you have any questions or concerns, please contact our support team at (818)735-3000.

 System Status:

• Internet Connectivity: Up
• DNS: Up
• Mail: Up
• Web: Up

 Planned Maintenance:

• [ 2008-04-07 08:18:03 ]
  No planned maintenance at this time.

 System Upgrades:

• [ 2008-03-04 08:01:49 ]
  No planned upgrades at this time.