Our fiber carrier, Zayo, identified two issues on the short-side of our fiber ring. The first identified issue was a bad fiber at their fiber hut nearest the Atlanta DC3 data center. Upon identification they remediated the bad fiber. The second identified issue was as a dirty fiber jack at the Atlanta DC3 meet-me room (MMR) panel. Having identified the issue, Zayo took action to clean the jack and reset the fiber cable connection. In sum, these remediation actions have successfully cleaned up the experienced signal loss thereby bringing the short-side of our fiber ring into specification again. Monitoring shows the short-side of the fiber ring has stabilized and we are closing this incident with no related issues reported in the past 12+ hours.
Posted Oct 17, 2018 - 11:25 UTC
Update
Our fiber carrier has isolated the issue between their fiber facility at the street and our Altanta DC3 data center. We are continuing to work directly with Zayo and are going to leverage an optical time-domain reflectometer (ORDR) to determine where the issue lies on the short-side of our fiber ring. Additionally, the long-side of our fiber ring is up and operational. We will continue to monitor the situation closely and will make further announcements as necessary.
Posted Oct 16, 2018 - 14:39 UTC
Investigating
As of 9:36 PM ET on 10-15-2018 our upstream carrier, Zayo, began experiencing a dark fiber outage of unknown origin impacting a small portion of our Atlanta-based customers. This outage was isolated to one side of our redundant fiber ring connecting our Atlanta DC3 and DC4 data centers. Due to this redundancy, Zayo was able to restore service at 10:19 PM ET on 10-15-2018. As Zayo continued to explore the origin of the fiber ring failure they experienced a complete failure of this fiber, at approximately 12:54 AM ET on 10-16-2018, causing both sides of the ring to collapse. This collapse has resulted in a larger portion of our Altanta DC3-based customers to experience a network outage. We continue to be in direct communication with Zayo and are actively working with them to resolve the incident.