Panthernet

What happened?

BFS lost all network services- Verizon (phones and inter-building connectivity) and Earthlink (internet service) on Tuesday afternoon August 5th.

The initial CAUSE of the outage is unknown.

Tuesday we called all vendors to inform them of the outage.

Wednesday we called all vendors to check the status of our complaint.

Thursday there were four Verizon technicians and one external vendor all working on the issue on-site on Thursday, August 7th.

Verizon discovered that our uplink at Willougby had been physically disconnected, that the digital card that activates the connection between the buildings had been deactivated AND the digital card that powers our lines at Pearl Street had been deactivated.

We discovered that a deactivation notice had been entered into Verizon's system for our Willoughby building but the origin of this is still under investigation. A deactivation order can only be generated by the customer or Verizon. We had made no such request. At this time Verizon denies generating such an order. We have a communications consultant who is investigating this issue.

Phone service and inter-building connectivity was restored between 3:00-4:00 Thursday, August 7th.

Phones were now working but there was still no connection to the internet.

The Verizon technician on-site told us that there was no connectivity coming into the Pearl Street building and that the root of our issue was at the central office.

Monday August 11th, our Head of School, our Director of Finance and our Director of Technology all called Earthlink to complain about the length of outage.

After much red-tape our case was escalated and we were able to 1)work directly with a senior account manager to help troubleshoot the internet outage in both buildings and 2)confirm that a technician had been re-dispatched to our location.

A Verizon technician showed up (even though we purchase DSL service from Earthlink Verizon owns all the lines in the ground that provide connectivity throughout the five boroughs) at Pearl Street this afternoon who was able to determine that the root of our outage at Pearl Street was due to a corroded cable at 383 Pearl Street (our neighbor) where our service originates from the Verizon primary cabling grid.

Internet service was restored to the Pearl Street building just before 4:00pm Monday, August 11th.

After troubleshooting the Willoughby outage further with our Network Manager and our external network vendor, Promenet, we hope to have solved the outage at Willoughby Street as well.

I cannot confirm this until I am able to verify the connection from inside Willoughby tomorrow.

We hope this information keeps you well informed about both the complexities of our network management and the cause of this six-day connectivity outage.

In friendship,

Kerri