Reacting To Network Timing Alarms – Do’s and Don’ts

Alarms Are Inevitable – Here’s How To Respond

Guidance Regarding Network Timing Maintenance

 

Alarm Do’s & Don’ts

I have personally been involved with the implementation and maintenance of Network Synchronization and its associated systems for over 25 years. It is interesting, timing and power are two of the most mission critical elements in any communication network, yet neither have received the appropriate level of attention they both deserve. Some years ago, an internal study was conducted to determine the most common causes for FCC reportable events. As you may have guessed, both power and timing were at the top of the list. The point of this post is to provide the reader with some guidelines when working on timing networks alarms.

Rob Jodrie

Rob Jodrie

Solutions Architect, Syncworks

Rob started working in the Telecommunications Industry with the Bell System in 1982. He had responsibility for Tier II Network Synchronization and Transport Technical Support at Verizon for fourteen years and has been working at Syncworks since 2015.

When properly designed, timing networks are resilient and robust. When any activity is planned, step one is what I call the “information collection stage.” It is important to understand the current state of the timing system prior to undertaking any activities that may negatively impact supported services. Steps associated with the information collection process include retrieving office records that indicate fuse panel and BDFB power assignments. Determining how the Building Integrated Timing Supply (BITS) clock is referenced and retrieving its current alarm status can be extremely useful. Cabling connected to a BITS clock should have cable tags indicating the location of the far end equipment to which it’s providing timing.

The key point is this: knowledge is power. The more you know about the timing system that you are about to work on, the better. Having established a well thought out plan may be the one thing that keeps you from making a tough situation worse. An understanding of the network’s current configuration and alarm status will allow you to develop the appropriate procedure. All of the aforementioned information can be obtained without impacting the system’s operation.  

SyncCare offers peace of mind for network alarms and more.

Takeaways

  • DO “Keep your hands in your pockets.” 
  • DON’T randomly unplug and swap out cards.That may cause the system to fail. When I was first trained in the operation and maintenance of a BITS clock, keeping my hands in my pocket was the advice I received from a very wise Engineer. He explained that the system is designed for redundancy. Even though the system may be reporting alarms, there may be NO impact to service due to its resilient design.
     
  • DO “Trust But Verify”
  • DON’T EVER connect an input reference to a BITS clock without first confirming its viability.When operating as designed, ALL BITS clocks should be traceable to a Stratum 1 reference. GPS Receivers and Cesium Atomic Clocks meet this criteria. In some cases, traffic bearing DS1 signals or OC-N Derived DS1 signals from SONET nodes may also be configured to perform at a Stratum 1 level. The quote “trust but verify” comes to mind. Just because you have a DS1 signal properly framed and error free, does NOT mean it meets the stringent requirements to reference a BITS clock.

  • DO have a plan – always!
  • DON’T go it alone.If you are on-site and notice alarms in a BITS clock, it is good practice to notify the NOC and make them aware of the current alarms and conditions. It is helpful as well to understand the scope of a timing trouble. The issue may simply be a failed plug-in that has switched to protection and requires replacement causing no impact to service. It may also be a situation where multiple or all network elements in the office are slipping, i.e., experiencing degraded input references. The latter is obviously a more critical situation, a second set of eyes may help you to accurately assess the potential impact to service.

Summary Takeaways

To ensure the synchronization network performs optimally, the best thing we can do is respond to alarms in a timely manner. When a BITS clock loses its input references and goes into holdover, most oscillators are of a very high quality and can continue providing accurate timing to network elements. The problems arise when this condition is left unattended for an extended period of time. Make it a habit to walk by the BITS clock when you’re in an office. Report any anomalies to the NOC and develop a plan to safely restore the system to an alarm free state.