Reacting To Network Timing Alarms – Do’s and Don’ts

Reacting To Network Timing Alarms – Do’s and Don’ts

Alarms Are Inevitable – Here’s How To Respond

Guidance Regarding Network Timing Maintenance

 

Alarm Do’s & Don’ts

I have personally been involved with the implementation and maintenance of Network Synchronization and its associated systems for over 25 years. It is interesting, timing and power are two of the most mission critical elements in any communication network, yet neither have received the appropriate level of attention they both deserve. Some years ago, an internal study was conducted to determine the most common causes for FCC reportable events. As you may have guessed, both power and timing were at the top of the list. The point of this post is to provide the reader with some guidelines when working on timing networks alarms.

Rob Jodrie

Rob Jodrie

Solutions Architect, Syncworks

Rob started working in the Telecommunications Industry with the Bell System in 1982. He had responsibility for Tier II Network Synchronization and Transport Technical Support at Verizon for fourteen years and has been working at Syncworks since 2015.

When properly designed, timing networks are resilient and robust. When any activity is planned, step one is what I call the “information collection stage.” It is important to understand the current state of the timing system prior to undertaking any activities that may negatively impact supported services. Steps associated with the information collection process include retrieving office records that indicate fuse panel and BDFB power assignments. Determining how the Building Integrated Timing Supply (BITS) clock is referenced and retrieving its current alarm status can be extremely useful. Cabling connected to a BITS clock should have cable tags indicating the location of the far end equipment to which it’s providing timing.

The key point is this: knowledge is power. The more you know about the timing system that you are about to work on, the better. Having established a well thought out plan may be the one thing that keeps you from making a tough situation worse. An understanding of the network’s current configuration and alarm status will allow you to develop the appropriate procedure. All of the aforementioned information can be obtained without impacting the system’s operation.  

SyncCare offers peace of mind for network alarms and more.

Takeaways

  • DO “Keep your hands in your pockets.” 
  • DON’T randomly unplug and swap out cards.That may cause the system to fail. When I was first trained in the operation and maintenance of a BITS clock, keeping my hands in my pocket was the advice I received from a very wise Engineer. He explained that the system is designed for redundancy. Even though the system may be reporting alarms, there may be NO impact to service due to its resilient design.
     
  • DO “Trust But Verify”
  • DON’T EVER connect an input reference to a BITS clock without first confirming its viability.When operating as designed, ALL BITS clocks should be traceable to a Stratum 1 reference. GPS Receivers and Cesium Atomic Clocks meet this criteria. In some cases, traffic bearing DS1 signals or OC-N Derived DS1 signals from SONET nodes may also be configured to perform at a Stratum 1 level. The quote “trust but verify” comes to mind. Just because you have a DS1 signal properly framed and error free, does NOT mean it meets the stringent requirements to reference a BITS clock.

  • DO have a plan – always!
  • DON’T go it alone.If you are on-site and notice alarms in a BITS clock, it is good practice to notify the NOC and make them aware of the current alarms and conditions. It is helpful as well to understand the scope of a timing trouble. The issue may simply be a failed plug-in that has switched to protection and requires replacement causing no impact to service. It may also be a situation where multiple or all network elements in the office are slipping, i.e., experiencing degraded input references. The latter is obviously a more critical situation, a second set of eyes may help you to accurately assess the potential impact to service.

Summary Takeaways

To ensure the synchronization network performs optimally, the best thing we can do is respond to alarms in a timely manner. When a BITS clock loses its input references and goes into holdover, most oscillators are of a very high quality and can continue providing accurate timing to network elements. The problems arise when this condition is left unattended for an extended period of time. Make it a habit to walk by the BITS clock when you’re in an office. Report any anomalies to the NOC and develop a plan to safely restore the system to an alarm free state.  

 

Syncworks Rob Jodrie Speaker at UTC 2024 Annual Conference

Syncworks Rob Jodrie Speaker at UTC 2024 Annual Conference

UTC Annual Conference to Feature Syncworks, Southern Company and Burns & McDonnell

These three leading critical infrastructure companies will replay their March UTC presentation “A Strategic Evolution to Packet-Based Timing: Leveraging Precision Time Protocol Across Your Network.” In the end, it was a smart move by Southern Company to invest in ePRTC and vPRTC as it ensured their security for years to come.

Rob Jodrie of Syncworks will be speaking on at Utilities Technology Council™ Annual Convention on Thursday, May 23, 2024 at 9:10 AM – 10:10 AM.  He will be joining Justin Hardy of Southern Company and Matt Kitchen of Burns & McDonnell to discuss the benefits of transitioning to a PTP timing architecture. This presentation will kick-off the UTC show and we are proud of the work they have done to present it to that audience.

Southern Company faced the challenge of a SONET to packet network transition and together with Syncworks and Burns & McDonnell, designed and deployed a new MPLS system to provide system-wide transport for critical applications. This initiative introduced stringent new timing requirements and mandated a departure from the frequency BITS clocks traditionally used to synchronize TDM environments. To provide the necessary accuracy and traceability, Southern Company deployed Precision Time Protocol (IEEE 1588 PTP), Synchronous Ethernet (SyncE), and Network Time Protocol (NTP) synchronization sources to provide frequency and time of day for the MPLS, SouthernLinc LTE, and corporate IT networks.

While the enablement of MPLS was the catalyst for IEEE-1588 PTP, it became clear that a highly accurate and reliable packet timing source could be broadly utilized across the network. This UTC annual conference session will explore the various use cases and mutual benefits of packet-based timing, including next-gen transport, mitigation of GPS vulnerabilities and Grid timing backup from a secure telecom core. Learn how to fully leverage your timing investments, plan for emerging applications and navigate the various stakeholders that will come to rely on your clocks.

To learn more about PTP deployment and this specific case study, contact Rob Jodrie here.

Rob Jodrie

Rob Jodrie

Solutions Architect, Syncworks

Rob Jodrie started working in the Telecommunications Industry with the Bell System in 1982. He had responsibility for Tier II Network Synchronization and Transport Technical Support at Verizon for fourteen years and has been working at Syncworks since 2015.

Syncworks Background

For over twenty-five years, Syncworks has been evaluating,testing, designing, and implementing GPS signals and timing networks for telecom, cable, utility, and enterprise customers in the US and the Caribbean. We are a well-known and trusted partner and critical supplier to major network operators. 

As a skilled integrator of multiple vendor products, Syncworks can provide options for the most performant, resilient, and economical timing network possible.

Syncworks’ Solutions Architect Mr. Jodrie is uniquely qualified to speak on this topic as he crisscrosses the USA every week enabling NPT, PTP and more. Not only is Mr. Jodrie on the cutting edge of PTP and NTP timing for telecom networks, utilities, and cable providers, he can also look ahead with ePRTC, vPRTC, and reach back into his long history with BITS clocks.

Utilities Technology Council Background

A Utilities Technology Council meeting serves to advocate and provide resources for utilities and critical infrastructure providers, aiming to advance the adoption and use of technology in the industry. UTC focuses on addressing the unique technology and telecommunications challenges faced by utility companies, including electric, gas, and water utilities.

Key Points:
1. Mission: UTC’s mission is to create a favorable business, regulatory, and technological environment for companies in the utility sector to enhance their efficiency, reliability, and safety through the use of information and communication technologies.

2. Members: The organization’s members include utilities, technology suppliers, and other stakeholders like Syncworks involved in the energy and utilities sector. UTC provides a platform for collaboration and knowledge-sharing among its diverse membership.

3. Advocacy: UTC engages in advocacy efforts to influence policies and regulations that impact the deployment and integration of technology in utilities. This includes participating in regulatory proceedings and working with government agencies.

4. Educational Programs: UTC organizes conferences, workshops, and educational programs to facilitate the exchange of knowledge and best practices within the utilities industry. These events often cover topics such as smart grid technologies, cybersecurity, and emerging trends in utility communications.

5. Industry Collaboration: The organization promotes collaboration and partnerships among its members, fostering an environment where utilities can learn from each other’s experiences and work together on common challenges.

6. Focus Areas: UTC addresses a range of technology-related issues relevant to utilities, including grid modernization, cybersecurity, spectrum management, telecommunications infrastructure, and the integration of advanced technologies into utility operations.

Southern Company Background

The Southern Company is a leading energy company that operates in the electric utility sector. It serves millions of customers across the southeastern United States. The company is known for its focus on generating and distributing electricity in a reliable, safe, and environmentally responsible manner.

Key Points:
1. Utilities: The Southern Company operates several subsidiary electric utilities, including Alabama Power, Georgia Power, Gulf Power, and Mississippi Power. Each subsidiary serves customers in its respective state.

2. Energy Mix: The company utilizes a mix of energy sources, including nuclear, coal, natural gas, and renewable energy, to generate electricity. It has been investing in renewable energy projects as part of its commitment to sustainability.

3. Infrastructure: The Southern Company maintains a vast infrastructure of power plants, transmission lines, and distribution networks to ensure a stable and efficient supply of electricity to its customers.

4. Innovation: The company has been involved in initiatives related to research and innovation in the energy sector. This includes exploring advanced technologies and solutions to meet evolving energy needs.

5. Environmental Stewardship: The Southern Company places emphasis on environmental stewardship and has taken steps to reduce its carbon footprint. It has set goals for cleaner energy production and reducing emissions.

Burns & McDonnell Background

Burns & McDonnell is a multidisciplinary engineering and construction company that provides a wide range of services across various industries. The firm is known for its expertise in designing and implementing complex projects, including those in the energy, aviation, water, and industrial sectors.

Key Points:
1. Services: Burns & McDonnell offers a comprehensive suite of services, covering engineering, architecture, construction, environmental consulting, and various technical consulting services.

2. Industry Focus: The company has a diverse client base and works on projects in sectors such as power generation, aviation, water and environmental, oil and gas, telecommunications, and industrial facilities.

3. Employee Ownership: Burns & McDonnell is known for its unique corporate structure, where a significant portion of the company is owned by its employees. This employee ownership model is often highlighted as a distinctive feature of the firm.

4. Project Diversity: The firm is involved in a wide range of projects, from the design and construction of power plants and infrastructure to environmental consulting and renewable energy initiatives.

5. Innovation: Burns & McDonnell emphasizes innovation in its projects and operations. The company often explores cutting-edge technologies and sustainable practices in its engineering and construction work.

6. Recognition: The firm has received recognition for its work and has been included in various industry rankings for engineering and construction companies.

About Syncworks

Syncworks is a the national leader in GPS security. Critical infrastructure in the US is a top priority at the highest level of government. Our mission is to enable, educate, and support efforts to become complaint with celestial and terrestrial GPS systems working together.
 
Our flagship product, the TimeProvider® 4500, is a gateway clock that accepts multiple inputs from Global Navigation Satellite Systems (GNSS), Synchronous Ethernet (SynE), and IEEE 1588 PTP Grandmaster Clock and E1/T1 digital transmission links.  

As of January 1, 2024, we have expanded our Field Services to include Antenna Installation and Entrance Facility Cabling, Legacy Equipment Decom and Traffic Migration, Disposal (hazmat) Services, Radio Commissioning (MW, P-LTE, CBRS), Enterprise Wi-Fi.

For more information, contact sales@syncworks.com or call (904) 280-1234

Resilient Timing for 5G

Resilient Timing for 5G

The Rising Role of Precision Time Protocol (PTP)

A key component to resilient timing for 5G is the security of the GPS input. GPS vulnerability has long been a topic of discussion. While it has been widely acknowledged to be a security concern by the power utility, transportation, and communication industries, we have only recently begun taking action to address this issue. The object of this paper is to discuss the evolution of the timing network starting with the distribution of DS1 and Composite Clock signals for digital transport equipment and ending with the deployment of PTP packet timing in support of 5G wireless service.

Evolution of Telecommunications Timing 

Since its inception, the need for increased bandwidth and capacity in our telecommunications system has been a constant driver of technological advances. In the early 1940’s, AT&T developed the L-carrier system. As our population grew, so did the demand for increased capacity and the improved performance of our communications network. L-carrier systems were frequency division multiplexers. That is, a common transport medium that could be used to support multiple users operating simultaneously at different frequencies. It was an efficient way to increase the capacity of existing infrastructure. In the 1960s, Bell Laboratories developed T-carrier. Rather than frequency-based multiplexing, T-carrier is a form of time division multiplexing (TDM.) As with L-carrier, T-carrier also allows multiple users to share a common transport. Thus, increasing the capacity and efficiency of the transport network. TDM, however, assigns a timeslot rather than a frequency to each channel. To maintain the proper sequencing of the channels, end-end synchronization is required. More on this when we discuss their similarities in 5G’s use of FDD and TDD.  

 

The 5G Paradigm

Today, the same demands for increased bandwidth, capacity, and performance that caused the development of earlier multiplexing schemes are now driving innovation and technological advances in 5G wireless service. And with it resilient timing for 5G. Two spectrum usage techniques used in 5G are Frequency Division Duplex (FDD) and Time Division Duplex (TDD). There is an interesting analogy between the comparison of FDD to TDD and L-carrier to T-carrier. Like L-carrier, FDD transmits uplink and downlink information at different frequencies. Whereas with TDD, uplink and downlink transmissions are assigned different timeslots in the same spectrum frequencies, not unlike T-carrier. TDD frames include different time periods and timeslots that allows for a more efficient use of the spectrum. TDD requires uplink and downlink transmission signals to be precisely synchronized. Failure to do so will result in poor RF performance causing corrupted data and dropped calls. The evolution from frequency to time comes at a price as it requires greater than a tenfold improvement in network timing accuracy. This can be achieved by deploying multiple and diverse technologies such as GPS, SyncE, and IEEE 1588.  

 

Resilience in Timing 

Now that we have established the key role that highly accurate and precise timing plays in 5G networks, we can discuss ways to increase its resiliency. As previously mentioned, GPS vulnerability is a major concern for our nation’s power utility, transportation, and communication industries. Executive Order 13905 titled Strengthening National Resilience Through Responsible Use of Positioning, Navigation, and Timing Services states that our national and economic security is hugely dependent on our critical infrastructure. GNSS is widely used globally as a timing source. Concerns for regional or global GNSS outages have caused enterprises to pursue ways to protect their networks from reliability issues and potential security risks. Because of its low cost and simplicity in deployment, GNSS has been the primary solution for accurate time distribution. The downside to this solution is the operational expense incurred when GNSS failures occur. Dispatching personnel to locations to troubleshoot and repair equipment is costly from both a service and an economic perspective. GNSS is ubiquitous and therefore it represents a single point of failure that mandates a more secure approach to the distribution of accurate timing. Standards have been set describing how accurate time can be transported over a packet network. These standards define profiles for the operation of precision time protocol (PTP) and Synchronous Ethernet (SyncE). IEEE 1588-2008 is the standard for timekeeping. 1588 Packet Timing Protocol is a series of standards developed specifically for the transport of time over packet networks. The Primary Reference Time Clock (PRTC) is the new generation of time distribution system that is widely deployed by wireless service providers. Using PRTC systems, packet time distribution is a way to provide timing to a mobile access point as a back-up to locally provided GNSS time delivery.  

  

 

Resilient Timing for 5G – Technological Solutions  

PRTC systems deployed in the core network use PTP to provide backup timing to local GNSS time distribution. However, since we have not eliminated the dependency of GNSS in the core, our reliance on GNSS remains. The PRTC standard G.8272 includes the requirements for time and phase over a packet network. It describes a clock that delivers <100ns phase and time performance traceable to UTC. With the ever increasing need to improve the performance of emerging mobile technologies and increase protection against GNSS outages, a new standard G.8272.1 called enhanced primary reference clock ePRTC was created. This standard describes a clock that delivers <30ns phase and time performance traceable to UTC. An ePRTC consists of a GPS, an Atomic Clock, and an ePRTC system. Its objective is to produce an independent timescale that is autonomous. It provides time, phase, and frequency calibrated by GNSS. The timescale is then maintained based on the stability of the Atomic Clock. A PRTC receives time directly from GNSS. An ePRTC generates its own timescale locally. The powerful attribute of the ePRTC is that by generating its own independent time it is NOT subject to attacks on GNSS such as jamming or spoofing. Per the standard, once in holdover due to the loss of GNSS, the ePRTC output can increase from 30 ns to 100 ns over a 14-day period. Therefore, subtending PRTC systems can meet their accuracy budget of 100ns when referenced to the ePRTC.  

Outlook  

Although GPS has served us well and is likely to remain an integral component of the timing network, addressing its susceptibility to both intentional and unintentional service interruptions is essential. GPS dependent PRTC systems provide timing over packet networks that can be used to back up GPS. With the advent of ePRTC, GPS is used to calibrate the system, but it then creates a timescale based on the stability of the Atomic Clock. Downstream PRTC systems can take advantage of the improved accuracy created by the ePRTC along with the benefit of a 14-day holdover in the event of a GNSS outage. The next step in establishing a highly accurate and secure time source is the deployment of virtual PRTC (vPRTC) which is a node with a high-performance boundary clock that meets the <100 ns requirements of PRTC without requiring a local GPS antenna system at the site.  A “virtual PRTC” network offers resilient, extremely stable, sub-100 ns time to any point on a DWDM network. The High-Performance Boundary Clocks create a vPRTC node that saves the operators time, expense, and the complexity associated with deploying GPS at all sites. That, along with the added protection and security that ePRTC and vPRTC provides, makes it a critical design consideration that will resolve the present and future concerns surrounding GPS vulnerabilities.  

 

Conclusion   

The more things change, the more they stay the same. While the evolution of the timing network has gone through many different phases, the drivers for the innovations remain the same. Our customers require and expect continual improvements in the services we provide. Governments and industries will continually demand more reliable, resilient, and secure networks. Fortunately, ePRTC and vPRTC technology enable service providers to meet these requirements in a secure and cost-effective manner, providing robust and resilient timing to critical infrastructures. 

Rob Jodrie

Rob Jodrie

Technical Support Engineer, Syncworks

Rob started working in the Telecommunications Industry with the Bell System in 1982. He had responsibility for Tier II Network Synchronization and Transport Technical Support at Verizon for fourteen years and has been working at Syncworks since 2015.

Timing Alarms and the FCC Network Outage Reporting System (NORS)

Timing Alarms and the FCC Network Outage Reporting System (NORS)

The Unnoticed Threat: Network Alarms and the Overlooked Role of Timing

In a January 2024 conversation with Rob Jodrie, Solutions Architect at Syncworks, Mr. Jodrie shed light on the often underappreciated realm of network alarms and the peculiar reality of why they don’t receive the attention they deserve. Alarms signify potential network failures, a critical aspect that historically tends to be overlooked. Despite new technology, the issues persists today.

Network engineers now have tools that make this oversight less likely to demand an entry in the FCC Network Outage Reporting System. However; there is still a threat looming over many network operators. Aging timing equipment is the most vulnerable weak link most likely to send out an alarm. Their contact closure-only method of alarming renders them a danger. They are called “idiot alarm.”

Despite the FCC NORS stigma and the unexpected costs associated with getting the network back to Stratum 1, network alarms continue to be overlooks. A back study that Mr. Jodrie participated in showed that “alarms and power were the two main causes found in the required NORS reports. With all this evidence pointing to a simple solution, there’s still no real good answer as to why they keep going unnoticed.” Mr. Jodrie said. “It’s an old story. It goes like this: “Pay attention to your timing alarms.”

FCC Network Outage Reporting System (NORS)

What is the Network Outage Reporting System (NORS)?

In 2004, the FCC implemented outage reporting rules to ensure swift, comprehensive, and accurate information about significant communication service disruptions with potential impacts on homeland security, public health, safety, and the nation’s economic well-being. Communication providers, including wireline, cable, satellite, wireless, interconnected VoIP, and Signaling System 7 providers, must adhere to these rules. Reportable network outages lasting at least 30 minutes trigger mandatory reporting in the Commission’s Network Outage Reporting System (NORS). Data submitted to NORS is considered confidential.

Depending on the provider type, notifications must be made within specific timeframes: preliminary information within 120 minutes, an initial outage report within three calendar days, and a final report within 30 days of outage discovery. Interconnected VoIP providers follow a similar process, with variations based on the outage’s impact. Covered 911 service providers, responsible for aggregating and delivering 911 traffic, have specific notification timelines and information-sharing requirements when an outage affecting a 911 call center occurs. Source

NORS and the Odd Reality of Timing

Rob, reflecting on his Tier 2 Technical Support career, highlighted the historical neglect of timing in network operations. He shared insights into the FCC reportable events, where outages of certain size, capacity, or duration required reporting to the FCC, often resulting in fines and root cause analysis. Surprisingly, a back study revealed that power and timing were the most common culprits behind these expensive events.
Rob Jodrie:

We used to have to do write ups on what they call FCC reportable events and those were, I forget the exact specifics, but if you had an outage of either a certain size, capacity, bandwidth was down or of a certain duration, length of time or both, you’d have to some in some cases report that to the FCC and it was not a good thing. Fines would come about and all this and they would ask for root cause analysis. Meaning you got to tell me what was the thing that caused this issue, you know. And then you were supposed to, you know, get a plan to say here’s how we addressed it.

But point is we did kind of a back study on that at one point in time and we found that the two most common issues causing FCC reportables were power and timing.

So it’s really interesting to me that you know these FCC reportables were not small events and they were very, very expensive events and it just has always surprised me how little attention that it gets.

Telecom Solutions and BITS Clocks
The conversation turned towards telecom solution BITS clocks, workhorses that have been running for decades. Dave reminisced about the discreet alarms in the early days, simple contact closures that provided minimal information. Even today, thousands of these systems are active in the U.S., relying on basic alarming methods that necessitate physical inspection to determine the issue.

Rob Jodrie:

The interesting thing is in the early days and this is still out here with these in today’s market, these telecom solution bits, clocks which are workhorses, they’ve been running for literally decades. We put our first one in Portland, ME in 1987, I think. But they have what they call discreet alarms, which means it’s a contact closure and all it does is sends a relay event, if you will, to what they call a scan point.

And it just detects is there a short across the line or is there an open and if there’s a short across the line, the words come up and it says “sync major”, “sync minor”, “sync critical.” That’s it. That’s all you know, you don’t know if you’re in hold over, if the whole thing is down, you have zero idea. It’s just a dumb alarm and you would see that alarm and you would have to get a pair of hands out there to stand in front of the box and look at the system and determine based on lights what do we got what’s going on. So get kind of kind of interesting and that is still the case the the next generation of gear thankfully you know is is intelligent right we can log on to it and we can retrieve information from it. We can find out is it in holdover do I have references you know power trouble what’s the story without rolling a truck.

Evolution of Alarm Systems
Despite the persistence of legacy systems, the next generation of gear brings intelligence to the forefront. With the ability to log in remotely, retrieve information, and diagnose issues without dispatching a truck, modern systems have come a long way. However, the conversation acknowledged that the vast number of existing systems that still rely on primitive alarming methods due to aging telecom timing equipment still in use. Specifically the Telecom Solution DCD.

Rob Jodrie:
So the NORS and alarms have come a long way but there are thousands and thousands of those Telecom Solution systems still out and active in networks in the United States. It’s hard to hard to believe but that’s what it is and and sometimes we would actually find that the alarming that piece of wire that would go from the bits clock to the scan point that would report back to the NOC center.

Sometimes that wire would get ripped out and there was no alarming. So we’d walk into an office look at the BITS clock see it’s an alarm call the NOC and say “why did you not respond to the alarm?” “We don’t see any alarm.” The lack of love for timing has been sort of a an interesting thing to note over over these years.

As time marches on, I have seen people becoming more aware of alarms and of the FCC Network Outage Reporting System at times but for the most part it usually takes a bit big time with a service-affecting outage to get their attention.

Only then is there a lot of attention given to it.

How Alarms Manifest Themselves Today

Rob Jodrie:
One of the most common alarms in use today is something called SNMP. It stands for Simple Network Monitoring Protocol and it’s an intelligent thing. What you do is you have an SNMP server located somewhere centrally and you have to point this intelligent equipment via an IP address scheme back to it. So when you have a device that’s intelligent and some problem comes in where the network has lost their GPS signal, it then sends an intelligent piece of information back to this SNMP server. Then that translates it and comes up and gives actionable and almost pinpoint information along the lines of “this piece of equipment at this location has this issue and now somebody needs to respond to that.” With the SNMP you have more to go on rather than sync major red light.

Now issues when someone calls our SyncCare support line and says “I’m in holdover, I’ve lost GPS.” those are noted in the alarm logs so you can go back and say when did this happen? Has this been a week, two weeks? What events were there? There’s just more information you can collect, but it’s all done usually via SNMP.

Rob Jodrie

Rob Jodrie

Technical Support Engineer, Syncworks

Rob started working in the Telecommunications Industry with the Bell System in 1982. He had responsibility for Tier II Network Synchronization and Transport Technical Support at Verizon for fourteen years and has been working at Syncworks since 2015.