After more than 50 years of product development and operation in utilities, one might expect the implementation of an Energy Management System (EMS) to be “plug and play.” Unfortunately, the complexity of power and network applications and the uniqueness of each utility’s power system make an EMS deployment anything but simple. However, understanding common pitfalls and best practices can significantly reduce risk and improve the opportunity for a successful deployment of this complex control system. This article identifies key activities in planning and implementation that illustrate lessons learned and best practices for deploying an EMS.
The planning phase should include:
- Defining the project management structure and governance processes;
- Finalizing project scope and vendor statement of work;
- Conducting work breakdown to define and schedule tasks, deliverables, and milestones;
- Developing the resource plan; and
- Defining system requirements.
While there are best practices in each of these activities, below we will focus on finalizing project scope/vendor statement of work and defining system requirements.
Finalizing Project Scope and Vendor Statement of Work
Developing a statement of work for a complex control system can be a complicated process, fraught with risk. Because product-specific expertise is required to review requirements and potential customizations, it is common to define multiple vendor statements of work aligned with key project phases or milestones. Within each statement of work, scope, schedule, responsibilities, assumptions, and deliverables must be clearly defined, including deliverable review and acceptance procedures. Furthermore, upon completing system requirements and defining associated customizations, a conformance matrix should be developed with the system vendor. A conformance table will assure all requirements have been addressed, agreed upon or not, and exceptions identified, thereby reducing the risk of contract changes due to ambiguity or gaps in coverage.
Key areas of scope that need to be fully vetted with the system vendor include database and display conversion and development, systems integration, dispatch training simulator, network application tuning, testing, change management, cutover, and support. And with today’s virtual infrastructure and converged architecture, it is not uncommon for utilities to provide infrastructure specifications and standards to the system vendor, creating additional complexities that must be considered during statement of work through planning and implementation.
Defining System Requirements
Requirements analysis is a key step in developing a successful partnership between the vendor and the utility. If both vendor and utility clearly define what each party requires, there will be less risk of unexpected costs and schedule delays. Requirements analysis includes defining the system applications functionality required and the associated technical (also called non-functional) requirements for performance, security, fault tolerance, usability, support and maintenance, and other related areas for implementation; e.g. code release and installation, testing, cutover, post cut-over support, etc.
The requirements conformance table can also be leveraged here. The conformance table requires a response to each section of the requirements with the vendor specifically stating whether they agree, have an alternative proposal, or take exception to the requirement. If an alternative is proposed, the vendor must provide a detailed explanation. Likewise, if an exception to the requirement is taken, the utility and vendor can work together on an alternative solution. During requirements analysis, all requirements and customizations will be reviewed and agreed upon and any alternatives or exceptions will be resolved to the satisfaction of both the utility and vendor. Consider this to be a very collaborative process between utility and system vendor, and is often facilitated by a systems integration partner.
Because of the age of EMS legacy systems, typically over 10 years old, a detailed review of customizations developed by the utility and how those customizations will be either integrated or absorbed by the new vendor software should also be included. This is a key part of the requirements analysis phase and should be given ample time and consideration to a wide range of utility SME experience needed. Key factors to consider include cost to absorb, schedule impact, complexity or risk to integrate, reliability, compliance, support, and maintenance. Many utilities are moving to a “no customs” approach. Instead, they are changing internal business processes, reporting and tools to support an out-of-the-box vendor deployment. To ensure a reliable BES, these decisions should be made with careful consideration to the potential downstream impacts
to transmission operations.
The implementation phase consists of work streams related to system applications and infrastructure design, configuration and build, testing, operational readiness, cutover and post cutover support, and many related core tasks. Below, we highlight three of the key areas for success: engaging the right team, database and display development, and testing.
Engaging the Right Team
Often, the utility team in charge of delivering the new EMS is the same team responsible for maintaining the legacy EMS day-to-day operations and support—there are only so many EMS and transmission operations subject matter experts (SMEs) to go around. It is easy to see how daily legacy system operational needs can override project needs, increasing project risk.
It is important to consider either engaging a systems integration partner or backfilling the legacy system operation and support team. Either will enable the utility SMEs to participate on the new system implementation without trying to balance the day-to-day concerns of supporting the most critical real-time system in the utility.
On the surface, a systems integration partner or backfilling the day-to-day utility staff can look like a significant additional cost to the project. However, an experienced integration partner can devote 100% concentration to the EMS project and engage consultants who bring recent experience, best practices, and lessons learned. In addition, an integration partner can provide the project management and key areas of subject matter expertise to do the “heavy lifting” on behalf of the utility. While utility participation is certainly required (e.g. requirements definition and approval, integration to legacy systems, application and technology changes, IT environments, system testing and tuning, knowledge transfer, etc.), an integration partner can bring significant value with a breadth of resources, project accelerators, lessons learned, and risk management.
Database and Display Development
Of all the issues that can delay a project, it is the timely development, migration, and cleanup of the database and displays that can have the largest impact. Database and display accuracy are critical for real-time operations. Operators must have confidence that the information presented is accurate and refers to the correct device to confidently execute supervisory control before accepting the new system.
Many vendors provide a database conversion service to migrate legacy systems to the new EMS. However, the converted database inevitably requires clean up as well as additional data to support advanced applications. Utilities often underestimate these efforts and the operator requirements for display quality. Automated display generators usually create an unfamiliar layout of busses and breakers. In the end, rebuilding the one-line diagrams remains a manual process requiring a team of resources and time. An integration partner or additional set of resources can add value by handling the scope of display building required.
With the continued advancement in NERC CIP compliance, the scope of factory testing has been reduced to focus on demonstrating core application functionality versus system performance. New compliance and cybersecurity requirements require that system performance be tested onsite at the utility, during site acceptance testing, where the utility server infrastructure, additional third party software, and secure network communications (e.g. firewalls, routers, etc.) are installed to meet utility NERC CIP compliance and cybersecurity specifications.
Factory acceptance testing has also evolved in some cases to the point where many of the applications are field tested and included as part of the standard system. The most important areas to concentrate on are testing the interfaces with field equipment (RTUs and IEDs), especially when new RTU communication protocols are implemented. In this case, an RTU simulator can provide significant value in testing various RTU interfaces and communication protocols. Additional key areas of site testing should include testing between control centers, testing to the ISO, retrieval of real-time and historical data, point-to-point checkout, integration interfaces between applications, and the user interface (e.g. alarms, one-line, and tabular display quality).
It is also critical to properly plan and schedule time for tuning advanced applications (e.g. power flow, state estimator, load flow, contingency analysis, voltage stability analysis). Like tuning applications, the time, roles, and responsibilities must be clearly defined for tuning and validating alarm design and configuration. Upgrading or replacing the EMS offers a great opportunity to improve alarm management and address limitations in prioritization, display real estate required, sorting, filtering, color schemes, descriptions, etc. Alarm management is a key consideration to improve operational efficiency, safety, and system reliability.
Delivering an EMS upgrade or replacement is clearly a complex undertaking. The good news is there are others that have braved this path, resulting in a source of lessons learned and best practices to help utilities avoid common pitfalls. Engaging the right team who can successfully apply those lessons to simplify the complexities and mitigate risks must be a utility priority. While the additional costs may seem a deterrent, utilities should weigh these costs against the likely cost overruns from vendor scope changes, schedule delays, SME constraints, and burn-out, as well as the impact of missing project commitments on such a critical system in the utility.