Fundamental 1: Plan for Incidents, Emergencies, and Disasters
Functioning as a non-profit, WaterISAC serves as a vital hub for information, offering resources and facilitating collaborations between the U.S. Department of Homeland Security's Cybersecurity and Infrastructure Security Agency, the FBI, the U.S. Environmental Protection Agency, state intelligence fusion centers, and other federal and state agencies.
We will cover the Plan for Incidents, Emergencies, and Disasters as part of the 1st of 12 Cybersecurity Fundamentals of Water and Wastewater Utilities recently released by Water ISAC, delving into key recommendations and insights to strengthen cybersecurity resiliency
Incident Response Planning
One might be surprised to find response planning at the beginning of a recommended practice. The Five ICS Cybersecurity Critical Controls highlight the importance of prioritizing incident response planning, particularly an ICS-specific Incident Response Plan. As these response plans are created, utilities can benefit greatly from identifying cybersecurity and continuity of operations gaps and implementing best practices.
Developing strategies for incident, emergency, and disaster response is crucial for a swift recovery.
Both IT and OT teams should focus on cyber incident response plans and disaster recovery plans as key components of overall business continuity or continuity-of-operations plans.
Plans should not be developed by a single department, but rather by a team across all departments, including external teams. The inclusion of emergency response and law enforcement authorities in the development of the plans will ensure a coordinated and unified response that reaches all organizational resources
In January 2024, CISA, EPA, FBI, as well as the federal government and WWS Sector partners developed an Incident Response Guide (IRG). This IRG provides water and wastewater sector owners and operators with information about federal roles, resources, and responsibilities for each stage of the cyber incident response lifecycle that help enhance their respective Incident Response plans and procedures.
Despite taking precautions, numerous organizations continue to face cybersecurity breaches. According to experts, it is not a question of if an organization will experience a compromise, but rather when. Those organizations that are most successful in handling these incidents are the ones that can promptly identify and respond to the intrusion (Fundamental 4). A well-crafted CIRP can minimize harm, enhance trust from partners and customers, and decrease recovery time and expenses. It is crucial for the incident response plan to be established beforehand and integrated into business continuity plans within the organization.
Two key suggestions for inclusion in the CIRP are:
It is important to prepare in advance for emergency operating procedures for industrial processes - at least as appendices or references for how, during an incident with potentially degraded capability, water operations staff will maintain system operations.
By following the Five ICS Cybersecurity Critical Controls, it is important to prioritize pre-planned steps for common or high-impact incident scenarios. These scenarios, such as a ransomware infection, compromise of remote access, or loss of a critical function, should be given special attention due to their likelihood of occurring again and real-world occurrence. By proactively considering the details of maintaining operations during these incidents in advance, response effectiveness can be improved, and impacts and recovery time can likely be reduced.
The Cyber Incident Response Team:
To improve their ability to react to cybersecurity incidents, organizations may want to create a dedicated cyber incident response team that oversees and handles the incident response procedure.
While the security operations center takes care of routine investigations, a distinct team should be established specifically for addressing major cybersecurity incidents.
This team would be responsible for establishing an incident response governance model (Fundamental 10), which would involve defining the different types and levels of severity for incidents that may occur.
There should be a comprehensive response to cyber incidents. The team should include internal and external stakeholders, as well as IT and OT security staff and operators.
The team composition should also include other staff such as executives, communications and public relations teams, human resources, legal, product, and engineering personnel.
Backups of Your System:
A system backup is critical to ensuring timely recovery and reducing the risk of data destruction or hindering system recovery after a cyber incident. In order to ensure effective recovery when needed,
backups need to be protected from corruption or destruction (such as during a ransomware attack), validated, and tested.
If restorations are made without verifying efficacy through test restorations, the results can often be costly.
It is also advised to take the time to create human-readable restorations. In the event that replacement devices cannot read the automated backup file, the human-readable version will increase the efficiency of restoration.
The Value of Cyber Insurance:
It is expensive to recover from a cyber incident. A cyber attack can cost tens of thousands of dollars for a small organization or millions of dollars for a large organization. Emergency support from vendors specializing in incidents can also be included in expenses, among others such as forensics and recovery, replacement of corrupted software, computers, and other hardware, complimentary credit monitoring, data stolen, notification of customers, lost productivity of employees unable to work until the system is restored, legal fees, etc. In addition to potential liabilities and public relations outreach, cyber insurance can be a valuable tool for building resilience.
It not only covers expenses related to cybersecurity incidents but also offers access to emergency support from knowledgeable experts and specialized vendors. However, it's important to note that policies may vary between insurers and it's crucial to thoroughly research and compare options before selecting the right policy for your needs.
For example, some policies may not cover pre-existing breaches, acts of war, or incidents caused by employee vulnerability to social engineering tactics. Additionally, most insurers require minimum security controls and risk assessments to be in place before granting a policy.
Plans for Disaster Response:
The America's Water Infrastructure Act requires drinking water systems to develop emergency response plans (ERPs) and update them every five years. The plans must include both cyber systems and physical systems.
Contents of Disaster Recovery Plan:
A list of major goals of the disaster plan.
Names and contact information of IT and OT personnel, vendors, and contract support, Roles and responsibilities.
Profiles of software and hardware used by the utility, including a discussion of which utility functions rely on each software and hardware item.
Service level agreements for outsourced services during a disaster.
Recovery time objectives, Maximum tolerable downtime
Backup procedures, Plans for mobilizing to temporary work locations.
Plans for backing up to a temporary site, Plans for restoring the home site.
Plans for testing and exercising the DRP.
Communication Out of Band as a Backup:
You should consider such critical response dependencies before an incident occurs if you have a backup communications plan. The internet may be down as a result of a DDoS attack - or worse, the incident could coincide with a larger regional incident that could cause widespread problems.
Have alternate means of data communications to the infrastructure via an out-of-band mechanism based on cellular connectivity
This OOB network needs to be securely designed, implemented and protected as it is a back-door access to your network
Resilience to Power Outages:
In order to protect their systems against the impacts of power outages, utilities must prioritize providing backup power to operate their IT and ICS equipment. In cases of emergency, on-site generation is necessary.
NIST recommends that utilities have an uninterruptible source of power on-site to bridge the gap between power loss and activation of emergency generators.
It is critical to plan for sufficient fuel to keep generators running during an emergency.
Water utilities can work with their local power company to prioritize critical facilities during power restoration efforts.
It Takes Practice to Become Proficient: The CIRP and DRP need to be operationalized, reviewed, practiced, and updated accordingly. To increase readiness, consider implementing a red team and/or blue team approach to the exercises. Red Teams represent the offensive side of cybersecurity while Blue Teams represent the defensive side of cybersecurity.
Tabletop Exercises (TTXs):
As previously mentioned, utilities are strongly encouraged to implement CIRPs through workshops and tabletop exercises (TTXs). There is a range of options available for conducting these exercises, from simple discussions to full-scale and coordinated functional drills. CISA provides multiple TTX options, including self-service resources and end-to-end support for exercise planning and execution, to assist utilities in evaluating their cybersecurity readiness.
CISA Tabletop Exercise Packages (CTEPs) are a comprehensive set of tools aimed at assisting stakeholders in conducting their own exercises. These packages can be customized and include template objectives, scenarios, discussion questions, as well as a variety of references and resources.
The scenarios covered within the CTEPs focus on different types of cyber threats, such as ransomware, insider threats, phishing, and industrial controls. They also incorporate physical scenarios and cyber-physical convergence exercises.
Moreover, the cybersecurity and cyber-physical convergence scenarios offer specialized exercises specifically tailored for Water and Wastewater Systems. In some cases, utilities may choose to involve external entities to aid in exercise planning, development, and execution.
Through its Stakeholder Exercises program, CISA offers full support for end-to-end exercise planning and conduct. This includes planning meetings, document creation, scenario development, facilitation, and after-action report development.
In today's digital age, safeguarding water and wastewater utilities against cyber threats is crucial. The guidance provided by WaterISAC illuminates the path toward enhanced cybersecurity resilience. By following fundamental principles like meticulous incident response planning and continuous practice through tabletop exercises, utilities can fortify their defenses. Collaboration, preparation, and diligence remain vital in protecting our water systems. Stay tuned as SAC explores further fundamentals to bolster cybersecurity practices across utilities.
Authored by:
Yash Deshpande
Analyst
Abhi Thorat
CTO and Founder