Business Continuity-Disaster Recovery Plan
1 Business Continuity-Disaster Recovery Plan
Version History
Version Date Author Reason/Comments
1.00 October 2022 Hunter Crosby Document Origination
1.1 October 2022 Hunter Crosby Specific Team Details Added Business Continuity-Disaster Recovery Plan
2 Table of Contents
Business Continuity-Disaster Recovery Plan.........................................................................................................1 Version History .................................................................................................................................................1 Table of Contents .......................................................................................................................................2 Introduction ..........................................................................................................................................................3 Design of the Plan.................................................................................................................................................4 Contact Information .............................................................................................................................................6 Roles and Responsibilities.....................................................................................................................................7 Chief Information Officer (CIO/CTO) ................................................................................................................7 Business Continuity Team (BCT).......................................................................................................................7 Business Continuity Manager.......................................................................................................................7 Business Continuity Team Members............................................................................................................8 Business Continuity Framework ...........................................................................................................................9 Phase I – Disaster Detection and Determination .........................................................................................9 Phase II – Notification of Personnel Responsible for Recovery....................................................................9 Phase III – Initiate the Business Continuity Plan.........................................................................................10 Phase IV - Disaster Recovery Strategy ........................................................................................................10 Phase V - Scope of the Business Continuity Plan........................................................................................12 Business Continuity-Disaster Recovery Plan
3 Introduction
Planning for the business continuity of the City of Ripon in the aftermath of a disaster is a complex task.
Preparation for, response to, and recovery from a disaster affecting the administrative functions of the Institute requires the cooperative efforts of many support organizations in partnership with the functional areas supporting the "business" of the City of Ripon. This document records the Plan that outlines and coordinates these efforts, reflecting the analyses by representatives from these organizations and by the City of Ripon Information Systems Network Administrator, Hunter Crosby.
For use in the event of a disaster, this document identifies the computer recovery facilities (hot sites and shell sites) that have been designated as backups if the functional areas are disabled The purpose of the Business Continuity-Disaster Recovery Plan is to allow City of Ripon to respond quickly and appropriately to service interrupting incidents.
Event Definition
Any abnormal observable loss of service or accessibility occurrence in system, network, environment, process, workflow, or personnel. Events may be negative in nature.
Adverse Events Definition
Events with a negative consequence. This plan only applies to adverse events that are caused by factors beyond the resolution capabilities of the Computer Incident Response Plan (IRP). Examples of adverse events outside of the scope of the IRP are: natural disasters, power failures, etc. will be covered by the Business Continuity Plan.
How to Use This Document
Use this document to learn about the issues involved in planning for the continuity of the critical and essential business functions at the City of Ripon, as a checklist of preparation tasks, for training personnel, and for recovering from a disaster. This document is divided into four parts, as the table below describes.
Part Contents
I Information about the document itself.
II Design of the Plan that this document records, including information about the overall structure of business continuity planning at the City of Ripon.
III General responsibilities of the individual City of Ripon Support Teams that together form the Business Continuity Management Team, emphasizing the function of each team and its preparation responsibilities.
IV Recovery actions for the City of Ripon Support Teams and important checklists such as the notification list for a disaster and an inventory of resources required for the environment. [Note: If a "disaster" situation arises, Section IV of the Plan is the only section that needs to be referenced. It contains all the procedures and support information for recovery.
Business Continuity-Disaster Recovery Plan
4 Audience
This document addresses several groups within the City of Ripon central administration with differing levels and types of responsibilities for business continuity, as follows:
• Administration
• Business Continuity Management Team
• City of Ripon Support Teams
• External Organizations
It should be emphasized that this document is addressed particularly to the members of the Business Continuity Management Team, since they have the responsibility of preparing for, responding to, and recovering from any disaster that impacts the City of Ripon. Part III of this document describes the composition of the Business Continuity Management Team in detail.
Distribution
As the written record of the City of Ripon 's Business Continuity Plan, this document is distributed to each member of the Business Continuity Management Team, including members of the City of Ripon Support Teams.
It is also distributed to members of the Administration, External Organization Coordinators and others not primarily involved with the direct recover effort but require advanced knowledge of planned actions and needs.
Design of the Plan
Part II describes the philosophy of business continuity planning at City of Ripon generally, and the kind of analysis that produced this Plan. It also provides an overview of the functions of the Business Continuity Management Team in implementing this Plan.
Purpose
The City of Ripon increasingly depends on computer-supported information processing and telecommunications. This dependency will continue to grow with the trend toward decentralizing information technology to individual organizations within the City of Ripon administration and throughout the organization.
The increasing dependency on computers and telecommunications for operational support poses the risk that a lengthy loss of these capabilities could seriously affect the overall performance of the City of Ripon. A risk analysis which was conducted identified systems as to risk Category I, belonging comprising those functions whose loss could cause a major impact to the City of Ripon within 72 hours. It also categorized the City of Ripon functions as Essential, or Category II - requiring processing support within 2-4 week(s) of an outage. This risk assessment process will be repeated on a regular basis to ensure that changes to our processing and environment are reflected in recovery planning.
The City of Ripon administration recognizes the low probability of severe damage to data processing telecommunications or support services capabilities that support the operations. Nevertheless, because of the potential impact to the City of Ripon, a plan for reducing the risk of damage from a disaster however Business Continuity-Disaster Recovery Plan
5 unlikely is vital. The City of Ripon’s Business Continuity Plan is designed to reduce the risk to an acceptable level by ensuring the restoration of Critical processing within 72 hours, and all essential production (Category II processing) within 2-4 week(s) of the outage.
The Plan identifies the critical functions of the City of Ripon and the resources required to support them. The Plan provides guidelines for ensuring that needed personnel and resources are available for both disaster preparation and response and that the proper steps will be carried out to permit the timely restoration of services.
This Business Continuity Plan specifies the responsibilities of the Business Continuity Management Team, whose mission is to establish Institute level procedures to ensure the continuity of the City of Ripon 's business functions. In the event of a disaster affecting any of the functional areas, the Business Continuity Management Team serves as liaison between the functional area(s) affected and other community organizations providing major services. These services include the support provided by Physical Plant, security provided by the Police, and public information dissemination handled by the City of Ripon Administration, among other.
Assumptions
The City of Ripon Plan is predicated on the validity of the following three assumptions:
• The situation that causes the disaster is localized to the data processing facility of Operations and Systems in the City of Ripon; the building or space housing the functional area; or to the communication systems and networks that support the functional area. It is not a general disaster, such as an earthquake or hurricane affecting the entire region.
It should be noted however, that the Plan will still be functional and effective even in an area-wide disaster. Even though the basic priorities for restoration of essential services to the community will normally take precedence over the recovery of an individual organization, the City of Ripon 's Business Continuity Plan can still provide for a more expeditious restoration of our resources for supporting key functions. • The Plan is based on the availability of the hot sites or the back-up resources, as described in Part IV. The accessibility of these, or equivalent back-up resources, is a critical requirement. • The Plan is a document that reflects the changing environment and requirements of the City of Ripon. Therefore, the Plan requires the continued allocation of resources to maintain it and to keep it in a constant state of readiness.
Development
The City of Ripon 's Information System Network Administrator, with assistance from key City of Ripon support areas, is responsible for developing the City of Ripon 's Business Continuity Plan.
Development and support of individual City of Ripon Team Plans are the responsibility of the functional area planning for recovery.
Business Continuity-Disaster Recovery Plan
6 Maintenance
Ensuring that the Plan reflects ongoing changes to resources is crucial. This task includes updating the Plan and revising this document to reflect updates; testing the updated Plan; and training personnel. The Business Continuity Management Team Coordinators are responsible for this comprehensive maintenance task.
The Business Continuity Management Team Coordinators ensures that a formal review is initiated whenever there is a significant change required for the plan. Annually, the Business Continuity Management Team Coordinators initiates a complete review of the Plan, which could result in major revisions to this document. These revisions will be distributed to all authorized personnel, who exchange their old plans for the newly revised plans.
Testing
Testing the Business Continuity Plan is an essential element of preparedness. Partial tests of individual components and recovery plans of specific the City of Ripon Teams will be carried out on a regular basis. A comprehensive exercise of the City of Ripon continuity capabilities and support by our designated recovery facilities will be performed on an annual basis.
Contact Information
Updated October 2022
The following is a list of roles to be referred to in a disaster recovery event.
Name Title Role Contact Information Escalation
(1-3)*
City
Administrator
Business Continuity
Manager
BC manager (209) 599-0235 1
IT Infrastructure Manager IR Manager (209) 599-0277 1
Chief Communications
Manager
BCT member (209) 599-0259 3
City Attorney Legal BCT member (209) 599-0214 3
IT Risk Manager BCT member (209) 599-0277 3
City Clerk HR Representative BCT member (209) 599-0217 3 Chief Physical Security
Representative
BCT member (209) 599-0259 3
WPNS 3
rd Party Support (209) 581-6789 3
Alliant Cyber Insurance
Provider
3 City Council Regulatory/Government
Reporting Body
3 *Escalation level determines order in which notification should occur:
1. Notify first, required on all incidents
2. Required on all moderate or high-severity incidents
3. Involve as needed
Business Continuity-Disaster Recovery Plan
7 Roles and Responsibilities
Chief Information Officer (CIO/CTO)
Seek approval from Executive Management for the administration of the Business Continuity Program.
Coordinate response activities with auxiliary departments and external resources as needed to minimize damages to information resources.
Provide updates on response activities to Business Continuity Team (BCT) and other stakeholders during an incident.
Ensure service level agreements with service providers clearly define expectations of the organization and the service provider in relation to Business Continuity. Ensure policies related to recovery management accurately represent the goals of the organization.
Review the Business Continuity Plan (“the Plan”) to ensure that it meets policy objectives and accurately reflects the goals of the organization. Seek Plan approval from BCT. Work with the Business Continuity Manager to periodically evaluate the effectiveness of the Plan and BCT.
Ensure Cyber Insurance is maintained as necessary and appropriate stakeholders are informed. (See Error! Reference source not found.)
Ensure lessons learned are applied/weighed based on risk for Business Continuity incidents.
Business Continuity Team (BCT)
Under the overall direction of the Business Continuity Manager, support is provided to assist a functional area's recovery by Infrastructure Support Teams. These teams work the problem condition to restore services and provide assistance at the Institute level. In many cases, the organizations comprising these support teams have as their normal responsibility the provision of these support services. This support is generally documented in a procedure’s manual for the organization. The Business Continuity Plan is an adjunct to that documentation and highlights the interfaces between the infrastructure level service and the Business Continuity Team operations requirements. In cases where the documentation in this Plan and the organization's documents differ, the organization's documentation has precedence. • Consists of legal experts, risk managers, and other department managers that may be consulted or notified during Business Recovery.
• Advise on recovery response activities relevant to their area of expertise. • Maintain a general understanding of the Plan and policies of the organization. • Ensure business continuity activities are in accordance with legal, contractual, and regulatory requirements.
• Participate in tests of the Business Continuity plan and procedures. • Responsible for internal and external communications pertaining to recovery activities.
Business Continuity Manager
The Business Continuity manager oversees and prioritizes actions during the incident. They are also responsible for conveying the special requirements of high severity activities to the rest of the organization as well as communicating potential impact to the CIO. Additionally, they are responsible for understanding the SLAs in place with third parties, and the role third parties may play in specific response scenarios.
Further responsibilities:
Act as a liaison for all communications to and from the CIO. Assemble a Business Continuity Response Team (CSIRT). Ensure personnel tasked with business continuity responsibilities are trained and knowledgeable on how to respond to incidents.
Business Continuity-Disaster Recovery Plan
8 Update Business Continuity Plan and procedures as needed based on results from testing, incident lessons learned, industry developments and best practices. Review the Business Continuity Plan and procedures at least annually. Initiate tests of the business Continuity Plan and procedures at least annually. Ensure team activities comply with legal and industry requirements for Business Continuity procedures.
Act as the primary Business Continuity Manager, responsible for declaring a disaster incident, managing team response activities, and approving close of recovery incidents. Be aware of Cyber Insurance Policies, contact mechanisms, and when to include providers. (See Error! Reference source not found.)
Business Continuity Team Members
The Business Continuity Manager is supported by a team of technical staff that work directly with the affected information systems to restore service to end users. Team members are typically comprised of subject matter experts (SMEs), senior level IT staff, third parties, outsourced security, or forensic partners.
Teams and responsibilities:
• Damage Assessment/Salvage Team. Headed by the Infrastructure Manager and activated during the initial stage of an emergency, the team reports directly to the Business Continuity Management Team, evaluates the initial status of the damaged functional area, and estimates both the time to reoccupy the facility and the salvageability of the remaining equipment. This team draws members from the Business Continuity Team as well as appropriate vendors supporting our environment.
o Following the assessment of damage, the team is responsible for salvaging equipment, data and supplies following a disaster; identifying which resources remain; and determining their future utilization in rebuilding the data center and recovery from the disaster. The members of the Damage Assessment Team become the Salvage Team • Transportation Team. A temporary Institute Support Team headed jointly by the Infrastructure Manager and Physical Security Manager responsible for transporting resources personnel, equipment, and materials to back-up sites as necessary. • Public Information. The interface with the media, the general public and faculty, staff and students who are not participating in the recovery process is handled by the Communications Manager, working closely with the CIO and HR Managers.
• Telecommunications Team. Headed by the Infrastructure Manager is responsible for establishing voice and data communications between the affected site and the remainder of the Institution.
TABLE 1: (COMPANY) BUSINESS CONTINUITY TEAM MEMBERS
No. BCT Member Role
1 IT Department Damage Assessment Team/Salvage Team, Telecommunications Team
2 Chief Public Information
Business Continuity-Disaster Recovery Plan
9 Business Continuity Framework
This section describes four required responses by the City of Ripon to a disaster, or to a problem that could evolve into a disaster:
1. Detect and determine a disasters condition.
2. Notification of Personnel Responsible for Recovery.
3. Initiate the City of Ripon 's Business Continuity Plan Activate the designated hot site.
Disseminate Public Information.
Provide support services to aid recovery.
Disaster Recovery Strategy
Emergency Phase
Backup Phase
Recovery Phase
Each subsection below identifies the organization(s) and/or position(s) responsible for each of these responses.
Phase 0 – Preface
In the event a major disaster, the first priority is to ensure the safety of every employee. In an emergency situation where life is threatened or staff is in danger of physical harm, it is imperative to take measures to protect all staff from danger. No one in the organization should place themselves into a dangerous situation or take unnecessary risks that may threaten their life or the lives of others unless properly trained to respond to such disasters.
Phase I – Disaster Detection and Determination
The detection of an event which could result in a disaster affecting information processing systems at the City of Ripon is the responsibility of Business Continuity Manager, Physical Security Representative, Infrastructure Manager, or whoever first discovers or receives information about an emergency developing in one of the functional areas or other building housing major information processing systems or about the communications lines between these buildings.
Phase II – Notification of Personnel Responsible for Recovery Personnel will follow existing procedures and notify the individuals who are acting as the Business Continuity Management Team Duty Persons (DP)). The DP on call will monitor the evolving situation and, if appropriate, will then notify the Business Continuity Management Team representative based upon a predefined set of notification parameters.
When a situation occurs that could result in interruption of processing of major information processing systems of networks on campus, the following people must be notified:
· The Business Continuity Management Team Coordinator (Network Administrator) Business Continuity-Disaster Recovery Plan
10 · The Business Continuity Team
Phase III – Initiate the Business Continuity Plan
Initiation of this Plan is the responsibility of the Business Continuity Management Team Coordinator or any member of the Business Continuity Management Team.
Activation of a Designated Hot Site
The responsibility for activating any of the designated hot sites or back-up resources is delegated to Chief Information Officer. In the absence of the CIO, responsibility reverts to the Business Continuity Manager.
Within 24 hours of the occurrence the CIO, or alternate, determines the prognosis for recovery of the damaged functional area through consultation with the Infrastructure Manager, Communications Manager and representatives from Operations and Systems, Telecommunications Systems and the functional areas affected.
If the estimated occupancy or recovery of the damaged functional area cannot be accomplished within 72 hours, the usual occupants of the designated back-up site are notified of the intention to occupy their facility.
Dissemination of Public Information
The Public Information Manager is responsible for directing all meetings and discussions with the news media and the public, and in conjunction with the BCT Manager and necessary BCT personnel not actively participating in the recovery operation. In the absence of the Public Information Manager, the responsibility reverts to the senior official present at the scene.
Recovery Status Information Number
A predefined number should be established as a voice mail information number for posting recovery status and information notices. All reports will be placed by the Business Continuity Manager or the Telecommunication BCT team leader.
Provision of Support Services to Aid Recovery
During and following a disaster, Institute Support Teams are responsible for aiding the BCT Teams. They operate under the direction of the Business Continuity Management Team through the Chief Information Officer.
Phase IV - Disaster Recovery Strategy
The disaster recovery strategy explained below pertains specifically to a disaster disabling the main data center. This functional area provides mainframe computer and major server support to the City of Ripon administrative applications. Especially at risk are the critical applications those designated as Category I (see below) systems. The BCT Operations and Systems (O&S) Team Plan provides for recovering the capacity to support these critical applications within 72 hours. Summarizing the provisions of the O&S Plan, subsections below explain the context in which the Institute's Business Continuity Plan operates. The Business Continuity Business Continuity-Disaster Recovery Plan
11 Plan complements the strategies for restoring the data processing capabilities normally provided by Operations & Systems.
This section addresses three phases of disaster recovery:
· Emergency
· Backup
· Recovery
Strategies for accomplishing each of these phases are described below. It should be noted that the subsection describing the emergency phase applies equally to a disaster affecting the Administration Building, other buildings housing computing assets, or the functional area that provides support for the maintenance of the critical system.
Emergency Phase
The emergency phase begins with the initial response to a disaster. During this phase, the existing emergency plans and procedures are implemented for direct efforts to protect life and property. Security over the area is established as local support services such as the Police and Fire Departments are enlisted through existing mechanisms. The Business Continuity Manager is alerted and begins to monitor the situation.
If the emergency situation appears to affect critical facility or services, either through damage to data processing or support facilities, or if access to the facility is prohibited, the BCT Manager will closely monitor the event, notifying BCT personnel as required to assist in damage assessment. Once access to the facility is permitted, an assessment of the damage is made to determine the estimated length of the outage. If access to the facility is precluded, then the estimate includes the time until the effect of the disaster on the facility can be evaluated.
If the estimated outage is less than 72 hours, recovery will be initiated under normal Information Systems operational recovery procedures. If the outage is estimated to be longer than 72 hours, then the Business Continuity Manager activates the BCT Members and notifies the Chief Technical Officer the Business Continuity Plan is activated. The recovery process then moves into the back-up phase.
The Business Continuity Management Team remains active until recovery is complete to ensure that the Institute will be ready in the event the situation changes.
Back-up Phase
The back-up phase begins with the initiation of the appropriate BCT Plan(s) for outages enduring longer than 72 hours. In the initial stage of the back-up phase, the goal is to resume processing critical applications.
Processing will resume either at the main data center or at the designated hot site, depending on the results of the assessment of damage to equipment and the physical structure of the building.
Business Continuity-Disaster Recovery Plan
12 In the back-up phase, the initial hot site must support critical (Category I) applications for up to 4 weeks and as many Category II applications as resources and time permit. During this period, processing of these systems resumes, possibly in a degraded mode, up to the capacity of the hot site. Within this 4 week period, the main data center will be returned to full operational status if possible.
However, if the damaged area requires a longer period of reconstruction, then the second stage of back-up commences. During the second stage, a shell facility is assembled and equipment installed to provide for processing all applications until a permanent site is ready.
Recovery Phase
The time required for recovery of the functional area and the eventual restoration of normal processing depends on the damage caused by the disaster. The time frame for recovery can vary from several days to several months. In either case, the recovery process begins immediately after the disaster and takes place in parallel with back-up operations at the designated hot site. The primary goal is to restore normal operations as soon as possible.
Phase V - Scope of the Business Continuity Plan
The object of this Plan is to restore critical (Category I) systems within 72 hours, and Essential (Category II) systems within 2-4 week(s) of a disaster that disables any functional area and/or essential equipment supporting City of Ripon systems or functions in that area.
The initial Risk Assessment of the computer applications that support administration assigned 4 systems to Category I Critical. This risk category identifies applications that have the highest priority and must be restored within 72 hours of a disaster disabling a functional area. Specifically, each function of these systems was evaluated and allocated a place in one of four risk categories, as described below.
Category I - Critical Functions
Category II - Essential Functions
Category III - Necessary Functions
Category IV - Desirable Functions
Note: Category IV functions are important to City of Ripon administrative processing, but due to their nature, the frequency they are run and other factors, they can be suspended for the duration of the emergency.
The administrative systems in Categories I - IV are those that provide Institute wide services. There are noninformation processing systems (such as telephone communications) that are also either essential for the Institute or the local area(s) they support. Recovery for these systems too must be based upon an assessment of the impact of their loss and the cost of their recovery.