Agenda Report
Fullerton City Council
MEETING DATE: FebRuary 4, 2025
TO: CITY COUNCIL / SUCCESSOR AGENCY
SUBMITTED BY: STEPHEN BISE, P.E., Public Works Director PREPARED BY: Richard Armendariz, Assistant Director of Public Works – maintenance SUBJECT: Cybersecurity Upgrades for City water Supervisory control and Data acquisition system
SUMMARY
Staff requests approval to procure new Supervisory Control and Data Acquisition (SCADA) software and related hardware and services for cybersecurity enhancements to the Water Division SCADA system.
PROPOSED MOTION
Approve exception to formal bid process, authorize Informal Bid Procedure and award $256,775.26 contract to Glass Box Technology, Inc. for hardware and software procurement and installation for City Water SCADA System Cybersecurity Upgrade Project.
Authorize Director of Administrative Services, or designee, to issue the following purchase orders and authorize City Manager, or designee, to execute and administer contract, in a form approved by the City Attorney. $15,176.31 to Paladin Technologies, Inc. for cabinetry procurement and installation $11,258.51 to Avidex Industries, LLC. for audiovisual equipment procurement and installation $4,663.70 to Wachter, Inc. for security system procurement and installation $34,975.00 to Fullerton Electric Co. for electrical work installation $95,500.00 to XV Solutions for professional programming services.
Authorize Director of Public Works, or designee, to approve change orders for additional support services within the approved project budget.
ALTERNATIVE OPTIONS
Approve the Proposed Motion
Other options brought by City Council.
STAFF RECOMMENDATION
Staff recommends the Proposed Motion.
CITY MANAGER REMARKS
The City Manager recommends approval.
PRIORITY POLICY STATEMENT
This item matches the following Priority Policy Statement:
Infrastructure and City Assets.
FISCAL IMPACT
City Council approved budget appropriation for the SCADA Hardware Upgrade Project in the Fiscal Year 2024-25 Adopted Capital Improvement Program Budget under Project 53781 Water System SCADA Upgrade in the Water Fund (Fund 44). Staff requests City Council approve retaining services and purchasing software and hardware for the SCADA upgrades. The following breaks down the costs:
Description
Vendor
Cost
Procurement and Installation
Glass Box Technology, Inc.
$19,250.00
Hardware
Glass Box through
Dell Technologies, Inc.
$237,525.26
Cabinetry
Paladin Technologies, Inc.
$15,176.31
Audiovisual Equipment
Avidex Industries, LLC.
$11,258.51
Security Access
Wachter, Inc.
$4,663.70
Electrical Upgrades
Fullerton Electric Co.
$34,975.00
Programming Services
XV Solutions
$95,500.00
Contingency (10%)
$41,834.88
Total
$460,183.66
Costs include a one-time $40,529.48 for SCADA software to monitor and enhance security measures for the SCADA system. The SCADA system will require approximately $20,000 annual license costs once completed, which staff will incorporate in future Public Works operating budget within in the Water Fund (Fund 44).
BACKGROUND AND DISCUSSION
The SCADA system helps manage and monitor water infrastructure, making it an essential component of public safety and resource management. Increasing sophistication and frequency of cyberattacks on public utilities necessitate robust cybersecurity measures as part of the proposed upgrade. The City installed the current SCADA system in 2008 and has not made significant upgrades, leaving it vulnerable to cyber threats. Legacy systems often lack modern encryption, secure access controls, malware protection and integration with updated cybersecurity protocols, making them attractive targets for cybercriminals.
Recent incidents across the United States highlight the risks associated with outdated SCADA systems:
Cucamonga Valley Water District Ransomware Attack (2024): A ransomware attack paralyzed the District’s phone system and prevented customers from making phone payments and potentially putting their customer’s account and payment information at risk.
American Water Cyberattack (2024): The largest investor-owned water utility in the U.S., which serves over 14 million customers in 14 states including California, was the victim of a cyberattack that resulted in unauthorized activity including disruptions to the utilities computer networks and billing systems.
Arkansas City, Kansas Cyberattack (2024): This municipal water system was the victim of a recent cyberattack, which resulted in the utility having to switch its water treatment plant from automated to manual operation for several days.
Camrosa Water District Ransomware Attack (2020): This California Water Agency was the victim of a cyberattack upon its file servers that resulted in the attacker gaining access to customers and employees names, social security numbers and checking and savings account information.
The proposed project includes new servers, supporting hardware, software programming and electrical upgrade installation.
The unique and proprietary nature of the SCADA hardware and software means only specific vendors meet stringent City cybersecurity and interoperability requirements. Staff determined the necessity to secure hardware and services as soon as possible due to the exigent nature and security risks posed to the City SCADA system. Staff does not recommend a competitive, formal bidding process due to the counterproductive amount of time it would take and it would not yield viable alternative solutions. Moreover, the increasing threat landscape targeting public infrastructure makes it essential to expedite the procurement process to implement enhanced cybersecurity measures without delay. Staff requests approval of an Informal Bid Procedure to perform Cybersecurity upgrades on the SCADA System project.
City staff coordinated with Glass Box Technology, Inc. to procure project hardware and provide professional services to standardize equipment citywide for consistent support and cost efficiency (Attachment 1). The City obtained three quotes through an informal bid process with Glass Box through Dell Technologies, Inc. providing the lowest costs at $237,525.26 (Attachment 2). Staff reviewed this quote and found it acceptable to award GlassBox to purchase hardware and services for this project.
Staff received additional quotes from vendors including $15,176.31 from Paladin Technologies, Inc. for cabinetry to house the new equipment (Attachment 3), $11,258.51 from Avidex Industries, LLC. for audiovisual equipment to operate the SCADA system (Attachment 4) and $4,663.70 from Wachter, Inc. to expand the citywide security system to the server room (Attachment 5). Staff requires this hardware and installation work to complete the project and procure in adherence to City purchasing procedures.
The server room will require electrical upgrades to power new hardware, including upgrading service from 110 Volt to 240 Volt. The City negotiated a $34,975.00 quote with Fullerton Electric Co. to perform these services pursuant to the City public projects procurement policy (Attachment 6).
City Council approved a Professional Services Agreement with XV Solutions on September 19, 2023 for On-Call Water Engineering Consultant Services. XV Solutions has extensive experience with the SCADA system. XV Solutions provided a cost proposal to provide programming services to integrate the new hardware into the existing system. XV Solutions specializes in improving SCADA cybersecurity to conform with industry standard guidelines adopted by the American Water Works Association. The scope includes incorporating advanced encryption capabilities, secure communication protocols and hardware with robust access control features (Attachment 7).
Adhering to these guidelines ensures the City can safeguard critical infrastructure from potential cyber threats and comply with national water system security requirements.
Some of the software procured for this upgrade will have annual license fees, specifically the AVEVA and VMware software. AVEVA allows staff to automatically monitor and control pumps, wells, tanks and other system demands in real time. VMware allows SCADA virtualization which mimics a physical server by running a virtual environment allowing for more efficient backups. The City would pay the first three years of the $40,529.48 AVEVA subscription license upfront and the approximately $14,000 subsequent years costs annually. The VMware annual subscription license fees cost approximately $5,500. Staff will include these ongoing operations expenses annual operations and maintenance budgets items.
Attachments:
Attachment 1 – Glass Box Technology, Inc. Proposal
Attachment 2 – Glass Box Technology, Inc. Quote for Equipment Attachment 3 – Paladin Technologies, Inc. Quote
Attachment 4 – Avidex Industries, LLC. Quote
Attachment 5 – Wachter, Inc. Quote
Attachment 6 – Fullerton Electric Co. Quote
Attachment 7 – XV Solutions Proposal & Professional Service Agreement cc: City Manager Eric J. Levitt
Water SCADA System Cybersecurity Upgrades
February 4, 2025 - Page 2