3 2
3 0
City Council
Staff Report
From: City Manager
Report Type: INFORMATION REPORT
Lead Department: Utilities
Meeting Date: December 18, 2023
Staff Report:2309-2084
TITLE
Informational Update on the Palo Alto Fiber Expansion Plan and Construction Alignment with Electric Grid Modernization
RECOMMENDATION
This memorandum and presentation are for informational purposes only; no action is requested at this time. Staff will return to Council in Q1 Calendar Year 2024 after the preliminary CEQA initial study is completed.
EXECUTIVE SUMMARY
Recently, staff presented to both the Finance Committee 1 and the Utilities Advisory Commission2, providing updates on the City’s efforts to both upgrade the City’s overhead lines in Palo Alto through electric grid modernization and piloting the build out of Palo Alto Fiber to the home. This report summarizes the staff work needed to advance these two efforts, which seek to support a reliable, sustainable and connected community.
When complete, residents can advantage of locally owned internet services in Palo Alto and use new whole home electrification technologies with ease like vehicle charging, home appliance electrification, solar energy generation and battery storage. Coupling these two efforts together saves money and time. It reduces community disruption by upgrading aging infrastructure that will also improve electric reliability and enhance safety.
Advancing the City Council direction from December 2022, staff is proceeding with the Fiber Expansion Plan to implement the Fiber Rebuild project and Phase 1 of the Fiber-to-the-Premises (FTTP) project. In Phase 1, FTTP would be built out in selected areas of the city, and expanded 1 November 7, 2023 Finance Committee Agenda
https://cityofpaloalto.primegov.com/Portal/Meeting?meetingTemplateId=13231 2 September 6, 2023 Utilities Advisory Commission Agenda https://cityofpaloalto.primegov.com/Portal/viewer?id=2948&type=0
3 2
3 0
gradually (Staff Report ID 14800)3. Construction of the fiber backbone in the Fiber Rebuild project and last mile infrastructure to provide FTTP broadband internet to the community in the FTTP project will be a significant undertaking for the City.
Priorities on the work ahead seek to manage project costs, minimize construction impacts to the community, and prevent major delays.
In summary, staff are advancing the following activities:
✓Pilot to evaluate alignment of fiber and grid modernization projects - Engineering make- ready work and construction for the grid modernization project will overlap with projects in the Fiber Expansion Plan. As reviewed with the Finance Committee and Utilities Advisory Commission, a pilot area has been identified to help inform on how alignment efforts impact costs, reductions to community disruptions, and project timelines. ✓California Environmental Quality Act (CEQA) - The City must analyze and evaluate the potential impacts of the project on various environmental factors and identify whether those impacts can be mitigated.
✓Joint Poles - The existing joint pole agreement with AT&T requires coordination between the City and AT&T to relocate 3 rd party telecom equipment on utility poles and provide space for new fiber attachments, while remaining in compliance with California Public Utilities Commission General Order 95 (GO 95) requirements for overhead electric line construction to insure electric utility service and secure personnel safety. ✓Contract amendments – Evaluation of the use of existing construction and engineering design contracts to expedite these for both FTTP and grid modernization for the pilot.
Existing contractors have the technical expertise and familiarity with the City’s construction standards, so construction may begin as soon as the power engineering design is complete.
Staff will return with specific actions associated with the efforts outlined to align the fiber expansion and electric grid modernization projects.
BACKGROUND
City Council considered three courses of action for a City-owned FTTP service and directed staff to proceed with the phased build approach. Under this approach, FTTP will be built out in selected areas of the city using $34 million from the Fiber Fund and $13 million from the Electric Fund, and the project will expand gradually from there to eliminate the need for debt financing.
Building the fiber backbone and last-mile infrastructure to provide FTTP broadband internet to the community will require significant planning, coordination, communication, and construction over the next 18 – 24 months.
3 Staff Report ID 14800 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/2022/12-19-2022-id-14800.pdf 3 2
3 0
On May 1, 2023, City Council approved Amendment Number 4 to Contract Number C20176363 with Magellan for Fiber program management, organizational change management, network operations and technical support, and utility pole electric make-ready engineering ( Staff Report 2303-12154). Although the phased build approach to FTTP will decrease the City’s financial risk and increase Council’s control over the velocity of the buildout, the City does not have the in- house staffing to fully pursue the Fiber Expansion Plan and efforts to coordinate with Electrification Grid Modernization. These programs require significant staffing and specialized skill sets over the next few years. While it is economically prudent to utilize available staffing resources, the City must also strategically invest in external resources for a successful roll-out.
On June 19, 2023, the City Council approved the FY 2024 CIP Budget with the new FTTP project, and Grid Modernization for Electrification Project. The approval of the electrification project accelerated efforts to align electrification and fiber construction, which impacted the Fiber Expansion Plan. Staff is deploying a pilot to determine how to align the grid modernization and the Fiber Expansion Plan to help minimize utility engineering pole make-ready work, pole replacements, noise disruption, and construction activity in neighborhoods.
Related actions to advance these efforts is noted in the Analysis section of this report.
ANALYSIS
Recently, presentations on the fiber to the premise and grid modernization projects including a pilot area staff are exploring to evaluate alignment of these projects were provided with the Utility Advisory Commission (UAC) and the Finance Committee. Below is a summary of the status and work underway that was discussed.
Align fiber and grid modernization projects
The City conducted a high-level electrification study to evaluate the impacts of projected electrification loads on Palo Alto’s distribution and substation transformers, primary/secondary distribution circuits, and to propose upgrades needed to mitigate overloads. The estimated cost to construct the necessary electric system upgrades for a 100% electrification scenario is between $220 million to $306 million, depending on the approach. The workplan for this project is a staged, multi-year approached designed to accomplish the upgrade of the electric distribution system to meet the City’s goal of being ready for full-scale electrification by 2030.
A pilot project is in progress to convert a 4kV area to 12 kV and upgrade the transformers and secondary networks to facilitate electrification. This work is expected to be completed by the middle of CY 2024, depending on other factors which impact project timelines, such as material lead times, pole make ready, and alignment with the fiber projects. The initial purpose of the pilot was to determine the feasibility of electrification design and construction methods and implement best practices to facilitate the most cost-effective deployment of resources. This 4 Staff Report 2303-1215 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/2023/05-01-2023-id-2303-1215.pdf 3 2
3 0
purpose has since been expanded to also test aligning the Fiber Expansion Plan with electrification efforts. Staff will be analyzing the cost savings, timeline, and resources (staffing and contractors) required for project alignment.
The design of the pilot area for the grid modernization (West Bayshore, Embarcadero Road, Louis Road, Colorado Ave) has been in development for several months because this is one of the remaining neighborhoods with a 4kV distribution system. About 80% of the City’s entire distribution has been converted from 4kV to 12kV to date. This conversion is necessary so customers can be switched from one circuit to another during emergencies. The recommended phase one area for FTTP and grid modernization will convert about 10% of the remaining 4kV systems in the City. The first fiber hut required for FTTP will be located near the Colorado power station. Based on the vicinity of the hut, predominantly aerial construction in the grid modernization pilot area, and timing of the CEQA study, there was an opportunity to align FTTP with grid modernization. FTTP build out must be constructed in a manner that provides a pathway from the fiber hut to the areas of higher interest.
Given the hundreds of miles of overhead and underground construction activity between FTTP and electric grid modernization, the pilot enables exploration of various construction phasing options to minimize disruption, construction activity, and construction costs within neighborhoods while avoiding prolonged deployments. In addition, aligning these projects in the pilot helps alleviate construction constraints and internal resource constraints to project manage, perform engineering make ready, and inspect construction for both projects in parallel.
California Environmental Quality Act (CEQA) Compliance
CPAU has retained a CEQA consultant who is preparing an Initial Study on the environmental impacts associated with the construction and operation of the potential citywide FTTP network, will cross much of the City of Palo Alto and include some new infrastructure (i.e. fiber hut, fiber cabinets, aerial and underground cables, utility vaults). The Initial Study is estimated to take about 30 weeks to complete, including internal and mandated review time such as 30 days for public review and comment. The preliminary assessment will be available to staff for review and comments by end of Dec 2023.
The preliminary assessment, or Initial Study, will evaluate the project’s potential impacts on various environmental factors and identify whether those impacts can be mitigated. If the project’s impacts can be mitigated to a less-than-significant level, then the City may issue a Mitigated Negative Declaration (MND) or Negative Declaration (ND). If the preliminary assessment of the project determines one or more significant impacts on the environment despite mitigation, a full environmental impact report (EIR) will be required, involving a more comprehensive and detailed analysis of the project’s impacts and alternatives.
The CEQA consultant is preparing a CEQA Initial Study in accordance with local, state, and federal statutes. The Initial Study identifies physical, biological, social and economic factors that might be affected by the fiber expansion project. Some environmental factors that could potentially be 3 2
3 0
affected by this project include aesthetics, air quality, geology/soils, hazardous materials, noise, utilities services, and hydrology/water quality. Specific examples of likely CEQA analysis include:
•Hydrological analysis to see if any proposed fiber lines would run through areas near bodies of water and show that there would not be significant impacts to waters of the state or waters of the US, as well as provide recommendations for how to protect nearby waterbodies even if there are no direct impacts (e.g. through proper stormwater protection measures)
•Air quality analysis (including understanding how to run claimed calcs for the proposed project to evaluate air quality emissions and GHG emissions) •Noise modeling (including analysis of potential impacts on nearby sensitive receptors from construction noise and vibrations)
•Hazardous materials analysis (including evaluation and mitigation if necessary, of potential impacts to construction workers and sensitive receptors especially if any of the dark fiber runs through areas of the city runs through a plume) Replacement of Jointly-Owned Poles
The City jointly owns 5,550 poles with AT&T. The current pole intent process (including billing) for managing joint pole replacements operates under a joint pole agreement executed between the City and AT&T in 1918. This process may not be feasible to accommodate the increased volume of pole transactions expected with the grid modernization and Fiber Expansion Plan.
CPAU began discussing with AT&T effective ways to coordinate and collaborate on the anticipated surge in pole replacement work and mitigating the impact to the community. One potential improvement is to identify third party resources authorized to work on both AT&T and CPAU facilities during pole replacement, make ready work, and/or inspection.
As part of the effort to streamline processes, the benefits of joining the Northern California Joint Pole Association (NCJPA) was evaluated. Members have “joint pole equity” and standardized cost-sharing methodologies for pole ownership, maintenance, use, setting, replacement, dismantling, relinquishment or removal of jointly owned poles. CPAU is not an NCJPA member as the volume of pole replacements were manageable under the existing process. The main objectives for considering membership with the NCJPA were to a) streamline the pole intent and billing process; and b) improve the process of recovering AT&T’s costs for joint poles replaced by CPAU. After review, membership is not recommended as the objectives for membership would not be achieved. For example, the current joint agreement with AT&T requires a response within 10 days vs. 45 days according to the NCJPA handbook, and the current agreement is based on actual replacement costs while NCPJA joint pole costs are based on average unit costs for all members. The City is considering amending the joint agreement with AT&T so the pole intent process can support the anticipated high volume of pole replacement.
Design and Construction Contracts for Pilot Area
Given the anticipated increase in pole work, CPAU will need to add resources to manage and perform a high volume of pole make-ready work and replacements. In the meantime, staff is evaluating how to leverage existing on-call construction and engineering contracts to perform 3 2
3 0
the engineering design, overhead and underground construction, and installation for the pilot area (approximately 409 poles and 1,224 homes). This will reduce the construction timeline by 6 – 9 months, which would otherwise be prolonged in the process of issuing multiple invitation for bids (IFBs) to perform the work for FTTP and grid modernization. The pilot area will be treated as a testbed to design and construct both projects in parallel to minimize community disruption and reduce shared construction costs. The pilot is intended to inform CPAU how to most effectively bid out the engineering, construction and installation services required for FTTP and grid modernization for the remaining phase 1 area (additional 1,241 poles and 5,560 customers). •VIP Powerline provides electric construction services including pole replacements, facilities upgrade on the electric distribution system, and customer connections. VIP also has some fiber construction experience such as installation of fiber cable and messenger wires. (Staff Report 2303-1119)5
•MP Nexlevel provides electric and fiber substructure installation and trenching services.
MP Nexlevel is also qualified to provide fiber construction and installation services including fiber installation, splicing, and testing. (Staff Report 13953)6 •Magellan provides multiple facets of implementing municipal fiber and broadband networks including business planning, engineering, program management, integration, and operations. In 2022, Magellan was acquired by Entrust Solutions, an experienced provider of engineering, consulting, data analytics, and automation services to the electric, gas, and telecom markets. (Staff Report 2303-1215)7 •Davey Tree provides clearing of trees and other vegetation from the overhead electric distribution system. Clearing vegetation from power lines is an on-going operation necessary to ensure the City provides safe, reliable power. (Staff Report 10883)8 Work is underway to identify the specific bill of materials (BOM) for electric and fiber specifically in the pilot area. Once complete, it is expected that this will be provided to current on-call construction and engineering firms for quotes. Ultimately, staff will bring forward appropriate recommended contract amendments for the additional scope of work including revised not-to- exceed financial limits.
Other Updates
Fiber Make-Ready Engineering
5 Staff Report 2303-1119 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/2023/04-10-2023-id-2303-1119.pdf 6 Staff Report 13953 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/2022/03-21-22-id-13953.pdf 7 Staff Report 2303-1215 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/2023/05-01-2023-id-2303-1215.pdf 8 Staff Report 10883 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes- reports/reports/city-manager-reports-cmrs/year-archive/2020/id-10883.pdf 3 2
3 0
Magellan completed the survey of approximately 6,000 utility poles in Palo Alto. The survey identified the current load and condition of each pole in preparation for the new fiber backbone and FTTP. The survey included detailed fielding and walk-out of all routes to validate running lines, existing utilities, and constructability. For aerial construction, Magellan provided make- ready engineering pole data to determine pole preparation and pole replacement, as well as timeframes for the work to be completed. For phase one of FTTP and grid modernization, CPAU will be passing 1,650 existing poles, attaching fiber to 1,300 poles, replacing 200 to 300 poles, and coordinating with third parties to move their telecom equipment lower on 325 poles. In addition to the fiber expansion project, CPAU has a recurring Wood Pole Replacement capital improvement project to replace approximately 100 deteriorated wood poles annually. Wood poles are used to support overhead utility lines throughout the City of Palo Alto. Though poles are inspected, tested, and treated to maintain their integrity, over time poles will deteriorate to the point that they are no longer in compliance with GO 95 requirements or safe for community and utility workers.
Fiber Hut Sites:
Fiber huts are usually a prefabricated building (10’ x 20’) containing electronics and network equipment that connects fiber to neighborhoods. The hut also has backup generators and HVAC systems to maintain equipment within its operating environmental specifications. Two fiber huts are recommended for the citywide network for Palo Alto Fiber. Staff and Magellan evaluated potential fiber hut sites and considered their proximity to strategic areas, planning/land use requirements, and existing infrastructure. As a result, the potential locations were narrowed to the Colorado substation and anticipated space at City Hall. The Colorado substation was previously identified as an ideal location due to its proximity to the area it serves and its current use for utilities. The Equinix data center in Palo Alto will serve as the second fiber hut site. City Hall was a potential option but Equinix provides a superior alternative due to timing, cost, construction, and future flexibility considerations.
Equinix Fiber Data Center:
Equinix is currently the hub for the City’s dark fiber service. The Equinix datacenter is vendor- neutral site where numerous internet service providers and content delivery networks can interconnect and exchange internet traffic between their networks. As the City expands its fiber footprint, more data center space will be needed. On November 27, 2023, Council approved a new license agreement with Equinix for a three-year term to secure space for the infrastructure required to operate the fiber and ISP for FTTP ( Staff Report 2310-2129) 9. A private cage will provide a secure space that meets the requirements to host the data center and fiber hut for FTTP.
9 https://www.cityofpaloalto.org/files/assets/public/v/1/agendas-minutes-reports/reports/city-manager-reports- cmrs/2023/11-27-23-id-2310-2129-mini-packet.pdf
3 2
3 0
Invitation for Bid (IFB) Construction Package:
Upon completion of the pilot, should opportunities arise to align FTTP and grid modernization, a public and open procurement process (IFB) for the construction of FTTP and grid modernization is expected. The consolidated IFB package will include construction-ready prints for fiber and power, construction details, splice details, pole make-ready details, construction standards as well as all documentation needed to complete the bid package. This shall include a written summary and scope of work, estimated costs to construct, as well as labor estimates, bill of materials, and the vendor list.
Staffing:
As the City solidifies staffing plans in parallel with contracted services, the Director of Information Technology will take on the responsibilities of an Assistant Director for Palo Alto Fiber on an interim basis. In the FY 2024 adopted budget, four (4) new FTE positions for the dark fiber expansion and implementation of FTTP were created. These positions will be recruited and filled as needed during the various stages of the project.
•Assistant Director - To provide the vision and strategy for the new fiber business and lead a high-performance team to quickly execute. Responsibilities include and are not limited to: overseeing fiber enterprise fund budget timelines and milestones (including the dark fiber optic business), managing the roll out and expansion of the fiber optic network, Internet Service Provider (ISP) operations, customer service operations, and business development.
•Manager Utilities Telecommunications / "Outside Plant Manager" - To lead the construction process, installation and repair service technicians, and ensure the outside plant processes run efficiently and smoothly. Responsibilities include and are not limited to: overseeing construction, managing engineers and contractors, achieving construction budget timelines and milestones.
•Manager Information Technology / "Sr. Network Engineer/Architect" – To lead the development process of the overall architecture of the broadband system and manage the system. Responsibilities include and are not limited to: managing the network and supporting technical needs across the organization.
•Manager Utilities Telecommunications / "Sales and Marketing Manager" - To develop the marketing strategy and lead marketing initiatives. Responsibilities include and are not limited to: meeting revenue objectives, cultivating strong partnerships, and maintaining retention rates.
FISCAL/RESOURCE IMPACT
This report is for informational purposes so there is no resource impact. Based on Council and UAC input, staff will return with specific actions associated with efforts to align the fiber expansion and electric grid modernization projects.
3 2
3 0
STAKEHOLDER ENGAGEMENT
Staff continues to engage with the community on these two efforts primarily through the City’s Finance Committee and Utilities Advisory Commission discussions which are open to the public to participate and share feedback.
On November 7, 2023, the Finance Committee was updated on the Fiber Master Plan, which includes a pilot to align FTTP with the Electric grid modernization project. The Finance Committee provided feedback on timeline, costs, and alignment of FTTP to grid modernization.
In addition, staff continues to share updates through the Palo Alto Fiber project webpage10, City communications channels such as social media 11, Medium.com12 blog and interactive Palo Alto Fiber Hub.
ENVIRONMENTAL REVIEW
A CEQA consultant is preparing a CEQA Initial Study for the Fiber-to-the-Premises project in accordance with the State CEQA Guidelines and the environmental regulations of the City.
ATTACHMENTS
Attachment A: Finance Committee Presentation
APPROVED BY:
Dean Batchelor, Director of Utilities
Staff: Dave Yuan, Strategic Business Manager
10 Palo Alto Fiber Webpage, https://www.cityofpaloalto.org/paloaltofiber 11 City Social Media Channels, www.cityofpaloalto.org/connect 12 Palo Alto Connect, https://medium.com/@paloaltoconnect