{{section.number}}e
CITY OF GLENDALE, CALIFORNIA
REPORT TO THE CITY COUNCIL
AGENDA ITEM
Report: Resolution Dispensing with Competitive Bidding and Authorizing a Dark Fiber and Colocation Master Service Agreement Extension.
1. Resolution to dispense with competitive bidding and authorize the City Manager, or his designee, to amend the existing Master Service Agreement (MSA) with Crown Castle Fiber, LLC (Contract No. C106498A) for dark fiber and colocation services and extend the term by five (5) years through June 30, 2029, not to exceed the annual budget, and authorizing one five-year renewal option.
COUNCIL ACTION
Item Type: Action Item
Approved for May 21, 2024 calendar
EXECUTIVE SUMMARY
Since 2017, the City has leased dark fiber and colocation services from Crown Castle Fiber, LLC at a carrier neutral facility equipped with tier 3 mechanical and electrical infrastructure. Dark fiber and colocation services allow the City to install, connect, and maintain electronics that serve as the backbone of the City’s connectivity to the internet.
This connectivity is vital to critical City operations such as Glendale Water & Power, Public Safety Dispatch, Public Works not to mention citywide services such as Microsoft Teams, Office 365, and email. The City’s current agreement is set to expire on June 30, 2024.
Given the importance of the City’s internet connectivity to core City operations, migrating to another colocation and dark fiber service provider is simply not feasible at this time due to the operational risk and cost. Establishing co-location and dark fiber services through another provider would involve a planned overlap period during which parallel services are in place and are fully functional with both the incumbent and future service vendors.
It would be extremely costly to procure additional network equipment, in addition to the professional services the City would require to establish the connectivity, not to mention the operational risk of migrating the service. Any service interruption in connectivity would adversely and significantly impact core City services.
2{{section.number}}e
5 1
6 9
RECOMMENDATION
That the City Council approve dispensing with the competitive bidding for dark fiber and colocations services, and authorize the City Manager to execute an amendment to the existing MSA with Crown Castle extending the term through June 30, 2029, authorizing one subsequent annual five-year extension option and not to exceed the annual budget.
ANALYSIS
After solicitation of competitive pricing from three potential vendors, on May 2, 2017, the City Council authorized the City Manager to enter into a MSA with Wilcon Operations LLC to provide dark fiber and colocation services for a five-year term and a not-to-exceed contract amount of $350,900. In September 2018, Wilcon Operations LLC completed the process of provisioning of the dark fiber circuits and services commenced.
In April 2019, establishment of colocation services within the Wilcon colocation suite located in the One Wilshire building carrier hotel began. Colocation services ordered included a half rack of equipment cabinet space, dual 120-Volt 20-Amp alternating current electrical service feeds and a multimode fiber optic cross connection to the City’s primary internet service provider, Los Nettos. This work was completed in June 2019 and in accordance with the terms of the agreement, invoicing for colocation services began. During this time, it should be noted that Amendment 1 to the Master Service Agreement was executed memorializing the merger of Wilcon Holdings, LLC with Crown Castle Fiber, LLC on December 31, 2018, and consenting to assignment of the agreement pursuant thereto.
The last phase of the project was to purchase the equipment needed to establish a three-node transport network interconnecting a pair of City data center locations and the colocation suite (Crown Castle formerly Wilcon Holdings) within the One Wilshire building. After solicitation of competitive pricing from multiple potential vendors, the City issued a contract to CCI Systems, Inc for planning, installation, configuration, testing, documentation, and remote training services of the transport network.
In May 2020, Crown Castle Fiber, LLC was contracted to perform the physical installation of the equipment within the colocation suite at the One Wilshire building necessary to establish the transport network. This action was taken to address the situation created during the initial outbreak of the COVID-19 pandemic during which domestic travel was discouraged and Executive Order M-33-20 was issued ordering all individuals living in the State of California to stay home or at their place of residence except as needed to maintain continuity of operations of the federal critical infrastructure sectors.
In July 2020, Crown Castle Fiber, LLC completed the process of the physical installation of the transport network equipment within the colocation suite at the One Wilshire. And lastly, in September 2020, CCI Systems, Inc completed the planning, 3{{section.number}}e
5 1
6 9
installation, configuration, and testing of the three-node transport network and routing of traffic to/from the Internet over this network commenced.
The City has expended significant cost and resources in establishing internet connectivity at One Wilshire. Critical city operations such as GWP, Public Safety and Public Works rely on this connectivity as do citywide services such as Microsoft Teams, Office 365, and email.
Given the importance of the City’s internet connectivity to core City operations, migrating to another colocation and dark fiber service provider is simply not feasible at this time due to the operational risk and cost. Establishing co-location and dark fiber services at another provider would involve a planned overlap period during which parallel services are in place and are fully functional with both the incumbent and future service vendors. Failure to plan and execute a migration of services which utilizes a staggered overlapping service approach would impact City services in the form of an extended outage of services and applications which rely upon internet services. Electronic mail, web browsing, collaboration, online application, remote access, and video\teleconferencing services are examples of services that rely upon internet services.
Additionally, since a capital investment in network equipment is necessary and these devices have an associated anticipated useful life span, shifting dark fiber and colocation service vendors should be timed to coincide with the periodic asset refresh cycle of the underlying transport network equipment. If the network equipment asset refresh cycle is ignored, the City would need to acquire replacement transport network equipment “out of cycle” and, thereby, increasing the cost of migrating dark fiber and colocation services to an alternative vendor. Procured in 2019 and provisioned in 2020, the cost of the equipment, configuration and support services for the transport network was approximately $615,000; it would likely be significantly higher if purchased today given the increases in technology components over the last five years.
Lastly, Glendale’s Charter, Article VI, Section 9, allows City Council to dispense with competitive bidding where the City Council determines that dispensing with competitive bidding is in the best interests of the City. Since Crown Castle Fiber, LLC is currently the City’s provider for dark fiber and colocation services, and given the foregoing reasons stated relating to the existing equipment and facilities, as well as to ensure the continuity of service, it is respectfully requested and recommended that the City Council dispense with competitive bidding pursuant to Article VI, Section 9 of the Glendale Charter and extend the City’s existing MSA with Crown Castle Fiber, LLC.
By extending the contract with the existing incumbent vendor, it will provide the city with immediate ongoing access to the needed services, while maintaining the existing terms and conditions currently in effect.
4{{section.number}}e
5 1
6 9
STAKEHOLDERS/OUTREACH
Dark fiber and colocation services are essential underlying infrastructure elements which facilitate providing highly available connectivity to and routing of network traffic to and from the Internet. These services can be leveraged to provide services to the following target use cases/customers identified within the Fiber Optic Business Plan:
• Commercial Businesses
• Community Anchor Institutions, including schools and medical facilities • Utility Functions
• Municipal Government Functions
These services are actively being utilized by the City for municipal government functions and GWP for utility functions, currently. An example of the application of these services is the Wi-Fi Internet service provided to staff and the community in non- library City facilities.
No public outreach for this dark fiber and colocation services contract is expected to be needed.
FISCAL IMPACT
The dark fiber and colocation master services contract extension will cost approximately $350,000 for the initial five-year term. There is one subsequent five-year term extension option with an estimated cost of $400,000.
No new appropriation is being requested at this time. All services ordered and provided under the MSA are subject to the appropriation limits approved by Council as part of the annual budget process. For the upcoming budget year, FY 2024-25, the estimated cost of services under this agreement is $57,526.51 and is included in the proposed GWP budget.
ENVIRONMENTAL REVIEW (CEQA/NEPA)
This item is considered a ministerial activity and, therefore, not subject to CEQA review.
CAMPAIGN DISCLOSURE
The names and business addresses of the members of the board of directors, the chairperson, CEO, COO, CFO, Subcontractors and any person or entity with more than 10% interest in the company proposed for contract in this Agenda Item Report are attached in Exhibit 1, in accordance with the City Campaign Finance Ordinance No. 5744.
ALTERNATIVES TO STAFF RECOMMENDATION
Alternative 1: Not approve the recommendation to dispense with competitive bidding and authorize the extension of the MSA with Crown Castle Fiber, LLC, and direct staff to consider other vendors. Under this alternative, staff would need a two-year extension with Crown Castle to provide ample time to evaluate the cost, request and obtain funding, plan, and execute the dark fiber and colocation service vendor migration simultaneously with replacement of the transport network equipment. It is highly likely that after the cost evaluation is conducted, it would be excessively cost 5{{section.number}}e
5 1
6 9
prohibitive to undertake an “off cycle” dark fiber and colocation service vendor migration.
Alternative 2: Consider any other alternative not proposed by staff.
ADMINISTRATIVE ACTION
Submitted by:
Mark Young, General Manager – Glendale Water and Power
Jason Bradford, Director of Finance/Information Technology Prepared by:
Varghese George, P.E., Senior Electrical Engineer/Fiber Services Manager Eric Brumm, Chief Information Technology Architect
Approved by:
Roubik R. Golanian, P.E., City Manager
EXHIBITS/ATTACHMENTS
Exhibit 1: Campaign Finance Disclosure for Crown Castle Fiber, LLC