Vers.: Aug. 5, 2019
AMENDMENT NO.1 TO CONTRACT NO. C22182051
BETWEEN THE CITY OF PALO ALTO AND EPI-USE LABS, LLC
This Amendment No. 1 (this “Amendment”) to Contract No. C22182051 (the “Contract” as defined below) is entered into as of August 1, 2025 by and between the CITY OF PALO ALTO, a California chartered municipal corporation (“CITY”), and EPI-USE LABS, LLC , a Wyoming limited liability company , located at 1155 Perimeter Center West, Ste 1100, Atlanta, GA 30338 (“CONSULTANT”). CITY and CONTRACTOR are referred to collectively as the “Parties” in this Amendment.
R E C I T A L S
A. The Contract (as defined below) was entered into by and between the Parties hereto for the provision of Hosting Services in AWS (Amazon Web Services) Cloud and SAP Basis Support Services for the CITY’s ERP system, as detailed therein.
B. The Parties now wish to amend the Contract in order to extend the term for a 1- year extension, changing the expiration term date of this Amendment from December 31, 2025 to December 31, 2026; and increase compensation by $939,675, changing a total not to exceed amount from $2,214,140 to $3,153,815.
NOW, THEREFORE, in consideration of the covenants, terms, conditions, and provisions of this Amendment, the Parties agree:
SECTION 1. Definitions. The following definitions shall apply to this Amendment:
a. Contract. The term “Contract” shall mean Contract No. C22182051 between CONSULTANT and CITY, dated June 21, 2022.
b. Other Terms. Capitalized terms used and not defined in this Amendment shall have the meanings assigned to such terms in the Contract.
SECTION 2. Section 2. “TERM” of the Contract is hereby amended to read as follows:
“The term of this Agreement shall be from the date of its full execution through December 31, 2026, unless terminated earlier pursuant to Section 19 (Termination) of this Agreement.
SECTION 3. Section 4. “ NOT TO EXCEED COMPENSATION” of the Contract is hereby amended to read as follows:
“The compensation to be paid to CONSULTANT for performance of the Services shall be based on the compensation structure detailed in Exhibit C, entitled “COMPENSATION,” including any reimbursable expenses specified therein, and the maximum total compensation shall not exceed Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Two Million Eight Hundred Sixty Eight Thousand Three Hundred Ninety Dollars ($2,868,390.00).
The hourly schedule of rates, if applicable, is set out in Exhibit C-1, entitled “SCHEDULE OF RATES.” Any work performed or expenses incurred for which payment would result in a total exceeding the maximum compensation set forth in this Section 4 shall be at no cost to the CITY.
Optional Additional Services Provision (This provision applies only if checked and a not -to- exceed compensation amount for Additional Services is allocated below under this Section 4.)
In addition to the not -to-exceed compensation specified above, CITY has set aside the not -to- exceed compensation amount of Two Hundred Eighty Five Thousand Dollars ($285,425.00) for the performance of Additional Services (as defined below). The total compensation for performance of the Services, Additional Services and any reimbursable expenses specified in Exhibit C, shall not exceed Three Million One Hundred Fifty Three Thousand Eight Hundred Fifteen Dollars ($3,153,815.00), as detailed in Exhibit C.
“Additional Services” means any work that is determined by CITY to be necessary for the proper completion of the Project, but which is not included within the Scope of Services described at Exhibit A. CITY may elect to, but is not required to, authorize A dditional Services up to the maximum amount of compensation set forth for Additional Services in this Section 4.
CONSULTANT shall provide Additional Services only by advanced, written authorization from CITY as detailed in this Section. Additional Servic es, if any, shall be authorized by CITY with a Task Order assigned and authorized by CITY’s Project Manager, as identified in Section 13 (Project Management). Each Task Order shall be in substantially the same form as Exhibit A -1, entitled “PROFESSIONAL S ERVICES TASK ORDER”. Each Task Order shall contain a specific scope of services, schedule of performance and maximum compensation amount, in accordance with the provisions of this Agreement. Compensation for Additional Services shall be specified by CITY in the Task Order, based on whichever is lowest: the compensation structure set forth in Exhibit C, the hourly rates set forth in Exhibit C-1, or a negotiated lump sum.
To accept a Task Order, CONSULTANT shall sign the Task Order and return it to CITY’s Project Manager within the time specified by the Project Manager, and upon authorization by CITY (defined as counter-signature by the CITY Project Manager), the fully exec uted Task Order shall become part of this Agreement. The cumulative total compensation to CONSULTANT for all Task Orders authorized under this Agreement shall not exceed the amount of compensation set forth for Additional Services in this Section 4. CONSULTANT shall only be compensated for Additional Services performed under an authorized Task Order and only up to the maximum amount of compensation set forth for Additional Services in this Section 4. Performance of and payment for any Additional Services are subject to all requirements and restrictions in this Agreement.”
SECTION 4. Section 19. “TERMINATION OR SUSPENSION OF AGREEMENT OR SERVICES ” of the Contract is hereby amended to read as follows:
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
19.1. “The City Manager may suspend the performance of the Services, in whole or in part, or terminate this Agreement, with or without cause, by giving thirty (30) days prior written notice thereof to CONSULTANT. If CONSULTANT fails to perform any of its material obligations under this Agreement, in addition to all other remedies provided under this Agreement or at law, the City Manager may terminate this Agreement sooner upon written notice of termination. Upon receipt of any notice of suspension or termination, CONSULTANT will discontinue its performance of the Services on the effective date in the notice of suspension or termination.
19.2. In event of suspension or termination, CONSULTANT will deliver to the City Manager on or before the effective date in the notice of suspension or termination, any and all work product, as detailed in Section 14 (Intellectual Property; Data), whether or not completed, prepared by CONSULTANT or its contractors, if any, in the performance of this Agreement. Such work product is the property of CITY, as detailed in Section 14 (Intellectual Property; Data).
19.3. In event of suspension or termination, CONSULTANT will be paid for the Services rendered and work products delivered to CITY in accordance with the Scope of Services up to the effective date in the notice of suspension or termination; provided, however, if this Agreement is suspended or terminated on account of a default by CONSULTANT, CITY will be obligated to compensate CONSULTANT only for that portion of CONSULTANT’s Services provided in material conformity with this Agreement as such determination is made by the City Manager acting in the reasonable exercise of his/her discretion. The following Sections will survive any expiration or termination of this Agreement: 14, 15, 16, 17, 19.2, 19.3, 19.4, 20, 25, 27, 28, 29 and 30.
19.4. No payment, partial payment, acceptance, or partial acceptance by CITY will operate as a waiver on the part of CITY of any of its rights under this Agreement, unless made in accordance with Section 17 (Waivers).”
SECTION 5. The following exhibit(s) to the Contract is/are hereby amended or added, as indicated below, to read as set forth in the attachment(s) to this Amendment, which is/are hereby incorporated in full into this Amendment and into the Contract by this reference:
a. Exhibit “A” entitled “SCOPE OF SERVICES ”, AMENDED, REPLACES PREVIOUS.
b. Exhibit “B” entitled “SCHEDULE OF PERFORMANCE”, AMENDED, REPLACES PREVIOUS.
c. Exhibit “C” entitled “COMPENSATION”, AMENDED, REPLACES PREVIOUS.
SECTION 6. Legal Effect. Except as modified by this Amendment, all other provisions of the Contract, including any exhibits thereto, shall remain in full force and effect.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
SECTION 7. Incorporation of Recitals . The recitals set forth above are terms of this Amendment and are fully incorporated herein by this reference.
(SIGNATURE BLOCK FOLLOWS ON THE NEXT PAGE.)
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
SIGNATURES OF THE PARTIES
IN WITNESS WHEREOF, the Parties have by their duly authorized representatives executed this Amendment effective as of the date first above written.
CITY OF PALO ALTO
City Manager)
APPROVED AS TO FORM:
City Attorney or Designee
(Contract over $25k)
EPI-USE LABS, LLC
Officer 1
By:
Name:
Title:
Officer 2 (Required for Corp. or LLC)
By:
Name:
Title:
Attachments:
▪ EXHIBIT A – SCOPE OF SERVICES, AMENDMENT NO.1 (AMENDED, REPLACES PREVIOUS) ▪ EXHIBIT B – SCHEDULE OF PERFORMANCE, AMENDMENT NO.1 (AMENDED, REPLACES
PREVIOUS)
▪ EXHIBIT C – COMPENSATION, AMENDMENT NO.1 (AMENDED, REPLACES PREVIOUS)
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 VP Services, Americas
Lyall Hinton
Kevin Mukheibir
Global TechOps Lead
Vers.: Aug. 5, 2019
EXHIBIT A
SCOPE OF SERVICES, AMENDMENT NO.1
(AMENDED, REPLACES PREVIOUS)
The CONSULTANT will provide Hosting Services in AWS Cloud and SAP Basis Support Services for the City’s ERP system, including the Services detailed below in this Exhibit A, entitled “SCOPE OF SERVICES.”
SCOPE OF SERVICES – OVERVIEW
This scope of services overview summarizes at a high level the Services to be provided, broken out into Tasks 1-5, which carry over to Exhibit B (Schedule of Performance) and Exhibit C (Compensation). Following this overview, the scope of services is provided in greater detail.
A. (TASK 1) Transfer to CONSULTANT the CITY’s existing Amazon Web Services (AWS) hosting account in AWS cloud, Amendment No.1 (Completed) CONSULTANT will transfer to CONSULTANT the CITY’s existing Amazon Web Services (AWS) hosting account for the CITY’s ERP system including but not limited to the following activities:
(1) Perform account transfer process in AWS cloud to CONSULTANT from CITY’s current third-party provider to enable the CONSULTANT to assume hosting and management and other services as detailed in this Agreement.
B. (TASK 2) Maintain account with AWS for hosting City’s infrastructure CONSULTANT will maintain the CITY’s account with AWS for hosting the CITY’s ERP system, including but not limited to the following account activities:
(1) Pay annual hosting fees to AWS for maintaining City’s infrastructure.
C. (TASK 3) Perform hosting and networking maintenance services CONSULTANT will provide hosting and maintenance services including but not limited to the following:
(1) Ensure uptime and security of all SAP related servers. (2) Offer Operating System support (SUSE Linux and Windows) on an as needed basis. (3) Offer Networking support on an as needed basis (e.g. opening of firewall ports, etc.). (4) Share AWS pricing details (including AWS billing invoices per month) for all hosted AWS services and help City realize any pricing discounts as and when offered by AWS.
(5) Share Cloudwatch report on a monthly basis and latest resource consumption. (6) Share AWS hosting bill on a monthly basis.
(7) Manage and configure firewall in AWS for Direct Connect and Internet access from AWS.
(8) Manage and configure Site to Site and Client to Site VPN for AWS access. (9) Ensure designated City staff members have full read access to the AWS administrator console for auditing and troubleshooting purposes.
(10) Help with Security Penetration testing services as and when requested.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
(11) Provide documentation and cross-training to City’s Operations staff as requested.
D. (TASK 4) Perform SAP Basis Services
CONSULTANT will provide SAP Basis Services including but not limited to the following:
(1) Ensure all SAP systems are available in AWS cloud and performing as per SLA. (2) Perform backups regularly [Server level and Database level]. (3) Restore servers and/or databases when necessary [All SAP servers are running on HANA database; one interface application servers are running on MySQL]. (4) Apply SAP Kernel patches as necessary [Once a quarter or six months]. (5) Apply Database patches as necessary [Once a year or so]. (6) Apply SAP HR legal patches and other required support packs/stacks as necessary [Usually once a year].
(7) Monitor SAP applications and resources using SAP Solution Manager [Solution Manager is already configured].
(8) Help with performance tuning of SAP application and/or databases. (9) Provide documentation and cross-training to City’s Basis staff as requested. (10) Provide access to City’s Basis staff at OS and Database level as needed basis and consider them as an extension to your own Basis team.
(11) Share resumes of all potential Basis resources that support City’s SAP environment.
E. (TASK 5 – Additional Services) Perform ad-hoc services on as needed requests that are not covered in the Scope of Services. These services constitute Additional Services under Section 4 (Not to Exceed Compensation) of this Agreement, implemented pursuant to City- approved Task Order as detailed in Section 4. Examples of such Additional Services include without limitation:
(1) Perform ad-hoc (Additional Services per Section 4) hosting and networking services such as enabling additional AWS services, building new application and/or database servers that are not included in the Scope of Services.
(2) Perform ad-hoc (Additional Services per Section 4) Basis services such as installation and configuration of new SAP applications and SAP HANA databases that are not included in the Scope of Services.
(3) Perform other ad hoc (Additional Services per Section 4) services that are not included in the Scope of Services as may be requested by CITY.
(4) Any Managed Services specifically requested by the City of Palo Alto outside normal business hours (except for P1 incidents and in-scope Managed Services production system work).
(THIS SCOPE OF SERVICES IS CONTINUED ON THE NEXT PAGE.)
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
SCOPE OF SERVICES – IN DETAIL
1. Key Contacts
CONSULTANT’s Key Contacts for this Agreement are as follows below. Day-to-day communications regarding implementation of the Agreement are to be directed to the appropriate Key Contact, as detailed in Section 13 (Project Management) of the Agreement.
Account Manager:
Lyall Hinton
Americas Services
+1 415 343 5368
[email protected]
Basis Service Delivery Manager:
Arno Groenewald
Service Delivery Manager
+1 404 528 4347
[email protected]
AWS (Cloud) Service Delivery Manager:
Sethu Manuel
Service Delivery Manager
+91 949 682 4501
[email protected]
Address:
2002 Summit Blvd,
#300 Atlanta,
Georgia 30319
2. Managed Services Overview
The purpose of this Scope of Services is to communicate the objectives, deliverables, and activities to carry out this Scope of Services for the provision of AWS Cloud and SAP Basis Support Services (also referred to herein as the “Managed Services”) for the City’s SAP systems, as defined in Section 14 (Intellectual Property; Data) of this Agreement.
The City SAP landscape is used to deliver business critical functions, and it is pivotal that these systems remain reliable and accurate. The Managed Services delivery model is core in supporting such services, which include:
• Cloud services (AWS), which include platform and infrastructure services. • Security services providing network and application security protocols to secure and protect valuable personal and City-specific information.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
• Basis services with specialized SAP platform maintenance services.
By using EPI-USE developed IP and products, the automated software applications provide the Managed Services team with the ability to proactively monitor, alert and resolve issues before they can cause delays to business-critical functions.
The service delivery framework is made possible with a variety of tasks occurring during a calendar year ranging from daily, weekly, monthly, quarterly, bi-annually, and annual activities. A mixture of run-the- services, proactive support, advisory services, and a centralized ticketing environment ensures this continuity can be achieved. These activities form part of the overall Managed Services solution provided to the City.
To highlight a few:
• 24/7 support, as remote services.
• Daily checks, monitoring, alerts, proactive maintenance and housekeeping. • Incident management.
• Service level agreement (SLA) tracking.
• Operating system, database and SAP system support packs and patching. • System refreshes to non-production systems; and
• Client Central for document repository.
To carry out this service, EPI-USE Labs will have the responsibility to determine and manage all qualified and skilled resources used to provide the solution as a service to the City. EPI-USE shall provide the City with Managed Services as outlined in this Scope of Services. On completion of a transition and onboarding period, EPI-USE will commence with the Managed Services offering.
Figure 1:EPI-USE Labs Managed Services Scope of Services for the City
3. Operating Model
EPI-USE Managed Services
Application Managed Services
Basis Managed Services
Security Managed Services
AWS Managed Services
Continuous
Improvement
Upgrades
Patches
Monitor
SLA
Resolve
Advisory
Self-Service
EPI-USE IP (Process, Product, People) as a Foundation
Technical Managed Services
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
The EPI-USE Managed Services offering provides basis and platform solution coverage across the SAP areas as depicted in Figure 1. The operating model centers around allocating a dedicated single point of contact (SPOC – Service Delivery Managers) per area (Cloud and Basis) to ensure continuity in communication and solution delivery standards. The team structure will consist of a variety of lead functions to manage architecture, risk, basis, and platform streams. Additionally, the team has been structured in an agile manner, ensuring the required delivery capability is readily available through peak periods.
To ensure continuity with minimal impact, the Managed Services team members will be selected from the large pool of EPI-USE Labs consultants who will become familiar with the City processes and systems.
Some of the main priorities of the Managed Services model will be on monitoring, housekeeping and incident management, governed by Service Level Agreements (SLAs). Incidents, changes, and requests will be raised by CITY with EPI-USE Labs via the EPI-USE ticketing system, Client Central.
The EPI-USE Support team will assess incidents quickly and recommend approaches to incident resolution or propose changes to the system.
Standby and after-hours support will be provided for production system emergencies and tasks that cannot be executed during normal business hours. An outline of Roles and Responsibilities is outlined in Addendum B, as some activities will be carried out by the City’s internal basis team as detailed therein. Furthermore, SAP basis administration on the Sandbox will be carried out by the City’s internal team, while at the same time, EPI-USE Labs will encourage knowledge transfer activities to the City’s internal Basis team.
3.1 Service Operating hours
Normal service operating hours relating to normal business hours is detailed below. This excludes US federal holidays.
Monday to Friday: 08:00 to 17:00 US Pacific Time
3.2 Governance
The governance structure for EPI-USE’s service delivery will include all stakeholders from EPI-USE and City’s IT department (or other staff as needed) in the service delivery model, along with an escalation process, which will be part of the Managed Services engagement itself. Monthly service review meetings (also referred to as steering committee meetings) will be held to monitor and share information about service delivery, for example, about performance, tickets, incidents, and the status of responses to incidents. The three escalation levels are as follows:
• Level 1 – Service Manager (also referred to as Service Delivery Manager) • Level 2 – Account Executive; and
• Level 3 – Executive Sponsor.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
The Service Delivery Manager will resolve most escalations as that role encompasses an overview of all the activities for the City under this Agreement. Issues the Service Delivery Manager cannot resolve will be referred up the escalation structure for resolution at a higher level.
The Service Delivery Manager will be the owner of the monthly service reviews and sit on the steering committee for monthly service reviews. Any additional scope (also referred to herein as Additional Services per Section 4 of this Agreement) items to this Agreement will be outlined and a way forward will be discussed in these meetings, to be implemented in accordance with the provisions of this Agreement, for example, pursuant to a Task Order per Section 4 or a contract amendment per Section
29.4. 3.2.1 Governance and Review reporting
The following reports will be provided at the monthly service review meeting, or when required:
• Platform performance metrics.
• End-user response times and trends.
• Network statistics.
• CPU performance.
• System uptime.
• Incident SLA reports.
• Total number of tickets handled (both AWS- and SAP-related) • Storage and usage.
• Backup/DR reporting; and
• AV reporting.
3.3 Service Levels (or “Service Level Agreements” or “SLAs”) EPI-USE will adhere to the Service Levels detailed below, providing timely resolution of incidents and technical issues, and ensuring that the City’s SAP enterprise resource planning (ERP) system continues to deliver efficient services to the City’ business units. A Service Level may also be referred to herein as a “Service Level Agreement” or “SLA”.
Figure 2:Service Levels (SLA)
Px Definition
Response
time after
Notification
Provide
Solution
Time
Target
Resolution
Time
Escalation Path
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
P1
Emergency:
• Entire organization is
affected.
• Core business process
cannot be carried out.
• Security violation.
1 hour 4 hours 1 day
Resolution > 1 day
Escalation path:
Email to Service
Manager (1st),
Account Executive
(2nd).
P2
Critical:
• Multiple users or
departments are
directly affected.
• Incident has serious
impact on critical tasks
and no workaround is
available.
• Customer affected.
4 hours 1 day 3 days
Resolution > 2 days
Escalation path:
Email to Service
Manager (1st),
Account Executive
(2nd).
P3
High:
• System cannot
function as designed or
installed.
• Small group of users
are directly affected.
• Compliance timeline is
affected.
1 day 3 days
Requires
resource
estimation,
City's
approval,
prioritization,
and
scheduling
P4
Medium:
• Small number of users
are affected.
• Isolated incident.
• Degraded performance
and/or is difficult to
use.
5 days 2-3
weeks
Requires
resource
estimation,
City's
approval,
prioritization,
and
scheduling
P5
Low:
• User requests general
information, service, or
consultation.
• Cosmetic
enhancements.
• Report enhancements.
5 days 4-6
weeks
Requires
resource
estimation,
City's
approval,
prioritization
and
scheduling
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
* P2-P5 tickets are subject to core business working hours and exclude out-of-hours, weekends, and public holidays, unless by mutual agreement.
3.3.1 System Availability Service Levels
System availability is defined as the availability of any SAP Service. EPI-USE Labs considers a system available, if it can operate the system within the bounds of the services EPI-USE Labs provides.
Table 1: System Availability
System availability (uptime)
Landscape Uptime*
Production 99.5%
Non-Production 99.0%
*Measurement of uptime excludes maintenance windows and other planned downtimes.
System Availability percentage is calculated as follows:
System Availability Percentage = Total Minutes in the Month − Downtime Total Minutes in the Month ∗ 100
System Availability (“SA”) determines the minimum system uptime commitment.
Downtime excludes Total Minutes in the Month attributable to:
• Scheduled downtime during a regular maintenance window. • Any other scheduled downtime, which the City has been notified of at least five business working days prior to such scheduled downtime; or
• Unavailability caused by factors outside of the Supplier’s reasonable control, such as unpredictable and unforeseeable events that could not have been avoided, even if reasonable care had been exercised.
Regular Maintenance
EPI-USE Labs would preferably like to set a weekly maintenance window for system and infrastructure. During this time, the City should not expect system uptime (downtimes are not SLA relevant) and should not schedule any jobs or processes. The window can be pre-agreed on prior to contract start. All maintenance downtimes (regular or irregular) will be pre-arranged and pre- approved via the change process.
Recommended maintenance window: Weekly, Saturday: 14:00 to 20:00 PST. (TBD) Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Irregular Maintenance
It is possible that certain intensive maintenance activities may not be possible to achieve within the regular maintenance window. In the case of a longer than regular outage, the supplier will provide five days’ advance notice and work with the City to find a time window that is most suitable for the City users.
Maintenance exceptions
While EPI-USE will endeavor to accommodate occasional requests for exceptions to these maintenance windows, EPI-USE cannot guarantee that exceptions can be accommodated. The shared nature of many data center facilities dictates certain maintenance requirements. In the case of urgent hardware replacement or security fixes that could otherwise negatively impact the City, as well as the ability of EPI-USE Labs to meet SLA commitments to the City, such changes to maintenance schedules may not always be possible. EPI-USE will consult with the City in the event of these exceptions.
3.3.2 Disaster Recovery Service Levels
A disaster recovery service is applicable to all the production virtual machines. Networking is designed to have a fast recovery time, but in the event of a physical network connectivity failure, the recovery of the production virtual machines can be a manual process. EPI-USE will recover the virtual machines, from replicated backups from the disaster recovery data center. The service levels below are based on the backup and recovery of the SAP systems in the event of a disaster.
Table 2: Disaster Recovery RTO and RPO
Disaster Recovery
Scope Time
RTO Production servers 24 hours
RPO Production servers 24 Hours
The recovery time objective (RTO) is the time it will take to recover to the recovery point objective (RPO). The RPO is the point in the past to which the supplier will recover the system.
3.3.3 Backup Frequency and Retention Levels
Backups are broken down into three sections:
• Virtual machine.
• File level; and
• Database.
Below is a summary of the retention policy.
Table 3: Backup and Retention
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Backup and Retention policy
Object Frequency Level Landscape Retention Period
Virtual machine (with
Database) Daily Full All 30 days
Virtual machine (with
Database) Monthly Full All 12 months
Virtual machine (with
Database) Yearly Full All 3 years (contract
period)
Database backups require the use of the database integration agent that is provided with the backup environment. EPI-USE Labs utilizes a snapshot of the virtual machine, complete with database, which will be taken and retained.
3.4 Managed Services - SAP Systems Inventory
EPI-USE Labs will provide the SAP Service platform in AWS utilizing a virtualized environment as specified below:
• Primary data center location: AWS US-WEST (Oregon)
• The supported City SAP Landscape servers in scope, which will form part of the Managed Services component, are listed in Addendum A: Managed Services SAP Systems Inventory. • Database: HANA; and
• Operating System: Linux, Windows.
Connectivity: The hosted landscape will be accessible over an encrypted point-to-point IPSec terminating on the City’s vWAN hub in AWS and from there into the City environment.
4. Platform (AWS) Managed Services Scope
EPI-USE will deliver a flexible Cloud Managed Service to help the City get the most out of the AWS platform, containing costs and managing complexity.
EPI-USE will be the single provider and administrator to help reduce subscription spend and increase the operational efficiency and effectiveness of the SAP landscape in the cloud.
4.1 Maintain account with AWS
EPI-USE will maintain the AWS account for the City’s AWS infrastructure. EPI-USE will also take care of paying the AWS fees on an annual or monthly basis and charge these fees back to the City at AWS Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
list prices. The exact AWS fees payable will only be available after EPI-USE reviews the services and associated costs in the City’s AWS account. EPI-USE will conduct a Well-Architected Review that includes a full assessment of the City’s AWS spend.
4.2 AWS Platform Support Services
The AWS platform administration will include:
• Subscription and tenancy administration.
• Resource group administration.
• Networking services administration.
• Virtual machine administration.
• Storage administration.
• Connectivity administration from AWS to the City network edge; • Platform, connectivity, monitoring and alerting
• Security configuration and administration; and
• Backup and disaster recovery administration.
4.3 Operating System Support Services
The service supporting operating systems administration will include:
• Monitoring the operating system.
• Improving system performance
• Operating system administration
• Capacity planning and monitoring
• Operating system patch management
• Antivirus management
• Backup and restore management
• Disaster recovery workload management and testing; and • General fault and configuration of operating system.
4.4 Deliverables and Frequency
Table 4 refers to the summary deliverable and frequency of platform services.
Table 4: Platform as a Service
Service Area Deliverable Frequency
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Cloud Services (Platform) Service desk (Client Central) Daily Incident management Daily
Change management (formal change
approval) Weekly
Request management (small change with no
approval) Daily
Problem management (RCA) Daily
Service level management Monthly
Event management Daily
Platform Housekeeping and monitoring Daily
Governance reporting:
Performance metrics.
End-user response times and trends.
Network statistics.
CPU performance.
Backup compliance.
AV compliance.
Vulnerability scanning (Quarterly).
Penetration testing (Annually).
Monthly
4.5 EPI-USE’s AWS Access
There are two operating modes within AWS Organizations: Consolidated Billing and All Features.
All Features will be enabled. The City’s usage and invoices are made available to EPI-USE, and EPI- USE will have full control over what the City’s End Customer Accounts can do in accordance with this Agreement including but not limited to Exhibit G (Addendum – Supplemental Terms – AWS Solution Provider Program for End Customers). Additionally, through AWS Single Sign-On integration with AWS Organizations, EPI-USE will also have access to Client Content. EPI-USE will provide access to their Program Management Account to the City and will work with the City to use only IAM to access the Program Management Account. Further, EPI-USE may choose to restrict its access to the Program Management Account via its root user credentials, by working with the City to implement multi-factor authentication and giving City sole control over one of the factors required for authentication. To access a copy of the AWS Organizations user guide, visit Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html (or any successor or related location designated by AWS, as may be updated by AWS from time to time).
4.6 AWS Support
In managing the City’s AWS account, EPI-USE will register the City for the support type the City has selected below and will include the City as a named support contact. The City’s named support contact person will be determined during the onboarding process, and, as needed thereafter, may be changed by written notice to EPI-USE from the City’s Project Manager or Chief Information Officer.
The City hereby selects the continued use of AWS Business Support for the performance of this Agreement. Signing this Agreement provides written confirmation of this selection.
4.7 [RESERVED]
5. Security Managed Services Scope
EPI-USE Labs will maintain ISO 27001 certification and on an annual basis an SSAE16-SOC 2 report, or their equivalent can be made available to the City on request. EPI-USE will handle the City data according to its information classification and handling standards. The City information is classified as Confidential and appropriate information handling, human resources and technical security measures are in place to reduce the risk of accidental or malicious information disclosure. EPI-USE have adopted the ISO 27001 information security framework, which governs all aspects of information security. EPI- USE have been certified as ISO 27001 compliant by external, accredited auditors.
5.1 Data and Security
EPI-USE Labs uses reasonable security technologies in providing the services. EPI-USE Labs will implement technical and organizational measures to secure personal data processed in the services, in accordance with applicable data protection laws.
The City is responsible for the City data and entering it into the SAP cloud systems.
EPI-USE will be connecting to the City’s AWS environment in two ways and both these processes are transparent as they are logged in detail.
• SSM method through AWS web console. With this method, the complete access is logged by AWS itself and city will have access to these log entries from the AWS console. • Through Openvpn, managed by EPI-USE. The Openvpn server logs these entries in great detail, which may be accessed through the Openvpn server admin web panel. Both services provide the City with complete access to the log records of connected users.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Security responsibilities – Table 5 specifies which of the parties are responsible for the various security areas.
Table 5: Security Responsibilities
Security Area EPI-USE
Responsible City Responsible
Firewall on AWS platform X
Intrusion Detection System on AWS platform X
Virtual Private Connection on AWS platform X
Virtual Private Connection at the City site X
Virtual Private Connection on end user devices X
Access Control and User Management on Operating
System X
Access Control and User Management on Database X
Application Access Control and User Management on
SAP X
Patch Management on Operating System X
Infrastructure/Database Access Control and User
Management X
Internal Vulnerability Scanning X
Manage AWS Security Services (Security Hub,
Guardduty, Inspector etc..) X
6. Basis Managed Services Scope
EPI-USE Labs will be responsible for SAP Basis Managed Services for the SAP landscape. The following services are included.
6.1 Daily Checks and Housekeeping
The purpose is to check and maintain to be proactive should any incident or requests arise.
Housekeeping will constitute preventative proactive management and various automation activities will be carried out where applicable. EPI-USE Labs will enable the City to have visibility of critical systems with the use of EPI-USE Labs software or 3rd party software, if required. EPI-USE Labs will carry out these tasks once either daily, weekly, or monthly and the check sheets/reports will be made Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
available to the City on the Client Central Workspace. Where possible, these tasks will be carried out automatically with the use of EPI-USE Labs or 3rd party software.
6.2 Monitoring and Alerts
EPI-USE Labs will monitor all hosted systems and alert on critical events, using EPI-USE Labs and/or third-party software. A critical event is defined as:
• SAP production outage which influences critical business functions.
For critical events, EPI-USE will be alerted (via a paging system to the standby team) of the defined critical event, triggering an automated P1 incident entry in Client Central. This in turn will, initiate the P1 incident response procedure.
The P1 response procedure in client central will add the applicable the City system owners (Defined in Client Central) as watchers to the ticket, which will in turn send an email to all watchers every time new information is added or changed to this P1 ticket. Response times and resolution times will adhere to the defined SLA times.
A contact list will be generated for contacting business unit owners (including a backup resource). This will be used to drive any communication via Client Central tickets that would arise during an outage or serious issue.
6.3 SAP Basis Landscape Advisory
EPI-USE will provide SAP landscape advisory services on an ongoing basis. These recommendations will be discussed in the monthly review meetings. This advisory service will pertain to SAP performance enhancements and short, medium, and long-term SAP landscape roadmap guidance. The following are:
• Advice on end of SAP support on SAP versions.
• Advice on target SAP versions available.
• Advice on target SAP kernel.
• Advice on target enhancement pack.
• Advice on latest database version.
• Advice on capacity planning.
• Proactive Basis process and setup improvements
• Large HANA database tables in production; and
• HANA Database sizes/growth.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
6.4 Support Pack updates
This function is deemed part-and-parcel of the SAP Basis managed services. EPI-USE Labs will apply the following:
• Database patches annually
• SAP Kernel updates bi-annually
• Security patches, where deemed necessary and as required • SAP service packs annually
• SAP HR legal patches annually
• SAP Notes on request
Major software (OS, DB, Application) release upgrades e.g. a full release upgrade and EHPs are excluded from the scope of work and will be handled as separate projects. Functional and application testing will not form part of the scope of these upgrades and is limited to the technical application.
6.5 System Refreshes (as-needed, upon request)
EPI-USE will help launch the CITY’s separately contracted-for Libelle system refresh tool once per quarter, as a part of the support services. Once launched, the City staff will complete the system refresh operation. SAP system client copies, using the standard SAP tools, are included under the support services.
To implement any additions to this Scope of Services, a Task Order for Additional Services pursuant to Section 4 (Not to Exceed Compensation) will be used. As detailed in Section 13 (Addendum C:
Change Request Form) below, a Change Request Form (shown in Addendum C) may only be used pursuant to Section 4 (Not to Exceed Compensation) of this Agreement, regarding Additional Services, as an Attachment A (Scope of Work) to an approved Task Order as detailed in Section 4.
6.6 Deliverables and Frequency
Table 6 refers to the summary deliverable and frequency of Basis Managed Services.
Table 6: Basis as a Service Deliverables
Service Area Deliverable Frequency
Basis Services Service desk (Client Central) Daily
Incident management Daily
Change management (formal change
approval) Weekly
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Request management (small change with
no approval) Daily
Problem management (RCA) Daily
Service level management Monthly
Event management Daily
Patches Annual
Basis Housekeeping and monitoring Daily
Governance reporting:
System uptime.
Incident SLA reports.
Performance metrics.
Storage and usage.
Monthly
6.7 Excluded from Scope
The following tasks are specifically excluded from the Scope of Services of this Agreement and, if desired, would need to be implemented via a Task Order as Additional Services pursuant to Section 4 (Not to Exceed Amount), or via contract amendment pursuant to Section 29.4, of this Agreement:
• Major SAP NetWeaver release upgrades.
• User access management, SAP Authorizations and Roles. • Although the technical transport management function is part of the responsibility of EPI-USE Labs, the change management process and approval of changes will remain the responsibility of the City.
• End user device support is excluded (e.g., printers, desktops, SAP GUI) • Additional SAP NetWeaver application implementations and the support of these environments after implementation.
• All other services outside the current SAP landscape.
6.8 General Assumptions
• Business days are Mondays to Fridays excluding federal holidays • Business hours are between 8AM and 5PM (PST)
• Support is offered during business hours, except for P1 Incidents, which are, supported 24/7 Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
• Measurement of service levels is suspended for the time that an incident is under the responsibility of the City or any third party or EPI-USE Labs • EPI-USE Labs will assign a Service Delivery Manager to the City who will be responsible for all support- related issues
• If standby is required for P2 to P4 incidents outside of business hours, the City can request this before the time
• After-hours work should be limited to 10 (ten) hours per month, excluding work for P1 incident
• All requests for after-hours support must be submitted 48 hours in advance. These requests must be directed to the applicable Service Delivery Manager • All service requests must be submitted in ticket form, regardless of verbal or email agreements. SLA’s are measured based on tickets only.
7. On-boarding, Transfer and Transitioning, Amendment
No.1 (Completed)
The transition and handover phase will be critical to the success of our support service. The onboarding process starts with the handover between the EPI-USE team and the City team of the existing SAP systems, AWS platform and AWS account. This phase should commence in the month prior to the actual ongoing Managed Services engagement commencing. Service transition will be crucial to ensuring that EPI-USE has a good understanding of:
• City’s latest system platform, landscape, and configuration • to transfer the existing AWS Account and billing responsibility to EPI-USE • the organization and how it operates
• the internal support team’s skills and capabilities
• other vendors that provide support (EPI-USE will require information) • any outstanding and open tickets.
By assessing these elements of the current support service, EPI-USE will be better placed to advise on service management processes, transition steps, and knowledge transfer requirements as part of the transition phase.
Transition activities will include the review of documentation, facilitated workshops with the City, and system familiarization.
The transition phase consists of three main areas: Assessment (information gathering), Design and Setup, and Handover, as further detailed below.
Assessment (information gathering)
During this phase, EPI-USE will spend time reviewing:
• all documentation (including but not limited to functional, technical, and business process documentation)
• the platform, landscape setup and configuration
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
• to transfer the existing AWS Account, and this will entail:
o EPI-USE have root access to the AWS account
o The AWS account has been un-linked from the incumbent outgoing provider’s AWS Master Payer account (if applicable)
o EPI-USE’s payment method has been updated in the account o An AWS support plan has been agreed for the AWS account • the current status of outstanding incidents and service requests • skills of the internal support and IT teams
• current support procedures (incident management, change management).
Design and Setup
Given sufficient information, EPI-USE will start designing the support service to be delivered from the start of the contract including:
• Adjusting the support methodology and processes to fit the City’s requirements. This will be done in collaboration with the City.
• Aligning with the City’s incident and change management processes and introducing and knowledge transfer of Client Central.
• Discussing and designing a skills’ matrix and recommending a knowledge transfer plan. • Assessing the gap – if any – in levels of documentation required (for business processes, configuration, and system landscape) and agreeing to an approach to fill this gap. • Agreeing on an approach to handover from the incumbent outgoing provider, ensuring that all outstanding incidents and service requests are addressed.
Handover
Handover to EPI-USE Labs is the last transition phase. EPI-USE will start to take on some elements of the support contract while allowing the outgoing supplier to close existing incidents and service requests. Normally, EPI-USE would recommend a period of joint operation when EPI-USE takes responsibility for the resolution of new issues. This transition period with the incumbent outgoing provider can be up to two weeks. The Design and Setup phase normally ensures that the responsibilities for all parties are clearly agreed and that all parties, including the incumbent outgoing provider, are in agreement with the approach. This is important to ensure continuity of service for the City.
The three-stage process is applied in conjunction with the fundamental principle of People, Process and Technology, which forms the foundation for successful service management. In order to transition smoothly from the City’s incumbent outgoing provider to EPI-USE, all three key areas need to be carefully considered as part of the transition planning.
8. Ad-hoc Services (as needed, upon request)
Perform ad-hoc services on as-needed requests that are not covered in the Scope of Services. These services constitute Additional Services under Section 4 (Not to Exceed Compensation) of this Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Agreement, implemented pursuant to City-approved Task Order as detailed in Section 4. Examples of such Additional Services include without limitation:
(1) Perform ad-hoc (Additional Services per Section 4) hosting and networking services such as enabling additional AWS services, building new application and/or database servers that are not included in the Scope of Services.
(2) Perform ad-hoc (Additional Services per Section 4) Basis services such as installation and configuration of new SAP applications and SAP HANA databases that are not included in the Scope of Services.
(3) Perform other ad hoc (Additional Services per Section 4) services that are not included in the Scope of Services as may be requested by CITY.
(4) Any Managed Services specifically requested by the City of Palo Alto outside normal business hours (except for P1 incidents and in-scope Managed Services production system work).
9. Client Central Service Desk
Client Central will be utilized as a service desk for the Managed Services support, with Client Central user and role management included. Further to the service desk (ticketing) offering, a knowledge base and downloads area will be allocated in the workspace for various shared documents and artefacts for e.g., platform information, technical documents, functional documents, how-to, solution suggestions, recorded video trainings and suggestions.
9.1 Service Desk Ticketing
Standard features included:
• Ticket feature to record all incidents, changes or activity with the date and time of change as well as the person responsible for the change.
• Attachments can be added to each ticket. This is stored on the ticket itself (unless removed) and is separate from the downloads mentioned above.
• The ticket records and displays the response and resolution time on the ticket, in relation to the defined SLA.
• Buttons are used to drive the workflow on a ticket and change it from one state to another, e.g., in progress, awaiting info or completed. They can be configured to require a comment or to simply progress the workflow. Buttons can be colored as required to highlight the purpose of the button (e.g., green for approval).
• States in a workflow have the following qualities:
o The name can be defined as required.
o The color can be used to better show the purpose of the status (e.g., green for approved). o The status can either be open (SLA is counting), waiting (SLA is paused, someone is waiting for feedback) or closed (the SLA has been stopped and ticket has been resolved).
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
• Tickets can be linked to each other to quickly navigate between relevant tickets. • Tickets can be filtered based on any of the fields in the ticket, assignee, status, SLA compliance, date updated, and status. This helps to easily find and report on individual tickets. • Big scale reporting can also be done on any of the above-mentioned factors. City can use the information to build graphs in Client Central to display specific data visually. • Custom filters can be built so that reports can be saved and reused as required
9.2 Ticket Type and Description
• Incidents: Incidents happening in the landscape that interrupts daily operation. Typically logged by the City’s users.
• Change request: Scheduled changes, usually preceded by a Service request. A single change request documents the changes made (as specified in a service request) per individual landscape tier as the date and time of each change is different. This results in one Service request and multiple change requests connected to it. • Service requests: non-incidents (typically) logged by Business to implement/change something in the landscape.
• Problem ticket: mainly used for keeping track of RCA investigations. Also used to keep track of performance/service improvement mini projects. Typically, all P1 Incidents require an associated Problem ticket, but multiple Incidents can be linked to the same Problem ticket. • Events: Tickets logged by the Managed Services team to communicate proactive monitoring incidents. This is usually connected to monitoring tool alerts or monitoring/daily health checks performed by the Managed Services.
• Internal KB: Requests for Knowledge base article creation on a certain subject. • Patch: Pro-Active patches/alerts (typically from 3rd party vendors like SAP, OS vendors, etc.) • Define-your-own wherein City can define its own ticket type/description.
9.3 Knowledge Base Area
The Knowledge base consists of documents of a word-type format that can be created by and shared with authorized users. They can also be added to tickets as required by pasting the article link on the ticket.
The structure can be configured by creating nested directories if required. City can search for a knowledge base article by using the search bar. The search bar works with a wildcard search technique to search for matching content in all articles, or City can search for tags that were added to the article during the article creation.
9.4 Downloads Area
Downloads is a dedicated document repository for business relevant documents that can be uploaded to, downloaded from, and shared with authorized users.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
The structure can be configured by creating nested directories if required. City can search for a file by using the search bar based on the file name or description.
10. Costing and Fees
Costing and fees for this Agreement shall be as detailed elsewhere in this Agreement including but not limited to Section 4 (Not to Exceed Compensation) and Exhibit C (Compensation).
10.1 AWS Fees
Estimated AWS fees are provided in Exhibit C (Compensation) of this Agreement, however, as noted in said Exhibit C, AWS consumption will be invoiced based on actual AWS spend and using AWS’s actual published list prices. AWS fees will be invoiced separately from the Managed Services fees and billed on a monthly basis.
Reserved Instance and Savings Plan purchases will be reviewed on an as-needed basis and these purchases are required to be approved by the City pursuant to a Task Order as Additional Services under Section 4 (Not to Exceed Compensation), or as a contract amendment pursuant to Section 29.4, of this Agreement. Depending on the commitment period these purchases will be paid by the City all upfront or amortized over the duration of the remaining Agreement term, pursuant to a Task Order as Additional Services under Section 4, or as a contract amendment pursuant to Section 29.4, of this Agreement.
Notwithstanding any information received to date; any additional clarifications which may have been provided, or the outcome of any discovery workshops conducted, EPI-USE has calculated its AWS pricing based purely on its current understanding of the City’s requirements. The City therefore acknowledges that, except for items which may arise solely because of EPI-USE’s actions during the engagement, EPI-USE shall be entitled to recover AWS fees for those items which would not have been known at the outset of the engagement, and which increases the AWS spend budget. Any changes outside of the AWS spend budget stated in this Agreement, shall be approved in writing by the City pursuant to a Task Order as Additional Services under Section 4 (Not to Exceed Compensation), or as a contract amendment pursuant to Section 29.4, of this Agreement.
If the City exceeds the usage limits applicable to the AWS Service subscription, the City agrees to pay for the fees and charges for such over-usage pursuant to a Task Order as Additional Services under Section 4 (Not to Exceed Compensation), or as a contract amendment pursuant to Section 29.4, of this Agreement. For any extension of a subscription term within the term of this Agreement or the provision of a Service upgrade, the then-current AWS price will apply. At the beginning of each new subscription term within the term of this Agreement, EPI-USE may, in consultation with the City, increase AWS fees to reflect increased AWS costs imposed by AWS. Not raising fees is not a waiver of EPI-USE’s right to do so if AWS raises its fees. EPI-USE will notify the City as soon as practicable of any changes in AWS fees by e-mail. Failure by the City to pay the fees when due and payable, if not timely cured by City after notice of the failure from Consultant, may result in suspension of access to, or rights to use the Services.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
11. Addendum A: Managed Services Systems Inventory
The AWS infrastructure services inventory to be included on the AWS managed services:
Table 7: AWS services inventory
Service Configuration summary
S3 Standard
50 TB per month
PRD
Amazon FSx for Windows File
Server
Desired storage capacity (1 TB)
STG
Amazon Virtual Private Cloud
(VPC)
Number of Site-to-Site VPN Connections (10)
QAS
Amazon Elastic File System
(EFS)
Data stored in Standard storage (5 TB per month)
DEV
Amazon S3 Glacier (Glacier API
only)
S3 Glacier storage (10 TB per month)
SBX
Amazon EC2 (r5.large) Quantity
- 12
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
PRD
Amazon EC2 (r5.xlarge)
Quantity - 8
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
STG
Amazon EC2 (x1.16xlarge)
Quantity - 3
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (4000 GB)
QAS
Amazon EC2 (t2.small) Quantity
- 2
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
DEV
Amazon EC2 (t2.xlarge)
Quantity - 1
Operating system (Windows), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
PRD
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Amazon EC2 (m5.large)
Quantity - 1
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
STG
Amazon EC2 (m4.2xlarge)
Quantity - 1
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
QAS
Amazon EC2 (r5a.large)
Quantity - 4
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (150 GB)
DEV
Amazon EC2 (r5.2xlarge)
Quantity - 1
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
PRD
Amazon EC2 (t2.xlarge)
Quantity - 1
Operating system (Linux), Storage for each EC2 instance (General Purpose SSD (gp2)), Storage amount (500 GB)
STG
The corresponding SAP systems relevant within the scope of Managed Services:
Table 8: SAP System inventory
City SAP systems
System type Version Tier Database Operating system
ECC EHP 8 PRD HANA Suse Linux
ECC EHP 8 STG HANA Suse Linux
ECC EHP 8 QAS HANA Suse Linux
ECC EHP 8 DEV HANA Suse Linux
ECC EHP 8 SBX HANA Suse Linux
BW 7.5, SP16 PRD HANA Suse Linux
BW 7.5, SP16 STG HANA Suse Linux
BW 7.5, SP16 QAS HANA Suse Linux
BW 7.5, SP16 DEV HANA Suse Linux
CRM 7.0, EHP4 PRD HANA Suse Linux
CRM 7.0, EHP4 STG HANA Suse Linux
CRM 7.0, EHP4 QAS HANA Suse Linux
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
CRM 7.0, EHP4 DEV HANA Suse Linux
Gateway PRD HANA Suse Linux
Gateway STG HANA Suse Linux
Gateway QAS HANA Suse Linux
Gateway DEV HANA Suse Linux
ESS/ MSS Portal PRD HANA Suse Linux
ESS/ MSS Portal STG HANA Suse Linux
ESS/ MSS Portal QAS HANA Suse Linux
ESS/ MSS Portal DEV HANA Suse Linux
Solution Manager PRD HANA Suse Linux
NWDI DEV HANA Suse Linux
BSI Taxfactory/Jump Server PRD HANA Suse Linux
Open VPM Server PRD Suse Linux
SAP Router PRD Windows
SAP AD PRD Windows
Fiori Frontend PRD HANA Suse Linux
Fiori Frontend STG HANA Suse Linux
Fiori Frontend QAS HANA Suse Linux
Fiori Frontend DEV HANA Suse Linux
HANA DB Server PRD HANA Suse Linux
HANA DB Server QAS
STG HANA Suse Linux
HANA DB Server DEV HANA Suse Linux
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
12. Addendum B: City and EPI-USE Labs Responsibilities
The following table will define the high-level responsibilities of the parties in a RACI format.
Table 9: RACI Table
ID
Service
R – Responsible, A - Accountable, C –
Communicated, I – Informed, P- Participate
Frequency EPI-
USE CoPA
1 Service Management
1.1 Account Management
1.1.1 Governance meetings Monthly RAC RAC
1.1.2 Platform capacity planning (usage-based) Ongoing R CI
1.1.3 Platform capacity planning (future project-
based) Quarterly R CI
1.1.4 Provide SLA and usage reports and
performance metrics Monthly RA CI
1.2 Service Request Management
1.2.1 Log service requests in Client Central Ongoing CI RA
1.2.2 Review, approve service requests Ongoing RA RA
1.3 Incident Management
1.3.1 Daily support desk operations Ongoing RA RA
1.3.2 Review, approve, resolve, escalate incidents Ongoing RA RA
1.3.3 Confirm resolved incidents/solutions Ongoing RA RA
1.3.4 Create incidents/solutions Ongoing RA RA
1.4 SAP Online Support Service
1.4.1 Create OSS users Ongoing CI RA
1.4.2 Assign OSS permissions Ongoing CI RA
1.4.3 Manage developer keys Ongoing CI RA
1.4.4 Manage object keys Ongoing CI RA
1.4.5 Support management of S-User ID Ongoing CI RA
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
1.4.6 Monitor remote support connection to OSS Ongoing CI RA
1.4.7 Open/close SAP remote support connections Ongoing CI RA 2 Infrastructure
2.1 Data Center Management
2.1.1 Infrastructure (AWS) all support Ongoing RA CI
2.1.2 AWS tenancy administration Ongoing RA CI
2.2 Network Management
2.2.1 Proactively monitor data center networks and
utilization Ongoing RA CI
2.2.2 Change data center network configuration Ongoing RA CI
2.2.3 All network maintenance Ongoing RA CI
2.3 Storage Management
2.3.1 Proactively monitor storage performance &
capacity Ongoing RA CI
2.3.1 Manage data files, file systems and disks Ongoing RA CI
2.3.1 Scale storage capacity (additional) Request RA CI
2.4 Operating System
2.4.1 Create/maintain OS users Ongoing RA CI
2.4.2 Monitor for/report security incidents Ongoing RA CI
2.4.3 Configure/change OS parameters Ongoing RA CI
2.4.4 Monitor/remediate OS, logs, file systems Ongoing RA CI
2.4.5 Interact with vendor to remediate OS issues Ongoing RA CI
2.4.6 Proactively monitor swap and paging Ongoing RA CI
2.4.7 Proactively monitor memory usage Ongoing RA CI
2.4.8 Manage OS patches Ongoing RA CI
2.4.9 Perform OS upgrades Ongoing Project CI
2.5 System Startup/Shutdown
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
2.5.1 Scheduled OS/hardware system startup &
shutdown Request RA CI
2.5.2 Shutdown/restart OS/hardware after
failure/maintenance Ongoing RA CI
2.6 Backup/Restore
2.6.1 Perform periodic file system and database
backups Ongoing RA CI
2.6.2 Requests for ad-hoc backups Ongoing RA CI
2.6.3 Monitor backup processes Ongoing RA CI
2.6.4 Perform periodic backup/restore testing;
include result in the relevant report Biannual RA CI
2.6.5 Perform restore and recovery of file
systems/databases/tables after failure Ongoing RA CI
2.6.6 Perform restore and recovery of file
systems/databases/tables upon request Ongoing RA CI
2.6.7 Perform verification of restored systems Ongoing RA CI
2.6.8 Perform functional verification Ongoing RA CI
2.7 Integration
2.7.1 Configure integration with customer directory
service Ongoing RA CI
2.7.2 Integration with SAP Cloud (Network
connections and VPN) Ongoing RA CI
2.8 SFTP Service
2.8.1 Provide file sharing on hosted systems Request RA CI
2.8.2 User and access management to hosted
systems Request RA CI
2.8.3 Create/maintain shared folders Request RA CI
2.8.4 Implement and maintain antivirus software
on servers Ongoing RA CI
2.8.5 Antivirus software licenses Ongoing RA CI
2.8.6 Backup data on shared folders Ongoing RA CI
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
2.8.7 Restore data on shared folders Request RA CI
3 Database Management
3.1 SAP HANA
3.1.1 Proactively monitor database health and
resource usage Ongoing RA CI
3.1.2 Monitor table growth to prevent operational
issues Ongoing RA CI
3.1.3 Partition/rebalance tables due to uneven
growth Ongoing RA CI
3.1.4 Plan and perform file system extensions Ongoing RA CI
3.1.5 Perform HANA housekeeping (manage logs,
traces, free up resources) Ongoing RA CI
3.1.6 Maintain HANA configuration parameters
based on best practices Ongoing RA CI
3.1.7 Start/stop database Request RA CI
3.1.8 Add hardware for scale-out configuration Request RA CI
3.1.9 Update/patch database Ongoing RA CI
3.1.10 Configure HANA Transports Management Request RA CI
3.1.11 Implement /maintain additional SAP tools (in
HANA context delivered by SAP) Request RA CI
3.1.12 Analyze and optimize expensive SQL
statements for improved performance Request RA CI
3.1.13 System troubleshooting to resolve issues and
return HANA to normal operation Ongoing RA CI
3.1.14 Create/maintain HANA Studio users Request RA CI
3.1.15 Management of database users Ongoing RA CI
3.1.16 Backups Ongoing RA CI
3.1.17 Recover database after technical issues Ongoing RA CI 4 Basic Operations
4.1 Monitoring
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
4.1.1 Configuration of basis monitoring tools Ongoing RA CI
4.1.2 Monitoring of alert consoles for different tools Ongoing RA CI
4.1.3 Logging of performance events Ongoing RA CI
4.1.4 Respond to events according to notification
and escalation procedures Ongoing RA CI
4.1.5 Identification, installation, and activation of
monitoring tools Ongoing RA CI
4.2 SAP Upgrades & Enhancement Packs
4.2.1 SAP Major release upgrades Request Project CI
4.2.2 SAP Enhancement Pack upgrades Request Project CI
4.2.3 Upgrade database Request RA CI
4.2.4 Perform functional changes and
developments Request CI RA
4.2.5 Testing of upgraded system and functionality Request CI RA
4.3 SAP Support Packs
4.3.1 Apply SAP kernel upgrades Biannual RA CI
4.3.2 Apply SAP patches Annual RA CI
4.3.3 Apply SAP Notes Request RA RA
4.3.4 Post-support pack testing Request CI RA
4.4 SAP Security Management
4.4.1 Create/maintain/review/audit roles, profiles,
authorizations Ongoing CI RA
4.4.2 Create/maintain/lock/unlock users, assign
roles Ongoing CI RA
4.4.3 Maintain users/roles/profiles in client 000 Ongoing RA CI
4.4.4 Provide customer access to client 000 Request RA CI
4.4.5 Monitor and share information about security
fixes published by SAP Ongoing RA RA
4.5 Client Copies & Refreshes (excludes
repository, to non-production targets) Maximum quarterly Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
4.5.1 Perform copy/refresh Request CI RA
4.5.2 Perform technical post-processing Request CI RA
4.5.3 Perform validation of copied/refreshed system Request CI RA
4.6 System Copies (to non-production)
4.6.1 Perform system copy Request CI RA
4.6.2 Perform technical post-processing Request CI RA
4.6.3 Perform validation of installed system Request RA CI
4.7 Change Control
4.7.1 Application development, customizing,
configuration, maintenance, support Ongoing CI RA
4.7.2 Planning change requests Ongoing CI RA
4.7.3 Apply SAP notes for applications Ongoing CI RA
4.7.4 Apply SAP security notes Ongoing RA RA
4.8 Printer Management
4.8.1 Create, change, lock, unlock, maintain
printers within SAP Request CI RA
4.8.2 Printers within SAP, configured on the OS
(due to security constraints) Request RA CI
4.8.3 Analyze printing and spool issues Request CI RA
4.8.4 Spool system housekeeping Ongoing CI RA
4.8.5 Design and implement print forms Request CI RA
4.8.6 Troubleshoot issues with printer hardware Request CI RA
4.9 Desktop Support
4.9.1 SAPGUI support Request CI RA
4.9.2 SAP SSO support Ongoing CI RA
4.9.3 Desktop and front-end user support Request CI RA
4.10 Additional Proactive Services
4.10.1 Analyze SAP EarlyWatch Alert reports and
consider technical recommendations Ongoing RA CI
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
4.10.2 Implement data archiving strategy and
processes Request CI RA
4.10.3 Monitor and maintain data archiving systems Ongoing CI RA
4.11 System Installations / Migrations
4.11.1 Install new systems / migrate from on
premise to cloud Request Project CIP
4.11.2 Technical configuration of all
installed/migrated systems Request Project CIP
4.11.3 Integration of new/migrated systems to cloud
operations environment Request Project CIP
4.11.4 Integrate all hosted systems with customer
Solution Manager in cloud Request Project CIP
4.12 Disaster Recovery
4.12.1 Implement disaster recovery setup according
to architecture blueprint Request RA CIP
4.12.2 Develop plan for disaster recovery (data
center and system infrastructure only) Request RA CIP
4.12.3 Manage and monitor disaster recovery
architecture and replication Ongoing RA CIP
4.12.4 Develop plan for disaster recovery of
applications and connectivity to cloud Request RA CIP
4.12.5 Test disaster recovery system (data center
and system infrastructure only) Quarterly RA CIP
4.12.6 Execute full disaster recovery failover test Annually RA CIP
4.12.7 Technical and connectivity testing for disaster
recovery failover Annually RA CIP
4.12.8 Application testing for disaster recovery
failover Annually RA CI
4.13 Management of Local Area Network
4.13.1 Provide and maintain network at sites Ongoing CI RA 5 NetWeaver ABAP Operations
5.1 General Operations
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
5.1.1 Proactive availability monitoring of SAP
instances Ongoing RA CI
5.1.2 Troubleshooting SAP instances Ongoing RA CI
5.1.3 Analyze systems logs and fix technical
failures Ongoing RA CI
5.1.4 Proactive monitoring of SAP processes Ongoing RA CI
5.1.5 Analyze update terminations and take
appropriate action Ongoing RA CI
5.1.6 Clean up terminated updates Ongoing RA CI
5.1.7 Analyze lock entries and take appropriate
action Ongoing RA CI
5.1.8 Proactively monitor ABAP short dumps Ongoing RA CI
5.1.9 Troubleshoot performance issues (excludes
ABAP and application issues) Ongoing RA CI
5.1.10 Maintain SAP system profile parameters Ongoing RA CI
5.1.11 Implement/update tools for SAP support
service readiness Ongoing RA RA
5.1.12 Management of web service runtime Ongoing CI RA
5.1.13 Configure RFC connections to SAP and non-
SAP systems Ongoing CI RA
5.2 Interface Administration
5.2.1 Create/execute/monitor/troubleshoot batch
input sessions Ongoing RA CI
5.2.2 Configure/monitor/troubleshoot interfaces
and related (IDOCS, scripts etc.) basis
configuration
Ongoing RA CI
5.3 Job Scheduling
5.3.1 Schedule and monitor standard housekeeping
jobs Ongoing RA CI
5.3.2 Schedule/monitor/change application batch
jobs Request CI RA
5.4 Change Control
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
5.4.1 Configure and maintain SAP Transport
Management System and routes Request RA CI
5.4.2 Release tasks and transport requests Ongoing CI RA
5.4.3 Import transport requests Request CI RA
5.4.4 Troubleshooting SAP Transport Management
System Ongoing RA CI
5.4.5 Address functional transport issues Ongoing CI RA
5.4.7 Testing of completed transport requests Ongoing CI RA
6.0 Sandbox Basis administration Ongoing RA P
6.1 Knowledge transfer to City basis on Sandbox Ongoing R P
13. Addendum C: Change Request Form
The Change Request Form provided below may only be used pursuant to Section 4 (Not to Exceed Compensation) of this Agreement, regarding Additional Services, as an Attachment A (Scope of Work) to an approved Task Order as detailed in Section 4. The Change Request Form, by itself, is not permitted to be used separate from a Task Order as detailed under Section 4 and any use of a Change Request Form that is separate from and not part of an approved Task Order as detailed under Section 4 shall be void and without effect.
Agreement ref: Originator:
Client name: City of Palo Alto Change request date:
Title of change:
Reason for change:
Include an outline of the reason for the change and the impact of not implementing it. Attach written change request.
Description of
change:
Include full details of the change including any specifications, areas of the Services impacted.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Impact on charges:
Include full details of any charges, if any, for the change, and any revision to the charges.
Impact on
agreement:
Describe the resulting updates to the Clauses and Schedules to the Agreement.
Implementation
schedule:
Assumptions /
additional
information:
Provide any assumptions made at the time of the change and any relevant supporting information.
14. [RESERVED]
15. [RESERVED]
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
EXHIBIT B
SCHEDULE OF PERFORMANCE, AMENDMENT NO.1
(AMENDED, REPLACES PREVIOUS)
CONSULTANT shall perform the Services so as to complete each milestone within the number of days/weeks specified below. The time to complete each milestone may be increased or decreased by mutual written agreement of the Project Managers for CONSULTANT and CITY so long as all work is completed within the term of the Agreement. CONSULTANT shall provide a detailed schedule of work consistent with the schedule below within 2 weeks of receipt of the notice to proceed (“NTP”) from the CITY. Exhibit A (Scope of Services) also contains deliverable and frequency information for the provision of Services under this Agreement.
Milestones
Completion
Number of Days/Weeks
(as specified below)
from NTP
1. Task 1: Transfer existing AWS hosting account in AWS cloud Completed
2. Task 2: Maintain account with AWS for hosting City’s infrastructure
Annual ongoing
3. Task 3: Perform hosting and networking maintenance services Annual ongoing
4. Task 4: Perform SAP Basis Services Annual ongoing
5. Task 5: Perform ad-hoc services on as needed requests that are not covered in the Scope of Services. These services constitute Additional Services under Section 4 (Not to Exceed
Compensation) of this Agreement, implemented pursuant to City-approved Task Order as detailed in Section 4.
As-needed basis with the
performance schedule as
provided in the approved
Task Order, as noted in the
checkbox below.
Optional Schedule of Performance Provision for On-Call or Additional Services Agreements. (This provision only applies if checked and only applies to on -call agreements per Section 1 or agreements with Additional Services per Section 4.)
The schedule of performance shall be as provided in the approved Task Order, as detailed in Section 1 (Scope of Services) in the case of on-call Services, or as detailed in Section 4 in the case of Additional Services, provided in all cases that the schedule of performance shall fall within the term as provided in Section 2 (Term) of this Agreement.
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
EXHIBIT C
COMPENSATION, AMENDMENT NO.1
(AMENDED, REPLACES PREVIOUS)
CITY agrees to compensate CONSULTANT for Services performed in accordance with the terms and conditions of this Agreement, and as set forth in the budget schedule below. Compensation shall be calculated based on the rate schedule attached as Exhibit C -1 up to the not to exceed budget amount for each task set forth below.
CITY’s Project Manager may approve in writing the transfer of budget amounts between any of the tasks or categories listed below, provided that the total compensation for the Services, including any specified reimbursable expenses, and the total compensation for Additional Services (if any, per Section 4 of the Agreement) do not exceed the amounts set forth in Section 4 of this Agreement.
CONSULTANT agrees to complete all Services, any specified reimbursable expenses, and Additional Services (if any, per Section 4), within this/these amount(s). Any work performed or expenses incurred for which payment would result in a total exceeding the maximum amount of compensation set forth in this Agreement shall be at no cost to the CITY.
(CONTINUED ON THE NEXT PAGE.)
Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
BUDGET SCHEDULE
TASK AMENDMENT NO.1
CONTRACT
YEAR 1
(PAID in
FULL)
CONTRAC
T YEAR 2
(PAID in
FULL)
CONTRACT
YEAR 3
(PAID in
FULL)
CONTRACT
YEAR 4
(Effective
August 1,
2025 –
December 31,
2026)
SUBTOTALS
Task 1
(Transfer existing AWS hosting
account in AWS cloud)
$4,500
(Paid)
--
--
--
$4,500
(Paid)
Task 2
(Maintain account with AWS for
hosting City’s infrastructure. Task 2
costs include estimated AWS costs.
If actual AWS costs are lower than
estimated, Consultant will charge
City the lower costs. If actual AWS
costs are higher, City may apply
Additional Services funds to cover
the overage per Section 4, or the
parties will amend the Agreement
per Section 29.4 for City to add
funds to cover the overage.)
$192,000
(Paid)
$144,000
(Annual)
--
$144,000
(Annual)
--
$144,000
(Annual)
--
$204,000
(17 Months)
$192,000
(One-time
Paid)
$636,000
(Annual fees)
Task 3
(Perform hosting and networking
maintenance services)
$86,400
(Annual)
$86,400
(Annual)
$86,400
(Annual)
$122,400
(17 Months)
$381,600
(Annual fees)
Task 4
(Perform SAP Basis Services)
$8,640
(Paid)
$372,600
(Annual)
--
$372,600
(Annual)
--
$372,600
(Annual)
--
$527,850
(17 Months)
$8,640
(Paid)
$1,645,650
(Annual fees)
Subtotals for Services (Tasks 1-4) $808,140 $603,000 $603,000 $854,250 $2,868,390 Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3 Vers.: Aug. 5, 2019
Reimbursable Expenses (if any) $0 $0 $0 $0 $0
Subtotals for Services and
Reimbursable Expenses (if any) $808,140 $603,000 $603,000
$854,250
$2,868,390
Task 5
(Additional Services)
(Perform ad-hoc services on as-
needed requests that are not covered
in the Scope of Services. These
services constitute Additional
Services, Amendment No.1 under
Section 4 (Not to Exceed
Compensation) of this Agreement,
implemented pursuant to City-
approved Task Order as detailed in
Section 4.)
As needed, per
Task Order,
per Section 4
of this
Agreement,
Not to Exceed
Amount at Far
Right Across
All Contract
Years
As needed,
per Task
Order, per
Section 4
of this
Agreement,
Not to
Exceed
Amount at
Far Right
Across All
Contract
Years
As needed,
per Task
Order, per
Section 4 of
this
Agreement,
Not to
Exceed
Amount at
Far Right
Across All
Contract
Years
As needed,
per Task
Order, per
Section 4 of
this
Agreement,
Not to
Exceed
Amount at
Far Right
Across All
Contract
Years
$200,000
(Years 1-3)
$85,425
(Year 4)
Amendment No. 1, Up to Year 4
Maximum Total Compensation
$808,140* $603,000* $603,000* $854,250*
$3,153,815
*Maximum Total Compensation Amounts per each contract year in the table above do not include the cost of Additional Services t hat may be approved by City per Section 4 (Not to Exceed Compensation); the cumulative amount of which across all contract years shall not exceed the maximum amount for Additi onal Services in this Agreement.
REIMBURSABLE EXPENSES
CONSULTANT’S ordinary business expenses, such as administrative, overhead, administrative support time/overtime, information systems, software and hardware, photocopying, telecommunications (telephone, internet), in -house printing, insurance and other ordinary business expenses, are included within the scope of payment for Services and are not reimbursable expenses hereunder.
Reimbursable expenses, if any are specified as reimbursable under this section, will be reimbursed at actual cost. The expenses (by type, e.g. travel) for which CONSULTANT will be reimbursed are: NONE Docusign Envelope ID: ED1CBE7F-8D59-424C-ABDD-289D79D7B4A3