title
CC - (1) Approval of a Professional Services Agreement with Utility Systems, Science & Software (US3) in an Amount Not-to-Exceed $352,769 to Replace and Install Equipment; Make System Improvements, Provide Electrical Engineering Services, Evaluate All Pump Stations, and Make Recommendations for Optimizing the Sanitary Sewer’s Emergency Notification System (ENS); and (2) Authorize the Public Works Director/City Engineer to Approve Amendment(s) to the Agreement up to a Contingency Amount of $27,231.
body
Meeting Date: September 13, 2021
Contact Person/Dept: Gabe Garcia/Public Works
Phone Number: (310) 253-5633
Fiscal Impact: Yes [X] No [] General Fund: Yes [] No [X]
Public Hearing: [] Action Item: [] Attachments: []
Commission Action Required: Yes [] No [X] Date:
Public Notification: (E-Mail) Meetings and Agendas - City Council (09/08/2021);
Department Approval: Charles D. Herbertson, PW Director/City Engineer (09/01/21)
_____________________________________________________________________
RECOMMENDATION
Staff recommends the City Council (1) approve a professional services agreement with Utility Systems, Science & Software (US3) in an amount not-to-exceed $352,769, to replace and install equipment, make system improvements, provide electrical engineering services, evaluate all pump stations, and make recommendations for optimizing the Sanitary Sewer’s Emergency Notification System (ENS); and (2) authorize the Public Works Director/City Engineer to approve amendment(s) to the agreement up to a contingency amount of $27,231.
BACKGROUND
On June 11, 2018, the City Council approved a five-year professional services agreement with US3 for the maintenance of the sanitary sewer pump stations’ ENS. This maintenance agreement does not include funding for equipment, installation, any system improvements, or consulting services. The project proposed in this staff report bridges the gap to provide equipment, its installation, system improvements, and consulting services for the work proposed in this staff report. Staff recommends the City Council approve a separate agreement with US3 for this one-time project.
The ENS utilizes Motorola’s Supervisory Control and Data Acquisition (SCADA) for monitoring and control with radio-to-radio communication to integrate the City’s five sanitary sewer pump stations into an intelligent technology network that assesses mission-critical components and functions and reports pump stations condition and malfunctions in their operation as alarms to City staff who in turn respond to prevent sewage spills.
The ENS also reports alarm incidents from the pump stations via Motorola radio-to-radio communication to a central unit and set of computer servers (system central) located at City Hall. The servers and assistant hardware then deliver text messages (SMS) to first responders’ mobile devices, and concurrently sends emails to responsible Public Works Department staff. This set redundancy is to ensure delivery of critical information to prevent sewage spills. The ENS is a mission-critical component of the City’s sanitary sewer Spill Prevention Program.
The ENS employs Trihedral Engineering Virtual Tagged System’s Supervisory Control and Data Acquisition (VT SCADA), which integrated with Motorola’s SCADA give intelligence to the ENS system in order to continually track system conditions, maintain databases, and notify staff of incidents requiring Public Works first responders’ action to prevent sewer spills.
Additionally, this integrated SCADA system provides the same state-of-the-art security features that Motorola provides for mission-critical SCADA systems in military and critical enterprise networks.
DISCUSSION
Given its technology and critical nature of its mission, the sanitary sewer system’s ENS is a 911-level critical system in ensuring the efficient operation of the City’s pump stations and the prevention of sewage spills.
Intermittent failures of critical equipment are becoming more frequent thereby requiring greater staff and consultant time to repair and bring such equipment back online. For example, two hot-swappable servers are more than 10 years old and have long exceeded their expected life along with legacy versions of the SCADA system, the operating system, drivers, and peripheral equipment including modems, monitors, and KVM switch. Additionally, staff and consultants have identified the need to institute new system control components to ensure that sewage flow in the pump stations are conveyed to the appropriate system pipes, and the need for new alarms in check valves are required.
Further as needs have arisen over time, some unique components have been introduced that tend away from uniformity. For the sake of efficiency, such unique equipment needs to be replaced with uniform components. For example, Human-Machine Interface (HMI), are dashboards on screens located in the control room of each of the pump stations, and are used to control and show complex pump and sewer flow volumes and other data into useful information for first responder staff in addition to logging history of operations and trending. The absence of uniformity from one HMI to another detracts from staff’s ability to respond and run the pump stations efficiently and to ensure the equipment is running appropriately.
Additionally, the need for further redundancy in critical components has been identified as necessary given the experience of previous failures. This includes having dual wet well level sensors, which provide data that cause the pumps to activate to reduce wet well levels and avoid a sewage spill. If a single level sensor is installed and fails, the risk of sewage spill is increased.
Given that the wet wells are corrosive environments, staff and consultants have identified the need to install stainless steel conduit and mounting hardware to protect and ensure data input from the equipment to the ENS is safeguarded.
The above experience by staff and consultants has informed the proposed project scope listed below.
Proposed Project
1. Provide and replace two hot-swappable redundant servers, the operating systems, Trihedral’s Virtual Tagged SCADA with capacity for 25,000 tags, and all supporting software and drivers. Create approximately 20,000 tags for tracking all ENS conditions, all required programming, calibrations, configure all software, and perform and pass site-acceptance testing to the satisfaction of the City;
2. Provide, install, connect, and configure a rack-mountable UPS to maintain powered servers and equipment in the system central at City Hall. This will ensure that when an electrical outage occurs, power to the servers is uninterrupted until City Hall’s emergency generator responds;
3. Provide, install and configure dual redundant modems, as well as two POTS (plain old telephone service) lines and internet communication line;
4. Provide, install and configure two complete workstations, one at City Hall and another at the Public Works yard including PC, monitors, and cabling to review the ENS real-time status;
5. Provide an app to function on any mobile platform for Public Works’ first responder staff to monitor the ENS status in real-time, view alarms, and acknowledge alarm conditions;
6. Install dual level sensors at each wet well in all pump stations, connect them to the SCADA system, calibrate them, program the HMI’s to reflect the two sensors, provide programmed functionality so that staff can change the active level sensor from their ENS app, and log which sensor is active;
7. For all sensors at all pump stations, replace all termination boxes, sensor support and mounting hardware, conduit/tubing, unistruts, nuts, bolts, and clamps with stainless steel;
8. Remove the HMI at the Bankfield pump station and return to staff, and replace it with the HMI from the decommissioned Overland pump station;
9. Implement new controls and data points (Hz motor speed, operational current, voltage, pump run time, and all data points to be determined by staff) for Bankfield, Braddock, Fox Hills, Bristol, and Jasmine pump stations, by installing conduit as required, adding new ethernet cabling, connecting to new I/O cards in the variable frequency drives, routers, and ACE3600 central processing units, and programming all required new SCADA tags. Add all new data points to each HMI at all pump stations and program all drives;
10. Create new alarms to log and report when one or more of the following conditions are present at the system central at City Hall or at the pump stations:
a. One or both hot-swappable redundant servers are off-line.
b. Communication line to provide ENS conditions to first responders is down.
c. Emergency bypass pumps at each pump station has been activated.
d. Emergency generators have been turned on or turned off.
e. Check valves are not closed while pumps not running.
11. Decommission in the fall of 2021 based on staff direction the Mesmer and Overland pump stations, salvage all ENS equipment, and provide to staff for storage to be used as spares when the need arises;
12. Secure and provide all US3 and third-party communications data contracts, services, subscriptions, equipment, and hardware for communication of alarm conditions and staff response. All ongoing costs for all communications shall be the responsibility of US3 for the remaining term of the Maintenance Agreement through the end of fiscal year 2022-2023; and
13. Evaluate all active pump stations’ ENS system components and communications, as well as City Hall’s ENS central and communications, inspect all connections, review current energy efficiency, and submit a report making recommendations for improvements needed to optimize the ENS system and ensure its proper operation, as well as to inform future maintenance needs and staff’s preventative maintenance schedule.
Competitive Bidding Requirements Do Not Apply
Pursuant to Culver City Municipal Code (CCMC) Section 3.07.055.A, formal competitive bidding is not required in this case, because the needed equipment, supplies or services are proprietary items of original equipment manufacturers and/or their authorized exclusive distributors. Further, this procurement meets the criteria for a sole source exemption, pursuant to CCMC Section 3.07.055.C (see Attachment 1).
Additionally, the introduction of a third party firm would create a gray area as to demarcation of responsibilities, in light of the existing maintenance agreement with US3, and may lead to contractors deflecting responsibility to each other.
FISCAL ANALYSIS
The cost for all project equipment required to meet the scope of work shall no exceed $152,624. The professional services by US3’s electrical engineers, to review, inspect, analyze, program, install, and integrate all components of the project shall not exceed $163,845. The professional services to assess the ENS system at all pump stations and City Hall central, evaluate all components, communications, inspect connections, review energy efficiency, and make recommendations, shall not exceed $36,300.
The aggregate scope of work described above comes to a total project cost not-to-exceed $352,769.
Staff is also requesting authorization for the Public Works Director/City Engineer to approve amendment(s) to the US3 agreement to cover unexpected costs in an amount not-to-exceed $27,231, if needed.
The combined project cost and contingency amount total $380,000. This project was adopted by the City Council in the Fiscal Year 2021/2022 Capital Improvement Budget as project PS012. If approved, the Professional Services Agreement would be funded by the Sewer Enterprise Fund, budget account number 20480000.619800.PS012 Other Contractual Services. It should be noted that there is sufficient funding available for this request.
ATTACHMENTS
1. 2021-09-13_ATT 1_Sole Source Letter
MOTION
That the City Council:
2. Approve a Professional Services Agreement with Utility Systems, Science & Software (US3) in an amount not-to-exceed $352,769, to replace and install equipment, make system improvements, provide electrical engineering services, evaluate all pump stations, and make recommendations for optimizing the Sanitary Sewer’s Emergency Notification System (ENS); and
3. Authorize the Public Works Director/City Engineer to approve amendment(s) to the agreement up to a contingency amount not-to-exceed $27,231; and
4. Authorize the City Attorney to review/prepare the necessary documents; and
5. Authorize the City Manager to execute such documents on behalf of the City.