Collaboration Moves Utah Toward Next-Gen 911

This is an article in Emergency Management Magazine about the phone project I have been working on for the last three years and the project that I won Utah APCO’s Technician of the Year Award.

Collaboration Moves Utah Toward Next-Gen 911

When the technology in several of Utah’s 911 centers was nearing obsolescence simultaneously, three of the state’s public safety directors decided to get creative.

“We thought, why not work together?” said Tina Scarlet, executive director of the Weber County, Utah, Emergency Services District. “Why not implement a shared model that various public safety answering points could use and that would be more efficient?”

In November 2011, Scarlet and two others — former Salt Lake Valley Emergency Communications Center (VECC) Executive Director William Harry and Utah Department of Public Safety Dispatch Center Manager Chris Rueckert — submitted a request for information for a multi-node, IP-based 911 call-handling solution.

“We wanted a system with advanced call-routing capabilities that could give us greater efficiencies and ultimately cost less than our various on-premise solutions,” Scarlet said. “The upgrade to an Internet-based system would also allow us to spread 911 calls around so that no one center had to put off a call when it was too busy.”

Today, the Greater Wasatch Multi-Node Project is the first IP-capable 911 call delivery system in Utah.

Multi-Node Advantages

Once the trio decided to explore the new strategy, they compared their current vendor’s solution to alternatives offered by Intrado and CenturyLink. Both companies offered key features they desired, including an emergency service number-based ESInet for all 911 wireline and wireless call handling and routing.

“We decided on the Intrado solution because it would allow both servers to be up and operational at the same time,” said Kevin Rose, statewide interoperability coordinator for the Utah Department of Technology Services. “Redundancy was paramount.”

The IP-based Intrado VIPER call processing equipment provides quadruple redundancy at VECC and Weber County, as well as a common database for call routing and mapping systems that displays the caller’s location.

In January 2013, VECC, Weber Area 911 and the Utah Department of Public Safety/Salt Lake Communications Center went live on the new ESInet and the Greater Wasatch Multi-Node Project was officially launched.

“About that time, several of the other public safety answering points (PSAPs) became intrigued about the direction this was going,” said Mark Whetsel, technical services manager at VECC.

The original three partners realized that the new system would allow them to host much of the new technology at two centers — VECC and Weber Area Dispatch — and give other centers that got smaller upgrades access to the full system via the Internet. That would allow them to combine resources and share the cost of fully upgrading the two centers instead of spending the same amount of money on all of them, Rose said.

“As we heard of other agencies getting ready to switch out their equipment, we asked if they wanted to join. We liked the fact that it reduced costs, but it was also about the ability to back each other up,” Scarlet said. “We are all large entities, and with the capabilities of the system and the way we can have the roaming login, we felt it would be a good fit to have others join in. Before we knew it we were adding additional members.”

Today, the Greater Wasatch Multi-Node Project supports six state-level and local PSAPs covering four counties in Utah, including the Bountiful City Police Department, Greater Salt Lake Unified Police Department and Salt Lake City Police Department.

The Multi-Node Project has two servers, one in Weber County and one in Salt Lake VECC.

“The servers are synchronized images of each other connected with T1 circuits,” Whetsel said. “Because we have a major fault line that runs over half the state, we wanted to create a geo-diverse network that would enable redundancy and resiliency.”

Whetsel said that because Weber County is about 60 miles north of Salt Lake City, it was a logical place to put the second node.

“If a major earthquake should hit, the chances of both of those centers being affected simultaneously is relatively remote,” he said. “And with the multi-node system, if one server goes down, the second server can pick up without missing a beat.”

Collaboration among the partners also generates cost savings through shared resources and equipment.

“Anytime you have multiple PSAPs working together it is a huge benefit because we can share information, training, etc.,” Scarlet said.

The Multi-Node Project also enables “agent roaming,” which lets users share call-taking positions across different PSAPs during times of high-call volume.

“A 911 call taker or dispatcher can log into any shared workstation, at any location, and receive and dispatch emergency calls as if they were at their own PSAP,” Scarlet said. “This collaboration of individual PSAPs and sharing of resources is unprecedented in Utah.”

 

Challenges and Unknowns

Naturally, forging new ground also came with challenges.

“There were a lot of unknowns because this was the first time such a system has been done in Utah,” Scarlet said. “There was a lot of pressure to ensure it succeeded. We had a great deal of confidence in the vendors, but the pressure was there.”

Scarlet said the partners agreed early on that this was a long-term relationship and it was imperative they were all on the same page. The team put together memorandums of understanding to establish how all system components would work and formed a governance structure including representatives from each partner agency.

In all, the system took 18 months from initialization to launch. A significant portion of that time was spent establishing network facilities and formalizing contacts with vendors. The three original partners also stipulated that the new system must be cost neutral, which threw another wrench into the plan.

“We couldn’t put new equipment in and have it cost the agencies more than their current budgetary allotment for that service,” Whetsel said. “Trying to maintain cost neutrality and keeping within those fiscal boundaries with the new equipment coming in was a challenge.”

Once the system went live, with a staggered start for each partner, the new dispatcher interface took some adjustment as well.

“Trying to coordinate training was time consuming,” Whetsel said. “Because it was a new deployment in Utah, we had to work closely with support technicians on the ground. We are still trying to muddle through the governance structure. We put together a panel of representatives from all the respective agencies, and we meet at least monthly and discuss what’s going on, direction, etc. We are working on formalizing the group, but it’s still in its infancy.”

 

Looking Forward

In addition to its other advantages, the IP capability of the Greater Wasatch Multi-Node Project has set the foundation for other next-gen 911 data services and applications, such as text-to-911.

“We are looking to have text messaging available by May 2014,” Scarlet said. “We’re working on establishing best practices for that now. Also when someone calls 911 now, there is a database that provides coordination between the phone number and the location of the caller. It is old school and requires a lot of manipulation. So we are beginning the process of converting from that old-school way of handling the auto number and auto location to a format that is more dynamic and uses geospatial data as opposed to a flat file.”

With the goal of being better prepared for next-generation 911 within two years, Utah is also working toward a CAD-to-CAD system that would allow for the electronic exchange of information between 911 operators. Utah also hopes to expand the backbone of the new 911 system throughout the state.

“There has been some discussion about applying this model throughout the state — to other regional or shared systems, so it’s given us some different options to explore,” Rose said. “We’d like to see if this model could work in other areas of the state and look at different options for the future that could save the state money and provide us more flexibility.”

Scarlet said she’s not sure how many other PSAPs in the state might emulate the Multi-Node Project. “In terms of how many others would do combined systems, that will be up to the PSAPs,” Scarlet said. “But I have heard that other PSAPs are talking about doing something similar. Anytime you can maximize your support and interoperability, it makes sense operationally and it makes sense for the citizens.”

While the project was the first of its kind in Utah, it could help set the stage for future initiatives.

“This was definitely something new to the state of Utah, but we got a lot of support for it from the surrounding areas,” Rose said. “Because it was so new we had to do quite a bit of educating to get buy-in. It was an eye-opener, and I think it will possibly influence other types of projects in the state down the road.”

You may use or reference this story with attribution and a link to
http://www.emergencymgmt.com/safety/Utah-Next-Gen-911.html

Utah APCO Technician of the Year

My boss nominated our team for the Utah APCO Technician of the Year award.  I have included the nomination below.  Last week we received notice that we won and will be honored on October 2nd at the APCO Fall Conference.
Utah APCO Annual Award. 
Technician of the Year
I feel that this year’s award should be presented to the Technical Staff at Salt Lake Valley Emergency Communication Center.  This year has been an extremely busy one for the TS team at VECC.
On November 28 at 03:00 hrs they began a process to upgrade VECC to a brand new phone system (Viper) provided  by Intrado. This upgrade was the first in Utah that provided end to end IP based phone delivery for over 900,000 residents in the Salt Lake Valley. Due to technical difficulties in the network the VECC TS team was required to put in a 20 hour shift.  They returned after 4 hours to complete the upgrade the following morning. VECC was only the first PSAP to move to the Viper system in December of 2012 members of the VECC TS team traveled to Weber 911 and assisted with the migration there. In April 2013 Unified Police joined the network and the VECC TS team once again traveled to UPD and assisted in their migration process. Bountiful Police Department came online in May of 2013 and VECC TS also assisted them. Their willingness to assist all of the agencies was inspiring
Once the NG9-1-1 systems were up and operational VECC TS personnel then took on a second extremely large project. This project began as a simple internet upgrade. VECC had been running for 12 years on a single T1 internet connection. Due to increased needs for web based applications it was determined that VECC needed to increase to a 20 megabit connection. On the surface it seemed like
a simple project but as it turned out it required that every IP address for every machine some 250 devices need to be touched and a new address entered. It also required extensive coordination between VECC vendors and user agencies to make certain that firewalls and other devices were updated. This procedure is very similar to removing all of the veins and arteries in the human body and replacing them with new, while maintaining the life of the patient.
All of this was accomplished while maintaining the day to day operational needs and requirements of the center. There are numerous other accomplishments the team has achieved during this year, too many to list here.
I believe that the dedication and resourcefulness of the entire VECC Technical Service Team are worthy of recognition by Utah APCO.
The Team:
Lin Shaffer, Eran Bair, George Priday, Brice Rawlings, Joe Borgione(Contract), Brandy Lucas
Mark Whetsel
Technical Services Manager  Salt Lake Valley Emergency Communications Center

How to Get VMware running on an HP DL380 G8

At my day job we are experimenting more and more with server virtualization. We have been playing with VMware as our platform. We currently run one physical server (host) with 6 different virturalized servers.

We recently purchased a new HP DL 380 G8 server with dual Intel® Xeon® E5-2603 (1.80GHz/4-core/10MB/80W) Processors and 64 GB (16x4GB) Dual Rank x8 PC3L-10600 (DDR3-1333) memory. This would supplement our current VMware server and give us redundancy and fail-over capabilities.

The first step in the process was to download the latest and greatest currently at VMware ESXi 5.1. I download it and its in the form of an ISO. Not a big deal I have a utility to burn it to DVD and make it bootable for install. Whoops the new DL380 doesn’t have a CD/DVD drive.

I find a tool on line called Linux Live USB Creator. It works great to create a bootable USB device using Linux. The tool allows you to include your ISO to start as well. The tool worked flawlessly. However I discovered that the VMware ISO I had was missing some HP Drivers.

I quickly searched the VMware site and found the HP specific ISO. Unfortunately the links for the HP stuff were broken.

Another quick search and I found another tool to create a HP specific ISO. The tool is the ESXI Customizer. It requires the installation of the VMware vSphere PowerCLI.

Linux Live USB Creator

ESXi-Customizer

HP Customized ESXi 5.0 ISO

HP LaserJet 4100n Printer Repairman

Being a jack of all trades in IT means that everyday there is a new challenge.  A few weeks ago our printer in the Technical Services department began making a terrible racket. It sounded terrible. It made an extremely loud popping noise.

It was near the end of our budget year and our budget was gone.  I knew that a service call to Les Olsen or similar repair company costs a bare minimum $200 just for them to come out and diagnose the printer.  I thought I might google and see if we could figure out the problem.  A quick search for HP LaserJet 4100n grinding noise and I find out its a common problem.  I watched a youtube video on how to repair it.

Brokenprinter.com has lots of how to videos.

I begin disassemble and find we have another part that needs replacing. I order both needed parts for around $60 total and with in days the printer is back up and operational.  This printer project took about 2 hours in research and repair. The parts were found online from several parts vendors.  I saved the company a couple hundred bucks and picked up some skills in the process.

HP LaserJet 4100n

 

Network Addressing Make Over

We had outgrown our IP allotment and it was time to upgrade. For the first 23 years of VECC we had enough IP addresses to do the things that we needed. We had used up our 254 pubilc IP’s. We didn’t own our own IP’s they were leased from Xmission.  We found out through Century Link that VECC was entitled get more public IP’s through a program that had been set aside for public safety.  We quickly filled out the paper work and got two class C public address ranges.  This was more than enough to do what we had in mind.

When I first started at VECC the software that we used required public IP addresses now this was no longer the case. We could now go to private addressing and VLAN out our networks. I was tasked with making this happen for VECC.

The goals were simple.

  1. Change all the IP addresses in our network to private including Windows and Unix servers
  2. Create VLANs to segregate our different needs
  3. NAT our private server addresses to public IPs.
  4. Make all the necessary routing changes at VECC and coordinate all the routing changes with our different agencies.
  5. Update our two Cisco ASA firewalls to the latest software.
  6. Install and configure Cisco’s AnyConnect VPN
  7. Less than an hour of down time.

 

I was able to accomplish all of this in just a couple of weeks.  I coordinated with all the different VECC agencies to make the switch on our Frame Relay connections, our Cisco to Cisco IPSec connections as well as our basic internet routing.

I created 5 VLANs for our various needs like servers, workstations, web servers, wireless internal and external. I configured all of our switches and the appropriate ports for the VLAN.  I mapped out the entire physical network, something that had never been done at VECC ever.

This project was a huge undertaking due to all the different agencies that connect to VECC.  My boss credited me with it being a huge success.