Virtualized Servers with Intelligent Monitoring Tools Maintain Service Levels Through Holiday Peak
State-of-the-art cloud-based information services require both robust infrastructure and real-time monitoring software that allows resources to be directed toward the needs of the moment and emergent issues to receive timely attention from the technical support staff. ShotSpotter’s cloud-based gunfire location and alerting systems have these needs and additional challenges that arise from the unique nature of the service.
Two times every year around New Year’s Eve and July Fourth, ShotSpotter experiences a peak in incident volume related to firework explosions and celebratory gunfire that multiplies the load on the computer systems that identify gunshots among the many other impulsive sounds like backfires and construction noise that are common in the urban environments of its customers. In 2014, the July 4th peak was further elevated because the holiday occurred on a weekend which is the time that illegal gunfire is most prevalent.
Planning for these 100x to 500x increases in server workload is in addition to the normal capacity planning that tracks the growth of ShotSpotter’s customer base which has roughly doubled each year.
By using server virtualization and real-time monitoring tools, ShotSpotter is able to manage the computing resources available to the busiest geographies during peak loads as well as the steady state increases in capacity associated with customer acquisition. The virtual machines that handle the incident processing are organized by time zone and geography. Large scale firework displays begin after sunset which is, of course, staggered in time across the country. The physical hosts supporting the virtual machines automatically shift processor and network resources to the busiest virtual machines. As activity falls off in easterly time zones, the resources are seamlessly shifted toward westerly time zones where activity is increasing. For both exceptional events like July 4th firework displays illustrated below and for illegal gunfire which is highest during late evening hours (10pm to 1am) on weekends the peak load follows the sunset. See this post here.
Eastern Time Zone
Central Time Zone
Western Time Zone
The physical host systems are large multi-core computers with large memory and multiple redundant network interfaces. These servers are backed by high-performance, tiered storage with caching, compression and de-duplication from Nimble Storage. The ShotSpotter environment uses Dell enterprise-class servers fully populated with memory and Westmere processors. Under normal loads, the physical hosts will support multiple virtual machines each using a fraction of the total capability with activity levels varying by time as noted above. Because the virtual machines may be dynamically reconfigured to have more virtual processors, memory and disk, it is possible to react in real time to exceptional increases by relocating virtual machines for less urgent functions, such as development systems or accounting systems, to lower performance hosts allowing the business critical gunshot locating systems to consume more of the available resource.
To enable the operations staff to make these judgments, the virtual and physical assets are under continuous monitoring. ShotSpotter employs Zabbix, an open source monitoring utility, monitoring tens of thousands of lightweight agents and thousands of event triggers. If a high impact condition is detected, the operations staff is immediately notified so that corrective action can be taken. The information is also summarized in trend charts so that planned responses are made to adapt for changing conditions.