JAMS Automation as Frequent as Every 4 Seconds: Southwest Power Pool

COMPANY BACKGROUND

Southwest Power Pool (SPP) is one of nine Independent System Operators mandated by the Federal Energy Regulatory Commission (FERC) to ensure reliable supplies of power, adequate transmission infrastructure, and competitive wholesale prices of electricity. The Arkansas-based company serves nine states and more than 100 power producers, marketers, and cooperatives 24 hours a day, 7 days a week, 365 days a year.

THE CHALLENGES

Southwest Power Pool operates much like an air traffic control system, synthesizing power transmission data from more than 100,000 data points, every 4 – 30 seconds. Real-time processing is an absolute necessity as their system connects the region’s power suppliers to businesses, municipalities and households that depend on them.

Prior to implementing JAMS, SPP adopted batch scheduling tools according to the technologies each business unit used. Windows groups used Task Scheduler. Linux groups used Cron. The company also implemented Automate, a proprietary scheduler, but it proved unreliable as it was not server-based and required an actively logged-in client to function consistently. It also lacked support for Windows clustering on virtual machines. (SPP operates more than 1,000 VMWare and ESX Servers.)

According to Senior Programmer/Analyst Brad Cowell, the mix of systems caused a support nightmare. Southwest Power Pool, being highly regulated, required an automation architecture with granular enterprise security settings on job definitions and user roles. Also, in order to deliver accurate numbers to the volatile energy transmission market, the team needed a solution that could reliably run 100-150 jobs every hour, with some recurring as frequently as every 4 seconds. (It takes a lot of jobs to solve a 33,650 x 80,548 matrix every two minutes!)

THE SOLUTIONS

Southwest Power Pool found JAMS Job Scheduler matched well with its process intensive environment.

The IT team selected JAMS for its rich feature set, architecture, reliability and cross-platform capabilities. Once they installed JAMS, a process the organization found easy, they successfully migrated all of their heterogeneous batch processes into JAMS.

In addition to built-in execution methods, Southwest Power Pool took advantage of the ability to run any kind of script in JAMS. They leveraged JAMS to run their batch scripts, SQL SSIS packages, PowerShell scripts and C# code all within the larger context of their enterprise workflows.

Once they established automated workflows, the team integrated JAMS alerts with its existing support system, giving agents and business users alike the ability to receive detailed notifications about the status of critical jobs.

Today, there are nearly 60 people across production, support, development, and various business units, leveraging JAMS.

THE BENEFITS

JAMS saves Southwest Power Pool valuable support time, especially for its after-hours support teams. Agents can now author scripts and easily set them to run during later maintenance windows.

The organization has also eliminated a manual system check that took place every night (including weekends) – one that typically took 20 – 30 minutes. With JAMS, the IT team uses a scheduled job to query one of the organization’s Oracle databases for certain significant records. If the job fails, the team receives an email alert and they can address the underlying issue first thing in the morning.

In another example Cowell shared, JAMS flags jobs that execute successfully, but that exhibit a particular error condition, e.g. an expiring certificate. The provision of a full log file through JAMS helps SPP agents address error conditions before they cause a job to fail. “When there’s a problem, we know about it quickly,” he says.

Cowell describes JAMS as “super flexible”, noting that its robust job definitions menu offer multiple methods of matching a job’s schedule with the organization’s business rules.

With heterogeneous workflows now centralized in JAMS, Southwest Power Pool has brought reliability and resiliency to the “always on” processes used to manage power transmission throughout the region.