Basics of Workload Automation

This transcript was generated from the recording and lightly edited for readability.

0:00 Welcome and agenda

All right, I think we are all set to go ahead and get started. So, yeah, just wanted to say good morning everyone and thank you all for taking a few minutes out of your day to join us. My name is Adam Hoover and I'm really thrilled to be here hosting today's webinar on the basics of workload automation. We've got a few topics to cover today. We'll start by just digging into a little bit of background on how we end up on the path to workload automation and where you might be along that journey. Talk a little bit about the importance of a long-term strategy and a few tips on how to get started. And then really the most exciting part is we'll cover some best practices about workload automation as well before we jump into some questions and make sure that we answer the things that are most important to you. And along those lines, I just want to mention that if you do have questions, please use the question and answer section of the Zoom. If you're if you're curious about something, you can post it there and that way we can see it. And so feel free to to ask those questions and as I mentioned, we'll get to them at the end of the webinar. All right. And then really quickly, why why are we the authorities on workload automation and why should you trust us?

1:23 Why JAMS on this topic

Well, really it's because here at JAMS, we are in the workload automation space intrinsically. We're proven at scale. We've got 35 years or more of experience in this space. And experience working with tons of customers of all shapes and sizes that all experience the same kinds of workload automation challenges that we're going to be talking about today. So really excited to dive into that more deeply. By way of just quick introduction, I'll reiterate that my name is Adam Hoover. I'm one of the product marketing managers here at JAMS. And just really can't say enough about how how exciting it is to continue to talk to you guys and share my experience working with customers like yourselves on the different kinds of challenges that that you're experiencing. And joining me today is Rob. Rob, >> how you doing? Rob Newman, solutions engineer here at JAMS. Had the pleasure of concentrating on workload automation for the past eight years with JAMS. So happy to be here and thank you. >> Yeah. Excellent. All right. So, first up, just a little bit of background and we'll talk about the path to workload automation and that really starts for a lot of folks in a number of different places and you may you may be at any number of different stages along this path but at a high level just in our experience we found that the path to automation maturity actually tends to follow a pretty predictable pattern.

2:56 How built-in schedulers turn into blind spots

It usually starts with something pretty simple. An initial request comes in, you know, this kind of very simple need where a process or a script just needs to run without human intervention and it's always going to be innocuous at first, right? And as a result, we end up with tactical solutions. So when when a request comes in, it's really natural to reach for the path of least resistance. You know, that's where you end up turning to built-in tools like Windows Task Scheduler, maybe SQL Agent, you know, for those of you who are managing Linux servers. You know, you you're probably using cron. You know, after all, these are free tools, right? They're native. They're already available and and really so the appeal becomes very clear. Now I think the important thing to highlight is that that simple ad hoc request that you've solved with one of those native tools it's going to breed more demand for more automation you know success always leads to to additional requirements and additional asks and so as those additional requests come in what we often end up seeing is that the volume of new disparate scheduled tasks can really quickly start to spread across various systems and servers. So if you're starting in Windows Task Scheduler, you may end up spreading across SQL, across Linux, you know, across your entire environment, and now you're using a bunch of different tools.

And what that really leads to is eventually, you know, and this is unfortunately, what started off as that simple one-off request will eventually lead to blind spots. And these may not even be noticeable at first, but over time what we find is that a lot of folks start to uncover unexpected inconsistencies in their data. And and that really, you know, is often attributable to somewhere down the line a job having failed silently. And if that goes unnoticed, the data remains in a bad state and can eventually lead to some pretty substantial effort to fix it. And as these kinds of issues start piling up, really the limitations of those free built-in tools, which started off so appealing, become pretty hard to ignore. And so this is where a lot of folks that we've worked with reach a tipping point, which usually comes when their team is actually spending more time babysitting jobs than focusing on really high value strategic work. And that's where a lot of folks will start to search for a more robust solution. You know, usually something that doesn't require them to stay up all night trying to, you know, manage these these tricky situations. And really, I just want to say no matter where you are on your journey, there's no shame in any of this.

5:40 Where teams are on the automation path

All those built-in tools, they exist for a reason. Reaching for them absolutely makes sense in those early stages, but as we just walked through, there's typically a point where those tools just really aren't designed to keep up with the requirements anymore, and that's why they end up falling short. So, let's talk about what mature workload automation might bring to the table instead. Before we do that actually would love to take a quick moment to see though you know we just talked about these six highle steps along the journey. We'd love to hear from you guys kind of where your organization has sits in that process today. So are you are you just dealing with your first couple automated tasks? Have your requirements started to grow? Are you uncovering new issues? Or you are actively in the process of seeking solutions. We'd love to hear from you in our poll. So, we'll give everyone a minute or two to to share. I see a lot of folks are kind of fairly well along the path here. Few folks in the expanding requirements category and a lot of people are in the process of where they're now uncovering issues. So that's really exciting to see because I think it means that you know there's a clear understanding that things need to change and often you know it's actually really helpful to be in the uncovering issues phase because it means that you can kind of get ahead of things before they reach a breaking point and also seeing a lot of people in the seeking solutions phase as well.

7:18 What workload automation changes

Great to see. Well, we're so happy you took the time to join us. We've got a lot of a lot of great content that Rob is going to cover in a moment. But before we get to that, I'll wrap up this poll. Hopefully everyone's had a chance to participate. Few more votes coming in. Great. All right. So, we talked about the journey. Let's let's talk a little bit about what workload automation really is and how it's different from those built-in schedulers. And so I think the main thing is that instead of being siloed like your native schedulers might be you end up when you're dealing with workload automation with a centralized solution. You know and it's really going to help you create a true orchestration layer that sits on top of all those different systems and tools that you're managing. And instead of being rigid rigid workload automation really focuses on being a dynamic and event driven instead of being sprawling we're focused on making sure that it's visible that everyone knows exactly what they need to when they need to. And that allows you to go from being reactive and dealing with issues as they arise to proactive and and really ensuring that your environment is robust ahead of time.

It also allows you to go from having f fragile processes that you know may end up requiring a lot of time to go debug or problem like troubleshoot takes you to a more resilient place where things are running on their own successfully. You're not having to babysit them. And then the last thing I'll mention is that it really helps take what is a disperate ungoverned set of tools and unify them into a secure and manageable really strategy. So with that in mind, we'll talk more about each of those capabilities in a moment, but I would love to also before we get into best practices and and a deeper dive on each of those topics to hear from you guys about what's most important to you. You know, what are the capabilities that you're most interested in or that matter most to your organization. So we'll do one more poll and then we'll get into the rest of our content. Give everyone just another minute to fill this out. And keep in mind it's multiple choice or multi select. So you can you can tell us it's just one thing or it's all of them. Seeing a lot of votes coming in for centralization of management for event driven.

I know. Yeah. I love event driven is a meaty topic and Rob I'm excited to to get your perspective on that in a few minutes. Proactive alerting also very popular. Excellent. Tells me there's some folks who are really interested in making sure they're staying on top of things and staying ahead of things. All right, give everyone just another 15 20 seconds and then we'll wrap this poll up and move on. All right, I think we can end that and I think it it looks like centralized management was the winner by a by a small margin. So that's great and we'll we'll dive into that in more depth in a moment. But before we did that, I wanted to just do one more step back and really talk about, you know, if you've reached this point where things are starting to become unsustainable, you know, I do think that in working with a lot of different customers, I've personally found that it can just be so tempting to really go and buy the first tool that's recommended by Chat GPT, right?

11:13 Building a long-term automation strategy

But before you do that, I want to say that it's important to take that step back and make sure you have a clear strategy in place just before you adopt something new. And that way you're not just ending up right back where you started. And so even though I talked about the benefits of mature workload automation, like I said, it's possible to end up running into similar issues in the future if you're not careful. So, it's important to to think about the long term you know and make a plan before you implement solutions. And I really think if you do that, there's four main benefits that you're going to find. And I know planning is not anyone's favorite part. You know, everyone wants to get right to okay, I've already solved all these problems and I'm a rockstar. But like I said, if you take this step back, you're going to find that things are more scalable. You know, by planning for growth and change for the start, your infrastructure will be designed from the get-go to incre to handle complexity and volume as that continues to grow down the road. And you'll also know that you're ready to support jobs and processes even if someone leaves, right? Or if systems change in the future. And that really helps you avoid tech debt for yourself. And so scalability a key benefit of actually taking the step back creating a long-term plan long-term strategy.

It also helps you align with other other organizations and aligning your teams internally, right? If you have standardized security policies, if you have standardized naming conventions and well-defined processes, that helps down helps break down silos and makes it so that development and support teams can function more efficiently together. And also helps new team members get up to speed much more quickly. So reduces that dependency on institutional knowledge. The last couple of course it's always good to be efficient and we think that when you have an automation strategy instead of tactically implementing you know new jobs as requests come in. You can really start to get a clearer understanding of the relationships between your systems, your jobs, the business processes that they support. And that's going to help you to really design new new logic that handles failures more gracefully, alerts the right people at the right time, you know, all that kind of good stuff. And the biggest one, you know, if you have a clear vision for the future and a clear understanding of where things are headed, that allows you to position yourself as a proactive partner to the rest of your business rather than a reactive kind of support function.

13:49 Finding high-value automation work

So by identifying those highv value automation initiatives you know the conversation really starts to shift a little bit instead of you know hey can you fix this or can you can you do this it becomes hey where are we going next and that's going to help you really function as that trusted partner for your business stakeholders. So at a high level where should you start? Well, in my mind, the the best place to start is really at the beginning and understanding, okay, what is my environment? Where are we going? And so that that really kind of encompasses five steps here. And each of these steps, I'll I'll just go through them very quickly, but each of these steps is meant to basically give you a clear picture of where you're going to be headed. So the first is we want to start by auditing our current jobs. You know, if you start by taking stock of everything that's already running and where your automated jobs live, you know, hey, are they spread across multiple servers and tools, how many of them are documented or just exist in someone's head. You know, a thorough audit is going to give you that baseline to make more informed decisions about what to keep, what to consolidate, and what to retire. And of course, not all jobs are created equal. So assessing their impact can also help you prioritize.

15:05 Tiering workloads by business impact

I always just recommend using a simple tiering system. You know, for example, like tier one can be payroll processing where we know a failure has immediate significant consequences, you know, and then maybe tier three is something a little bit more like a monthly log cleanup, which, you know, can wait a little bit if if we're doing a failover and we need to, you know, prioritize that. So, knowing which jobs are mission critical really helps you make smarter decisions about where to invest your time and and resources. Then we want to map out our infrastructure. You know, think about okay, where is everything running? Do we have to deal with file transfer tools? Maybe, you know, the organization is thinking about bringing in new cloud tools or an ERP system, you know, and so knowing what infrastructure requirements you have on your hands, having that mapped out clearly is important as well. And then the last two, identifying regulations and policies. This one always starts off low on folks list and then creeps up and becomes quickly the highest priority you know once once you know other teams get involved and and that's because automation really just doesn't exist in a vacuum. So understanding the broader compliance and security landscape, you know, where where's your industry headed, you know, is really helpful, you know, allows you to plan for the future and make sure that you're designing a strategy that's taking into consideration all these requirements and then document, document, document.

I think everything that you learn needs to be put somewhere that you can reference it moving forward. And yeah, so that's five highle steps to take to kind of assess your situation, capture the requirements that you think you're going to need to support in the long term, and then hopefully make sure that you're really thinking about the big picture as you move forward. But I know that what we really are all here and interested in is digging into some really key best practices. And so I am really excited to hand things off to Rob to tell us a little bit more about that. >> Thanks Adam, that was great. And let's dig a little bit deeper into some best practices. Let's start off talking about orchestration because you don't want to guess when your job should run. They should run as soon as they can and only when they should. This ensures timely execution as well as correct execution. Now this can be accomplished by moving beyond those static timebased schedules. We often call that pillow processing to real time eventbased triggers. Event triggers can include things like job completion events, you know, success or failure, file arrivals or dependencies, process queries, APIs, and email triggers.

17:51 Event-driven automation: triggers and dependencies

And these events are often spread across those desperate systems within your infrastructure, whether it be on premise or cloud-based. Yes, IT environments are often fragmented with infrastructure designed around disparit technologies. We often call this IT silos. Correct execution often requires dependencies across these systems across your infrastructure and you need to make sure that your reports don't run before your data is loaded. Workload automation is what allows you to connect across these systems using event-driven orchestration to ensure the timely and correct execution of your processes. Event-based triggers and cross-platform dependencies work together to enable real time automated workflows. This combination shifts workflows from reactive to proactive, reduces manual intervention, and creates a more agile integrated environment. Now, let's take a look at some of what this might look like. Triggers and dependencies are typically configured on individual jobs as well as a sequence of jobs as well as your workflows. And here you can see some available triggers.

So, triggers will submit a job every time the trigger criteria gets satisfied. Here we have schedule triggers, successor, job failure triggers. We have file, email and variable triggers. We also have dependencies. So dependencies are for jobs that are submitted that need to wait for maybe a job to finish, success or failure. Maybe wait for a file, a variable, email or resource to become available. And another easy way to build this out is to build out a sequence of jobs or a workflow running independent jobs. And this is where those crossplatform dependencies and those complex processes can be broken into smaller reusable jobs that incorporates both sequential and parallel processing. And for a more complex automation process, we can build out workflows that will incorporate any level of rules or resultsbased logic. Workload automation should also provide visibility and be proactive.

20:29 Centralized monitoring, alerting, and recovery

This all starts with centralized monitoring. You need a single pane of glass for that endto-end visibility that is critical to building reliable processes and making your IT environments manageable. The less time you spend fighting fires, the more time you spend developing solutions. This centralized view acts as a single source of truth covering your hybrid, your cloud, and your onremise environments. Let's stop using all those desperate siloed tools. Your workload automation should be proactive, not reactive. You'll want immediate notification to the appropriate team. This will reduce time to resolution and minimize impacts on your business. You can configure automatic alerts, retries, recovery options, and escalation processes. The goal here is to minimize intervention. Workload automation alerts should be actionable. They must include a clear description and relevant job context. They should be targeted. Let's avoid mass emails. Sending alerts only to the team that is capable of taking action. And they must be prioritized. Making sure escalations will only wake up on call staff for those critical issues.

So, real time monitoring improves workload reliability. It can drastically reduce delays caused by bottlenecks. And this is especially important for those SLA critical workloads. Combine that with alerts, notifications, and recovery options, and you are now reducing errors, improving consistency, and enhancing overall productivity. Now, what does that single pane of glass look like? In this monitor view, that single pane of glass, you can see everything that is currently happening across your infrastructure. Jobs that are currently running, jobs that are about to run, maybe jobs that are pending, prerequisite, and even recently failed jobs. This view should provide a lot of important details about your running jobs or processes like how long it's running, when it's expected to run, when it's expected to complete, who's running it, what ID it's running under. You should also have complete control from this view with options like cancelling, releasing, putting on hold, and rescheduling jobs.

23:03 Monitoring in JAMS Scheduler

You can even edit jobs. And if you want to dig a little deeper into finding out exactly what a job is doing, you should be able to do that as well. This view will allow you to see exactly what is going on with your processes and in fact give you complete control over that process right from this view as well. And if we jump back into the editor for that sequence job, you can see here I have escalating event handlers, email failure alerts, event failure alerts, notification jobs, and recovery jobs. And all of this together will ensure that the appropriate and expedient level of notifications and alerts will be sent out. Risk mitigation. Your workload automation should be both resilient and secure. You're going to want to start by breaking those complex monolithic tasks into smaller, manageable chunks.

24:22 Parallel tasks and reusable jobs

Independent tasks can run concurrently rather than sequentially. This will drastically reduce the total execution time of complex processes. Those independent tasks can be used across multiple sequences or workflows. This will reduce overall maintenance and technical debt. Recovery options will prevent monolithic failures, making it easier to isolate and rerun only the failed segment rather than that entire process. These options can range from simple retry and restarts from the point of failure to implementing recovery jobs that execute when issues arise or thresholds are met. These should be automatic responses to known issues like restarting a service or running a remediation script. And you'll want to use historical data to set realistic SLA thresholds. Rather than just reacting to failures, let your WLA tool anticipate SLA breaches before they happen. And of course, all this should be done before alerting a person. And lastly, workload automation should be secure. Security and compliance are obviously critical in any IT environment. You need to make sure that your workload automation does not introduce any security vulnerabilities.

25:41 Securing the automation platform

The tools typically store connection details, passwords, scripts, and other resources that need to be locked down. This should be done with role-based access controls to ensure users only have access to what they need for their job. The developers need access to production connections and credentials. Should HR have access to SFTP connections and credentials? Yeah, probably not. And being able to identify who has access to what and who made changes to what and when ensures a continued secure implementation. Auditing and version controls can help identify gaps that need to be addressed. Process optimization not only makes troubleshooting easier, but can significantly reduce failure risks and resource waste. Role-based access controls make it easy to appropriately lock down every folder, job, credential, and ideally every object within your workload automation tool. Now, let's jump back into that sequence. And here we can see a complex process that's broken down into independent chunks.

26:57 An example workflow

You've got jobs that are being reused across multiple sequences within the process and there are independent tracks running in parallel. This makes it very easy to troubleshoot and retry or continue past failures within your execution. And here I can simply retry this failed step or maybe I just need to continue on past it and let the rest of the processes complete. Role-based access controls allow you to lock down everything within your environment. This ensures you have a secure implementation. This is should allow you to lock down not only your jobs but your folders, your credentials, your connections, and in fact every piece within your workload automation tool. And with that, back to you, Adam. Sorry, I was just muted there.

28:13 Wrap-up and resources

All right. Well, thanks so much, Rob. I think that was really helpful to see. I'm just going to bring this up here. Great. So yeah, I know it was really helpful to me certainly to see how all the concepts that we talked about can actually work in the real world. Now as much as I hope that everything we've talked about today of course has given you a clearer understanding of workload automation and maybe what your next steps will be, I also know really that it's impossible to cover everything in just one webinar. And so that's why I just really encourage you to join us in our new discussion group, our discourse group. This is a brand new space that we put together for anyone with questions hopefully like yourselves to be able to to post questions and start discussions with their peers across industries. You know, just everyone really in the workload automation space. And of course our staff of subject matter experts will also be an active part of that community. So ask a question, start a discussion and that way you can benefit from from their insights and experience. Also you'll find links here and in the slides for some free tools that you can take advantage of today including a Windows diagnostic script. So if you are still in the process of kind of trying to wrangle some of those scheduled tasks that you've built out on Windows servers for example this is a really helpful tool to assess how things are going.

And then for those who are familiar with cron, we also have a free cron library which will really help you to understand any existing cron jobs that you have that you're managing as well as to build new ones in a more effective and comprehensive way. All right, with that I will open it up to questions. Let's see folks the question and answer section of the Zoom is where to go. I know we're at the top of the the session here and we've only got a maybe a minute left, but I just want to see if anyone does have questions. We're happy to talk about it. I think we must be up against time because we're not not seeing too many questions popping in. But if if you do have anything that you want to cover, as I mentioned, the the Discord group is a great place to get started. So, I'll just the Discord group I'll say. So, I'll just put the link back up on the screen. And with that, say thank you guys so much for taking the time to join us.