RBMAS Screenshot - resized

Solving WebReports scheduling congestion

In this blog we’re going to talk about the WebReports scheduling feature. Apart from a primer on how this feature works, we’ll address a very common issue with this feature (if you guessed scheduling bottleneck, you’re in the right place), and some of the solutions to this problem that partners and customers have created over the years.  

Of course, we’ll also tell you about the Ravenblack Multi-Agent Scheduler, which was designed to solve this problem, as a logical, safe extension to the original WebReports scheduler. 

Why is Ravenblack qualified to discuss this?  
Good question! What gives us the authority to discuss this problem and evaluate solutions?  Well to start with, Ravenblack is made up of several members of a company called Resonate KT that originally invented, evolved, and supported WebReports including yours truly (the founder and technical architect of all things WebReports). In fact, the Multi-Agent Scheduler was developed by me and the original developer of the WebReports Scheduling feature. In our Ravenblack incarnation, we work with customers and partners to support them with WebReports development, so we get a broad view of the current state of the WebReports customer base, and all of our products are designed to solve typical customer problems.  

What does WebReports scheduling do?  
Here’s a quick introduction to the existing WebReports scheduler, as it was designed many years ago by our team. The WebReports module provides the ability for any WebReports developer to set up any WebReport to execute at a predefined time using the context of whichever user set up the schedule. This can be used to defer execution, or to set up a regularly occurring task that repeats a defined number of times, or indefinitely. This functionality has been used for decades to automate application (business) tasks, and to automate a variety of useful administration tasks.  
What problem are we solving?  
As mentioned above, in addition to building our own WebReports functionality, we work with customers and partners to support their WebReports development, so we fully understand common issues —like this one. 

The problem with the original scheduling functionality (that has emerged over the years) is that all scheduled tasks (WebReports) are executed (run) by one single automated agent, which means that all tasks due to run are executed sequentially.

This also means that every task waiting to run is dependent on any other tasks that are running ahead of them, and how long they take to complete.   

This situation can lead to sensitive application tasks such as sending emails, creating documents, initiating workflows, managing integration tasks, etc. being delayed for varying amounts of time due to other tasks that could run for several minutes (or even hours for heavy weight jobs).  

For additional context, watch this Ravenblack video about how our Multi-Agent Scheduler product provides the next generation of managed WebReports Scheduling.

What does the Ravenblack solution do?  
As a starting point, let’s review what the Ravenblack Multi-Agent scheduler (RBMAS) does.  Where the original WebReports scheduler has only one automatic “agent” to handle all of the task, the RBMAS allows the creation and management of multiple agents to share this work (as configured by an administrator).  While this might sound like a simple concept, the implementation is not that straightforward. To use an analogy, if you hired multiple people to perform actions on a task list, you would need some way of making sure that everyone knew what tasks they were performing and when. No tasks getting repeated, and no tasks getting missed. I have seen customers with knowledge of Content Server configuration create (deliberately or accidentally) multiple agents without doing anything to manage the workload sharing, leading to chaotic results. Thus, our product introduces a software mechanism to solve the basic management of tasks across multiple agents. 

Another key function, among others, that our product provides is an interface that allows administrators to easily create, edit or delete agents, a function that otherwise becomes an error prone manual configuration task. In addition to managing agents, this user interface (UI) also provides a way for users to dynamically assign any WebReport to any agent (without a restart). This powerful UI allows customers to create multiple purpose-built agents to ensure that business critical tasks can be isolated from other potentially large tasks.  

Alternative approaches 
Over the years, and before we created the Multi-Agent Scheduler, our customers and partners have come up with a variety of workarounds to alleviate this problem.  
We can’t claim to have seen all of these workarounds but here are four approaches that we are aware of, with our initial thoughts on suitability: 

Approach  Comments

Build a unique Oscript based solution (on a per customer basis).   

Many customers are resistant to Oscript based solutions for a variety of reasons. To our knowledge there are no such solutions that are built as a multi-customer product. 

Note that while the Ravenblack solution does provide a new version of the OScript script for the scheduling agent, it is primarily a WebReports-based application, minimizing the need for Oscript objects and an Oscript module. Additionally, the Ravenblack solution is created as a multi-customer, commercial offering, supported by some of the original team that created WebReports.  

Buy, configure or create client software that runs on a schedule, initiating requests to the scheduled WebReports on the server.    

This particular approach is often deployed as a point solution to segregate a particular, problem WebReport that is "misbehaving" and slowing up all the other scheduling tasks.  While it does address a specific issue, it also creates a scenario where a heavy weight task is being performed on an adhoc basis, external to the core system, requiring different tools (and sometimes different personnel) to manage and support.  It is also not a holistic solution as additional problem WebReports (or sensitive WebReports) may also need solutions that could result in a number of externally scheduled WebReports in addition to the organically growing system based WebReport schedules.  For an administrator, this can become complicated, and very difficult to manage effectively, particularly if there is a need to troubleshoot any of the non-standard schedules. 

Our description of the Ravenblack solution (below) speaks in more detail about the benefits of using our built-in extension to the original scheduling solution, but by way of comparison addresses the disadvantages of this alternative approach in more depth. 

Use Workflow scheduling 

This adds unnecessary complexity to any solution, making it harder to maintain and support. Additionally, while this approach uses distributed agents through Workflow, it does not allow the ability to dedicate a specific agent and thread to specific applications or tasks (as the Ravenblack solution does). 

Use WebReports triggers 

The WebReports trigger feature does not provide a direct replacement for scheduling, but it does provide an alternative to solving certain business problems. Typically, this involves a decision as to whether any processing work should be done in immediate response to an event (such as a user adding a document), or at a later time when a scheduled WebReport is set up to perform the work.  Each approach has pros and cons, and often a balanced approach (column A and column B) is the right answer; however, some customers and partners have solved scheduling congestion by increasing the use of triggers to offload the scheduling tasks.  

In some cases, this could be the right solution, but in other cases this could lead to a misuse of WebReport triggers, causing other issues such as excessive user wait times instead of solving the root issue.  

Our point of view is that the Ravenblack Multi-Agent Scheduler is preferable to these approaches, which is of course why we developed this solution. As the original designer of WebReports scheduling, we are comfortable saying that this is phase two of the original architecture and is an improvement that extends the original architecture in a robust and secure way.   

The benefits of using Multi-Agent Scheduler 
To speak to some obvious benefits of our approach, it’s a relatively minor change to the existing, reliable, common approach to WebReports automation. Everything runs within the Content Server environment, without the need for any external hardware and or extra client software running.  
More specifically: 

  • Virtually no moving parts are required, as each agent runs on the admin server and directly invokes WebReports through the purpose built, internal WebReports API.   
  • No external login is required as each agent adopts a pre-defined user context for each WebReport with code running directly on the server.  
  • If the server is running – the agents are running; no other processes to start, run, or manage.  
  • Thanks to the use of co-located server agents:  
    • No external client (and possibly additional hardware) is required. 
    • No external scheduling software is required. 
    • No external monitoring is required (to ensure continuity).  
    • No internet connectivity is involved: 
      • Better reliability. 
      • Better security (user context created internal to the server). 
  • Content Server request software is bypassed as the user’s original request context is used for each execution.  
  • Uses the extremely robust and reliable agent system that conforms to the existing Content Server monitoring software and logs system. No additional supervision or monitoring required.  
  • Uses a dedicated thread for each agent as opposed to random thread assignment which allows for dedicated, easily identifiable log files (thread, trace, connect) for each scheduling agent that the administrator creates. 
  • Scheduler information logs are merged with normal WebReports logs allowing easy debugging where necessary–everything in one place! It also allows for easier timing and analysis of performance regarding scheduling performance versus WebReport performance.  
  • Existing customer support personnel can easily support this approach (no additional support personnel, or training required).  
  • Covered by existing support protocols for server uptime and monitoring.  
  • Managed using existing admin pages and a similar paradigm to the original scheduler.  
  • Works in conjunction with the native WebReports destination and scheduling code.  
  • All scheduling follows the same paradigm and code—no bespoke, non-core software or components required. 

This diagram illustrates some of the differences between our extension of the existing WebReports solution, and a client-based scheduling solution.  

Comparison Diagram

Note that each one of the internal agents has a dedicated thread.  Any issues with WebReports that have been assigned to a given agent are easily analyzed through the log file dedicated to that agent, with all the output for the scheduler as well as the WebReport combined in the output from the single thread.  

Scheduled WebReports in this model are never in competition with any users and are usually running on the admin server. Obviously, they are in competition with any WebReports assigned to the same agent but that is completely under the control of the administrator.

In addition to these core benefits, the Multi-Agent Scheduler also provides some other extensions to the existing Admin functions. Under the current UI feature set, it is not possible to edit a schedule that was created by another user.  The new UI allows any schedule to be edited, and makes enable/disable and delete functions more interactive. The display has been enhanced in various ways, but more importantly it includes a mini traffic light to indicate the status of the agents, and a separate “maintenance” page that provides for several configuration options including the ability to initiate a restart from the browser.

In summary, the Ravenblack Multi-Agent  Scheduler runs directly on the server using native technology and Oscript to run any scheduled WebReports, using an assortment of uniquely created agents to manage the workload in an optimum fashion.  
Robust, secure, and reliable! 

Multi-Agent Scheduler Product Page