queue management

Rob O'Connell

We need to revisit our queues and I'm curious how others manage cross queue dependencies and prevent queues from running during planned outages (like deployments). Thanks!

Allen Roth

@Rob OConnell
For all scheduled queues we start the name with “SCHEDULED” to clearly distinguish those from ones that are run one-off. We then start the description with what the schedule is such as “DAILY, WEEKLY, MONTHLY” or the specific day of the week followed by the time that queue job is run prior to the actual description. It then is a very easy task to check which queue job(s) will be affected by a scheduled maintenance or outage.

Rob O'Connell

@Allen Roth
Hi Allen. Yes, we also include the frequency in the queue name (Daily, Weekly, Monthly). The problem is that we have a couple daily queues that are running during the weekly deployment window, however, we don't want to block out the 1-hour window each day since the deployments are weekly (if there is a deployment to production). Just curious how others have managed similar situations.

We also need to re-work our queues to make sure dependent steps are scheduled prior. An example is making sure static query refreshes are completed before the query is used (static child queries, static queries used for smartfield calculations, etc.). I'm thinking our queues need to be organized into different processes (Static query refresh, SmartFields, Global Changes, etc.). I'd be interested to hear how folks have their queues organized. Thanks!

Will Mercer

@Rob OConnell
We use a similar naming convention as well. We don't have a weekly deployment window, but if you do and it involves down time it does sound like it would be easier to just arrange the schedules around that knowledge? If there isn't a way you can do this, then you could always write something to disable / enable queue jobs and then run this on either side of your window.

We re-organised our queues a few years ago so we have a few master queues and then a series of sub-queues which are executed by the master queues. This has helped make things easier to understand and admin, but I don't have any specific advice on how to get the order right.

David Wanless

@Rob OConnell

We also include the schedule in all our Queue names, e.g., Daily 08:45, or Monday 07:30.

We don't have cross-queue dependencies - if B depends on A, we put B in the same Queue as A, but further down.

Kind regards,

David.

Rick Root

@David Wanless
We don't have any queues that run during the maintenance windows, so that's one way we avoid that as a problem! Nothing even really runs close to our maintenance window.

Many of our queues have the schedule (sort of) in the queue name.

Daily Every 5 Minutes Queue
Daily Every 30 Minutes Queue
Daily Evening Queue
Daily Midnight Queue
Daily Morning Queue
Weekly Receipting Queue (Monday/Wednesday)
etc….

We handle some dependencies by putting depending items in sub-queues. For example, each recognition program has a global change and then the recognition process. If the global change fails, we don't want to run the recognition process, but we still want it to move on to other recognition programs, so each recognition program has its own subqueue, and each of those is added to the main recognition program queue.

David Wanless

@Rick Root, yes, we use subqueues extensively.