Scheduling hope and best practices?

hpl123

Hi all,
I use the scheduler in Aware extensively for all kinds of things and long long ago, I thoroughly tested the scheduler and it then always executed the processes at the right times i.e worked as it should. After that, I just relied on it working and standing in hope it always works (i.e I don´t have any control mechanisms in place to actually control if the processes execute daily, when they should etc. etc.). Today, I out of coincidence figured out that 1 of the processes that should have been executed this morning at 10 am did not execute and a chill went down my spine, could it be the scheduler doesn´t always work? If so, I have used it for years and maybe with suboptimal results? I know of course once in a blue moon something happens on a server, it restarts or whatever but that should be once in a blue moon and not a regular occurrence.

A couple of questions to aware people and/or support:

I have multiple processes e.g sending emails set at the same time say 10 am e.g 1 process that sends various emails related to 1 app, another process sending emails related to something else, a third process sending SMS and email related to a whole other thing etc. etc.. Is this a bad practice, is it better to spread them out timewise?
Could Aware miss some of all of the processes scheduled for execution say at 10 am if 1 of the process say takes over 1 minute to execute or are all processes executed in parallel?
Is it better to have 1 main process with many sub processes if I want to have all of the sub processes executed at the same time say 10 am?
Any one experiencing or have experienced issues with the scheduler?
Is it a good idea to create a flag/logging mechanism logging execution of the scheduled processes and then look at that from time to time to make sure all is running as it should? Anyone doing this an have some ca % about how well it works?

hpl123

hpl123 wrote
Is it a good idea to create a flag/logging mechanism logging execution of the scheduled processes and then look at that from time to time to make sure all is running as it should? Anyone doing this an have some ca % about how well it works?

Just built this to take a look in the coming weeks and months.

ACDC

I am working on some critical processes that are scheduled to run every day. You have shined a light on something I never thought of. Past experience with the scheduler have been good and I have assumed it to be very reliable, but I have never audited its process.

So yes I agree, it makes sense to extend each process in the scheduler to fire an audit rule on the last step, I am going to build that in. In fact this is a must for any critical automated background unattended process.

PointsWell

If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS

Pass the AuditLogBO to the Process

In this case I would ensure that your Processes are appropriately branched into sub processes. You could use the END PROCESS function I suppose but my personal opinion is that this is an inelegant way to end the process. Because you have added the AuditLogBO to the process you could add in logging by AuditLogBO.Audit=AuditLogBO.Audit+CURRENT_TIMESTAMP+' Step in process started'

When your process has come to an end the last process step should set the AuditLogBO.endtime

The only risk I suppose is if the process takes so long that the second scheduled transaction starts and finds the set of BOs that the first process is running against. You could prevent this by associating the target BO with the AuditLogBO for the period of the process and then remove it once the process ends

Example BSV

BobK

Henrik and others,

Here are my observations about Scheduling and some of my practices (which may or may not be BEST practices). Henrik, I think you will find answers to a couple of your questions in the following drivel.

In general:
I look at the Scheduling as a special Process that runs automatically about every 60 seconds.
Each Business Rule is processed, in order, 1 at a time until all Business Rules have been processed.
If it takes longer than 60 seconds for all Business Rules to be processed, the first Business Rule will be processed a second time even if the last Business Rule has not been processed yet. Just to be clear, the last Business Rule will still get processed. Just because the Scheduling has started another run, the first run of the Scheduling will still complete processing all of the Business Rules.

My System.
I have 20 processes that are essential to my system that run via the Scheduling.
Each process is contained in its own Business Rule, so the Scheduling has 20 Business Rules.
I want each of these processes to run as often as possible, so the Business Rules do not have any conditions and just 1 Action to execute one of the processes.
Some times, 1 or more of these processes take more the 60 seconds to complete.
If a process is currently running, I do not want this same process to be started again until the current run completes
To prevent a particular process from running multiple times concurrently, I have a BO (ControlRecord) that is used to keep track if process is running or not.
The system has 20 instances of the ControlRecord (1 for each process). The ControlRecord has the following attributes: "Type" which contains the process name, "Switch" which is a Yes/No attribute and indicates if the process is running or not and "TimeStarted" which is the date and time when the process was last started.
The first thing a process does is FIND the ControlRecord for this process and check the Switch. If the Switch is No, it gets set to Yes, the TimeStarted is set to the Current time and a sub-process is called to do the actual processing.
The last thing the process does is reset the Switch back to No.
I have a Query for the ControlRecord, so I can easily see which processes are currently running and when the last time each ran.

Some observations from the above.
Multiple processes can be running at the same time, which is OK as long as they are all different process types.
If a process is currently running the next time the Scheduling checks to see if it should run again, it will be skipped. In this system skipping a process is OK as long as it is currently running.
In this system, having a process run at a particular time down to the minute is dangerous because it is possible that the condition may not be checked during the 60 seconds when the condition would evaluate to true, or if the time was the only condition the process could run more than once because the condition was checked multiple times during the 60 seconds that the condition evaluates to true.
I also have another processes that needs to run once per day early in the morning. To accomplish that, I have the Business Rule for this process before the above Business Rules and this process has the following conditions that need to be true before the process run: IF CURRENT_HOUR >= 2 AND SystemSettings.LastTimeRun < CURRENT_DATE.
SystemSettings.LastTimeRun is the last time this process ran and gets updated to the CURRENT_DATE when this process starts. So this process will run anytime after 2 AM but will only run once a day.

Some issues I have run into
1) On rare occasions a process has run multiple times concurrently. This could happen if for example Proc01 is running, the Scheduling starts its next round and skips Proc01 and starts Proc02, but then Proc01 completes and Proc02 is started a second time because the Switch for Proc02 was checked before the Yes value was saved to the database.
2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.

joben

BobK wrote
2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.

What exactly is ControlRecord? Is it a built-in switch or is it a attribute you created? If it's built-in, how do we read its value?

Interesting thread. We have had issues for some time with scheduled processes not always running. We have created logging for when a process starts and when it ends. Some of our systems are heavily dependent on scheduling, and we have even went as far as using the Task Scheduler in Windows Server to launch a browser with a URL containing the command to launch a specific process. This is not only bad practice, but it doesn't always work.

Our approach right now is that we have rewritten our processes so that they can be run each day instead of weekly, or monthly, etc. It's the only solution we can come up with right now.

hpl123

PointsWell wrote
If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS

Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?

hpl123

BobK wrote
.... IF CURRENT_HOUR >= 2 AND SystemSettings.LastTimeRun < CURRENT_DATE.

Will try this instead, this should work better I would think and most of my processes doesn´t need to be executed at an exact time.

hpl123

Today all processes executed as scheduled and I will keep monitoring this, can´t believe I never thought about this before or set up some control mechanism before :oops: .

When it comes to the problem yesterday, the strange thing is I have 3 processes scheduled to run at 10 am and 2 of them did run but not the third one and is very strange, as some processes are executed the scheduler works and is doing what it should but it misses a process?

hpl123

BobK wrote
I look at the Scheduling as a special Process that runs automatically about every 60 seconds.

Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.

PointsWell

hpl123 wrote
PointsWell wrote
If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS

Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?

That is why I am suggesting that the scheduled process does only one thing - create a BO. Once the BO is created it is BO business rules running which have nothing to do with the scheduled process.

BobK

joben wrote
BobK wrote
2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.

What exactly is ControlRecord? Is it a built-in switch or is it a attribute you created? If it's built-in, how do we read its value?

ControlRecord is a BO I created and is found like any other BO. For example the first two Business Rules in Process01 are:

FIND ControlRecord WHERE ControlRecord.Type = 'Process01'

IF ControlRecord.Switch = 'No' Then
	ControlRecord.Switch='Yes'
	ControlRecord.TimeStarted=CURRENT_TIMESTAMP
	SubProcess01

The first Rule in Process02 would be:

FIND ControlRecord WHERE ControlRecord.Type = 'Process02'

and so on...

hpl123 wrote
PointsWell wrote
If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS

Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?

Not exactly. If a process runs a long time, that scheduler "thread" IS still active, but I would not say that anything is locked. When the long running process completes, the scheduler will move to the next rule. Also about 60 seconds after the first scheduler "thread" started, another scheduler "thread" will start processing all the rules one at a time. After the last rule in the scheduler completes, that scheduling "thread" dies.

If the scheduler only has 1 action and it runs a long time (more than 60 seconds), the second scheduler thread could start that 1 action again while the first thread was still executing that action. Unless the rule had a condition preventing the the second thread from starting it again.

BobK

hpl123 wrote
Today all processes executed as scheduled and I will keep monitoring this, can´t believe I never thought about this before or set up some control mechanism before :oops: .

When it comes to the problem yesterday, the strange thing is I have 3 processes scheduled to run at 10 am and 2 of them did run but not the third one and is very strange, as some processes are executed the scheduler works and is doing what it should but it misses a process?

For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.

BobK

hpl123 wrote
BobK wrote
I look at the Scheduling as a special Process that runs automatically about every 60 seconds.

Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.

Because of the issues with the logs, I do not use them much. But I believe that to get the rules executed via the scheduler to appear in the logs you need to select "Level of logging" of either "All server events excluding SQL" or "All server events including SQL"

Rem

BobK wrote
For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.

This is interesting ...

Question: are you referring to three different processes that, using the scheduler, are set to start at the same time or one process which starts another and then yet another?

If the first scenario, then this seems odd because in my world the processes would run simultaneously?
If the second scenario, why should the nested process bother about the clock (and so on)..

BobK

Rem wrote
BobK wrote
For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.

This is interesting ...

Question: are you referring to three different processes that, using the scheduler, are set to start at the same time or one process which starts another and then yet another?

If the first scenario, then this seems odd because in my world the processes would run simultaneously?
If the second scenario, why should the nested process bother about the clock (and so on)..

The processes I am referencing are Henrik's, so I am not exactly sure how the are setup. My answer assumes they are set up in the schedule as:

Rule 1
IF CURRENT_TIME='10:00' THEN Process1

Rule 2
IF CURRENT_TIME='10:00' THEN Process2

Rule 3
IF CURRENT_TIME='10:00' THEN Process3

hpl123

BobK wrote
hpl123 wrote
Today all processes executed as scheduled and I will keep monitoring this, can´t believe I never thought about this before or set up some control mechanism before :oops: .

When it comes to the problem yesterday, the strange thing is I have 3 processes scheduled to run at 10 am and 2 of them did run but not the third one and is very strange, as some processes are executed the scheduler works and is doing what it should but it misses a process?

For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.

Yes, all rules are scheduled to run at 10 am so I have 3 of these:

If CURRENT_TIME='10:00'  then Process

Again the strange part is 2 out of 3 processes executed yesterday, why would Aware do that? Each of the 3 processes could take over a minute to run if it has to execute many operations (send various emails, SMS etc.) and that was the reason I wondered if these were executed after each other instead of in parallel. If executed in parallel, all process would run at the same time but if not, the first process may take 40 seconds which means the second process would "make it" for scheduled execution at '10.00' but if that process then also takes 40 seconds, the third process would not "make it" as when the second process is done, the time would be '10.01'.

hpl123

BobK wrote
hpl123 wrote
BobK wrote
I look at the Scheduling as a special Process that runs automatically about every 60 seconds.

Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.

Because of the issues with the logs, I do not use them much. But I believe that to get the rules executed via the scheduler to appear in the logs you need to select "Level of logging" of either "All server events excluding SQL" or "All server events including SQL"

I tested this last night and think I had full logs activated and the scheduled events did not get entries in the logs. Trying tonight again with full logs just to make sure.

aware_support

Aware IM executes timer events in different threads. However, if a CPU is quite busy executing previous timer handlers and/or user requests, it is possible that the timer handler waits for the CPU to become free and misses its 60 seconds window.

So if you have a rule that checks for the exact time, it may miss an event occasionally. You need to handle this at the application level - the solutions outlined in this forum thread are quite valid.

hpl123

hpl123 wrote
BobK wrote
hpl123 wrote
Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.

Because of the issues with the logs, I do not use them much. But I believe that to get the rules executed via the scheduler to appear in the logs you need to select "Level of logging" of either "All server events excluding SQL" or "All server events including SQL"

I tested this last night and think I had full logs activated and the scheduled events did not get entries in the logs. Trying tonight again with full logs just to make sure.

Look like it does show scheduler events when you have full logs with SQL etc. enabled.