Scheduling hope and best practices?

If you have questions or if you want to share your opinion about Aware IM post your message on this forum
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Scheduling hope and best practices?

Post by hpl123 »

Hi all,
I use the scheduler in Aware extensively for all kinds of things and long long ago, I thoroughly tested the scheduler and it then always executed the processes at the right times i.e worked as it should. After that, I just relied on it working and standing in hope it always works (i.e I don´t have any control mechanisms in place to actually control if the processes execute daily, when they should etc. etc.). Today, I out of coincidence figured out that 1 of the processes that should have been executed this morning at 10 am did not execute and a chill went down my spine, could it be the scheduler doesn´t always work? If so, I have used it for years and maybe with suboptimal results? I know of course once in a blue moon something happens on a server, it restarts or whatever but that should be once in a blue moon and not a regular occurrence.

A couple of questions to aware people and/or support:
- I have multiple processes e.g sending emails set at the same time say 10 am e.g 1 process that sends various emails related to 1 app, another process sending emails related to something else, a third process sending SMS and email related to a whole other thing etc. etc.. Is this a bad practice, is it better to spread them out timewise?
- Could Aware miss some of all of the processes scheduled for execution say at 10 am if 1 of the process say takes over 1 minute to execute or are all processes executed in parallel?
- Is it better to have 1 main process with many sub processes if I want to have all of the sub processes executed at the same time say 10 am?
- Any one experiencing or have experienced issues with the scheduler?
- Is it a good idea to create a flag/logging mechanism logging execution of the scheduled processes and then look at that from time to time to make sure all is running as it should? Anyone doing this an have some ca % about how well it works?
Henrik (V8 Developer Ed. - Windows)
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Re: Scheduling hope and best practices?

Post by hpl123 »

hpl123 wrote: Thu Apr 22, 2021 6:05 pm - Is it a good idea to create a flag/logging mechanism logging execution of the scheduled processes and then look at that from time to time to make sure all is running as it should? Anyone doing this an have some ca % about how well it works?
Just built this to take a look in the coming weeks and months.
Henrik (V8 Developer Ed. - Windows)
ACDC
Posts: 1138
Joined: Sat Jun 30, 2007 5:03 pm
Location: California, USA

Re: Scheduling hope and best practices?

Post by ACDC »

I am working on some critical processes that are scheduled to run every day. You have shined a light on something I never thought of. Past experience with the scheduler have been good and I have assumed it to be very reliable, but I have never audited its process.

So yes I agree, it makes sense to extend each process in the scheduler to fire an audit rule on the last step, I am going to build that in. In fact this is a must for any critical automated background unattended process.
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Scheduling hope and best practices?

Post by PointsWell »

If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS

Pass the AuditLogBO to the Process

In this case I would ensure that your Processes are appropriately branched into sub processes. You could use the END PROCESS function I suppose but my personal opinion is that this is an inelegant way to end the process. Because you have added the AuditLogBO to the process you could add in logging by AuditLogBO.Audit=AuditLogBO.Audit+CURRENT_TIMESTAMP+' Step in process started'

When your process has come to an end the last process step should set the AuditLogBO.endtime

The only risk I suppose is if the process takes so long that the second scheduled transaction starts and finds the set of BOs that the first process is running against. You could prevent this by associating the target BO with the AuditLogBO for the period of the process and then remove it once the process ends

Example BSV
Last edited by PointsWell on Fri May 21, 2021 3:23 am, edited 2 times in total.
BobK
Posts: 544
Joined: Thu Jan 31, 2008 2:14 pm
Location: Cincinnati, Ohio, USA

Re: Scheduling hope and best practices?

Post by BobK »

Henrik and others,

Here are my observations about Scheduling and some of my practices (which may or may not be BEST practices). Henrik, I think you will find answers to a couple of your questions in the following drivel.

In general:
I look at the Scheduling as a special Process that runs automatically about every 60 seconds.
Each Business Rule is processed, in order, 1 at a time until all Business Rules have been processed.
If it takes longer than 60 seconds for all Business Rules to be processed, the first Business Rule will be processed a second time even if the last Business Rule has not been processed yet. Just to be clear, the last Business Rule will still get processed. Just because the Scheduling has started another run, the first run of the Scheduling will still complete processing all of the Business Rules.

My System.
I have 20 processes that are essential to my system that run via the Scheduling.
Each process is contained in its own Business Rule, so the Scheduling has 20 Business Rules.
I want each of these processes to run as often as possible, so the Business Rules do not have any conditions and just 1 Action to execute one of the processes.
Some times, 1 or more of these processes take more the 60 seconds to complete.
If a process is currently running, I do not want this same process to be started again until the current run completes
To prevent a particular process from running multiple times concurrently, I have a BO (ControlRecord) that is used to keep track if process is running or not.
The system has 20 instances of the ControlRecord (1 for each process). The ControlRecord has the following attributes: "Type" which contains the process name, "Switch" which is a Yes/No attribute and indicates if the process is running or not and "TimeStarted" which is the date and time when the process was last started.
The first thing a process does is FIND the ControlRecord for this process and check the Switch. If the Switch is No, it gets set to Yes, the TimeStarted is set to the Current time and a sub-process is called to do the actual processing.
The last thing the process does is reset the Switch back to No.
I have a Query for the ControlRecord, so I can easily see which processes are currently running and when the last time each ran.

Some observations from the above.
Multiple processes can be running at the same time, which is OK as long as they are all different process types.
If a process is currently running the next time the Scheduling checks to see if it should run again, it will be skipped. In this system skipping a process is OK as long as it is currently running.
In this system, having a process run at a particular time down to the minute is dangerous because it is possible that the condition may not be checked during the 60 seconds when the condition would evaluate to true, or if the time was the only condition the process could run more than once because the condition was checked multiple times during the 60 seconds that the condition evaluates to true.
I also have another processes that needs to run once per day early in the morning. To accomplish that, I have the Business Rule for this process before the above Business Rules and this process has the following conditions that need to be true before the process run: IF CURRENT_HOUR >= 2 AND SystemSettings.LastTimeRun < CURRENT_DATE.
SystemSettings.LastTimeRun is the last time this process ran and gets updated to the CURRENT_DATE when this process starts. So this process will run anytime after 2 AM but will only run once a day.

Some issues I have run into
1) On rare occasions a process has run multiple times concurrently. This could happen if for example Proc01 is running, the Scheduling starts its next round and skips Proc01 and starts Proc02, but then Proc01 completes and Proc02 is started a second time because the Switch for Proc02 was checked before the Yes value was saved to the database.
2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.
Bob
joben
Posts: 221
Joined: Wed Nov 06, 2019 9:49 pm
Location: Sweden
Contact:

Re: Scheduling hope and best practices?

Post by joben »

BobK wrote: Thu Apr 22, 2021 9:40 pm 2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.
What exactly is ControlRecord? Is it a built-in switch or is it a attribute you created? If it's built-in, how do we read its value?

---

Interesting thread. We have had issues for some time with scheduled processes not always running. We have created logging for when a process starts and when it ends. Some of our systems are heavily dependent on scheduling, and we have even went as far as using the Task Scheduler in Windows Server to launch a browser with a URL containing the command to launch a specific process. This is not only bad practice, but it doesn't always work.

Our approach right now is that we have rewritten our processes so that they can be run each day instead of weekly, or monthly, etc. It's the only solution we can come up with right now.
Regards, Joakim

Image
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Re: Scheduling hope and best practices?

Post by hpl123 »

PointsWell wrote: Thu Apr 22, 2021 9:37 pm If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS
Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?
Last edited by hpl123 on Fri Apr 23, 2021 9:45 am, edited 1 time in total.
Henrik (V8 Developer Ed. - Windows)
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Re: Scheduling hope and best practices?

Post by hpl123 »

BobK wrote: Thu Apr 22, 2021 9:40 pm .... IF CURRENT_HOUR >= 2 AND SystemSettings.LastTimeRun < CURRENT_DATE.
Will try this instead, this should work better I would think and most of my processes doesn´t need to be executed at an exact time.
Henrik (V8 Developer Ed. - Windows)
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Re: Scheduling hope and best practices?

Post by hpl123 »

Today all processes executed as scheduled and I will keep monitoring this, can´t believe I never thought about this before or set up some control mechanism before :oops: .

When it comes to the problem yesterday, the strange thing is I have 3 processes scheduled to run at 10 am and 2 of them did run but not the third one and is very strange, as some processes are executed the scheduler works and is doing what it should but it misses a process?
Henrik (V8 Developer Ed. - Windows)
hpl123
Posts: 2579
Joined: Fri Feb 01, 2013 1:13 pm
Location: Scandinavia

Re: Scheduling hope and best practices?

Post by hpl123 »

BobK wrote: Thu Apr 22, 2021 9:40 pm I look at the Scheduling as a special Process that runs automatically about every 60 seconds.
Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.
Henrik (V8 Developer Ed. - Windows)
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Scheduling hope and best practices?

Post by PointsWell »

hpl123 wrote: Fri Apr 23, 2021 9:40 am
PointsWell wrote: Thu Apr 22, 2021 9:37 pm If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS
Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?
That is why I am suggesting that the scheduled process does only one thing - create a BO. Once the BO is created it is BO business rules running which have nothing to do with the scheduled process.
BobK
Posts: 544
Joined: Thu Jan 31, 2008 2:14 pm
Location: Cincinnati, Ohio, USA

Re: Scheduling hope and best practices?

Post by BobK »

joben wrote: Fri Apr 23, 2021 7:46 am
BobK wrote: Thu Apr 22, 2021 9:40 pm 2) If a process crashes because of an internal error, the Switch for that process never gets set back to No, so this process would never run again. For this situation, I have another process that runs about every 15 minutes and checks each ControlRecord. If the ControlRecord Switch is Yes and the ControlRecord TimeStarted is more than 30 minutes ago, the Switch is reset to No. Under normal processing, my processes should not take 30 minutes to complete.
What exactly is ControlRecord? Is it a built-in switch or is it a attribute you created? If it's built-in, how do we read its value?
ControlRecord is a BO I created and is found like any other BO. For example the first two Business Rules in Process01 are:

FIND ControlRecord WHERE ControlRecord.Type = 'Process01'

IF ControlRecord.Switch = 'No' Then
ControlRecord.Switch='Yes'
ControlRecord.TimeStarted=CURRENT_TIMESTAMP
SubProcess01

The first Rule in Process02 would be:

FIND ControlRecord WHERE ControlRecord.Type = 'Process02'

and so on...

hpl123 wrote: Fri Apr 23, 2021 9:40 am
PointsWell wrote: Thu Apr 22, 2021 9:37 pm If your concern is that the scheduler starts a process that runs too long and prevents the scheduler starting another process then I would approach this by decoupling the scheduler from the process.

You could create an Audit log BO. Give it a start and end time attribute and any logging that you want.

When your scheduled process is to run have it do only one task CREATE AuditLogBO WITH AuditLogBO.start=CURRENT_TIME

Then have a rule on the AuditLogBO IF AuditLogBO IS NEW THEN RUN PROCESS
Is this how it works? If the scheduler executes a process with a single action in it, isn´t the scheduler "thread" still "active" and locked until all rules, subprocesses etc. etc. are executed?
Not exactly. If a process runs a long time, that scheduler "thread" IS still active, but I would not say that anything is locked. When the long running process completes, the scheduler will move to the next rule. Also about 60 seconds after the first scheduler "thread" started, another scheduler "thread" will start processing all the rules one at a time. After the last rule in the scheduler completes, that scheduling "thread" dies.

If the scheduler only has 1 action and it runs a long time (more than 60 seconds), the second scheduler thread could start that 1 action again while the first thread was still executing that action. Unless the rule had a condition preventing the the second thread from starting it again.
Bob
BobK
Posts: 544
Joined: Thu Jan 31, 2008 2:14 pm
Location: Cincinnati, Ohio, USA

Re: Scheduling hope and best practices?

Post by BobK »

hpl123 wrote: Fri Apr 23, 2021 9:44 am Today all processes executed as scheduled and I will keep monitoring this, can´t believe I never thought about this before or set up some control mechanism before :oops: .

When it comes to the problem yesterday, the strange thing is I have 3 processes scheduled to run at 10 am and 2 of them did run but not the third one and is very strange, as some processes are executed the scheduler works and is doing what it should but it misses a process?
For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.
Bob
BobK
Posts: 544
Joined: Thu Jan 31, 2008 2:14 pm
Location: Cincinnati, Ohio, USA

Re: Scheduling hope and best practices?

Post by BobK »

hpl123 wrote: Fri Apr 23, 2021 9:52 am
BobK wrote: Thu Apr 22, 2021 9:40 pm I look at the Scheduling as a special Process that runs automatically about every 60 seconds.
Makes sense this is how it works and things like this should ideally be logged somewhere i.e logged internally in the Aware logs. Is this logged somewhere i.e when the scheduler checks and executes scheduled processes? From what I can see, these processes and not logged in the regular production log and the server logs are very difficult to look at and see things that happened in the past. It would be very useful if we either knew how to look at a scheduler specific log or if scheduler events were shown in the production etc. logs as well.
Because of the issues with the logs, I do not use them much. But I believe that to get the rules executed via the scheduler to appear in the logs you need to select "Level of logging" of either "All server events excluding SQL" or "All server events including SQL"
Bob
Rem
Posts: 216
Joined: Wed Oct 03, 2007 8:58 am
Location: Sweden
Contact:

Re: Scheduling hope and best practices?

Post by Rem »

BobK wrote: Fri Apr 23, 2021 1:43 pm For the 3 processes scheduled to run at 10 AM are all 3 in a different rules and each rule has a condition like IF CURRENT_TIME = '10:00''?

There is no guarantee that the scheduler starts exactly on the 00 second, for example it could start at 10:00:20.
If the scheduler starts at 10:00:20 and your first process runs for 25 seconds and the second process runs for another 25 seconds, the time will be 10:01:10 when the condition on the third process is evaluated. The times will not match and the process will not run.
This is interesting ...

Question: are you referring to three different processes that, using the scheduler, are set to start at the same time or one process which starts another and then yet another?

If the first scenario, then this seems odd because in my world the processes would run simultaneously?
If the second scenario, why should the nested process bother about the clock (and so on)..
Regards
Rune

Image
Post Reply