Updating Records

[ab042]

I have a process that has only two rules. The client file has around 15,000 records and is running on a machine with 2gig RAM.

#1) FIND ALL Client
#2) If Client.Company CONTAINS ('DESK') THEN
Client.Type = 'FURNITURE' ELSE
IF Client.Company CONTAINS ('BATH') THEN
Client.Type = "BED&BATH' ELSE
IF Client.Company .........

This process almost totally locks up the machine with JavaW taking over 98% of the process time. It never appears to finish.
------------
If I change the #1 RULE to:
FIND Client WHERE (Client.Company CONTAINS 'Bath')
and run a rule with 3 different options with IF and IF THEN it doesn't update all the records as the rules define.
------------
If I make the following Process everything works, but with the following will take me a lot of Processes to do what I need.
Rule #1: FIND Client WHERE (Client.Company CONTAINS 'Bath')
Rule #2: Client.Type = "Bed&Bath'

===========
I guess I have a few questions:

#1) Should a FIND ALL with 15,000 records and ONE rule with Several (9 I believe) IF THEN hang the server?

#2) Does IF THEN work in Processes, I could be doing something wrong. I am new? But then again if it doesn't work should an error be displayed?

#3) During all of this I found myself looking at "Active Processes" menu option and CANCEL didn't appear to work when the system was high CPU and the Progress % never did show anything other than Zero.

[aware_support]

The simple solution would be to add "IN BATCHES OF 15000" at the end of FIND action. This should fix the problem.

If you do not specify the size of the batch the default batch of 1000 is used, so it processes 1000 records first, commits the transaction, then processes the next 1000 etc. It shouldn't hang the server, though - it should suspend and resume the process 15 times. What does the LogViewer display?

IF ... THEN ELSE is perfectly fine within a process - your rules are correct. It appears that batch operations and not behaving properly but we need to know what the log says. Then again if you specify the batch size of 15000 you shouldn't have a problem.

[aware_support2]

It may be better to move the rules from the process to the Client object. There would be no need to use ELSE, each rule should be entered separately, for example:

If Client.Company CONTAINS ('DESK') Then Client.Type = 'FURNITURE'
If Client.Company CONTAINS ('BATH') Then Client.Type = "BED&BATH'
.....

The system will automatically execute the rules on affected Client instances whenever Client.Company changes.

In general, it is the recommended approach with Aware IM to place rules on objects whenever possible and avoid creating unnecessary processes. The business logic is more clearly visible when all processing rules are attached to applicable objects - not standalone processes.

[ab042]

I added the IN BATCHES command and the log showed where that did find the records at the very start of the process.
-22
-2
-16
-33 Found xxx objects

It then repeated the following:
-6 Evaluating Condition Client.Company CONTAINS 'Desk'
-7 Condition Evaluated to False John Doe and Company
.... and then the next Company Name etc etc
-----------------
I left it run for around 15 minutes and the ONLY way to abort it was to shutdown the server from the control panel.
FYI: CANCEL on the active process screen does nothing and it also showed ZERO progress the complete time.
-----------------
Since the objects where not appearing in the log in any order and there was no way to know if the process was repeating or working I also tried adding a ORDER BY in the FIND command and ran it again.

The objects still appeared in random order in the log and I'm not sure if that command worked or not and still couldn't figure it out.

On your "Move the Rules" suggestion. I do understand what your saying here. But this was to be a ONE TIME process to update an IMPORTED client list and get the Type field defined. I do not want this automated selecting when a user Add's or Edit's a client record.

I figured this would be a 3 minute EASY ONE TIME process, but after 5-6 hours, I finding that not to be the case.

Hope this helps.

[aware_support]

Looks like you didn't wait long enough.

Could you try a smaller batch size - for example, 20? It could be much faster. You could also try turning off logging temporarily. Give it enough time.

[ab042]

I have a few questions please. And it should be noted that I have tried changing several of the memory -Xmx type settings in StartAwareIM.bat and the startoptions.props and it didn't appear to help with this.

Logging was already off (Or set to Minimum). Except to get the info you requested. I assume thats as off as it gets?

Changing it to a batch size of 20 did complete the process in 7 minutes for around 12,000 records.

#1) Why was it in Batches of 15000 then Batches of 20? I understand that it processes X # of records and then commits and continues. But why is it that I'm having to set this and what makes 20 work better than 15000 and how would I know what to set it to.

#2) What is the answer for other update processes that I will be creating? (20 or 15000 or maybe 500 or 100000??) Shouldn't the default work or at the very least not lockup a system and require AwareIM server to be shutdown?

#3) Why doesn't the Active Processes work? It shows the Performance % as ZERO, when in fact I should assume something is happening. And why can't I kill the process if it hangs? Not that it should be hanging!

#4) If the expected performance during a VERY SIMPLE batch process takes the CPU to over 98% on a machine with 2gig of RAM. You've got a major flaw in performance design somewhere.

#5) Can we change the % of process used during batch processes or something? Does reports do the same thing? My users won't be happy if 3 users start some kind of report or batch process and everything comes to a stops.

Please advise

[aware_support]

#1) Why was it in Batches of 15000 then Batches of 20? I understand that it processes X # of records and then commits and continues. But why is it that I'm having to set this and what makes 20 work better than 15000 and how would I know what to set it to

If you don't set the value the default of 1000 will be used, so strictly speaking you do not have to set it. You do need to think about the batch size if performance/memory is an issue. Aware IM loads all found records in a batch into memory and then performs rule processing in memory. If your batch size is large and each business object has lots of attributes this can take up a lot of memory. However, if further processing is simple, it may be faster than processing small batches. On the other hand, if processing is complicated large batches may take up CPU time (and memory). Most of the time we recommend small batches. Aware IM commits transactions after every batch.

2) What is the answer for other update processes that I will be creating? (20 or 15000 or maybe 500 or 100000??) Shouldn't the default work or at the very least not lockup a system and require AwareIM server to be shutdown?

The default is good when the number of found records is less than 1000, which is the overwhelming majority of cases. If it is greater than 1000 like in your case you need to change the default to a smaller value like 20 or 50 or 100 (you can experiment which value gives the best performance)

#3) Why doesn't the Active Processes work? It shows the Performance % as ZERO, when in fact I should assume something is happening. And why can't I kill the process if it hangs? Not that it should be hanging!

The performace of the process is a function of how many transactions have been committed. If you specify a large batch size, so that everything happens within one transaction you won't be able to see the performance or cancel the process - another reason to use small batches if processing of a large batch takes a lot of time.

#4) If the expected performance during a VERY SIMPLE batch process takes the CPU to over 98% on a machine with 2gig of RAM. You've got a major flaw in performance design somewhere

Updating records en masse is never simple. If you have thousands and thousands of records that you want to process and update you need to fine tune your batch size.

#5) Can we change the % of process used during batch processes or something? Does reports do the same thing? My users won't be happy if 3 users start some kind of report or batch process and everything comes to a stops.

Reports work differently. Starting a batch process that updates thousands of records is never a good idea if the system is under a heavy load. It is recommended to do such batch processing overnight or whenever very few users are working with the system.

[ab042]

THANKS.

I have the javaw in startawareim set at -Xmx128m and the business object is only normal Business Card type stuff so its not that big.

I have a good size server with lots of memory. If this simple batch brings it to a halt may I suggest you take a look at the logic if someone doesn't key a Batches of command. Something doesn't appear right.

That being said the Batches of 20 did work and I thank you for the fast response and will use that until I here otherwise.

Can you tell me where I change the DEFAULT batch size so I don't have to change it everywhere?

[aware_support]

Unfortunately at the moment you cannot control the default batch size so you have to set it explicitly in every FIND action where the default of 1000 is not good enough.

[denisv]

I have a very large amount of data that I am importing into Aware and the performance is oor. Can I use the In Batches for this sort of operation ? Would it help there ?

[aware_support]

Performance of import largely depends on whether validation is on or off. The batch size may be important as well (particularly if validation is on)