When AwareIM dies

[ACDC]

My logic, the database is number 3 on the list of priorities. If the two java processes are running well within the memory allocation then the DB needs to be looked at.

When you reset the control panel it resets the connection to the DB and all is well until the concurrent threshold is reached again

MySql has no explicit memory setting. It sets up a memory ceiling based on its configuration, so if its not configured correctly it wont use available memory and will run very inefficiently

The amount of users you have and the size of memory thats being utilised by the DB seems odd to me unless your app is very small. Hence the focus on the DB

[kklosson]

This is informative. Thanks.

[hpl123]

Kklosson,
If you figure this out and find potential bottlenecks or configuration misses in the default Aware installation please share a comment or two.

One thing I have thought of looking into is more advanced server monitoring tools that could monitor server (CPU, memory), MySQL and Java etc. processes for a server. There exists tools that actually monitor all of these in 1 tool and then one can see performance ETC. over time. If you or anyone else either looks into this or already know of tools we could use please share the knowledge.

[aware_support]

The most important thing is to collect the output of Tomcat and the Aware IM server AS SOON AS the system dies. The post-mortem analysis depends on what's in this output. The rest is pure speculation and guessing in the dark.

By the way, one of the latest builds of 6.0 introduced more robust handling of the situations when Tomcat lost connection with the server for whatever reason. If this happened the entire system used to go down. Now it doesn't - it just reconnects and carries on. It's quite possible that you are experiencing the same issue and update to the latest version should fix the problem.

[kklosson]

Good news. I'm about to update the production system to 6.0 and I'll report back on this thread.

[kklosson]

I continue to have issues of my server simply dying - it just stops responding to anything. The memory usage of the two java processes is fine, cpu consumption is fine, there is really no reason for the server to stop responding but is does. I have to restart the AwareIM stack then all is well. I cannot believe that the system is overwhelmed - can't me more that 20-30 users. Running build 2028, 64 bit JDK, latest MySQL connector. Very frustrated...

[aware_support]

You need to capture the output of Tomcat and Aware IM Server when "death" happens - it can be million of reasons not related to Aware IM. We do it by running Aware IM as a service (albeit in 32-bit), in which case all the output is in the logs.

[kklosson]

To be clear, you are referring to the Tomcat and AwareIM logs available at the AwareIM Control Panel, yes?

A. I'm afraid I don't understand the significance of running AwareIM as a server relevant to the logs.
B. I normally examine both logs when the stoppage occurs and normally don't see anything that appears to be a problem. A recent stoppage produced the attached in the Tomcat log that was abnormal....

[aware_support]

The log and service is relevant because often you are not present when something happens so you cannot capture the output. You have to be present right at the moment when the server dies, otherwise output may become overwritten with stuff that happens after the problem. With the logs captured in a file you can look at the entire history - especially if you roughly know the time of the problem.

[kklosson]

So I have a situation where I cannot login nor can some others. But as I look at the logs there is considerable activity occurring. It seems that something has reached its max capacity. Plenty of RAM and CPU overhead available. Nothing in either the AwareIM or Tomcat logs that say anything useful. Then things slow down, less log activity, and I assume the system has simply died. Restart AwareIM control panel to restore operations. There's almost no question that I am failing under load of users but can't figure out what to change to make it better.

I have:
Build 2028
Windows server 2008 R2
AWS EC2 Intel Xeon CPU ES-2650 @ 2.00GH
8GB RAM
2048 on AwareIM
2048 on Tomcat
64 bit JDK
Latest MySQL connector
Stock Tomcat installation

[Rennur]

Right now I have 25 users logged in and the memory AwareIM server is consuming is just under 2,600,000k.
System usage is already above your allocated settings of 2048 with only 25 users. My BSV is 3.3mb.

No performance issues but it may be due to a low number of logged in users.

[Rennur]

Have you cons

Have you considered Tomcat 64 bit and replacing the activeMQ with the latest version.

As per support, ActiveMQ processes everything so it would be a good idea to be running with the latest version.

[kklosson]

First time I have heard of it. Will study.
Thanks.

[customaware]

Is there anything in Context History table Kingsley?

If so..... delete all those records and try again.

I had a Process that was behaving badly and it filled that table up to the point where nothing would work.

Support found the problem and removing the Context History records fixed it.

[kklosson]

I have a tabled called execution_contexts but it's empty.