Aopertionware Server freezes after 2 - 3 days

rocketman

Tearing my hair out because I do't know whee to start looking. After a couple of days of faultless operation the AwareIM will suddnly freeze and prevent further logins.

The Aware Log says

FAULT: ERROR: org.openadaptor.adaptor.jms.JMSSink - txnCommit() failed with JMS Queue/Topic PublishException: org.openadaptor.adaptor.jms.PublishException: org.openadaptor.adaptor.jms.JMSPublisher - commit() failed with JMSException: org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

ERROR: transaction commit failed (will call rollback) , ERROR: org.openadaptor.adaptor.jms.JMSSink - txnCommit() failed with JMS Queue/Topic PublishException: org.openadaptor.adaptor.jms.PublishException: org.openadaptor.adaptor.jms.JMSPublisher - commit() failed with JMSException: org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

2011-02-08 08:04:17,534 ERROR DLGC#403 -Error sending message Error processing output message

Send failed exception thrown Sending failed;

aware_support

Most likely reason is dynamic change of the server's IP address or some other similar change in the network. You have to make sure that the IP address is static.

rocketman

Not in this case. I have a hosted VPS with static IP address - which I use- and have used today to log in using remote desktop to restart AwareIM

Any other clues would be appreciated - and thanks for the quick response

aware_support

Please check with your VPS provider if there were any changes in their network at the time of the failure. It could be just one of these one-off things. If this is the case just keep an eye on this.

rocketman

Apparently reports are coming in that it has happened about 6 times during the two weeks I've been on holiday. I will certainly check with the ISP later.

My worry i s they may be implementing some of the modern green switches that go into sleep mode when there is no activity on a link and was wondering how this might affect AwareIm.

What actually does it take for AwareIM to lose connectivity? If I'm right are there any techniques or workarounds to help keep a link alive.

Of course, I may be way off the mark here but I've now had more than 12 months of (almost) flawless service - so something is going wrong somewhere

rocketman

PS ... To AwareIM support.

The reason I asked the question what does it take to "lose connectivity" is so that I can ask specific questions of the ISP rather than a woolly "Have you changed anything recently" question

Thanks in advance

rocketman

This may have a bearing on this post. One of my members has just been kicked off the server with the following message - which I've never seen before.

Connection failure status:504 status text;Gateway Timed-out

rocketman

It would seem this problem was related to an earlier problem I had relating to the Tomcat Maxthreads setting. I had totally forgotten that when upgrading to a new version of AwareIM, the server.xml file in C:\AwareIM\Tomcat\conf get overwritten.

Have now changed maxthreads to 300 and maxsparethreads to 200. All seems to be fine now. No freezing for over a week.

It does beg the question, since AwareIM ships with maxthreads set to 150 and no-one else seems to have the problem, whether there is some inefficiency in my BS or my MySQL is not releasing threads in a timely fashion.

If there are any Tomcat/MySql experts out there who could offer advice on fine tuning these parts of the system or how to monitor thread usage in real time to help me spot any areas of weakness in my BS it would be much appreciated.

For the technical, Server is VPS with 2Gb memory and 8 cores. Also running IIS6 hosting 7 standard websites, three of which also use the MySql server. Of these, two have been developed using a website development package that stores the html pages in Mysql tables. In total, they get about 30,000 page impressions/month

AwareIM has 512Mb allocated and Jboss and Tomcat both have 128Mb
Session timeout is set to 60 minutes which cannot be changed for a number of reasons

AwareIM has 5600 member logins in 90 days (avg 62/day)

One thing I have noticed - which I have reported to my ISP is that at certain peak times CPU usage on my hardware node can shoot up to 99%

nlarson

Rocket did was max threads the root issue? I am running into this on GoDaddy VPS. For me, it seems any prolonged period of inactivity will close the connection. I've got default settings all around and see nothing in my event logs to indicate the issue. I saw similar behavoir on my dev pc when it would sleep, but obviously me server is not set to power saver mode so I am at a lose as to why this happens.

rocketman

Still not bottomed this one out. but it must be coming down to either tomcat, Mysql or possibly jboss - which I know nothing about - not releasing resources.

AwareIM can stay good for anything from 1 - 4 days then suddenly stop accepting new connections although users who are already in and working are able to continue working not knowing there's a problem.

stopping and starting AwareIM (NOT a full server restart) cures the problem and strangely any users who were idle at the time of the restart don't even know I've done a restart, they just seem to be able to carry on, which is strange because I would have thought at least the session states would have been destroyed by the restart. However if they attempt to do anything whilst the restart is in progress they get a 404 page not found message as you would expect

Full investigation on a live system is difficult because when the user calls - they usually want it back up like yesterday so I never seem to have time to monitor the logs.

Sorry I can't be more helpful. If you find anything do let me know. In the meantime I'll keep on investigating

nlarson

Now that I am paying attention I am seeing this in as little as a few hours, but I have only 1-2 users connecting at this point. There is no set time period for me either, but I can't imagine # of connections is the root of my issue. Perhaps we can compair the logs to see if it is exactly the same? Mine is below. I assume TS is Tomcat Server, but that is just a guess. I am wondering if others using VPS have this too.

Thanks.

Aware IM Server started...

Process status: Aware IM Server: Active and functional

Process status: Not started

Overall status: Aware IM system is active and functional

INFO: TS terminating

INFO: Timer_NOTIF_0.Controller - Source TS is exiting

INFO: Adaptor terminating with error code 1

INFO: Timer_NOTIF_0.Controller - Sources are dry; terminating

INFO: Timer_NOTIF_0 - Controller thread is dead - run() returning

log4j:WARN No appenders could be found for logger (BUSINESS_LOGGER).

log4j:WARN Please initialize the log4j system properly.

INFO: Timer_NOTIF_0.Controller has commited 100 transactions

INFO: TS has made 100 polls

INFO: InputAdaptor3.Controller has commited 100 transactions

TRACE: org.openadaptor.adaptor.jms.JMSListener - closing JMS connection - JMSException: org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

TRACE: org.openadaptor.adaptor.jms.JMSListener - connection.stop()

TRACE: org.openadaptor.adaptor.jms.JMSListener - consumer.close()

TRACE: org.openadaptor.adaptor.jms.JMSListener - consumer.close: JMSException: [org.jboss.mq.SpyJMSException: Cannot unsubscribe to this destination; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]]

TRACE: org.openadaptor.adaptor.jms.JMSListener - consumer.close: JMSException: [ - during cleanup after - JMSException: org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]]

TRACE: org.openadaptor.adaptor.jms.JMSListener - session.close()

TRACE: org.openadaptor.adaptor.jms.JMSListener - connection.close()

ERROR: org.openadaptor.adaptor.jms.JMSSource PublishException org.openadaptor.adaptor.jms.PublishException: org.openadaptor.adaptor.jms.JMSListener - close() failed with JMSException: org.jboss.mq.SpyJMSException: Cannot unsubscribe to this destination; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

during cleanup after - JMSException: org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

TRACE: org.openadaptor.adaptor.jms.JMSListener - connection.stop()

TRACE: org.openadaptor.adaptor.jms.JMSListener - consumer.close()

TRACE: org.openadaptor.adaptor.jms.JMSListener - session.close()

TRACE: org.openadaptor.adaptor.jms.JMSListener - connection.close()

during cleanup after - JMSException: org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected) [java.io.IOException: Client is not connected]

INFO: TS has made 200 polls

INFO: org.openadaptor.adaptor.jms.JMSSource Retry [=1] reconnecting to JMS Queue/Topic

TRACE: org.openadaptor.adaptor.jms.JMSListener - closing JMS connection null

TRACE: your JMS vendor implementation of QueueConnectionFactory.createQueueConnection() returns a class which is an instance of TopicConnection as well as an instance of QueueConnection

INFO: org.openadaptor.adaptor.jms.JMSSource Resume processing JMS Queue/Topic notifinqueue

INFO: org.openadaptor.adaptor.jms.JMSSource Retry [=1] reconnecting to JMS Queue/Topic

TRACE: org.openadaptor.adaptor.jms.JMSListener - closing JMS connection null

TRACE: your JMS vendor implementation of QueueConnectionFactory.createQueueConnection() returns a class which is an instance of TopicConnection as well as an instance of QueueConnection

INFO: org.openadaptor.adaptor.jms.JMSSource Resume processing JMS Queue/Topic defaultinputqueue

Once the error starts it periodically repeats.