When AwareIM dies

johntalbott

Any luck?

kklosson

Since my last post, I had not had any failures until last night. It went down around 11:50pm. I restarted both services and it went down again in about an hour. I am currently reviewing the logs from those events. It's very difficult. There are a number of bad events but the logs seem to also report a lot of benign failures. I'll give them a good look and then I will engage with Support to see what he can figure out.

kklosson

I've posted this zip file which contains wrapper logs from 6 recent events. The file names represent the date and time of the failures. I have a hard time understanding them. In most cases, it seems that the system is suddenly unable to communicate on port 9000:

INFO | jvm 1 | 2016/04/01 01:21:52 | com.bas.connectionserver.server.ConnectionException: Unable to open TCP/IP socket for server localhost on port 9000
INFO | jvm 1 | 2016/04/01 01:21:52 | Please make sure that port 9000 is unblocked.

There are also numerous references to the detection of situation that can cause a memory leak.

"Apr 01, 2016 1:21:22 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
INFO | jvm 1 | 2016/04/01 01:21:22 | WARNING: The web application [/a] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:"

Here are the logs... 8MB
http://www.esdevllc.com/tomcat_wrapper_logs.zip

What I can't seem to detect it what user actions or processes may be occurring as a trigger to any of this.

aware_support

You must capture not only Tomcat logs but Aware IM server logs as well - Tomcat logs on their own are useless for this kind of problem.

ACDC

This problem has persisted for so long, have you thought of relocating to another neighborhood , it is possible that the VM environment may be messing with your setup. I am thinking memory management , Port Availability etc etc .

It may be a long haul but eventually you are going to have to do this if it persists

kklosson

I have considered it but since I am unsure of the cause, I could do all kinds of things trying to chase it. The logs are indicating that it is failing to connect on port 9000. I have a port viewer connected and will monitor the port status during the next failure. I do see that up to six "unknown" connections occur and dissipate as I monitor it. I am waiting for the next failure to see if the java processes drop their ports or something.

rocketman

Just jumping in here because I was looking for a solution to a problem I had last night. The Aware server didn't seem to die - it just wouldn't let anybody in. Being on a windows server, I used task manager to check memory usage and saw that it was a lot higher than normal. Did a server restart, checked the running processes of all my clients and discovered a total of 36 processes stuck in the "waiting" state spanning 2-3 weeks. Three of these were processes that have a very long execution time and the users may have aborted (something for me to look at)

Bottom line - I was able to kill these running processes (from within AIM - but you could just delete "Execution Contexts" (the data not the table). Did another server restart for safety. Went to bed. Server is stable this morning and memory usage looks to be ok.

I am not a Tomcat geek, so would be very interested to learn how to set the unloadDelay mentioned in the below quote - or whether Aim Support could recommend any other way of programatically clearing out running processes on a regular scheduled basis

Apr 01, 2016 1:21:22 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
INFO | jvm 1 | 2016/04/01 01:21:22 | WARNING: The web application [/a] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:"

rocketman

OK - for info, my own stupidity caused this particular problem. In order to conserve bandwidth over 4G, I developed a rapid entry version of a particular part of my system which, rather than doing dynamic lookups on a surname, relied on inputting unique identifiers for pilots and aircraft. These are sent to a temporary file and the server then processes this information, looks up the correct information from the relevant BO's and created a flight record. My problem was I was concatenating a name string which in the temporary file was set at 40 characters into a attribute in the main BO that was 20 characters. Whilst I had trapped for all common user entry errors, I had not checked that the attribute values in the two BO's matched (Well, I couldn't possibly make such a basic mistake now could I)

Being a process running on the server, AIM did not notify the user of the problem, it just hung with the spinning ball and "executing" message. The users only recourse was to close the browser down and restart the app - which left the process hung in the "waiting" state

kklosson

As an update to this long and important thread, since updating to V7 and performing a few "optimizations", I have experienced excellent performance and reliability. As part of my optimization, I speculated that one potential cause of meltdown occurs when a user attempts to upload a large file, maybe 50mb or so. I saw a lot of references to "upload actions" in the log files when reviewing after a meltdown. While my business rule for the object limits the file to 5mb, the file has to be fully uploaded for the rule to evaluate the file. Based on Support's recommendation, I commissioned JT (forum member) to write a client-side Javascript to check the file size before uploading. I recently updated AwareIM and forgot to replace the script file and about a day later I experienced a meltdown. Interestingly, I also received a support ticket from a user who said he had trouble uploading a 50mb file. This confirmed my suspicions that large file uploads were unhealthy at least in my case.

Secondly, when I updated to V7, I decided to keep it in 32 bit just to set a baseline with the thinking that I would change to 64 bit if needed. At this point, I really don't see the need and I don't want to improve on something that is working. Both Java processes run at around 500-700MB memory. I don't have a good metric for concurrent users, but my access logs show 800 to 1,000 logins per day.

My only remaining issue, and it is one that has cropped up since going to V7, is that users report a dramatic slow-down in performance after about 20 minutes of solid use. Note that these users are in the system persistently doing lots of normal transactions - it's their daily business application. Pressing F5 to refresh the browser restores normal responsiveness. They report that the application just slows down in UI actions like pressing a drop-down and it takes a few seconds to do anything. But the problem affects all components. So its seems to clearly be a client-side issue. I don't want to speculate what's happening because I really don't know.

So this is where I am at this time and it's a better place.

hpl123

Kingsley,
Good to hear things are better and BIG thanks for sharing progress/update. Any chance you could share the upload check plugin or having Awaresoft integrate it in Aware IM core functionality IF they are willing to?

weblike

Hi all,

I receive complains from users that the apps are slow, especially when they click o a drop-down list, it takes about to 5-10 seconds to open large drop-downs.

Has anyone clue how to optimize this?

Thank you

tford

I'm assuming the drop down is a reference attribute.

A couple things to explore / experiment with:

Are attributes you are sorting by indexed?
Is the attribute marked as "fetch all records at once"?

weblike

tford wrote
I'm assuming the drop down is a reference attribute.

Yes

A couple things to explore / experiment with:

Are attributes you are sorting by indexed?

Do you mean in DB?

- Is the attribute marked as "fetch all records at once"?

Yes. Can I limit to fetch only specific number of records?

tford

- Are attributes you are sorting by indexed?

Do you mean in DB?

I have a BO called Rally with a reference attribute Student. Student is a drop down which sorts on Name_Last and Name_First.

So in my case, in the Student BO, the Name_Last and Name_First attributes are indexed.

- Is the attribute marked as "fetch all records at once"?

Yes. Can I limit to fetch only specific number of records?

I don't think so, but you can fetch records dynamically as the user types.

RLJB

Yes, we can guarantee this is an issue that causes system slowness, can also cause system crash if you have enough users and enough instances in drop down. It’s not 8.1 issue. It’s an 8 issue.

Solution is to reconfigure dropdown to show the filter only, so they have to type something for the options to appear. Not always what you want but better than a prod server falling over!

(Based on real life experience 😃 )

Jaymer

RLJB
Do you have any opinion on using Badges that constantly are hitting the server to refresh?
Does this have any negative effect on a large production system?

weblike

RLJB wrote
Yes, we can guarantee this is an issue that causes system slowness, can also cause system crash if you have enough users and enough instances in drop down. It’s not 8.1 issue. It’s an 8 issue.

Solution is to reconfigure dropdown to show the filter only, so they have to type something for the options to appear. Not always what you want but better than a prod server falling over!

(Based on real life experience 😃 )

I only have maximum 10 concurrent users..This is a huge number? I use other programming languges like PHP and there can handle thousands of users and do not fail.

🙁

RLJB

No, 10 isn't a big number of users in my opinion, even 1000 isn't a big number re users. But configure around it, or do a BSV and send it to Support they might be able to fix it.

Re the counts on badges I do wonder about that, see it in the logs all the time, haven't had time to do a test on it. We have increased the time between refresh rates on them "just in case" though.

« Previous Page