NOT AwareIM issue but WTF?

[kklosson]

My app is hosted on AWS (Oregon region). All good. I have customers all over the US, Alaska, and Hawaii, and I am in Virgina. All are using the same multi-tenant application. One customer (my largest) is in Las Vegas. They are routinely seeing very poor performance even when others, including myself, are experiencing perfectly normal performance. They say that at about the 20-minute mark, the system becomes slow and unusable. They punch out and log back in and experience normal performance for another 20 minutes. Makes no sense. I have run some long sessions trying to duplicate but it doesn't happen.

One thing is that for some reason, my SSL cert doesn't validate for them. Big red line through the cert. Nobody else experiences that. I have no clue how that can be. My cert is absolutely valid.

They say that when they go home and are on their own Cox or whatever network, they experience similar performance. I don't know yet about the cert.

They are the only customer with these issues. They are the biggest customer and the vast majority of the system data is attributable to them. But I have created a user within their account and have yet to experience any of their issues. Runs like champ.

Could there be some trouble between Oregon and Vegas?
Could the broken cert be a symptom of an underlying issue?
Could it be a disturbance in the Force?

I would like to do some ping testing from my server to their network. But again, most of the data points I have so far are in conflict.
I'll take your wildest guess.

[johntalbott]

It seems like there are a number of things to investigate/rule out. Have you been in contact with AWS support?

[customaware]

It is because all the data that flows through Vegas needs to be Laundered Kingsley.

Bu Seriously, sounds to me like it is definitely something their end.

Best case.... if you know someone in Vegas....(you have to go yourself if not. )

Get them to log in and test if they are seeing the same behaviour. In not then it is something to do with the Company network or their ISP.

[mrbdrm]

do they all use the same ISP? do they have issues with other websites or web applications?

[customaware]

Another idea.....

Take a snapshot of your EC2 and spin up another instance in - US West (N. California) us-west-1.

Apply the snapshot and get the app up and running on that.....

See if it makes a difference....

[intra]

Before jumping to conclusions , it would be best to establish if they can reproduce the '20 minute' issue in a reliable fashion.

Which browser are they using?

Who's the ISP connecting them to your AWS instance?

[kklosson]

I have verified that the symptoms are not just this one account. Generally, the user has a good time for about 20 minutes, then their session starts to bog down and it's not long before it just isn't usable. If they simply kill their browser session and log back in, they are happy again for about 20 minutes. I can see nothing on my server. There are no measures whatsoever that indicate anything is in extremis.

[BenHayat]

I have verified that the symptoms are not just this one account. Generally, the user has a good time for about 20 minutes, then their session starts to bog down and it's not long before it just isn't usable. If they simply kill their browser session and log back in, they are happy again for about 20 minutes. I can see nothing on my server. There are no measures whatsoever that indicate anything is in extremis.

I was talking to another Aware developer and he said his customers say the exact thing as above.
That sounds like the session memory gets too big on server side and Java runtime and GC get bogged down.
I had seen that in the old ASP.Net when session memory would get too large. User's response would get very slow until logging out and logging back in with new session.

[kklosson]

The question is, is there anything I can do about it?
I am currently running 32 bit as a means to rule things out regarding other server issues. Tomcat and AwareIM are running just under 600MB each, so Java heap memory shouldn't be in trouble. But session memory? Is this something I can allocate?

[BenHayat]

The question is, is there anything I can do about it?
I am currently running 32 bit as a means to rule things out regarding other server issues. Tomcat and AwareIM are running just under 600MB each, so Java heap memory shouldn't be in trouble. But session memory? Is this something I can allocate?

I'm not saying it's definitely session memory issue, but the symptoms you brought up, was very similar to old ASP.Net webForm, were session instance would get larger and larger with every request.
It does not affect heap memory, but lots of time Java Garbage Collector doesn't kick in to release old memory, so the OS spends a lot of time writing and creating virtual memory on disk and that's when it gets bogged down, before responding back to user. Java 32 can only use up to 1 gig, so it runs out of room much earlier.

When we had this problem with ASP.Net Webform that users would get slow and we watched server CPU, memory and Disk I/O, the Disk I/O and CPU were utilized 100%. We would logout from test client and server would become normal.

Because of this problem, the new ASP.net CORE was completely re-written to deal with session data size and memory footprint.

The above issue could also exist with Java platform and Aware's server.

[BenHayat]

Another suggestion I have, which I did in ASP and solved the problem, is to create another smaller app that uses a different Aware server and this app is mainly for file uploads or downloads by user. This will take away all the memory consumption from your main app.

Once we did that (in ASP), we didn't have so much problem with main app memory and slowness. Large image and PDF uploads, kills server memory, until everything is uploaded and then saved to disk.

[CalD]

Not disagreeing with Ben, But why would session memory be a problem in Vegas and no where else? earlier in the thread it sounding like you had isolated the issue to them being in vegas - I.e you try to reproduce it and you cant?
Do you know they are using same browser and version as you?

[RocketRod]

I would ask for remote access to one of their PC's so you can see the issue first hand. Get them to install TV or similar. Then investigate different browsers and browser settings. I suggest it's their end.

Cheers Rod

[RLJB]

" Get them to install TV or similar."

agreed, easiest 1st step to try and reproduce

[kklosson]

I've concluded that the problem is not just one account. I have confirmed it on one other account. The similarities are that both accounts have 10 or 20 users in the same office basically using the system as their primary business tool throughout the day. So it's possible that the symptom is simply exposed under this level of use. I tried one thing. In the StartAwareIM.batch file, I added -XX:-UseGCOverheadLimit.

start %JAVA_HOME%\bin\javaw -Xmx192m -Xms20m -cp %CLASSPATH% com.bas.controlcenter.ControlCenterApp -XX:-UseGCOverheadLimit

This did not help much, if at all. I also note that this is a new issue since updating to V7.

Bottom line, something is causing the session to start caving in after a certain amount or duration of use. Restart session, all good.