[SOLVED] ActiveMQ Issues - Out of Memory - Server crashing

[Jaymer]

Jump to the solution here: https://www.awareim.com/forum/viewtopic ... 7&start=15

We're in final testing of an 8.1 application we are about to roll out.

Its basically a Lead tracking system.
Each lead has 7 possible ps_xxx references (owner of the lead, referrer, 2nd_owner, source, etc.) + a couple of om_xxx lists.

All has worked fine with lots of Leads loaded and Queries to search & browse worked fine.
Then, all we did for one set of data (about 200 out of 50,000) was fully attach/point to all the ps_xxx fields. (its not mandatory that all 4 be DEFINED).

1st thing we noticed was a significant delay in paging during Queries.
Screens that were almost instant to go to the next page, now took 1 second, for example... long enough to see the spinner flipping.
(Going to another tenant WITHOUT these references filled out was still fast. That was the only difference.)
Almost simultaneous with this was out of memory errors... which sent me on a search of the forum to no avail.

Getting these primary errors:
java.lang.OutOfMemoryError: Java heap space
Out of memory error encountered in the source OS

Code: Select all

java.lang.OutOfMemoryError: Java heap space

2018-06-19 14:13:49,714 org.apache.activemq.broker.region.Topic  -Usage(Main:memory:topic://defaultoutputtopic:memory) percentUsage=222%, usage=149466463, limit=67108864, percentUsageMinDelta=1%;Parent:Usage(Main:memory) percentUsage=222%, usage=149466463, limit=67108864, percentUsageMinDelta=1%, 
Usage Manager memory limit reached for topic://defaultoutputtopic. 

Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. 
See http://activemq.apache.org/producer-flow-control.html for more info

2018-06-19 14:25:31,039 org.apache.activemq.broker.region.Queue  -Usage Manager Memory Limit (67108864) reached on queue://defaultinputqueue. 
Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. 

See http://activemq.apache.org/producer-flow-control.html for more info

Out of memory error encountered in the source OS

I started to read about this on the Web.
One thing I found was a critical bug in ActiveMQ 5.8, and per this thread https://www.awareim.com/forum/viewtopic ... emq#p44219
I downloaded 8.12.1, renamed the .jar file, copied it into place, relaunched Aware and then the query still hung with out of memory error, PLUS this:

Code: Select all

2018-06-19 14:56:07,384 org.apache.activemq.broker.TransportConnection.Transport  -Transport Connection to: tcp://127.0.0.1:49812 failed: java.io.EOFException

which totally Fubars TCP and you have to relaunch Aware again.

So the system works fine on queries of the Lead table that dont involve as much system resources.
The more indepth queries (ie. have more joins to return simple reference values, like a Name to display in a pulldown) generate the Out of memory error and the user cannot do anything.

I'll keep researching.
Anyone else seen this?

[Jaymer]

posts online reference their activemq.ini file, where there are memory settings.

i don't see this file in AIM's install.
I wonder how you adjust these parms?

Code: Select all

<pendingMessageLimitStrategy> 
   <constantPendingMessageLimitStrategy limit="1000"/> 
</pendingMessageLimitStrategy> 

System usage is: 

<systemUsage> 
   <systemUsage> 
       <memoryUsage> 
           <memoryUsage percentOfJvmHeap="70" /> 
       </memoryUsage> 
       <storeUsage> 
           <storeUsage limit="100 gb"/> 
       </storeUsage> 
       <tempUsage> 
           <tempUsage limit="50 gb"/> 
       </tempUsage> 
   </systemUsage> 
</systemUsage> 

[Jaymer]

bin/startAwareIM.bat
start javaw -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Xss512k -cp %CLASSPATH% com.bas.controlcenter.ControlCenterApp

bin/startupOptions.props
AWAREIM_SERVER_STARTUP=java -Xmx1024m -Xms512m -classpath ../Tomcat/lib/*;../CustomJars/* com.bas.newcp.ServerStarterECP

bin/startupOptions.props
TOMCAT_STARTUP=java -Xmx512m -Djava.endorsed.dirs\=../Tomcat/common/endorsed -classpath ../Tomcat/lib/tools.jar;../Tomcat/bin/bootstrap.jar;../Tomcat/bin/tomcat-juli.jar -Dcatalina.base\=../Tomcat -Dcatalina.home\=../Tomcat -Djava.io.tmpdir\=../Tomcat/temp org.apache.catalina.startup.Bootstrap start

ConfigTool/eclipse/ConfigTool.ini
-Xmx384m

[ACDC]

I am not on 8.1 yet, but in 7.1 there are heap settings in the awareim.CONF file.
Have you tried increasing them

[Jaymer]

No Joy

I saw 2 parms. 1 was 20Mb, forget the other.
Made both 1024Mb.

Screen Shot 2018-06-19 at 5.50.21 PM.png (21.21 KiB) Viewed 26068 times

same results:

Code: Select all

Exception thrown com.bas.connectionserver.server.BASServerException Unknown system error
com.bas.connectionserver.server.BASServerException: Unknown system error
	at com.bas.connectionserver.server.ConnectionFactory.sendMessage(ConnectionFactory.java:271)
	at com.bas.connectionserver.server.ConnectionFactory.sendMessage(ConnectionFactory.java:229)
	at com.bas.webapp.common.WebAppUtils.sendMessageToExecutionEngine(Unknown Source)
	at com.bas.webapp.common.WebAppUtils.sendMessageToExecutionEngine(Unknown Source)
	at com.bas.webapp.thin.handlers.F.K.A(Unknown Source)
	at com.bas.webapp.thin.handlers.F.K.A(Unknown Source)
	at com.bas.webapp.thin.handlers.F.K.A(Unknown Source)
	at com.bas.webapp.thin.handlers.XMLRequestHandler.A(Unknown Source)
	at com.bas.webapp.thin.handlers.XMLRequestHandler.handleURLRequest(Unknown Source)
	at com.bas.webapp.thin.servlets.WebInterfaceURLServlet.doPost(Unknown Source)
	at com.bas.webapp.thin.servlets.WebInterfaceURLServlet.doGet(Unknown Source)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:621)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Web server returned error: Unknown system error

[aware_support]

You don't need to change ActiveMQ settings, you need to increase memory to the Aware IM Server - the more you give it the better. Don't know what you are doing in your application, but if you are doing something crazy (like allowing the system to return a thousand of huge records), give it 16Gb if your machine has them.

[Jaymer]

Thx for reply Vlad

Not doing anything weird.
Only returning 40-50 records per Query.

Odd thing is, my "All" query find 186 records, but only brings 50 into the grid each page.
You can next/prev pages, but every now and then it will get out of Memory.

But Another menu button finds only 31 out of the 186 - with a VERY similar query... just one to narrow the list down.
But it always hangs - the recs can be found easily/quickly in SQL tool.
Another Query finds 16 recs and it usually works, but still sometimes just hung.

Thats it. No other users. No activity.
Another Tenant that doesn't have all the 8 reference fields populated works fine bringing hundreds back to the grid (still only 40-50 per page).

[aware_support]

If you are sure that it's a query that returns a few records and nothing else, then your memory settings must be very low - maybe you are using default values.

[ACDC]

I don't know if this could be related to your problem, but keep in mind if you have a reference on an object that is set to "Fetch All Records at once" it loads everything into memory

So for example, if you have 50 000 product items and on the invoice line item the product reference was set to "Fetch All Records at once", every time a user selects a product item it will load 50 000 into memory potentially consuming all available memory .( multiply this by more than one user, things could get very nasty)

[weblike]

If you are sure that it's a query that returns a few records and nothing else, then your memory settings must be very low - maybe you are using default values.

Same issues here....
Production server:
1. Having 20 concurrent users max.... (RAM goes up to 8GB)...In PHP I could have 300 users with half of memory...
1.b. DB size (table with no more 8000 rows takes few seconds to load (25 records per page). I'm seeing the spinner...spinner...spinner
2. Same VPS which worked fine with 7.1 (24GB RAM, Intel XEON 16 vCPU) => is crashing with 8.1
3. Configurator is blocking randomly ("not responding" messages in Task Manager)...screen freeze....

Selection_673.png (20.68 KiB) Viewed 25949 times

Selection_670.png (8.33 KiB) Viewed 25949 times

IF we are doing wrong, please tell us how to do it right!

Thank you.

[chris29]

We have a similar issue. Found that you need to ensure that all log outputs are closed and put the following in the cp.ini at the end

-vmargs
-Xmx1024M

[rocketman]

"IF we are doing wrong, please tell us how to do it right!"

Your memory allocation seems very high for a 24GB server If it's a windows server - keep in mind you have to leave some space for the OS services plus a chunk for MySQL. Try halving them - and you shouldn't need anywhere near that much for the configurator surely?

[RLJB]

I'm very interested in learning more about this thread - we have some very slow searches, we noticed a difference moving from V7 to V8 and it sounds worse in 8.1 (we haven't deployed any prod servers to 8.1 yet).

George - for a 24 GB server, we would set:

Config tool: to 1 GB or less (assuming this is a production server and you don't do config on it, you don't need to allocate it memory, all our prod servers run with Config tool at 256mb - no issues, we publish through the browser anyway)

Tomcat: We were told to set Tomcat to 1gb. We had a java expert review our set up and tell us anything more than this is wasted and not needed. I didn't entirely believe him, so we set it to 4GB. Tomcat never falls over on our prod servers anymore.

Server: We give this 8GB or 16GB depending on the server size and the user numbers we have. I have no proof that the 16GB runs any better/faster than the 8GB - there are too many variables at play (BSVs, usernumbers, etc).

This leaves 4GB for the OS, and we have DB running on different server. Hope that helps.

[RLJB]

Hey Chris (hope you're well mate)... what does this do, vmargs etc? And also what do you mean by "all log outputs are closed " is that the Aware logging on the control panel?

We have a similar issue. Found that you need to ensure that all log outputs are closed and put the following in the cp.ini at the end

-vmargs
-Xmx1024M

[chris29]

Hey there, yes all good thanks

Anything after -vmargs are passed as arguments to the JVM in Eclipse. We found that if you don't set max memory on the cp and the logging windows open you will eventually run out of server memory.

Since moving from 7.1/8 to 8.1 we have had to increase memory settings. In V8 we resorted back the the old control panel.