Poll: Concurrent users?

hpl123

Hi all,
We are quite a few developers who has a lot of developed apps (internal/external) and after the problems Kingsley (kklosson) is facing with a server not being able to handle more than 50 concurrent users, I would be interested in hearing how many concurrent users you guys have had or have regulary in your apps?

I myself have small apps currently only with a small number concurrent users per app BUT with plans on trying to scale and trying various new product ideas and I need to know if there are general issues with 50+ concurrent user apps because that greatly affects how I think about and plan my apps (and I am sure this goes for most of us).

If you could share just a number i.e. you don´t have to share what the app does (if you don´t want to), domainname, company name etc. etc.. Just the number would help a lot to get a general feel and consensus on what the potential limits are.

Thanks in advance

kklosson

This is worthy information to share. As we all know, the answer to how many users your app can support is "it depends". I am also interested in knowing how you determine the concurrent user count and what actions you have taken if any to optimize your application. It's also important to know what your application does that goes beyond CRUD functions such as aggregate expressions that tax system memory. Regarding my own issue, my real quandary is that I incur memory heap errors when neither the AwareIM or Tomcat servers are consuming anywhere near their allocated memory. So regardless of the cause, why do I incur a memory heap error when I have so much available. The result of all this is that I have no ability to scale this application whatsoever. I can't employ a load balancer because the load on the machine is zip. I am interested in your responses on this topic.

RentProperty

We've frequently have about 35 concurrent users on one of our apps... But... This thing is running on a monster R3.large machine with 15GB memory, and an M4.large on the DB side which has 8GB memory.

We had a lot of performance issues when we were still on a T2.medium(4GB), but then we upgraded and optimized some processes and at the same time and I must say since then the servers have has been performing quite well... Hope we can push this setup till we have at least 80 concurrent users.

However... I would love to hear that there are people who have reached 200 or 300 concurrent users on a system, as we have now started developing a system for End users(No Businesses) so when we launch this we probably get to those figures quite quickly and I'm crossing figures we won't run into too much trouble.

customaware

Codrin told me late last year that they have stress tested to 1,000 users and it was fine.

Not sure what his hardware config is.

johntalbott

Does Codrin frequent the forums? It would be super helpful to get more details.

As I've read the various posts about scaling concerns .. it's leave me concerned.

Has anyone (or support) done any actual load testing?

customaware

John,

I would drop Codrin and email support@idronic.com or give a call +61 7 3123 0290 and have a chat.
He is more than happy to discuss his setup and the lengths he has gone to stress test Aware.

hpl123

Hopefully Codrin sees this and can share some wisdom here in this post instead 🙂. I sent him a PM about the post as well.

codrin

Hi all,

A few years ago we have carried out a number of tests to establish what would be the best deployment option for our cloud app. Our business model deploys a new database for each user/company. The entire process is automated from a sales BSV integration with our website where the user picks the plan they wish to sign up for , pay online with a AwareIM / Braintree integration and once successful their BSV is deployed on the closest server to their location.

Originally we tried to push one server ( QuadCore 3.5Gz, 16GB Ram SSD ) to its limits and the best we could achieve was 100 BSV's running on a single server instance before issues appeared. However I need to stress that this was done when Jboss was the channel of communication and as you all remember there were serious stability issues with it. The main issue we had at that time was the Jboss database was growing exponentially and fast leading to a crash about once a month and there was no mechanism that we found that could prevent that. Also note that in average we have 10 users per BSV and no more than 100 logged in at one point in time.

So because of if and after monitoring closely the activity on the server we've concluded that for us the safest and best performance arrangement was to allocate around 100MB of RAM on a SSD server per BSV. These days we build a server to about 50 BSV's and then the install script flags that server full and deploys a new instance.

Not sure if any of this info will assist any of you , personally if our app will increase in popularity we intend to investigate running proper scaled up instance based on the model presented by Vladimir at the conference in Bali. Also in my opinion the biggest failure that we have as developers in AwareIM when it comes to an efficient deployment is to not focus enough on optimizing our rules and processes.

weblike

Hi all,

I have few apps, small ones. The "biggest" has about 130 users subscribed, but not more than 10 concurrent users same time.

codrin wrote
Also in my opinion the biggest failure that we have as developers in AwareIM when it comes to an efficient deployment is to not focus enough on optimizing our rules and processes.

Regrading codrin's post, would be great if we would know which are the slowest processes, queries, bo's rules, etc So we could optimize better.
Maybe Vlad's opinion on this would be very helpful, because we all dream on thousands and thousands of users. 🙂
cheers,

johntalbott

codrin wrote
Originally we tried to push one server ( QuadCore 3.5Gz, 16GB Ram SSD ) to its limits and the best we could achieve was 100 BSV's running on a single server instance before issues appeared. However I need to stress that this was done when Jboss was the channel of communication and as you all remember there were serious stability issues with it. The main issue we had at that time was the Jboss database was growing exponentially and fast leading to a crash about once a month and there was no mechanism that we found that could prevent that. Also note that in average we have 10 users per BSV and no more than 100 logged in at one point in time.

Just to confirm ... 100 concurrent users has been the max, yes? What happens if more than 100 concurrent users are signed in?

aware_support

This has been discussed many times and it is terribly frustrating sometimes to repeat the same things over and over again. Still:

Aware IM itself imposes NO RESTRICTIONS on the number of concurrent users
However, it does not mean that your specific app will support millions of concurrent users without you having to do anything about it. The ability to support concurrent users DEPENDS ON HOW WELL YOUR APPLICATION IS STRUCTURED. Asking how many concurrent users Aware IM supports is like asking how many users C++ supports. It can support millions or it can hardly support one - depending on how you write the code.
You need to optimise every operation in your application if you are serious about supporting many concurrent users:
a) Make sure that rules are optimised - there is a thread about it on this form. Inspect a rule log and see what's going on there
b) Make sure there are no CPU extensive operations in your app. If you have to have them be smart about them.
In kklosson's app, for example, there is an operation that generates a 100 pages report! This report probably has to go through the entire database many times and is guaranteed to bring the CPU to its knees (no matter how powerful and how much memory there is). How many concurrent users do you think will be able to use his system when this report is running? The answer is zero. The right thing to do is to schedule such reports to run overnight or get a separate server to handle them on the background. This requires smart solutions on the developer's part.
c) Identify performance bottle-necks in your app and optimise them. Sometimes you may need to change rules or even replace a query with a stored procedure. This is very important for the number of concurrent users.
d) And of course add other servers if need be to scale it up. Scaling/load balancing IS supported in Aware IM.

Concurrent users will only be supported properly if the CPU does not have to work hard on any operation that the user performs there!!

Bottom line: Aware IM is not a silver bullet as far as concurrent users are concerned. Developers need to invest heavily in any serious application that intends to support many concurrent users. This involves careful planning of application behaviour, optimisation of rules, optimisation of queries and so on.

johntalbott

aware_support wrote
This has been discussed many times and it is terribly frustrating sometimes to repeat the same things over and over again. Still:

Aware IM itself imposes NO RESTRICTIONS on the number of concurrent users

However, it does not mean that your specific app will support millions of concurrent users without you having to do anything about it. The ability to support concurrent users DEPENDS ON HOW WELL YOUR APPLICATION IS STRUCTURED. Asking how many concurrent users Aware IM supports is like asking how many users C++ supports. It can support millions or it can hardly support one - depending on how you write the code.

You need to optimise every operation in your application if you are serious about supporting many concurrent users:
a) Make sure that rules are optimised - there is a thread about it on this form. Inspect a rule log and see what's going on there
b) Make sure there are no CPU extensive operations in your app. If you have to have them be smart about them.
In kklosson's app, for example, there is an operation that generates a 100 pages report! This report probably has to go through the entire database many times and is guaranteed to bring the CPU to its knees (no matter how powerful and how much memory there is). How many concurrent users do you think will be able to use his system when this report is running? The answer is zero. The right thing to do is to schedule such reports to run overnight or get a separate server to handle them on the background. This requires smart solutions on the developer's part.
c) Identify performance bottle-necks in your app and optimise them. Sometimes you may need to change rules or even replace a query with a stored procedure. This is very important for the number of concurrent users.
d) And of course add other servers if need be to scale it up. Scaling/load balancing IS supported in Aware IM.

Concurrent users will only be supported properly if the CPU does not have to work hard on any operation that the user performs there!!

Bottom line: Aware IM is not a silver bullet as far as concurrent users are concerned. Developers need to invest heavily in any serious application that intends to support many concurrent users. This involves careful planning of application behaviour, optimisation of rules, optimisation of queries and so on.

We're so sorry to keep bothering you with such foolish questions. When is the AwareIM for Dummies book coming out? I'd like to pre-order.

kklosson

The hard part in my case is determining what is failing. I have more than enough CPU and memory. I'm running 64-bit with 2GB each (of 16) allocated to the AwareIM and Tomcat servers. I've never seen the memory usage get above 1 GB for either service. So if it was just a matter of the machine CPU or heap memory running out, I'd say okay, that's all my app can support. In the position I'm in, I cannot implement a load balancer because it would view the machine as good to go. In fact, when the server stops responding, the machine CPU and memory look fantastic. I always expected that when my system overloaded, I would see the CPU or memory hitting the roof but I don't. It breaks internally. I'm getting better at understanding the logs but they are so overly verbose that it is difficult. As Support implies, there is no doubt that aspects of my app need to be optimized, but I simply cannot gain enough information to figure out where the problem lies.

baskosi

Reading about your problems it looks like you should rethink your fast server. Sometimes fast servers can cause problems. We all probably know too little about your solution but I would certainly look into that 100 page report mentioned by support.

One of my experiences involving fast servers was with an application server processing requests too fast for the database server. This caused too many opened db connections. The result was unresponsive app server that had enough free memory and idle CPU.

It‘s hard enough to find an error in your code when you‘re looking for it; it‘s even harder when you‘ve assumed your code is error-free. – Steve McConnell, Code Complete

BenHayat

baskosi wrote
One of my experiences involving fast servers was with an application server processing requests too fast for the database server. This caused too many opened db connections. The result was unresponsive app server that had enough free memory and idle CPU.

Very interesting post!

kklosson

Me too. I hate this because I've got enough red herrings to chase. While I don't doubt anything, all of my senses tell me a computer can never be "too fast". But the guy makes a point I have to consider.

johntalbott

kklosson wrote
.... all of my senses tell me a computer can never be "too fast".

Your senses are right.

Have you noticed a large number of db connections being open when the app pukes?

kklosson

Where would I look for that?

johntalbott

MySQL Workbench http://dev.mysql.com/doc/workbench/en/wb-performance-dashboard.html

MySQL Notifier looks it could provide some useful real-time db monitoring. http://www.mysql.com/why-mysql/windows/notifier/

baskosi

Regarding my own issue, my real quandary is that I incur memory heap errors when neither the AwareIM or Tomcat servers are consuming anywhere near their allocated memory. So regardless of the cause, why do I incur a memory heap error when I have so much available.

Heap errors may often occur due to small memory allocation errors (not releasing the memory). If code that doesn't release all of the allocated memory is called often, it will lead to heap errors and crashes.

I'm again thinking about that mentioned 100 page report. Such a report would definitely cause heap errors if there is a memory allocation problem in the reporting tool.

Another suspect might be your database server. If going through a lot of records while preparing the report (or any other process) and not having all the required indexes, the database server will do a lot of full table scans to filter records. This in itself could exhaust available heap memory allocated to the db server. Have you looked at db error logs?

I had many (non Aware IM) programs crash due to exhausted heap. The only way to find out which part of the code caused them was to print used/available memory before and after code I suspected to eat too much memory. They were almost always called in long running loops.