Server is dead in water

BenHayat

kklosson wrote
Thanks Ben. That's good info.

One other info I left out. He still uses EC2 only for cases when a single tenant wants his own private database. So he asks client to signup with AWS and then he uses the VPS for a single client for their use. He said AWS EC2 is perfect for single tenant, but for multi tenant SaaS, you need dedicated server.
After my first six months of production time, I plan to go dedicated server as well.

kklosson

So an update. Moving to a dedicated server is a bit of a chore because I would probably have to move the database server as well. So I bumped up the the app and db servers to big monsters. I get only slightly better performance, which was not really the issue. Ran well all day then melted down for now particular reason. The logs barely said the word "Error". The app server has 16GB ram with 6GB allocated to AwareIM. The CPU was in this config barely gets above 5 percent ever and the two java processes hold between 5 and 6 mb. It's absolutely dumbfounding. I have engaged support again. This time I don't think it can be blamed on memory allocation of the virtual machine.

Rennur

Just a stab in the dark,
Check if there are are any processes heavily involved with or mak use of SystemSettings which can cause the innodb database [MySQL] to lock the table and make the app non-responsive.

I've experienced this 'locking' once but I don't know the exact cause. I suspect it was the database problem.

Table Locking Issues
https://dev.mysql.com/doc/refman/5.6/en/table-locking.html

kklosson

Would this not produce a database error of some kind?

kklosson

And I have no processes that write to the systemsettings table.

Rennur

Would this not produce a database error of some kind?

It is not an error as such, it is just part of normal database behavior.

The issue is when the database does not release this lock on the table that performance is affected. My guess is when a process tries to write to it during lockout the app stops responding.
In my case it was the systemsettings table, but it could happen to any other table the process is trying to write to.

I diagnosed the issue through 'Navicat for MySQL' when I tried to update the table directly and got the lock timeout error. AwareIM did not log any errors about this. I'm no expert but maybe check your mysql log for lock or timeout errors to your database tables.

kklosson

Thanks. I have Navicat. Can you tell me how to look for this?

Rennur

Thanks. I have Navicat. Can you tell me how to look for this?

If this is in fact the problem, the hard part would be to pinpoint which table locked.

When the system hangs or becomes unresponsive:
Check tables that are in context such as LoggedInBO that a process might use first. In navicat, change the table value and see if you can save it. it will give you a lock error or lock timeout release error if it is the cause and the changed value would not save.

Check out this info on deadlocks:
InnoDB Standard Monitor and Lock Monitor Output
https://dev.mysql.com/doc/refman/5.5/en/innodb-standard-monitor.html
How to Cope with Deadlocks
https://dev.mysql.com/doc/refman/5.5/en/innodb-deadlocks.html

kklosson

Thanks. Most of this goes to how AwareIM is programmed so I will not have much insight. But I can review table locks at the command line to see if it's an issue. In the scheme of things, my app is pretty simple. People open forms and save data. I do have one fairly wide table that is updated regularly.

« Previous Page