AWS Cloudwatch Help Please

RLJB

We have a pretty big install, that seems to semi-regularly fail at about 11am on a Wednesday (in a completely different timezone - which is nice). We're trying to track down "why?" using all the usual things:

AwareIM Logging files for review
AWS RDS MySQL logging for review

We use AWS and Cloudwatch a little, but haven't really gotten into it. My question is, has anyone set this up properly with Aware? We use Linux OS on our ESC2 instances, what I would really like to be able to track is similar to the "top" linux command and specifically the 2 x "Java" processes that is Aware and the CPU/Memory that is being consumed on a minute-by-minute basis.

One thing I can't get my head around...

Cloudwatch reports "CPU Utilisation" and it never peaks over 37% (so this seems fine), but when I look at the CPU/Memory utilisation of the two Java processes (using linux "sudo top" command) we often are peaking at 130%, so they seem to report quite different things.

Any tips/learnings for how to set this up would be greatly appreciated?

intra

Have you requested custom metrics from AWS so you can see sub-minute data ?

I had similar issues with temp files chewing up space out of /tmp.

I use a custom script to report MEM, Disk Space usage in cloudwatch and set an alarm around the 80% mark so i can plan downtime for maintenance. Are you sure its CPU giving you grief?

RLJB

Pete - those 'custom scripts' are they the Python ones from the AWS documentation?

No, not certain it is CPU/Memory, it's a funny one, not certain about much here. Logging everything, but nothing much of concern is showing anywhere. Just users complaining of slowness and then failure.

intra

Yep, those scripts will give you memory and space (put) into cloudwatch.

In my case, i had similar complaints from users. I configured metrics to be sent to cloud formation and noticed /tmp was filling up with odd log files from background processes. When i killed the process those log files were cleaned up and things went back to normal.

Since then I've put in a cloudwatch alarm at the 80% mark and then schedule maintenance windows to clean up processes.

Hope this helps.