Server issues - Aware Restart Needed

customaware

I have a server that has been running quite happily for the last 4 years.

I have recently upgraded it to version 8.6 (latest build)

Since then, we are having issues with something freezing up and users not being able to log in.
The only way to fix is to restart Aware on the server.
It is happening between 2-4 times a day.

Even though it stops users logging in, I can see the Aware IM Log and the Tomcat log still running.

There is plenty of memory (6Gb for Aware and 6Gb for Tomcat)
There is plenty of hard disk space.
Hardly any load on the CPU or memory.

Tried running and a Service and from CP with the same issue.

I have no idea what the issue is. But, until I can pin it down, I need some way of either....

A) Being notified as soon as it happens so that I can RDP in and restart Aware

B) Some way on the server to recognise that Aware is not accepting logins and having Aware automatically restart.

Any ideas or suggestions would be hugely appreciated.

UnionSystems

Hi Mark,

For issue (A) we use https://uptimerobot.com/ for monitoring public facing URLs. Primarily it checks on our clients websites but we do have it checking a public facing AwareIM server as well. It cheap and reliable.

Be interested in anyone’s suggestions for (B).

Jaymer

Yes dave, I do a similar thing. I don’t recall the service. I set up a REST endpoint and have some service out there that hits that endpoint and looks for data. He gets it every 5 to 10 minutes. It definitely will alert us. I’m not the one who set that up, but I could find more details, but it’s similar to what you posted.

Jaymer

Mark
Sorry, that is annoying.
One thing I have seen in my 8.6 tomcat logs is some Yahoo making a request that throws an error with a rest servlet.
In this particular server, I have nothing using rest and I’ve never set it up. I can’t tell from the log what data the offender is sending, it appears null/empty in the log.
Tim stamp shows that something did come in at the exact time of the error in the server/tomcat output.
But aware sees something in that request and then complains that the REST.props file was not found.

For me though, it’s not forcing a restart.
I think I posted a thread several months back that some guy posted something to me and it caused a tomcat issue, I’ll have to research that.
Version 8.5 or 8.6 moved us up to tomcat 9 and maybe some internal request is getting through that’s crashing tomcat.

I think it’s possible that if it is coming from a hacker, it’s something in the data stream that’s coming to your server doing this.
And that same data stream/exploit isn’t coming to others and that’s why we’re not seeing it across the board.

Jaymer

Fyi

Something hit Tomcat & crashed Port 9001 (comment)

rocketman

Had an identical issue back in 7.9 and it looked for all the world like it was MySQL refusing the connection.

Tore my hair out for weeks, but when I went to 8.0 everything returned to normal. I know that’s not much help but it might just point to the fact that it might not be AIM or Tomcat or some corruption in Tomcat.

If memory serves, I think I upped the allowed connections in MySQL which mitigated the issue to days rather than 2-3 times a day, but I always had the feeling I’d just stuck a band aid on an abscess