Application HANGS on IMPORT

ab042

To try and get some real world samples of our data and how your app will work.

I am trying to IMPORT a 12 meg file with 5 fields per record.

I've created the Business Object with the 5 fields NO RULES. In fact, the only thing I've done is create the Business Object and trying the IMPORT feature.

It works fine on smaller databases but hangs around the 70% mark for the large 12meg file.

Any suggestions?

Your application HANGING in a MAJOR ISSUE with our testing. Please help!

ab042

To provide a little more information

The process that appears to be hung is JAVAW.EXE at 97% cpu
Its using 60,716k of memory on a P4 w/2gig RAM

aware_support

Please export your business space version into the BSV file and e-mail this file along with the 12Mb CSV file to support@awareim.com (please zip up the CSV file or send it in chunks)

We need to reproduce this problem in-house.

Thanks

ab042

Since my last post. I have removed JAVA and rebooted and reloaded it with the latest version and it does the same thing.

The data is company confidential so I can't send the file.

But creating the Business Space couldn't be easier.

EMailLog

Attributes
--- Date: Date (mm/dd/yy)
--- EMail: Plain Text
--- Provider: Plain Text
--- RefNo: Number
--- Time: TimeStamp (hh:mm)

Publish it, select IMPORT. Somewhere around the 70% or 170,000 records it takes about all the CPU % and never recover's or passes this mark.

aware_support

We have just successfully imported a CSV file containing half a million records (10Mb).

There must be some record in your file that causes Aware IM to hang. We need to find this record. There must be a file in your server's temporary directory whose name starts with "log" and the extension is "tmp". If you can find this file open it with the text editor and look up the last message which should identify the last record the system has been processing. Find this record in the file (just to be sure include 5 records before and 5 records after) and send these records to us (hopefully you can disclose a few records?).

Also:
1) What database are you using?
2) What are the parameters used during import - are these default parameters (validation is off and batch size is 1000)?
3) How much free memory does your server have?

ab042

MySQL / No validation and the batch size is 1000
At the point of hanging it shows 235k available physical memory

I took approximatly 1200 records before the error and 1200 records after the error and imported that without an issue.

In both cases it hung at after showing 165000 records have been processoed and nothing after that in the LOG files.

So the problem must have something to do with memory and not an item in the file.

Any suggestions?

aware_support

How much memory does your server have?

One thing to try is to specify more memory to be allocated to the Aware IM Server on startup. This is what you need to do:

1) Shutdown Aware IM if it is running
2) Open the file startAwareIM.bat (or startAwareIM.sh if you are running Linux or Mac) and find "-Xmx96m"
3) Replace "96m" with a bigger value, for example "-Xmx256m"
4) Re-start Aware IM and try import again

Let us know if this is indeed the memory problem.

aware_support

Forgot to mention that the file "startAwareIM.bat (or .sh)" is in the BIN directory of your Aware IM installation.

ab042

I made the suggested change an it still stops in the same spot.

After I setup 256m on the javaw line and after it Locked at 70%, I checked and there is 384k Physical Memory available.

Any other suggestions?

aware_support

Can you watch the memory during import with the Task Manager? Does it keep growing?

Also, can you print out the ouput of the Aware IM Server when it hangs?

Can you also split the file in two and import both parts separately ?

aware_support

One other thing. Can you check how much memory MySQL uses before the import and after the hang?

aware_admin

It does seem that MySQL is leaking memory. We are investigating the issue now. Will post to this forum as soon as we find something out.

ab042

Any update on this?

aware_support

We are communicating with MySQL on this. We have run a simple MySQL test outside of Aware IM which seems to leak memory as well. At first MySQL said it was their bug and then said the behaviour is expected. So we are still investigating.

In the meantime:
1) Which version of MySQL are you using?
2) Are you able to check the memory used by MySQL process before and after the import (in TaskManager if you are running Windows)

ab042

Ok, I just tryed downloading the latest GA version of MySQL (v5.0) and I just downloaded your latest build 838 on a NEW MACHINE.

When I tryed the import it still hangs and goes to 99% CPU at 70%. I checked the physical memory and the machine had over 1 gig available at the time of the lockup.

Its the javaw thats eating the CPU not MySql. I then tryed changing the startupoptions.props in the BIN directory after I copied it and changed it to allow more memory 384 and again at 512 and it still locks in the same spot.

I'm not sure what it is but you do have a problem here. And I'm very concerned that if it locks up on a simple file import, that under heavy use it may lockup also. And that could be a major problem.

However, I've only been working with the app a few long days and your response to the questions and answers on these forums have been nothing short of fantasic and I thank you for it.

I also trust you'll find a solution soon. Its going to take a lot longer for us to build the app then it will take you to find this issue. Thanks again.

aware_support

I checked the physical memory and the machine had over 1 gig available at the time of the lockup.
Last time you said there were only 256K available memory left at the time of the lockup? Or was it on a different machine and on the current machine the situation is different?

We can't reproduce the problem on our systems. As I said we have imported half a million records on a mcahine with 1Gb of RAM and the memory used by javaw did not change ! MySQL memory did change and we thought this is where the problem was. Apparently not.

So there is still a chance that the problem is specific to your data and without having access to your data our task becomes very difficult. Please help us find this problem and fix it. First of all, let's find out for sure whether the problem is data related or not. Certainly the fact that it stops at the same spot NO MATTER WHETHER YOU INCREASE AVAILABLE MEMORY OR NOT seems to point to the data problem. Therefore, questions:

1) How much memory is used up by javaw when the system locks up?
2) Does the memory used by javaw keep increasing and is not released during import?
3) How much memory is used by the MySQL process during the lockup?
2) Have you tried to split the CSV file in two and import each half?
separately?

Your answers to EACH of the question may help us find the problem.

Best Regards