Application HANGS on IMPORT

ab042 · Post by **ab042** » Tue Jul 18, 2006 1:49 am

To try and get some real world samples of our data and how your app will work.

I am trying to IMPORT a 12 meg file with 5 fields per record.

I've created the Business Object with the 5 fields NO RULES. In fact, the only thing I've done is create the Business Object and trying the IMPORT feature.

It works fine on smaller databases but hangs around the 70% mark for the large 12meg file.

Any suggestions?

Your application HANGING in a MAJOR ISSUE with our testing. Please help!

ab042 · Post by **ab042** » Tue Jul 18, 2006 2:14 am

To provide a little more information

The process that appears to be hung is JAVAW.EXE at 97% cpu
Its using 60,716k of memory on a P4 w/2gig RAM

aware_support · Post by **aware_support** » Tue Jul 18, 2006 3:38 am

Please export your business space version into the BSV file and e-mail this file along with the 12Mb CSV file to [email protected] (please zip up the CSV file or send it in chunks)

We need to reproduce this problem in-house.

Thanks

ab042 · Post by **ab042** » Tue Jul 18, 2006 4:30 am

Since my last post. I have removed JAVA and rebooted and reloaded it with the latest version and it does the same thing.

The data is company confidential so I can't send the file.

But creating the Business Space couldn't be easier.

EMailLog
- Attributes
--- Date: Date (mm/dd/yy)
--- EMail: Plain Text
--- Provider: Plain Text
--- RefNo: Number
--- Time: TimeStamp (hh:mm)

Publish it, select IMPORT. Somewhere around the 70% or 170,000 records it takes about all the CPU % and never recover's or passes this mark.

aware_support · Post by **aware_support** » Tue Jul 18, 2006 6:06 am

We have just successfully imported a CSV file containing half a million records (10Mb).

There must be some record in your file that causes Aware IM to hang. We need to find this record. There must be a file in your server's temporary directory whose name starts with "log" and the extension is "tmp". If you can find this file open it with the text editor and look up the last message which should identify the last record the system has been processing. Find this record in the file (just to be sure include 5 records before and 5 records after) and send these records to us (hopefully you can disclose a few records?).

Also:
1) What database are you using?
2) What are the parameters used during import - are these default parameters (validation is off and batch size is 1000)?
3) How much free memory does your server have?

ab042 · Post by **ab042** » Tue Jul 18, 2006 6:46 pm

MySQL / No validation and the batch size is 1000
At the point of hanging it shows 235k available physical memory

I took approximatly 1200 records before the error and 1200 records after the error and imported that without an issue.

In both cases it hung at after showing 165000 records have been processoed and nothing after that in the LOG files.

So the problem must have something to do with memory and not an item in the file.

Any suggestions?

aware_support · Post by **aware_support** » Wed Jul 19, 2006 12:23 am

How much memory does your server have?

One thing to try is to specify more memory to be allocated to the Aware IM Server on startup. This is what you need to do:

1) Shutdown Aware IM if it is running
2) Open the file startAwareIM.bat (or startAwareIM.sh if you are running Linux or Mac) and find "-Xmx96m"
3) Replace "96m" with a bigger value, for example "-Xmx256m"
4) Re-start Aware IM and try import again

Let us know if this is indeed the memory problem.

aware_support · Post by **aware_support** » Wed Jul 19, 2006 12:26 am

Forgot to mention that the file "startAwareIM.bat (or .sh)" is in the BIN directory of your Aware IM installation.

ab042 · Post by **ab042** » Wed Jul 19, 2006 6:16 am

I made the suggested change an it still stops in the same spot.

After I setup 256m on the javaw line and after it Locked at 70%, I checked and there is 384k Physical Memory available.

Any other suggestions?

aware_support · Post by **aware_support** » Wed Jul 19, 2006 6:36 am

Can you watch the memory during import with the Task Manager? Does it keep growing?

Also, can you print out the ouput of the Aware IM Server when it hangs?

Can you also split the file in two and import both parts separately ?

aware_support · Post by **aware_support** » Wed Jul 19, 2006 7:01 am

One other thing. Can you check how much memory MySQL uses before the import and after the hang?

Post by **aware_admin** » Wed Jul 19, 2006 11:33 pm

It does seem that MySQL is leaking memory. We are investigating the issue now. Will post to this forum as soon as we find something out.

ab042 · Post by **ab042** » Sat Jul 22, 2006 1:11 am

Any update on this?

aware_support · Post by **aware_support** » Sat Jul 22, 2006 1:38 am

We are communicating with MySQL on this. We have run a simple MySQL test outside of Aware IM which seems to leak memory as well. At first MySQL said it was their bug and then said the behaviour is expected. So we are still investigating.

In the meantime:
1) Which version of MySQL are you using?
2) Are you able to check the memory used by MySQL process before and after the import (in TaskManager if you are running Windows)

ab042 · Post by **ab042** » Sat Jul 22, 2006 9:55 pm

Ok, I just tryed downloading the latest GA version of MySQL (v5.0) and I just downloaded your latest build 838 on a NEW MACHINE.

When I tryed the import it still hangs and goes to 99% CPU at 70%. I checked the physical memory and the machine had over 1 gig available at the time of the lockup.

Its the javaw thats eating the CPU not MySql. I then tryed changing the startupoptions.props in the BIN directory after I copied it and changed it to allow more memory 384 and again at 512 and it still locks in the same spot.

I'm not sure what it is but you do have a problem here. And I'm very concerned that if it locks up on a simple file import, that under heavy use it may lockup also. And that could be a major problem.

However, I've only been working with the app a few long days and your response to the questions and answers on these forums have been nothing short of fantasic and I thank you for it.

I also trust you'll find a solution soon. Its going to take a lot longer for us to build the app then it will take you to find this issue. Thanks again.