About Folder/SubFolders for Attachments. strategy?

[Jaymer]

We're not talking huge numbers here, but lets say an average company is going to get 5-20 new jobs a day into its shop.
Each job may have some images.
Aware is going to generate several attachments over the course of the job.
There's an Estimate, an Invoice, etc. that get printed as PDFs and attached to the job.

200 days per year so lets say 2000 jobs per year.
20,000 jobs in 10 year lifespan.

What strategy do you use to organize the filesystem for all these various docs & images that need to get stored?

And, we've all done the tests...
IF NOT EXISTS (some calculated directory path) THEN CREATE_DIR (another calc path).
And of course this gets done in several places...
1st time you add an image, the dir may not be there.
1st attachment, the dir may not be there.

If chances are that any job is going to need the \AttachmentDir at some point anyway, why not just go ahead and ALWAYS create that Directory/SubDir structure in the FS WHEN THE JOB IS CREATED - and then you don't ever need to check and have all that redundant code in the sub-processes.

So, if its 20,000 folders in the \AwareIM\Tomcat\webapps\AwareIM\Jobs\ directory, then is 40,000 ok? 100,000 ok?
just wondering...

[ACDC]

Why have a folder for each job ?
I have a file store folder that holds 295,000 files each referenced against the Main Document object ID+reference
Also, the file store is outside the Aware installation on another drive, I think that's a better option

[gijsvb]

Hi,

Choosing between one or many folders depends largely on the filesystem structure your system uses. Default linux ext3/ext4 filesystems get to become very slow with over 100.000 file entries in one folder. There are solutions for linux system to overcome this problem, but in a default configured linux system I wouldn't put a 100.000 files in one folder. In your example I would create subfolders per year holding the 2000 jobs of a year.

Afaik, for a windows default file systems this isn't a real problem.

[Jaymer]

ok, thx for that.
I'm mostly thinking Windows... and one of my pre-aware clients has a Win network and a mapped drive fluctuates with 75-100k file in a directory. Sure makes that sluggish when accessing that graphically in Windows Explorer.

Afaik, for a windows default file systems this isn't a real problem.

Well, I guess thats relative, isn't it?
Technically it may not be a problem, other than making the UI dreadfully slow and hard to find files.

[Jaymer]

Also, the file store is outside the Aware installation on another drive, I think that's a better option

Please elaborate as to "better option".

Since some of our attachments are Images, having them "under Aware" benefits from improvements to serve them directly to the browser without copying.

And Robin makes a linked directory so they are physically on a diff. Volume, but logically still under Aware.

[gijsvb]

ok, thx for that.
I'm mostly thinking Windows... and one of my pre-aware clients has a Win network and a mapped drive fluctuates with 75-100k file in a directory. Sure makes that sluggish when accessing that graphically in Windows Explorer.

Afaik, for a windows default file systems this isn't a real problem.

Well, I guess thats relative, isn't it?
Technically it may not be a problem, other than making the UI dreadfully slow and hard to find files.

For me, a folder managed by my application shouldn’t be browsable via file explorer at any cost. Users shouldn’t be able to move files around without my application knowing about it. What i meant is retrieving one file out of a 100.000 is less a problem for Windows systems as for Linux Systems.

[Jaymer]

For me, a folder managed by my application shouldn’t be browsable via file explorer at any cost. Users shouldn’t be able to move files around without my application knowing about it.

well, yes, thats very true.
i have one client who has their own server in a closet.
they create folders on the network for Jobs and dump photos in there. Only the owner can access that shared folder.
I used that structure to add my own attachments, etc. since the folders were already there. They started in v7 so those files aren't under Tomcat root.

Rather than the physical location of all the files, its probably more important that the FILENAME contain some way to re-attach it to its "owner", wouldn't you say, in the event of a database corruption.
Maybe thats always there in the back of my mind... if they are physically in a folder/dir then you know who they belong to at least, for recovery.

[customaware]

For ALL of my attachments, I append the Record ID to the Filename. That way... if I ever really need to, I know where it belongs.

[Jaymer]

For ALL of my attachments, I append the Record ID to the Filename. That way... if I ever really need to, I know where it belongs.

yes, good idea.
and i found out something that Robin had mentioned in Porto. He said he randomized his filenames - and shortly after I heard that, I experienced why, because the browser will think it has the image cache'd EVEN if you upload a new photo - the random part keeps that from happening.
(The older method of the images NOT under Tomcat root didn't have that issue.)

I use this to add a few digits: GENERATE_PWD(4,4,0,4,0)

[rprinzing]

All of the replies make sense to me. My personal preference (sanity only) is that my SAAS has a separate folder for each tenant (doctor). In their folder, each client has their on folder. In each client folder, there is a root, \docs, and \xrays. All of that is handled by the values found in SystemSettings. When a new client is added, the client folder is automatically created and the pointer is stored (firstname-lastname-ID which is always unique because of ID) in the client record. The \docs and \xrays are created then as well. All pics go in the root of the client (of the doctor), printed PDFs and Word/Excel documents go to \docs, and uploaded Xrays go to \xrays of the client. My server has two drives, so AwareIM is running from c:\ and the AwareIM_Docs is on d:\. As Jaymer pointed out, using powershell, I create a virtual linked folder that appears under the AwareIM application. Here is the powershell script (make sure the destination is created before you run this):
ni C:\AwareIM\Tomcat\webapps\AwareIM\DrJones -i SymbolicLink -ta "D:\AwareIM_Docs\DrJones"
After you run this script, a DrJones folder shows up under C:\AwareIM\Tomcat\webapps\AwareIM\.
For piece of mind, any doctor can ask about any patient and any FileSystem stored file, and I can get to it and answer questions quite fast. If a doctor wants to drop subscription to the SAAS, then I can zip up their folder, give it to them, and delete it off the server. Same with the DB.