|
PureBytes Links
Trading Reference Links
|
Hello, Everyone:
The old message below was no joke, even though it is dated April 1. ;^)
I had asked Tomasz, at that time, why I was having trouble Saving my large
databases - large in the sense of a *large number of files* (stocks). As I
built up my first big database, I found it had nearly 2,000 files. The time it
took to Save became annoyingly long. I was getting *really* worried when a
Save grew to the 55 minute mark, because it starting cutting into time doing
other necessary things, such as sleeping!
Another worrying thing was that, during a Save, the drive thrashed
*constantly*. It took so long to Save a file and thrashed the drive head so
much with each one that I started worrying about my drive. The poor old drive,
a Syquest 44Mb, sounded like a buzz saw! (This thing makes an incredible
racket, at the best of times, but it has never caused me an ounce of trouble.)
This was around the time Tomasz wrote the old message, below.
Fast forward to when I got a new hard drive. It was also around the time that
2.60 was released. If I recall correctly, 2.60 had a speed-up of about 10% for
Saves. After I put my data onto the new drive and updated AmiBroker, the
Load/Save times dropped signifiantly (less than half the Save time)! I had
*NO* idea why (and neither did Tomasz), then, but since everything was working
so much better, I promptly chalked the experience up to one of those weird
computer things or the new Save code and didn't research it anymore (but I
*didn't* forget about it).
Today, I decided to do a test, and I formatted one of these old-tech removable
disks using the DC-FFS, as Tomasz had recommended. I had copied my
second-largest database (2,300 files) into memory on another computer (so that
it would be as quick as possible), linked by a slow old-tech ARCnet network. I
started copying the database over to the newly formatted disk. Everything
looked just great, so I went to have supper. An hour and a half later, the
copy was still going on! The drive is thrashing like crazy again and it's
taking 11 seconds to copy one file! Bingo, the buzz saw had returned!
I stopped the copy process - only 1300+ files had been transfered, so far.
This was only a bit more than half of the database! The disk was reformatted
using the basic FFS, and the copy process was restarted. It only took ~28
minutes to finish! It took AmiBroker <28 minutes to reSave this same database.
So, Tomasz, there's the answer: if you have a large databse, do *NOT* use
DC-FFS. This file system may be OK for a very small database, but as the
number of files in the directory increases, the problem magnifies
exponetially.
Maybe my experiences can help some-one else on the ML from having similar
troubles; nobody who has not been through something like this can even imagine
how relieved I am! Other AmiBroker users might want to check the file system
of their own data partitions.
Does anyone have any other suggestions to help with handling a large number of
files, such as using a different file system cache or a non-Commodore file
system?
--
---===///||| Donald Dalley |||\\\===---
The World of AmiBroker Support
http://webhome.idirect.com/~ddalley
Member: ICOA and Team AMIGA
On 01-Apr-98, Tomasz Janeczko wrote:
> Hi Donald,
> > > Well, it's nice to hear that my code is not that bad :-)))
> > > But seriously, on pretty old IDE drive (420MB Conner/transfer <1.2M/sec)
> > > and 030/50MHz CPU AmiBroker
> > > manages to load 200 stocks (2.5MB) database in less than 10 seconds
> > > (when the drive is not fragmented).
> > > Fragmentation can slow down the thing to about 20 sec.
> >
> > Really? My biggest database is well over 9 times that and still
> > takes about 15/12 *minutes* for a Load/Save!
> >
> > I wonder why there is such a big difference?
> The difference is due to the fact (I think) that you have big number of
> stocks in your database.
> Amiga filesystem (FFS) handles upto 100 files in single directory with ease.
> Accessing about 200 files
> is not that much slower, but if you try to load 2000 files things start to
> be really slow. You can
> check how long it takes to get the directory listing using dir command.
> Situation is better when you use Directory Cache (DC-FFS) filesystem
> introduced in OS3.0.
> I have got all my partitions formated in DC-FFS and the results are as
> above. You may have far better results when using AFS or other advanced
> third-party file systems for Amiga.
> You can also try to ADDBUFFERS to your drive and check if it helps.
> Check also how long it takes to simply copy your database from one location
> to another using
> your favourite disk utility or DOS COPY command. Loading time should not
> exceed twice copy time.
> (On my drive loading takes about the same time as copying from one directory
> to another located in the same partition)
> Filesystem is one thing, the second one is time spent on memory allocations
> - this is at least 20% of loading time.
> > > Due to the program changes you should get at least +10% speedup on IDE
> > > drive (on SCSI drive the result may be
> > > much better since SCSI uses DMA and IDE uses CPU).
> >
> > It's kind of hard to tell when I go away and have supper while a
> > Save is going on. FOMCL
> Well, so then 1 or 2 minutes more or less don't make a difference :)
> Best regards,
> Tomasz Janeczko
|