Re: [Clamav-users] XML and large file scan performance

GiM wrote:
> To ClamAV users ML in message 'Re: [Clamav-users] XML and large file scan performance' wrote:
>> 7MB XML file in 0.3s ?
>>
>> let's do some math:
>> 7*1024^2 * 10 /3 / 1024^2
>> 23.3 MB/s
>>
>> Do you have SCSI drive ?
>> Cause, correct me if I'm wrong the fastest transfer rate on ATA is about
>> 17MB/s and 40MB/s on SCSI device.
>
> I've checked, that UDMA6 should be able to do 133M/s and
> SATA even 150M/s, but this depends on both hardware and system.

It has more to do with the type of test (sequential or random), then with the
environment. Take a look at the difference, using a Western Digital WD1200JB
(7200 RPM, 8MB cache, UDMA5) with Sandra:

Benchmark Breakdown
Buffered Read : 80 MB/s
Sequential Read : 45 MB/s
Random Read : 7 MB/s
Buffered Write : 89 MB/s
Sequential Write : 42 MB/s
Random Write : 13 MB/s
Average Access Time : 8 ms (estimated)

The 133MB/s # you mentioned is the bus speed, or what you can do ideally if you
only make small transactions which stay in cache. In practice, you top out at
90% of the IDE bus speed. And a real world virus scanner is going to always be
dealing with new incoming data, most probably with multiple scanner processes or
threads going on (so I/O is heavily multithreaded), all of which means numbers
around 10-15 MB/s for IDE drives are realistic. :-)

> I'm quite curious what was the enviroment you were making your tests on.

Something like this on Unix:

dd if=/dev/disk of=/dev/null bs=8k

...is a reasonable benchmark (at least for ballpark # purposes), so long as you
read enough data to exceed any caching in RAM.

[ Now, if you've got enough RAM, perhaps you can do your virus scanning entirely
via a pipeline or socket, without ever hitting disk, which could explain a
faster time you saw, but the Subject of the thread suggests we're dealing with a
file on disk. Besides, the MTA shouldn't confirm receipt of a message until
it's written to permanent storage. ]

-- 
-Chuck
_______________________________________________
http://lurker.clamav.net/list/clamav-users.html