Some time ago two guys wrote a patch for clamav
to use a different filter that works with the
collaboration of bm matcher,
in brief bloom av gives no false negative
but may have false positive, the file then is passed to
the bm matcher.
The attached patch is rather old, with new changes to the
engine I don't known if it still works,
and how it's easy to adapt.
it also needs to be tweaked to support scan with offset
(right now I've made as a false positive so the scan is passed to bm)
bloom av is faster than bm, the overall scan speed
is improved since the hypothesis is that
non virus files are a lot more than virus files.
I've attached also a profiled scan
look the detail:
[bm + ac]
54.63 166.07 166.07 8012 0.02 0.02 cli_bm_scanbuff
22.41 234.20 68.13 139428866 0.00 0.00 cli_findpos
15.26 280.59 46.39 8012 0.01 0.01 cli_ac_scanbuff
[(bloom | bm) + ac]
27.85 67.22 67.22 139428866 0.00 0.00 cli_findpos
27.31 133.15 65.93 245 0.27 0.27 cli_bm_scanbuff
19.52 180.26 47.11 8012 0.01 0.01 cli_ac_scanbuff
and
2.20 217.76 5.31 8012 0.00 0.00 cli_bloom_filter_scanbuff
so we gain 8012 - 245 bm scans, replaced by 8012 bloom scans that are faster
the logic of the overall scan is:
bloom first, if positive bm,
then ac as normal flow
the patch has not yet included gpl header, but the guy gave me the permission
to distribute it as GPL
Hope this helps
- --
Gianluigi Tiesi <
sherpya@netfarm.it>
EDP Project Leader
Netfarm S.r.l. -
http://www.netfarm.it/
Free Software:
http://oss.netfarm.it/
[bm + ac]
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
54.63 166.07 166.07 8012 0.02 0.02 cli_bm_scanbuff
22.41 234.20 68.13 139428866 0.00 0.00 cli_findpos
15.26 280.59 46.39 8012 0.01 0.01 cli_ac_scanbuff
5.54 297.43 16.84 14514 0.00 0.00 body
0.49 298.91 1.48 582 0.00 0.00 cli_html_normalise
0.24 299.64 0.73 5481172 0.00 0.00 cli_hex2int
0.19 300.21 0.57 149806 0.00 0.00 cli_readline
0.17 300.72 0.51 5188 0.00 0.00 cli_vermd5
0.16 301.21 0.49 5125122 0.00 0.00 html_output_c
0.16 301.70 0.49 39173 0.00 0.00 cli_hex2str
0.10 301.99 0.29 35677 0.00 0.00 cli_parse_add
0.07 302.20 0.21 3442 0.00 0.00 cli_filetype
0.07 302.40 0.20 41457 0.00 0.00 cli_chomp
0.06 302.58 0.18 52229 0.00 0.00 cli_strtok
0.05 302.72 0.14 5188 0.00 0.06 cli_scandesc
0.04 302.83 0.11 7 0.02 0.02 cli_bm_free
0.03 302.92 0.09 107609 0.00 0.00 html_tag_arg_add
0.03 303.01 0.09 4512 0.00 0.00 cli_hex2si
0.03 303.09 0.08 33367 0.00 0.00 cli_bm_addpatt
0.02 303.15 0.06 3442 0.00 0.06 cli_magic_scandesc
0.02 303.21 0.06 2477 0.00 0.00 cli_dequeue
0.02 303.27 0.06 2144 0.00 0.00 is_tar
0.02 303.33 0.06 7 0.01 0.01 cli_bm_init
0.02 303.38 0.05 57115 0.00 0.00 html_tag_arg_free
0.01 303.43 0.04 129846 0.00 0.00 cli_calloc
0.01 303.47 0.04 2727 0.00 0.00 zzip_file_open
0.01 303.51 0.04 22 0.00 0.25 cli_scanzip
[(bloom | bm) + ac]
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
27.85 67.22 67.22 139428866 0.00 0.00 cli_findpos
27.31 133.15 65.93 245 0.27 0.27 cli_bm_scanbuff
19.52 180.26 47.11 8012 0.01 0.01 cli_ac_scanbuff
7.02 197.21 16.95 14514 0.00 0.00 body
4.11 207.13 9.92 71896936 0.00 0.00 ba_value
2.20 212.45 5.32 993325 0.00 0.00 lookup_in_rb_table
2.20 217.76 5.31 8012 0.00 0.00 cli_bloom_filter_scanbuff
1.27 220.81 3.06 1026692 0.00 0.00 rb_find
1.04 223.31 2.50 13984528 0.00 0.00 compare_ints
0.94 225.59 2.27 29493596 0.00 0.00 ba_assign
0.88 227.72 2.13 15407376 0.00 0.00 cli_hex2int
0.73 229.48 1.76 582 0.00 0.01 cli_html_normalise
0.73 231.24 1.75 3465471 0.00 0.00 fast_hash
0.63 232.76 1.52 7 0.22 0.54 init_bloom_filter
0.60 234.21 1.45 105907 0.00 0.00 cli_hex2str
0.49 235.39 1.18 13878630 0.00 0.00 xor_hash
0.45 236.48 1.09 2109916 0.00 0.00 sdbm
0.25 237.09 0.61 149806 0.00 0.00 cli_readline
0.21 237.60 0.52 5125122 0.00 0.00 html_output_c
0.17 238.03 0.42 3432104 0.00 0.00 is_possible_match
0.14 238.37 0.34 5188 0.00 0.00 cli_vermd5
0.13 238.68 0.31 52229 0.00 0.00 cli_strtok
0.12 238.97 0.29 35677 0.00 0.00 cli_parse_add
0.11 239.24 0.27 4512 0.00 0.00 cli_hex2si
0.10 239.47 0.23 3442 0.00 0.00 cli_filetype
0.06 239.60 0.14 41457 0.00 0.00 cli_chomp
0.06 239.75 0.14 7 0.02 0.02 cli_bm_free
0.05 239.88 0.13 106208 0.00 0.00 cli_rndnum
0.05 240.00 0.12 7 0.02 0.02 rb_create
0.05 240.11 0.12 30516 0.00 0.00 rb_probe
0.05 240.22 0.11 33367 0.00 0.00 cli_bm_addpatt
0.05 240.33 0.11 33367 0.00 0.00 insert_into_rb_table
0.05 240.44 0.11 22 0.01 0.26 cli_scanzip
0.04 240.54 0.10 5188 0.00 0.04 cli_scandesc
0.03 240.61 0.07 3442 0.00 0.04 cli_magic_scandesc
0.02 240.67 0.06 2477 0.00 0.00 cli_dequeue
0.02 240.72 0.05 7 0.01 0.01 cli_bm_init
0.02 240.76 0.04 196580 0.00 0.00 cli_calloc
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla:
http://bugs.clamav.net
(text/plain)
