Anti-Virus Comparative report
What is wrong with Anti-Virus Comparative reports?
10 April 2009 by Jean De Daumier-SmithBased on Anti-Virus Comparative No. 21, February 2009 report
On-demand detection of Malicious Software
http://www.av-comparatives.org
All 17 antivirus product included in the test are already “very good anti-virus software” with relatively high (>90%) on-demand detection rates. So, straight from the beginning: this report is about software prodcuts that all do the job of virus detection and removal efficiently. Some well-known names are listed as Norton, Mcafee, Kaspersky, etc. There are also some strange names like Chinese Kingsoft and Command by Authentium. We don’t see there Trend Micro or Panda Antivirus; those probably have good reason not to participate in the test.
First you see the colorful tables of results with “award” indicated: TESTED, STANDARD, ADVANCED, and ADVANCED+. The awards represent a balance between detection rate and number of false positives detected. So, for example AVIRA AntiVir Premium 8.2 with detection rate 99.7% got ADVANCED comparing to ESET Nod32 v.3 which got ADVANCED+ award with detection rate 97.6% just because Nod32 showed less false positives. [There is a detail analysis of all false alarms where all “detections” are listed. I think there are two main reasons for these false alarms: (1) some programs are considered to be suspicious for corporate use (e.g. special network administration tools), even though they are not malicious, and (2) some packers/code samples used in executables are similar to those used to masquerade/construct viruses. The second reason when applicable may be used to judge antivirus quality.] Yet it is not clear till the very end. Logic sometimes suffers in the report:
“The Awards are not only based on detection rates – also False Positives found in our set of clean files are considered. A product that is successful at detecting a high percentage of malware but suffers from false alarms may not be necessarily better than a product which detects less malware but which generates less FPs.” I’m surprised they put such a statement at the end of the report. It is like — shall I read this all over again?
Here’s another good one: “We suggest to consider products with the same award to be as good as the other products with [the] same award.” Really?
I like also this: “Even if we deliver various tests and show different aspects of anti-virus software, users are advised to evaluate the software by themselves and build their own opinion about them.” Fair enough, good hint.
And the last one: “Not many Chinese vendors are eligible to participate in our international tests.”
Excuse my sarcasm, but the report feels like something very much homemade.
So, what is actually wrong with the report? Nothing, it is just all about very good anti-virus software. This report is “neither cold nor warm”, that is a problem. Besides, it is poorly designed and structured in spite of recent revamp. What I mean here is 3D graphs that should be 2D, boring colors, ugly results tables (too many borders, font misuse (italic, bold, all caps, aligned differently, underlined, etc.). Again, professional designers/writers were not involved. At least I would recommend the following to read before making such a report: The Visual Display of Quantitative Information by E. Tuft.
Who is the best finally according to Av-Comparatives? McAfee, Kaspersky, ESET, and Symantec. However, all the others are almost as good. For details please refer to the report.
Having said that, I’m looking forward to the retrospective test, “which evaluates how well products are at detecting new/unknown malware”. That’s should be interesting, as it shows how well an antivirus detects (would detect) a 0-day threat.
P.S. Testing methodology
http://www.av-comparatives.org/seiten/ergebnisse/methodology.pdf
Av-Comparatives get paid for its testing by antivirus vendors. They claim that being as non-profit organization (i.e. all their staff have fixed salaries) make them uninfluenced by money proposed and the test results are totally independent. Let’s believe in that.
Tags: Security, Antivirus