Big Antivirus Test at your place
12 May 2008
See why no security program is the best
Apart from the interface design that I'm obsessed with, there is one important question—should we trust antiviruses? I did the test below that anybody can repeat at home. It is not a comprehensive test, but it is easy to see that "the perfect antivirus" simply does not exist.
The driving force behind this article is one scary opinion [link to video with subtitles in Russian 30 Mb], that viruses can be easily modified that no antivirus will identify them. I'm not going to use Assembler and modify the source code (maybe next time), I will use ready-made solution: ASPack 2.12 — "an advanced Win32 executable file compressor." This program will compress the executables leaving their functions intact. So if the executable is a virus, the virus will act in exactly the same way being packed. Good antivirus is supposed to use advanced emulators to see through packers, as some packers hide the true code very well.
Setup
I've found a helpful link to online virus scanner virusscan.jotti.org which will let you scan any file with 20 (!) antivirus engines at once. The listed providers are: A-Squared, AntiVir, ArcaVir, Avast, AVG Antivirus, BitDefender, ClamAV, CPsecure, Dr.Web, F-Prot Antivirus, F-Secure Anti-Virus, Fortinet, Ikarus, Kaspersky Anti-Virus, NOD32, Norman Virus Control, Panda Antivirus, Sophos Antivirus, VirusBuster, VBA32.
The downside of this website [screenshot] is that we don't know how old the virus databases are, but for the purpose of this test it is not critical. I will mostly use well-known viruses.
![[screenshot]](img/viruses/virusscan.jotti.org.png){kind=link}
In total I used 21 virus samples, each in two variants—original and packed by ASPack:
First five seems-to-be virus samples I took "from the wild", from the Mininova.org torrent news website. You know, when you see something like this, you can be almost sure it is a virus.
The other 16 proved viruses I used from my archive.
Test
Here's the Excel table with complete test results to download [format Excel 2007].
First I checked five suspicious red-hot programs from the internet: "Windows_Ultimate_Keymaker.exe", "Vista_Developer_Activation.exe", "Rapidshare_Premium_Points.exe", "Norton_360_Keygen.exe", and "virus_1.EXE". I modified the names but kept the meaning.
The first one was obviously a new virus, detected by 8 out of 20 antiviruses (my installed Avast 4.8 detected it as well, but in case of virusscan.jotti.org, Avast virus database seemed to be old). Here came the first surprise, after ASPack treatment number of positive results shrank to three!
The other four samples were unlikely viruses, but what we see that some programs "recognized" them only in packed stage, i.e. typical example of false positive: Ikarus, Avira, ClamAV.
Then I started feeding the proven viruses. I was surprised there were number of cases when ASPack simple modification could make the virus "clean" for detection. To summarize, I put the results measured in my "Efficiency Rating" in the table below. For each successful detection, when the identified virus before and after ASPack was the same, I put 1 score. If it was nothing detected, or not seen after packing, or virus was identified differently before and after ASPack, or detected only after ASPack (false positive) I put zero for the respective antivirus program.
Please look at the table. The first is Dr.Web — Russian-made antivirus solution. There are some well-known names a little bit behind. I was surprised to see Panda and BitDefender performing so-so. Again, table with full details is here.
| Antivirus | Official website | Efficiency Rating |
| Dr.Web | www.drweb.com | 15 |
| AVG Antivirus | www.grisoft.com | 14 |
| Kaspersky Anti-Virus | www.kaspersky.com | 14 |
| NOD32 | www.nod32.com | 14 |
| VBA32 | www.anti-virus.by/en | 14 |
| CPsecure | www.cpsecure.com | 13 |
| ClamAV | www.clamav.net | 12 |
| F-Secure Anti-Virus | www.f-secure.com | 12 |
| Ikarus | www.ikarus.at | 12 |
| Avast | www.avast.com | 11 |
| AntiVir | www.avira.com | 10 |
| BitDefender | www.bitdefender.com | 9 |
| Panda Antivirus | www.pandasoftware.com | 9 |
| Sophos Antivirus | www.sophos.com | 6 |
| Fortinet | www.fortinet.com | 3 |
| F-Prot Antivirus | www.f-prot.com | 2 |
| VirusBuster | www.virusbuster.hu/en | 2 |
| A-Squared | www.emsisoft.com | 1 |
| Norman Virus Control | www.norman.com | 1 |
| ArcaVir | www.arcabit.com | 0 |
Outcome
Of course, we don't have Norton, McAfee, TrendMicro, and OneCare Live here, but I believe the results would be somehow similar. It seems any antivirus can be cheated by manipulating either the source code of a virus, or—what I did—the executable itself. That means new virus outbreak is just clicks away.
What to recommend? You can safely use any program in the first part of the list (yet there is nothing to say to excuse the other ones). Just remember that there is no silver bullet for fighting malware. Use your brain and don't rely on single solution when you know you can be exposed.
All five first programs in my table above seem to have excellent emulators. Yet, I'm glad Dr.Web has won this micro race.
Popular Searches
Windows Vista Antivirus