How Important Are False Positives in Measuring the Quality of an Antimalware Engine?

Posted by Emma Ban on 2015-05-21 11:35:00

Antivirus false positives are not always attributed the importance they deserve.

Picture this scenario: you download a program from a legitimate source and when you try to install it, your antivirus stops you saying it’s potentially malicious. What do you do?

Sometimes, depending on the antimalware product you use, the installation is blocked and the program put to quarantine. In other cases, your antimalware product asks you if you want to continue with the download even though it thinks it’s risky. In this case, you can either abandon the installation and look for another source, or proceed with it since the source is well-known and thus, must be reliable.

If you decide to play it safe and go for the first option, you may have just spared yourself a malware infection. But you may have also gotten a feeling of uncertainty and distrust with regards to the legitimacy of the source you thought was safe.

If you decide to go for the second option, you may indeed have to deal with malware of some sort, which hopefully your antimalware solution will be able to neutralize. But it may also turn out that the program is harmless, and your product is guilty of detecting a false positive. If this happened once in a blue moon, it might not affect you that much. But if it happened often, you might just learn to ignore malware detection alerts, and eventually find yourself in a ‘Boy-who-cried-wolf’ type of situation.


False positives can cause significant issues…

...not only to regular end users, but also to enterprise customers, software developers and even the antivirus vendors themselves. Let’s analyze the issues from each perspective.


1. False positives and consequences for consumers. 

According to Virus Bulletin, an antivirus false positive occurs when anti-malware software erroneously labels a clean file as malware, and can be either “a clean file mistakenly labelled as malware” or “legitimate email incorrectly labelled spam.” And clean files can be any type of file, from those already installed on your computer, to new ones you want to download. 

If a security software blocks access or deletes a file that’s vital to the proper functioning of some system programs, those may become unusable and, in some cases, the deletion may render a system unstable. For example, if a Windows system file is falsely detected as malware and removed, the entire operating system will have to be restored. Think of this happening in a large corporate environment, and you can imagine the consequences.

At the same time, if a piece of legitimate email is mistakenly labelled as spam, it may be the case of an important business document not reaching its intended recipient. Which, again, may have consequences for that business person and their company.

2. False positives and consequences for businesses

Since we’ve moved from consumer to business environments, let’s look at things from a higher-level perspective – that of a system administrator or an IT security team: when a malware alert is triggered pointing to a certain file, for example, they have to investigate the issue in order to ensure the file is clean. If the alarm turns out to be a false one, then their time is wasted. Which is why, the more false positives they receive, the greater the chance they ignore the next alarm. And when that happens in a business environment, things can get quite messy. Case in point: the Target security breach – their IT security team ignored malware alerts they got before they realized their systems had actually been breached, leaving some 40 million credit and debit cards and personal data of 70 million customers exposed to hackers. Granted, the malware alerts in this case were part of far more complex detection systems than your ordinary antivirus software. And the reasons for them failing to take immediate action are not entirely known, but they may vary from lack of security resources, to lack of a structured mitigation plan. However, regardless of the detection systems and risk mitigation policies they have in place, the fact of the matter is that false positives cost companies a great deal of money. A recent report conducted by the Ponemon Institute shows that companies spend around 21,000 hours annually analyzing false negatives and/or false positives. Which means they spend roughly $1.3 million per year on activities triggered by inaccurate or erroneous intelligence.

3. False positives and consequences for software developers

As mentioned above, some less known software developers may also be affected by false positives, if some security software falsely detected their legitimate programs as malware. Needless to say, such false alarms would drive away potential customers.

4. False positives and consequences for antivirus vendors

Last but not least, antivirus vendors may also be affected in a number of ways. If their product detects false positives frequently, or one false positive has a widespread impact, their Technical Support team may get overwhelmed with complaints and they may even get bad press.

If the antivirus company uses third-party antispam or antivirus technology, they won’t be able to control the situation directly, but would have to rely on their provider for improvements.

 Guide to Antivirus technology licensing



The quality of an antivirus engine can be measured by its false positive rates

Because of all the issues end-users and organizations face when dealing with false positives, any reputable independent test agency puts high emphasis on low false positive rates.

AV-Test sees false positives as one of two different factors that influence the Usability of a product. For Virus Bulletin, “the 'no false positives' rule is one of the main requirements for certification in the VB100 test process”. While AV-Comparatives considers false positives “an important measurement for AV quality,” and an important factor in determining the reliability of a product, besides its detection capabilities. VirusTotal, the free Google-owned online malware scanning service has also addressed this issue recently.

Even end-users themselves, have started to put more emphasis on low false positive rates. A recent security survey among end-users conducted by AV Comparatives shows that 55.7% of them consider low false positive rates among the most important characteristics of a security product. More than half of those surveyed.


So, 100% malware detection rate OR 0 false positives?

Clearly, malware detection is the most important criterion when assessing the quality of an antivirus product. It’s its raison d'être. So the higher the detection rate, the better. But even if a solution has the highest detection rate, if it has a great number of false positives, it’s as useless as a solution with no false positives and a low detection rate.

Just think about it: if your security product detected all malware and kept you safe from it, but frequently screamed “malware-malware,” even when not the case, would you still trust it?

So, false positives rates go together with detection rates. Any change in the detection rate would imply a change in the number of false positives, unless measures are implemented to control them. Such measures require significant hardware and human resource investment, so only companies dedicated to security can justify those expenses.



Final thoughts

The industry is becoming more aware of the importance of false positives, and so should you. Whether you’re an end user looking for a better antivirus product, an organization looking for the best security solution to protect large infrastructure, or a company looking to license antimalware technology – either by integrating an antimalware SDK into your own solution, or selling an existing solution under your own brand – do not overlook the antivirus false positives issue. False positives test results conducted by independent testing companies like the ones mentioned above are always the best way to start a product or technology evaluation.



Just keep this in mind: for an antimalware engine or product to be eligible, the product test results must demonstrate both good detection and low false positives.



Find me on:

Emma Ban

Emma Ban is a Content Writer at Bitdefender. Having worked in the industry for more than three years, in both B2C and B2B areas, she has a deep understanding of the online threats that put at risk the security of both consumers and corporations. Thus, her main focus is to provide insights into security technology trends that enable safe environments for companies and their employees. She thoroughly enjoys traveling and has a special interest in fashion technology.

Topics: OEM Business, Endpoint Security, Technology