Anti-Malware Testing Standards Organization

The Anti-Malware Testing Standards Organization, commonly known as AMTSO, is an international nonprofit association focused on improving the quality, relevance, and objectivity of anti-malware testing. It was established in 2008 in response to concerns that security-product evaluations were not always methodologically consistent, sufficiently transparent, or closely aligned with the threats and protection capabilities being assessed. Rather than selling endpoint-security products, AMTSO provides a forum in which security vendors, independent testing laboratories, academics, reviewers, publications, and other interested participants can discuss how anti-malware and related technologies should be evaluated. The organization’s stated role combines standards development, education, and practical support. Its objectives include facilitating discussion about testing anti-malware and related products; developing and publicizing objective testing standards and best practices; increasing awareness of testing issues; and supplying tools and other resources that can help evaluators conduct standards-based assessments. The scope is broader than traditional malware-detection tests and can include security features, cloud-assisted protection, performance considerations, mobile protection, and other aspects of modern security software, where relevant guidance is available. AMTSO grew out of discussions among security-product manufacturers and testing organizations. Its membership model also permits participation by academic institutions, reviewers, publications, and individuals. Entity members receive the full benefits of membership, while individual members generally receive comparable benefits without voting rights. The organization’s membership has included prominent testing laboratories and security companies, although the participation of vendors has periodically generated debate about independence and governance. Some testers and observers have argued that a strong vendor presence can create perceived or actual conflicts of interest, while AMTSO has continued to present itself as a forum intended to improve testing practice through participation from both sides of the testing relationship. Its governance initially relied on an elected, unpaid board supported by an advisory board and several committees. A significant infrastructure and management change followed in 2012, when an executive structure including chief executive, technology, financial, marketing, and strategy roles was added alongside the board. By 2023, the board was described as evenly divided between vendor and tester representatives, an arrangement intended to balance the interests of organizations that create security products with those that independently evaluate them. AMTSO’s practical output includes guidance documents, testing resources, references to relevant work outside the organization, and workshops held approximately three times each year. Workshops provide a setting for debate and for developing or refining guideline documents. One publicly available resource, the Security Features Check, is designed to test whether a device or endpoint implements basic anti-malware protection behavior by attempting to retrieve a harmless simulated malicious file. The organization therefore occupies a standards and coordination role within the cybersecurity ecosystem rather than a conventional consumer-brand role. Its influence is most visible among security vendors, testing laboratories, reviewers, and other professionals concerned with the credibility and comparability of anti-malware evaluations.

AMTSO was created in 2008 as an international nonprofit response to perceived weaknesses in the way anti-malware and related security products were tested. At the time, the security industry included numerous vendors and independent laboratories, but testing methods could differ substantially in scope, terminology, sample selection, measurement, and interpretation. AMTSO’s founding purpose was not to operate as another product-testing laboratory. Instead, it sought to create a common forum where the participants in the testing ecosystem could discuss methodology and work toward more objective and useful practices. The organization’s charter established four broad areas of activity: discussion of anti-malware testing; development and publication of standards and best practices; education and awareness; and provision of tools and resources for standards-based testing. This remit allowed AMTSO to address both technical and institutional questions. Technical questions include how to test protection against different types of threats and how to assess security features. Institutional questions include transparency, comparability, independence, and the responsibilities of vendors and evaluators when results are published. In its early organizational form, AMTSO was administered by an elected and unpaid board of directors. An advisory board provided strategic and other input, while committees handled areas such as membership, fees, public relations, and operational matters. In 2012, the organization undertook a major infrastructure change and added an executive team with roles including chief executive, chief technology, chief financial, marketing, and strategy responsibilities. This change supplemented rather than eliminated the board-based governance model. Membership expanded beyond the original community of vendors and testing organizations. Academic participants, reviewers, publications, and individual members could also take part. The organization later operated a two-level membership model: entity members received full membership benefits, including voting rights, while individual members received most benefits without the right to vote. The fee structure was a recurring practical issue because full membership costs could be difficult for individuals and small organizations to absorb. AMTSO’s membership also became a subject of discussion within the wider security community. Because security vendors represented a substantial part of the membership, some observers and testing organizations questioned whether the organization could be perceived as sufficiently independent. Some tester members left and, in some cases, later rejoined. These debates reflected a broader tension in cybersecurity evaluation: vendors possess important technical knowledge and resources, but their commercial interests can appear to conflict with the goal of impartial testing. AMTSO’s later board composition was described as evenly divided between vendor and tester representatives, an arrangement intended to provide formal balance. The organization’s work has included repositories of guidance documents, links to relevant external research and resources, and practical tools. Its Security Features Check is a freely available test intended to verify basic protective behavior on desktop or Android environments by attempting to download a simulated malicious file. AMTSO also organizes workshops around three times per year. The workshops support discussion and commonly produce or refine guideline documents that can assist both established and aspiring testers. AMTSO has also faced legal controversy. NSS Labs, a testing organization that had been a member, sued AMTSO together with CrowdStrike, ESET, and Symantec in an antitrust action. The case was dismissed, and NSS Labs later stopped operating. The litigation became part of the continuing debate about the organization’s membership structure and the competitive relationships among testing laboratories and security vendors. By 2023, the board was reported to consist of eight directors with equal vendor and tester representation. Luis Corrons of Gen and Simon Edwards of SE Labs were identified as co-chairs. Other listed directors represented SecureIQLab, ELLIO Technology, SKD Labs, Gen, Testing Ground Labs, and Microsoft. AMTSO therefore continued to function as a professional standards and coordination organization whose principal outputs are guidance, testing resources, education, and industry dialogue rather than commercial security products.

1. 2023Board is reported as evenly split between vendors and testers

   The listed board composition gives equal representation to vendor and testing-organization members.

   Source
2. 2012Executive management structure is added

   A major organizational infrastructure change introduces executive roles alongside the existing board of directors.

   Source
3. 2011Lower-cost individual membership is introduced

   AMTSO adds a less expensive membership option aimed at individuals and smaller organizations, without full voting rights.

   Source
4. 2008AMTSO is established

   An international nonprofit organization is formed to address concerns about the quality, relevance, and objectivity of anti-malware testing.

   Source

An industry-wide, nonprofit standards and coordination body intended to make anti-malware and related security-product testing more objective, relevant, transparent, and methodologically consistent.

• Security Features CheckTesting utility

  A freely available AMTSO resource intended to verify that basic anti-malware protection functions correctly on a test device. It attempts to download a harmless simulated malicious file rather than actual malware, allowing evaluators and users to check whether the relevant security controls respond as expected. The resource has been described for desktop and Android environments.

• AMTSO Guidelines and Best PracticesStandards and guidance

  A collection of methodological guidance intended to help security-product testers design and conduct more objective, relevant, and reproducible evaluations. The material addresses testing practice across a range of anti-malware and related security topics and is developed through organizational discussion and workshops.

• AMTSO WorkshopsProfessional forum

  Recurring industry workshops, held approximately three times per year, provide a forum for vendors, testing laboratories, and other participants to discuss testing methodology and develop or refine guidance documents. They are a central mechanism through which AMTSO turns debate within the testing community into practical resources.

• 2012Add an executive management structureStrategy

  AMTSO had previously relied primarily on an elected unpaid board, advisory input, and operational committees.

  What changed. The organization introduced executive roles covering chief executive, technology, finance, marketing, and strategy functions alongside the board.

  Aftermath. The change created a more formal operating infrastructure while retaining board governance.

  Source
• 2011Create a lower-cost individual membership tierStrategy

  The cost of full membership could discourage individuals and small organizations from participating.

  What changed. AMTSO introduced a cheaper subscription option that provided membership benefits but excluded full voting rights.

  Aftermath. The two-tier approach allowed broader participation while preserving distinct rights for entity members.

  Source

• Antitrust litigation brought by NSS LabsControversy

  NSS Labs sued AMTSO and several security vendors in an antitrust case. The action was later dismissed. The dispute was significant because it involved the structure and competitive implications of relationships between testing organizations, vendors, and an industry standards body.

  Source

• 2023AMTSO described with a balanced vendor-and-tester board

  The organization’s board was described as having equal representation from security-product vendors and testing organizations, with Luis Corrons and Simon Edwards serving as co-chairs.

  Leadership change
  Source
• 2011AMTSO introduces a lower-cost individual membership option

  The organization introduced a less expensive subscription category for individuals and smaller participants. The category provided access to membership benefits but did not include full voting rights.

  Other
  Source

1. Anti-Malware Testing Standards Organization