Taking The Significant Alerting Lead in MITRE ATT&CK® Evaluations’ Latest Round

With zero configuration changes, our AMDR rose far above the competition to deliver the most alerts of any vendor.

A leader in artificial intelligent automated SOC operations and endpoint security, announces our results from round two of the MITRE ATT&CK® Evaluations. We received the most alert detections out of the 21 participants validating our world-leading thoroughness, accuracy, and results-oriented focus.

Number One in Alerting

In achieving its goals to be fast, accurate, simple, and thorough, our AI generated the most accurate and thorough alerting among the evaluation participants. With alerts on 90 substeps in the evaluation, including General, Tactic, and Technique detections as per MITRE distinction, we provided the most complete alerting against the Advanced Persistent Threat 29 challenge. Alerts show that rather than just passively detecting adversary behavior, a vendor is capable of prioritizing that behavior and communicating it to the SOC operations team so they are able to act on it.

Cybots MITRE Evaluations Substep Alerts
“Alerts constitute the basis for all meaningful action in a SOC as operators need clear, concrete insight into their sea of security data to know where the ongoing and potential fires are,” said Chad Duffy, CyCraft’s Global Product Manager. “It is crucial that security providers are able to rapidly and accurately alert operators to prevent attacks escalating from discovery to breaches. If there is no alert, then it doesn’t really matter if you detect an attack or not, as the information is lost in an ocean of data. Equally important is doing so without requiring configuration changes to generate the detections that lead to the alerts, as SOC operators don’t have time to mess with settings when they are under siege. Plus, how would SOC operators even know what changes to make when they can’t find the detection in the data–without an alert, they don’t even know what to look for to drive the config change.”

Zero Configuration Changes

User- and results-focused, we generate all of the alerting with zero configuration changes.

A configuration change is when the vendor engages in onsite manipulation of their configuration to register detections.

A New Perspective on Telemetry

Further, CyCraft quickly and autonomously generated a complete storyline of the attack across systems, allowing security practitioners the ability to fully digest the entire attack and cyber situation concerning the ATT&CK emulated adversary. Because of this focus on context and enrichment to make results meaningful, and due to the AI on the CyCraft’s sensor as well as in its CyCraft AIR cloud platform, CyCraft does not generate much in the way of typical low-value telemetry detections, which are often in the form of raw, unprocessed, easily overlooked data requiring expert review, but instead registers them at higher levels on the MITRE classification: General, Tactic, and Technique. This is part of CyCraft AIR’s auto investigation, which takes what would be the raw telemetry data and auto investigates it via a proprietary AI engine.


The adversary, APT29, is a group that cybersecurity analysts believe operates on behalf of the Russian government and compromised the Democratic National Committee starting in 2015. MITRE used its ATT&CK knowledge base to examine the products’ ability to detect the tactics and techniques used by APT29 when emulating the group.

“We view the evaluations as a collaborative process to help the participating vendors improve their products, which ultimately makes cyberspace safer for everyone,” said Frank Duff, ATT&CK Evaluations lead. “Taken as a whole, the results indicate that the participating vendors are beginning to understand how to detect the advanced techniques used by groups like APT29, and develop products that provide actionable data in response for their users.”

Since the ATT&CK APT29 Evaluation of 2019, we further automated and sped up its alerting with a new version of its MDR AI engine and released its enterprise prevention and protection platform with NGAV for real-time blocking of suspicious and known threats. We will be participating in the next round of evaluations against emulations of financial institution-targeting adversaries Carbanak and FIN7.

“We see the ATT&CK Evaluations as a great playing field leveler. Finally, there is a place for vendors to go head-to-head in a transparent way that is meaningful to buyers, and the rest of the industry—a veritable blue-team cyber colosseum for leading products around the world to benchmark their true capabilities. End users are often overwhelmed with marketing buzzwords and frustrated in the dearth of concrete info when comparing products to avoid redundant, weak, or non-existent capabilities. With the MITRE evaluation and accompanying matrices, end users, vendors, buyers, and the industry at large now have a lexicon and a map to best spot, detect, respond to every move, and communicate effectively when facing sophisticated attacks,” 

About ATT&CK

ATT&CK® was created by MITRE’s internal research program from its own data and operations. ATT&CK is entirely based on published, open-source threat information. Increasingly, ATT&CK is driven by contributions from external sources. Cybersecurity vendors may apply to participate in the next round of the ATT&CK Evaluations, which will feature the Carbanak and FIN7 threat groups as the emulated adversaries.


MITRE’s mission-driven teams are dedicated to solving problems for a safer world. Through our public-private partnerships and federally funded R&D centers, we work across government and in partnership with industry to tackle challenges to the safety, stability, and well-being of our nation. www.mitre.org

get the latest threat intelligence and cybersecurity news

Subscribe to our newsletter to get updates on our latest analyst reports, webinars, whitepapers and case studies related to the cybersecurity world.

more cybersecurity updates

Irma Group Company Annual Meeting 2022

October 2022 – IRMA Group gathered key staff in Malacca (Malaysia) to share their corporate activities, plans and explore areas of cross-entity synergy that could be leveraged on to benefit the Group. Over four days of intense meetings, staff from Cybots, Ark Insights and Irma Insights shared ideas, achievements and aspirations. This was punctuated by some great recreational activities and meals.

The meetings served to create pathways and connections for cross-entity business synergy. It also included identification of new business opportunities and areas of expansion.

The event was a great success and plans are already unfolding for the next one.

Read More »