There are multiple avenues open to you to evaluate MDR vendors—each with its own advantages and shortcomings. Here are a few places to get started.

Analyst Reports

There are many research and advisory firms for technology in the world; however, none have reached the impact that Gartner has.

While we will focus on Gartner for this article, there are others we also recommend, including Forrester Research, IHS Markit, HfS Research, Ponemon Institute, IDC, Everest Group.

The Gartner Magic Quadrant

Since the 1980s, the Gartner Magic Quadrant (MQ) and its accompanying reports have been providing leadership with insights into a growing market’s trends, maturity, direction, niche players, challengers, visionaries, and leaders.

Gartner MQ provides a snapshot of vendors in the market but also affects the market as well, with vendors in the leadership quadrant gaining much attention. A separate industry complete with books, webinars, and snake oil exists solely to aid vendors in moving their dot on the MQ up and to the right.

However, do not make the mistake of ignoring the other 3 quadrants. One Niche Player could exclusively focus on your industry and geographic region, easily making them worth looking into. Also, if you consider yourself to be early adopters of tech, then the bottom half of the MQ should prove more interesting to you—with late adopters focusing on the upper half.

Peer Market Review & Market Guides

While the MQ model has seen much use across multiple sectors, it does have its own shortcomings. For example, if a market proves to be too immature (such as vendors offering low-Earth-orbit satellite services or quantum computing), the MQ model would have too little data with vendor dots rapidly shifting quadrants. Whereas with a mature market, vendor dots would rarely move, if at all, as Leaders would most likely have been in the upper right quadrant for quite some time. In addition, the MQ model takes 6 months to a year to compile and, in doing so, becomes a snapshot of the market from a year ago.

Gartner started publishing annual Market Guides with a list of “sample vendors” as opposed to the MQ’s method of categorizing vendors. (Forrester has also followed suit with their annual Tech Tide reports.) Gartner Market Guides give insight into current market trends and future projections while keeping them in the context of the buyer’s size, with small, medium, and large enterprises.

In 2015, Gartner launched Peer Insights, allowing buyers and leadership to (hopefully) cut through marketing wordsmithing or potential analyst bias and hear directly from fellow buyers who have used the tools and services of a vendor.

MITRE Engenuity ATT&CK Evaluation results

In 2018, the MITRE Corporation launched the MITRE ATT&CK Evaluations, where MITRE evaluates the efficacy of cybersecurity products using an open methodology based on their own publicly available ATT&CK (Adversarial Tactics, Techniques & Common Knowledge) Framework — a living, the growing framework of common tactics, techniques, and procedures (TTP) used by advanced persistent threats (APTs) and other cybercriminals. Everything a hacker can do on a victim’s system can be uniquely represented in the ATT&CK Framework.

The ATT&CK Evaluations are extremely useful to end-users of cybersecurity solutions as it provides transparency and publicly available data to the true efficacy of some of the leading cybersecurity products in the world.

ATT&CK Evaluation results also provide screenshots of cybersecurity solutions at work, granularly detail what is happening in each screenshot, and provide insight into a cybersecurity solution’s approach to security.

Each year (or “round”) of the ATT&CK Evaluations has cybersecurity vendors pitting their solutions against MITRE team-created emulations of known APTs (whose names somehow get progressively cooler with each round).

Round 1 (2018) Emulation — APT3
Round 2 (2019) Emulation — APT29
Round 3 (2020) Emulation — FIN7 & Carbanak
Round 4 (2021) Emulation — Wizard Spider & Sandworm

HOW DO I INTERPRET THE RESULTS FROM MITRE ENGENUITY’S ATT&CK EVALUATIONS?

As mentioned earlier, each round MITRE Engenuity emulates a real-world threat. Additionally, the last two rounds and the upcoming 4th round each focused on a differently motivated threat group. Depending on which active and emerging threats you’d like to focus on, you may have more interest in researching one round over another.

Each evaluation round has had different metrics gauging the efficacy of each vendor’s product, so a cursory understanding of each round’s adversary would be necessary.

In general, pay close attention to vendors that achieved more General, Tactic, and Technique detections out of the box with zero configuration changes as this shows what you can actually expect from a vendor as opposed to those that rely substantially on telemetry detections or configuration changes, as this is not what you will experience using their services and tools—especially if your focus is detecting and preventing intrusions, as opposed to just being really good at post-intrusion incident response.

If your MDR vendor can’t detect something in an actionable way, then no one can respond to it, leaving you vulnerable, despite the vendor’s claims otherwise.

“Small and mid-sized organizations often do not have the resources to fully address the complexity, variability, speed, and sophistication of modern cyber threats. These organizations often face the same cyber threats that large organizations do, putting them at a distinct disadvantage. To address those challenges, small and mid-sized organizations are increasingly adopting cost-effective MDR.”

Whitepaper: Securing Small, Medium, and Large Orgs for the 2020s,

22 December 2020

In terms of cybersecurity and this particular context, the size of your enterprise isn’t as critical as your level of security maturity. For example, in the last decade, we saw plenty of examples of large enterprises suffer the aftermath of a breach due to immature security standards and practices.

Small-Sized Enterprises / Security Maturity is Not High

These SOCs/IT teams are familiar with cybersecurity industry terms and typically rely on MSSP and/or MDR due to their limited team of analysts.

Use the Technology Comparison tool on attack sub-steps. Try following the line of thought below to help narrow down vendor solutions that best suit your environment.

1. Vendors with Technique or Tactic Detections
2. General Detections indicate the detection of suspicious behavior
3. High Telemetry Detections help with post-intrusion incident response. See if they offer data retention services to store telemetry and for how long as APT-level attacks have been known to take months to detect. The less time telemetry data is retained, the less useful the telemetry data becomes.
4. If a vendor does not have any of the above, strongly consider excluding them from your list of potential candidates.

Medium-Sized Enterprises / Security Maturity is Above Average

These SOCs have full-time analysts. The more detailed, contextual, and actionable information given to them, the better! They typically have sufficient technical knowledge. Some of the SOC analysts on their team may want super-detailed information, and some may consider that installing free tools like Microsoft’s Sysmon are efficient enough and can save money.

Try following the line of thought below to help narrow down vendor solutions that best suit your environment.

1. Decide which substeps are a priority for you.
2. Overlay vendor detection coverage over your prioritized substeps. What vendors provide coverage over your prioritized substeps?
3. Use the Technology Comparison tool to focus on substeps your team has prioritized. What vendors provide useful detection data?
4. Look into each vendor’s individual results and find a product interface your team likes and can understand.

The third point is particularly important. If you only see the commands (process cmd-line, PowerShell cmd-line, etc.) but do not see the system-level behavior (API Call, file event, etc.), it is relatively meaningless for data collection.

Large-Sized Enterprises / SecOps with High Maturity

Your full-time team of information security analysts operates in an ocean of alerts where drowning is not an option. Analyzing raw data 24 hours a day isn’t ideal either—at least, not for humans; however, automated alert triage and automated alert validation are only part of the answer.

Large-sized enterprises could have hundreds of thousands of endpoints. The ability to detect and verify malicious activity on anyone endpoint is great, but being able to correlate malicious activity across the entire network is what you really need. This allows your team to gain the full context of the attack from initial access to attack operation objectives.

Try following the line of thought below to help narrow down vendor solutions that best suit your environment.

1. Identify vendors that can provide MSSP / MDR as their solutions may be better at triage.
2. Identify vendors that can generate more Technique and Tactic detections as these detections can greatly reduce MTTD and MTTR.
3. Use the Technology Comparison tool to focus on substeps your team has prioritized. What vendors provide automated General, Technical, and Tactic detections?
4. Identify a product interface that your team likes and can understand.
5. Does the solution correlate all malicious activity across all endpoints into one attack storyline stemming from the root cause of the attack to potential attack operation objectives? Does the solution provide end users with a straightforward, easily digestible visual representation of the attack, or is it a wall of text?

The MDR market can be tricky to navigate. It’s easy to get distracted by all the smoke and mirrors produced from clever marketing buzzwords, biased reports, fear-focused advertising, and hype. Hopefully, this article provided you a decent layout of the MDR cybersecurity landscape and an actionable roadmap to success. If you should ever find yourself lost out there, contact us and let us know. We’re here to help. Happy hunting.