Analysis of Competing Hypotheses
Analysis of Competing Hypotheses (ACH) is a structured analytic technique that systematically considers the different hypotheses regarding an intelligence question. Its creator, Richard Heuer, aimed to help analysts deal with psychological biases and ambiguous or incomplete data/information that often permeate and interfere with intelligence analysis. Initially, ACH was intended for physical and political intelligence. However, its usefulness makes it an excellent resource for the CTI setting.
At the ACH’s core lies a valuable scientific concept, namely, the idea of disproving a hypothesis. While counterintuitive, it tests the explanatory power of a proposed hypothesis, whether it’s the analyst’s favored one or not. Every hypothesis receives a “fair shake,” with the analyst verifying evidence alignment to each one. So, ACH acknowledges the presence of biases (in this case, a form of selection bias, i.e., highlighting only the evidence that proves the chosen hypothesis) and, through rigorous methodology, seeks to circumvent them.
ACH adopts other aspects of scientific methodology. Much like a scientific article, it demands registering the rationale process adopted by the CTI team by consolidating and cross-examining hypotheses and evidence through some recording procedure, like a spreadsheet. Therefore, ACH provides credibility through an audit trail of the CTI team’s due diligence in the intelligence analysis process. Such technique (1) gives the client a perspective on the team’s logic, even if he disagrees with the conclusions, and (2) provides a reference for analysis replication attempts and reconsidering previous conclusions in light of new evidence.
Another benefit is the freedom that ACH gives analysts to be wrong. It’s a natural part of the analytical effort to propose hypotheses that evidence will eventually invalidate. ACH assumes this will be so, which is helpful in a setting where CTI teams may be or feel pressured never to commit analytical errors.
The CTI analyst follows seven stages when executing ACH: Hypotheses, Evidence, Diagnostics, Refinement, Inconsistencies, Sensitivity, and Conclusions and Evaluation.
Hypotheses
A hypothesis is a proposed explanation of the causes of an event. The CTI team must generate hypotheses through formal (e.g., Zwicky Box) or informal (e.g., brainstorming) procedures. No hypothesis is out of consideration, which shows that ACH gives space for free thinking and mistakes, both necessary to reduce bias and for a thorough analysis. This stage may involve different analysts with divergent perspectives, optimizing hypothesis generation.
Evidence
When compared to forensics, ACH has a looser definition of what constitutes evidence. It includes technical data and reports but encompasses less strict elements, like logical arguments and professional assessments.
Diagnostics
As mentioned, ACH aims to examine the validity of different hypotheses through their alignment with evidence. This procedure reaches a non-intuitive finding: evidence that validates the analyst’s favorite hypothesis may also do so for one or more of the alternative hypotheses. Such a situation happens due to different levels of evidence diagnosticity.
Heuer defines diagnosticity as how much a piece of evidence helps validate a specific hypothesis while disproving the rest. For example, the technical evidence that a spear phishing attack targeted some of the company’s employees can point to a cybercriminal group as the culprit but, alternatively, may suggest a nation-state, considering that social engineering is such a prevalent attack method. On the other hand, messages on an Internet Relay Chat (IRC) channel of a specific cybercriminal group discussing the previous attack are less ambiguous, advocating for one hypothesis while rejecting alternates. Therefore, the former evidence has less diagnosticity than the latter. This doesn’t mean less diagnostic evidence isn’t valid, but the analyst must consider this throughout the evaluation process. Heuer’s central argument is that analysts must prioritize evidence diagnosticity rather than its availability or volume.
At this stage, the CTI team draws a matrix with hypotheses at the top row and evidence at the farthest left column. Then, both sets are associated, indicating whether the evidence validates a hypothesis. The team uses symbols to demonstrate this (e.g., using C for consistent, I for inconsistent, and N for neutral, i.e., when evidence doesn’t validate or invalidate a hypothesis). Consider, for example, a simple case where the CTI team aims to determine the probable threat actor responsible for an attack against an oil company:
Refinement
In this phase, the CTI team can add new hypotheses that it didn’t consider initially and gather new evidence. Additionally, low diagnostic evidence may be discarded or replaced by others. Diagnosticity levels are essential for such assessment, but data validity measures (e.g., Police 5x5x5 and NATO Admiralty Code) also play a fundamental role. The analyst must discard evidence with low diagnosticity and credibility, which may hinder reasoning.
Inconsistencies
The CTI team critically evaluates the evidence and “draws tentative conclusions about the relative likelihood of each hypothesis.” In other words, the analyst seeks to eliminate the least likely hypotheses. It’s important to note that this stage demands a more cognitive or rational approach beyond gathering evidence. It involves a specific rationale, such as assessing if clustering some of the hypotheses together is possible.
Sensitivity
After establishing the least disproved hypotheses, the CTI team reflects on some critical questions about evidence. What will be the impact on the analysis outcomes if the evidence used proves to be false? What steps can the team take to ensure the evidence’s credibility? What’s the weight of primary/technical evidence to invalidate various hypotheses simultaneously? Ranking evidence according to their sensitivity levels (i.e., how much each piece of evidence contributes to the conclusion) is a good visualization and logical procedure. Naturally, basic source analysis provides key references for critically evaluating evidence sensitivity.
Conclusions and Evaluation
The CTI team finishes the analysis by drawing conclusions based on the least disproved and critically assessed hypotheses. Special attention must be paid to the logic and terminology used in the communication. Although rigorous and scientifically sound, “least disproved hypotheses” is a concept that most intelligence consumers won’t understand. Alternatively, the analyst can adopt a narrative model with an exposition, progressively building tension, a climax, and a resolution. The story places each piece of evidence into an explanatory framework/logic, painting a cohesive scenario that, in the end, considers and discusses the likelihood associated with each different hypothesis.
Bonus: Milestones and Courses of Action
Although not often present in the CTI context, it’s valuable to set up indicators that may signal a change in the scenario drawn by the analyst and to propose specific conducts (i.e., courses of action) that the company may follow to adapt to the changing circumstances.
