The key variables to consider in live phishing tests
What does “counting clicks” in live phishing tests tell you? There are really a lot more questions than answers.
Here are some of the variables that are rarely considered when reporting on live phishing tests:
1. Did the employee guess suspect a test based the subject line?
2. Did security software block messages from reaching users?
3. Were some employees away from work during the test period?
4. Did some recognize and click on a test message “just to see it”?
5. What was the message content’s difficulty compared last time?
6. Was the subject line more (or less) compelling than last time?
7. Did somebody tell others about a suspected a phishing test?
8. What could have impacted the number of “detected opens”?
9. What can you infer about people who reported AND clicked?
Photo by Ryan Putra on Unsplash
For just $325 USD, you can run a 6 week, automated program for gamified phishing awareness training and challenges. (Limited time offer. Normally valued at $999 USD)
Use Promo Code: 6WEEKS
On top of these variables, you should also consider how many data points you actually collected regarding “clicks” and “reports”. Phishing test scores usually focus on “how many clicked”, as a percentage of all messages sent. This is usually below 25%.
Ideally, you also get a count of “how many reported” a message. There is often less focus on this number, and it is often lower than those who clicked. But this doesn’t tell us anything about “how well they understood” how to analyze a phishing message.
Live tests can provide some useful information, but they have so many variables, the data is often very unreliable. This is all aside from the other ethical, cultural and educational issues that are also associated with live phishing tests.
Phishing education and remediation needs to be more supported by a more consistent, robust and inclusive methodology to provide the kinds of results executives need.
Scott Wright is CEO of Click Armor, the gamified simulation platform that helps businesses avoid breaches by engaging employees to improve their proficiency in making decisions for cyber security risk and corporate compliance. He has over 20 years of cyber security coaching experience and was creator of the Honey Stick Project for Smartphones as a demonstration in measuring human vulnerabilities.