In March of 2021, the marketing agency ICF Next ran a “live phishing assessment” on over 400 of its workers to test their vulnerability to phishing attacks, and ostensibly, to teach them a lesson about being careful with suspicious messages. It’s a common technique used by many IT Security teams, but this time, the organization may have learned a bigger lesson.

The problem with the ICF Next assessment was that they decided to use an offer of free COVID vaccinations as a lure to entice people to click on the link in the test email. Well, it worked, and 30% of the employees who received the message clicked on the link to sign up for the vaccine. But when they found out that it was just a test, there was a lot of negative sentiment being displayed, both internally and externally.

 What is the basic value proposition of “live phishing assessments”?

The increasingly common use of “live phishing exercises” as a phishing assessment methodology has a number of challenges, which are starting to become evident. These are also called “mock phishing assessments”, “phishing simulations”, “phishing tests” or “phishing campaigns”. Some people feel it is fair to test people this way, since that’s what attackers will do. They don’t play fair. But is this approach really giving the best value in terms of outcomes when the whole thing creates such a revolt?

The main advantages of using “live phishing assessments” is their potential for providing a measurement of how vulnerable a group of employees is based on what percentage of them click on the link in a simulated phishing message (called a “click-through rate”), and the immediate “teachable moment” that presents itself when employees do click.

There are a lot of assumptions around the use of live phishing assessments for vulnerability metrics. Among them is the fact that only those who click provide any data. Those who do not click are assumed to be “smart enough” not to fall for a phishing email. However, there are many reasons why employees may not click on link in a particular test. So, because there are so many variables, the value of this method for measuring vulnerability needs to be put in context with the variables that apply.

There is also a problem with the assumption that people will actually learn how to avoid clicking on suspicious links in future, based on the idea that they will “learn from their mistakes”. The effectiveness of the teachable moments when employees do click on a link in a phishing assessment also needs to be considered. People are often surprised and maybe embarrassed about being caught. So, their emotional state is not predictable, nor is their ability to learn and retain knowledge at that exact moment.

All of the good things that are expected to come from doing a live phishing assessment can be negated when the test is remembered more for its infamy than for its actual educational value.

This is not the first incident of employee backlash

In 2020, both the Chicago Tribune and Godaddy were publicly outed by their employees who felt that the topics of their live phishing assessments were unfair and insensitive, especially during the pandemic. The Tribune used a “pay bonus” theme, and Godaddy used a “holiday bonus” theme. So, the ICF Next example is just another illustration of how hard it can be to do phishing assessments in a way that provides a return on the investment (or how easy it can be to cause a PR disaster).

I’m sure it won’t be the last incident of this type, as organizations will continue to push the limits of this method to more closely simulate the deceptive tactics of the attackers. The intent is good, but the method is becoming quite problematic.

The natural progression of live phishing tests has limited potential value

As a security consultant several years ago, I had been conducting live phishing assessments for an organization of 10,000 employees. My methodology was to manage the design of campaigns for my clients to make sure that the results were meaningful and not counter-productive.

On more than one occasion, despite trying to limit the variables in each test, my client requested that I run more difficult tests that would cause a higher click-through rate, and catch more people. So, ultimately, it became more about having a story that illustrated the risks than about having valid vulnerability data or effective training outcomes.

The limit was reached when I proposed a simple topic around a payroll system change, which happened to invoke the name of the system that everyone knew had problems. It was a virtual certainty that not only would a significant portion of the workforce click on the link, but many would actually call their management to complain, before they would figure out that it was not a legitimate message.

So, even though it was very plausible that an attacker would likely use that same pretext, I was not allowed to simulate it. However, the management didn’t want the story that badly, apparently.

If live simulation of phishing emails has limited value, how are we supposed to effectively assess employee vulnerability?

The use of live phishing assessments grew primarily because the method of delivering the assessment (i.e. email) happened to match the method that attackers used (i.e. phishing), and it was very scalable. But now the limits of that method are becoming clearer, and the operational risks are starting to outweigh the information security benefits.

However, that doesn’t mean we can’t simulate phishing attacks. We can achieve similar goals within a scalable, gamified environment, where employees know they are being assessed. Although it doesn’t have the operational realism, it doesn’t really need to.  This method can be as effective as live phishing assessments because employees can still experience what the real threats will look like while they are in a normal state of mind, without causing any operational risks. The assessments can also be designed to be robust and consistent across all employees in a group, with few uncontrolled variables. So the data is much more reliable, while the cultural impact is more positive and learning outcomes are more clearly defined and measurable.

Live phishing assessments can still be used in an “audit” capacity, with moderately challenging pretexts that won’t cause major backlash. We can use them to check if people still know how to analyze a message in an operational environment. So it doesn’t even have to be done with live email messages every month.

So, rather than having limited scope and negative impacts on culture, we are able to use constructive methods that reinforce learning on an ongoing basis. There is no need to use increasingly unfair scenarios to test employees on their cyber security skills. Unfortunately, we are likely to see more Tribunes, Godaddys and ICF Nexts before employers start learning these lessons themselves.

Photo by CDC on Unsplash