Motivation

Here I would like to explain one of my projects about a comparative usability study of FDIO2 passwordless authentication. So, what is FIDO2? FIDO2 is a new authentication standard that was jointly developed by the FIDO Alliance, an open industry association that includes Google, Facebook, Microsoft, Amazon, VISA, plus more than 200 other organizations, and the World Wide Web Consortium (W3C), the main international standards organization for the web.

In a nutshell, FIDO2 consists of two protocols, WebAuthn and CTAP2, that together allow a relying party¹, like a web service, to make use of hardware authenticators, like security keys, for user authentication. FIDO2 supports two-factor and multi-factor authentication, but also single factor authentication in which the hardware authenticator is the only factor. This single factor, passwordless authentication has recently received a lot of attention. For instance, the media has declared FIDO2 passwordless authentication as the killer of password and the future king of user authentication on the web. So, we were wondering if FIDO2 is the kingslayer of web authentication?

More concretely, we wanted to find answers to the following research questions:

1. Do users accept FIDO2 passwordless authentication?

2. How do users perceive FIDO2 passwordless authentication in terms of usability?

3. What thoughts and concerns arise in the users’ minds when using FIDO2 1FA?

To answer these questions, we conducted the first large-scale comparative user study of FIDO2 passwordless authentication. We recruited 94 participants, which we randomly distributed between two groups: a password control group (Group$_{\textrm{Pass}}$ with 48 participants) and passwordless authentication group (Group$_{\textrm{1FA}}$ with 46 participants). While the control group (Group$_{\textrm{Pass}}$) used regular text-based passwords for web authentication, the passwordless authentication group (Group$_{\textrm{1FA}}$) used a Yubico Security Key as the single factor for authentication.

Methodology

The exact methodology of our lab study was as follows, and we explain the individual steps for each group below.

Welcome and Topic Introduction

For both groups, after we welcomed our participant and gave study instructions, the participant watched a short introduction video about password-based authentication in which we recapped well known benefits and drawbacks of text-based passwords.

Afterwards, each group went through a dedicated steps.

Group$_{\textrm{Pass}}$

After the introduction video, our control group (Group$_{\textrm{Pass}}$) proceeded directly to a hands-on task where they had to register accounts on mock websites and login to those accounts by using regular text-based password for web authentication. Our mock websites Fakebook and Schmoogle were based on the Facebook and Google Mail sites to provide our participants with a familiar site and reduce the impact of confounding factors when learning to deal with a new website.

Afterwards, we asked participants to reflect on this experience in a survey. The usability and the acceptance of the authentication mechanisms as well as user-specific factors that may affect these variables were measured using standardized methods like SUS score from Brooke or Affinity for Technology Interaction scale from Franke et al. In order to get a more complete picture of user perception, we also used free text questions to capture the ideas, benefits, drawbacks, and concerns that our participants see.

Group$_{\textrm{1FA}}$

Our passwordless authentication group (Group$_{\textrm{1FA}}$) proceeded differently after the topic introduction. Prior studies have shown that a lack of clarity about functionality and security benefits of authentication methods leads to a lower security rating, lower acceptance, and reluctance to switch to a new authentication method. We decided for our study design to address this lack through an introduction of passwordless authentication and showing how to use the security key before letting our participants try out passwordless authentication themselves. For the hands-on task we used again our mock fakebook and schmoogle websites, but this time supporting only paswordless authentication with FIDO2. The reason we decided to use mock websites is simply that at the time in 2018 when we conducted this study no real service offered passwordless authentication with FIDO2. Let me give you some details about the videos for our passwordless authentication group.

Introduction Video of 1FA

For introducing passwordless authentication, we provide publicly available information from vendor sites like Yubico or from technology-related blogs and news about FIDO2. Such as:

• No more passwords are needed and a security key can be reused instead for different accounts.
• The key costs 40 Euros, which corresponds to around 40 Dollars, the median price between low- and high-end security keys on the market.
• No secret data, like a password, needs to be stored on the web service.
• Compromising the client does not leak any secrets either.

In the video below, the first 2:39 min provide the general introduction of text-based passwords, followed by the introduction of Passwordless authentication method (1FA).

Setup Video of 1FA

Afterwards, the participants in Group1FA were provided with a setup video that explained the setup process for FIDO2 with a Yubico Security Key as passwordless authentication method. The content of the video was a step by step guide through the registration and authentication process using the Yubico Security Key on two demo sites that supports FIDO2. Here is the video:

Hands-on Task for Group$_{\textrm{1FA}}$

Like our control group Group$_{\textrm{Pass}}$, Group$_{\textrm{1FA}}$ had to register a new account on our mock websites and login to those accounts. In contrast to the control group, they used a Yubico security key as 1FA instead of regular text-based password.

Survey

Lastly, like our control group, our passwordless authentication group also answered the survey in which we asked them to reflect on their experience with passwordless authentication and tell us about the ideas, benefits or drawbacks they see, or concerns they have. As a result, our collected data allowed us to evaluate the usability and acceptance of FIDO2 passwordless authentication and to gather user concerns and feedback about the paradigm shift to passwordless authentication.

Results

Now, let me summarize the results of our study.

Demographics Results

First, the demographics of our study met the expectations for study conducted in a university setting:

56.4% identified as female

Most with university degree

On average 25 years old

We did not find any differences in the demographic composition between our control group and our study group (using two sample t-test and Fisher’s exact test depending on the variable, i.e., frequencies or the scale mean):

No relevant differences between groups

Quantitative Results

We quantitively measured the usability of the authentication methods with the SUS score and acceptance of the authentication methods with the scale from van der Laan et al. Our results show that:

Passwordless authentication is perceived as more usable and was more accepted than traditional password-based authentication

Qualitative Results

Let me go through the main points we identified in our qualitative data.

First, the vast majority, 79% of our participants, mentioned the effort associated with the usage of the authentication method that they used.

Participants in the control group Group$_{\textrm{Pass}}$ found the creation of secure and unique passwords but also the memorization of an ever-increasing number of passwords a difficult and demanding task.

For our participants in the passwordless authentication group Group$_{\textrm{1FA}}$ cognitive effort was not an issue. In fact, the reduction of cognitive effort compared to password-based authentication was a great if not the greatest advantage of passwordless technology.

“No recalling of the password. For [a] new account, one need not [to] worry to come up with a password and remember it for later use.” (P92, Group$_{\textrm{1FA}}$)

However, participants in this group criticized that passwordless authentication requires carrying a device to be able to authenticate. It was seen as problematic and annoying that it is not possible to use web services if the security key is not present, which restricts spontaneous and ad-hoc use. Comparing both authentication methods, the switch from password-based to passwordless authentication was associated with a clear shift in the participants’ perception from cognitive to physical effort, which reflects the paradigm shift underlying the switch to FIDO2 single factor authentication—away from ‘something I know’, over to ‘something I have.’

“I think the only problem with this kind of authentication system is that the user[s] have to carry their Yubikey [Yubico Security Key] everywhere with them […].” (P62, Group$_{\textrm{1FA}}$)

Participants from both groups thought about factors and problems that could affect the security of their accounts, but the prevailing threat models differed greatly.

Participants in the control group Group$_{\textrm{Pass}}$ were primarily worried that weak passwords, password reuse, or phishing attacks could lead to an attacker gaining access to their accounts.

Participants of our study group Group$_{\textrm{1FA}}$ were mainly afraid that someone else could gain access to their accounts with a lost or stolen security key. Several participants raised the question how to “revoke” and “recover” account access in such a case and they wanted an additional layer of protection to protect the security key against unauthorized use, such as biometrics. The concerns about losing access to the accounts went so far that participants expressed a desire for a backup authentication method.

“I just have one concern: What if someone steal[s] my Yubikey [Yubico Security Key]? Does that mean he can access all my accounts just inserting it [to] his computer?” (P66, Group$_{\textrm{1FA}}$)

“There should be a way to use your accounts without the yubikey [Yubico Security Key]. Otherwise you would be very dependent on it.” (P50, Group$_{\textrm{1FA}}$)

One very interesting aspect, however, was pointed out by one of our participants: the biggest advantage of passwordless authentication is the implicit guarantee that no one else can access the account as long as one is in possession of the security key. In this way, the disappearance of the security key from one’s own possession immediately warns of a potential unauthorized access to an account— something that passwords simply cannot offer.

Another major problem that has arisen in relation to passwordless authentication are situational barriers.

Participants in our study group Group$_{\textrm{1FA}}$ complained about technical incompatibilities of the USB-based security key with mobile devices, like smartphones or tablets, that do not provide fitting USB interfaces.

“Nowadays an USB dongle seem to be a bit old, new computer doesn’t have this port, also probably most of the authentication on these days are done in mobile devices…” (P70, Group$_{\textrm{1FA}}$)

Participants from the control group Group$_{\textrm{Pass}}$ came up with cases of authentication in which passwords seem to be superior because of their flexibility. They mentioned the ability to spontaneously delegate accounts via telephone or the usage of specially protected computers that do not provide physical access to standard interfaces, such as a public computer in a library.

"…If necessary, you can also help relatives via telephone or Internet by changing something in their account or doing something for them if they are prevented from doing so." (P84, Group$_{\textrm{Pass}}$)

Lack of access to the necessary interfaces also prevents the usage of an authenticator device like a security key.

Many statements of the participants also described aspects connected to the mental migration process from passwords to passwordless authentication. This shift means a break with the well-established habits and traditions of users.

Over the course of our study, it became very clear to us that our participants have a clear mental model of password-based authentication. They know the pros and cons and have a certain understanding of the factors responsible for the security of a passwords.

For passwordless authentication, on the other hand, such mental models must first be established in the users’ minds. Although our videos seem to be a helpful introduction to this new technology from the participants’ point of view, obvious misconceptions in the free-text responses show that their mental models are only rudimentary, which lead to a lack of trust.

“Is it possible to track my exact location once I insert the Yubikey [Yubico Security Key]?” (P52, Group$_{\textrm{1FA}}$)

On the other hand, the majority of participants in our study group Group$_{\textrm{1FA}}$ described the authentication with the security key as a fun, pleasant, and exciting new user experience, in contrast to traditional passwords that were described as monotonous, boring, and annoying.

“It was overall very nice and pleasant. I found it very intuitive to use.” (P52, Group$_{\textrm{1FA}}$)

Lastly, and specific to our study group Group$_{\textrm{1FA}}$, there were doubts about the robustness and maturity of the authenticator device as well as complaints about the price of the device.

“Once the Yubikey [Yubico Security Key] didn’t react and I didn’t know if I had to press it or it’s enough to just hold my finger on it.” (P52, Group$_{\textrm{1FA}}$)

". . . I don’t want to spend money on the key [Yubico Security Key]…" (P52, Group$_{\textrm{1FA}}$)

Willingness to (not) use passwordless authentication in Group$_{\textrm{1FA}}$

Willingness		Most Frequent Arguments
Yes	(16 participants)	- Easy/Secure/Memorywise-effortless
Yes, but	(13 participants)	- Fear of losing access to own account - Fear of account access by others
Rather not	(11 participants)	- Fear of losing access to own account - Mistrust
No	(6 participants)	- Mistrust - Annoying to carry extra devices

We also asked our participants in our study group Group$_{\textrm{1FA}}$ if they would be willing to use passwordless authentication in their private lives. 16 participants were unconditionally willing to use it and explicitly highlighted the ease and convenience of passwordless authentication over password. The remaining 30 participants had different kinds of concerns. Among those concerns the fear of losing access to the own account, the fear of illegal access by someone else, and mistrust were mentioned most frequently.

Influence of introduction video

We were wondering if our introduction video influenced our participants. To evaluate the stability of our findings, we repeated our study with a third group of participants Group$_{\textrm{1FAControl}}$ (47 participants), which forms our control group for single factor authentication. In regards to the demographics, this group did not substantially differ from the other two groups.

Changes to methodology for Group$_{\textrm{1FAControl}}$: Participants in our new, third group still performed the hands-on tasks to register and login to our mock websites with a security key as the single authentication factor. However, in contrast to the original passwordless authentication group, we did not show them any of the introduction or explainer videos. But, since users are very likely not familiar with using a security key, we added technical guidance to the mock websites, which mimics the guidance of the original facebook and google sites for setting up two-factor authentication with a security key.

Here is an example for the information we provided on Schmoogle.

There is merely a hint about the absence of the password field and an option to get more information. In case a participant clicked for more information, a modal dialog designed after the original Google instructions gave a three step guidance about how to handle the security key.

Quantitative results

Even without a detailed introduction, passwordless authentication was perceived as more usable and was more accepted than password-based authentication.

Qualitative results

In terms of qualitative data, our two single factor authentication groups were pretty much the same.

$$ \textrm{Group}_{\textrm{1FA}} \approx \textrm{Group}_{\textrm{1FAControl}} $$

There were only few differences:

• Control group Group$_{\textrm{1FAControl}}$ was more worried about account access by others in case the authenticator is stolen or lost and mistrusted the security key more, stating more often that they needed more information.
• Differences in the willingness to use passwordless authentication, where the participants in the control group Group$_{\textrm{1FAControl}}$ were mostly only under certain conditions willing to switch to passwordless authentication. Mistrust was mentioned more frequently as a reason to abstain.

Takeaways

While FIDO2 passwordless authentication has great potential to replace text-based passwords, we identified some obstacles on the road to adoption, for which we also tried to point out potential future work or recommendations.

User support and guidance for scalable recovery is needed

A predominant concern among the participants in our passwordless authentication group was the loss of the security key and hence access to their accounts. What is particular to the setting is that the reuse of a single authenticator across multiple websites, which is considered a strong point of FIDO2 authentication, amplifies the recovery problem. In case of device loss, all affected accounts must be recovered, and currently there is no proper support or guidance for users for a scalable recovery.

Support for authenticator revocation without prior recovery is needed

A few participants raised concerns about device theft and account access by the thief. In comparison to large-scaling problems like phishing campaigns or server breaches, those concerns are small-scaling and targeted with need for physical access to the victim. However, at the end of the day, the subjective views of users will determine the adoption of passwordless authentication. Thus, we think it is worthwhile to investigate solutions how users can securely revoke access to their account without the need to first recover access.

Users need to be made aware of (individual) corner cases to adapt their authentication strategies

In contrast to passwords, which can be entered anywhere, token-based authentication will currently always have corner cases in which it is not applicable. Users should be informed about corner cases in which they cannot make use of token-based authentication and allow them to adapt their authentication strategies. For instance, depending on the devices on which an account must be accessed, users choose a suitable authenticator device.

Great opportunity to tailor authenticator form and features for a personalised authentication

A few participants pointed out problems with the authenticator we used in our study, a Yubico Security Key. Most of those concerns were about the limited connectivity and hence lack of support for other client devices, like mobile phones with NFC or Bluetooth. Other concerns were about the price of the device, its robustness and usability, or more generally about the need to carry an extra device. Since FIDO2 does not define the form of the authenticator, just its capabilities and protocols, we think this is a great opportunity to tailor authenticator form and features to user demands, maybe avoiding the need to buy and carry dedicated devices and to offer a form of personalized authentication.

Establish mental models by drawing from existing models about physical keys

Finally, during our study, we noticed that our participants identify “authentication” automatically with “passwords” and they naturally did not have a mental model of how passwordless authentication with a security key works. Some participants expressed mistrust into the hardware token, mostly due to a lack of transparency.

Thus, the transition to FIDO2 passwordless authentication requires establishing mental models that see authentication more systematically. One path forward in this direction would be to draw from existing models about physical keys. For instance, that possession of the key means no one else can access the account, that spare keys can & should be used, or to associate every account with the right physical key.

---

Scientific publication