Hello everyone. My name is Aikaterini Kanta and I’m a PhD student with University College, Dublin and the European Commission’s Joint Research Centre, and I’m very glad to be here today at DFRWA USA 2021 to present to you our paper: ‘How Viable is Password Cracking in Digital Forensic Investigation? Analyzing the Guessability of Over 3.9 Billion Real World Accounts.’
The contribution of this work is three fold. First we have conducted what we believe is the largest and most comprehensive analysis of real-world passwords to date. This analysis of the passwords’ pattern of construction is done after we split the passwords into meaningful component fragments. And then we look at the most common semantic classes in those fragments. Then we perform an analysis on the strength and the crackability of those passwords.
But first, a little bit about authentication. The average number of passwords users need to remember was found to be 27 in one study, to over 191 in another one. And the reason for this is because we use passwords for everything. We use them when we log into our social media, we use them all when we go to the bank, we use them when we log into our computer. They can be of different types. They can be patterns, they can be numerical or alphanumerical passwords.
And lately we have seen that for mostly for critical websites, multi-factor authentications being used more and more. And that is two out of something that you know, for example, a password or a PIN number, something that you have, like a bank card or a security token, something you are, like a fingerprint, as your voice, or your face or something, or somewhere that you are, like a GPS localisation or an IP address. This type of authentication is used mostly for critical sites, like a bank. But still the most common type of authentication remains the single password.
The single password is still also sometimes the weakest point of entry to a system. And this is something that password cracking techniques have tried to leverage for many years. Traditionally the most common password cracking technique is a brute force attack, which is an exhaustive search that is guaranteed to work if you have infinite amount of time and resources.
But for passwords that are complex or long, it’s a technique that is not useful. In practice, if I consider an NVIDIA 3800 GPU card that can guess 54 times 109 passwords per second, it would take about two days to explore a set space of eight character passwords. And this is why the US National Institute of Standards and Technology recommends that at least eight characters are used in passwords and especially recommends LUDS-8, which means lowercase, uppercase, digit and special characters to be used together in your password.
Another traditional technique for password cracking is the dictionary attack, which uses word lists in conjunction with mangling rules. Mangling rules are rules that are replacements that try to mimic user tendencies, for example, a user might replace I with one or might replace S with the dollar sign.
And then most recently we have seen machine learning and AI techniques emerge, and this something happened due to the larger amount of data breaches in the last few years, which means that these techniques can leverage this huge amount of leaked passwords that are available.
So given that as a knowledge, this password on the screen, that used to be a pretty good password. It’s a password that is too long and complex to be able to be cracked by brute force and the dictionary attack would not work very well against it either because it’s not a dictionary word. And indeed the time to crack this password is a lot longer than the age of the universe. So it does seem like a pretty good choice, but if that password ended up being part of a data breach and it has been reused to more than one services then it’s not secure anymore. So data breach with the phenomenon of password reuse are a serious threat.
So if you wanted to know whether or not your passwords or your credentials have been compromised, you could do that by going to haveibeenpwned.com. This website was created by web security expert Troy Hunt, because he wanted to highlight the seriousness of data breaches. He wanted to serve as a black list of passwords, where companies would not allow users to use passwords that have already been part of data breaches and as well as help people know that their accounts have been compromised.
Another reason to use this website would be for academic purposes, which is what we have done. Indeed, we have used the Have I Been Pwned list Version 5, which contains 3.9 billion real-world accounts corresponding to about half a billion unique passwords. Because the passwords in Have I Been Pwned are stored in SHA-1 format, we have gained the plain texts of the passwords from the online community hash.org, which is a collective of people dedicated to password cracking.
And then with this list, what we have done is taken some statistics on the lengths, the makeup and the strength of the passwords, and also what we’re calling our advanced analysis which is splitting the passwords into their constituent fragments with the help of Óđinn and classifying them using WordNet, which is a lexical database of English that contains contextual synonyms that are called synsets.
So the analysis of Have I Been Pwned contains the number of breached accounts listed in the last few years. And you can see that the number has skyrocketed in the last five years, between 2015 and 2019, and continues to grow. The top 25 passwords in Have I Been Pwned can also be seen on this slide, and they’re not surprising considering the fact that most of them already belong in most used or worst password lists. We see a lot of keyboard sequences, keyboard walks, a number of sequences, a number of repetitions and common words like password.
Then we wanted to look a little bit into the lengths and the strength of the passwords in the Have I Been Pwned and for that we see here the lengths of the passwords, most of them being eight digits long which is something that is not surprising given that most password policies recommend and don’t accept passwords that are below eight characters long.
For the strength analysis we have used a tool called zxcvbn, which was developed and used by Dropbox, that classifies passwords into five classes, according to their strengths, so with zero being the least secure and four being the most secure.
We can see here that about half of the passwords belong to class 2, which in first glance seems like a pretty good result, but as you can see that on the other figure, distribution of password length per class, you can see that class 2 passwords are mostly of 9 digits and below, and that means that they could be more easily recovered if you consider a fast hash function based on the analysis of the previous slide at the beginning.
Another example for you is looking a little bit more into class 4, which contains the most secure passwords. We can see here that most of the passwords in class 4, all of the passwords in class 4 are of 10 characters or longer, with most of them being 11 to 16 characters.
But if someone were to know, as we do in our case, a little bit more about the internal structure of the passwords, they could make some decisions that can help them crack them faster. For example we know that 42% of the passwords in class 4 of Have I Been Pwned contain only lowercase and number characters. This means that the set space can be a lot smaller with this information.
Another example that I have here is that if we consider only 15 digit passwords and by this, I mean, those passwords only contain numbers and those represent 11% of the total of passwords in class 4. It would take I would say NVIDIA 3080 GPU about 12 hours, if we consider a fast hash function like MD5 and with a slower hash function like BCRYPT, it would take 650 years. Of course, that number can be a lot smaller, if we have more than one GPU device.
Then we looked into the internal structure of the passwords. And as you can see on the pie chart, the most common category is loweralphanum, which means passwords containing lowercase, alpha characters and numbers. The other two popular categories are just lowercase characters and just numerical characters.
But then if we wanted to look a little bit more into the way that these are stacked inside the password, we’re looking at what we call in password cracking a mask and that is the character categories of string, which is comprised of lower, uppercase and mixed numeric and special. We can see that stringdigit is the most popular category, which means any string followed by a number, whereas the category digitstring, which means one or more numbers followed by a string is number four.
Then we move on to our advanced analysis, where we split the passwords into their constituent fragments. For example, the password manchester.2019 would be split into three fragments, manchester, which is a letter fragment, dot, which is a special fragment , and 2019, which is a number fragment. And then this with WordNet would be classified as a city, a special character and a year. We see here the number of fragments per category and the total number of fragments which is 1.5 billion.
And if we go back to the number of unique passwords in Have I Been Pwned, which is half a billion, we see that this number is three times as much, which means that the passwords in Have I Been Pwned are constructed of more than one type of fragment. We see here as well, the ten most popular segments per category.
In the letter category, we have expected candidates, for example articles, pronouns, common words. For numbers we have sequences of numbers as well as sole numbers. And for special characters, which are in order of magnitude smaller than the other two categories we see that we have only sole special characters in the top 10.
Then we’re looking at the most common fragment categories as were classified at WordNet. We can see here that the most common category is number with almost 81% of the total, followed by common number and year. So all top three of these categories contain number fragments.
The difference between common number and number is that a common number would be a number that is meaningful to humans. For example, 314 which is the number for pi, 22, or 1, 2, 3, 4, which is a sequence. And the other category that is just numbers contains all of numbers that have not been classified as a common number, or a year or something that’s meaningful.
Then we can see that some of the most popular categories contain names like feminine and masculine names. They contain city names, keyboard walks, they contain animals family, computers, food, emotions, sports which are categories that offer contextual information, hinting once again at the fact that people tend to use meaningful words when they create their passwords.
Then we look at the most frequent fragment combinations, where again combinations of common numbers are very common combinations of numbers. And then it’s very interesting that we have a variety of names followed by numbers, digits, or years which suggests that, as we saw before, that the most popular mask is stringdigit, a string followed digit.
We see here that it’s very popular to have a name followed by a number, or by one single digit, or by a year. Also x + year and x + common-number means a fragment that has not been classified as anything but [x followed by a year or a common number, which suggests that is quite common to use numbers at the end of a password. For example, common number, which would contain sequences like 1 to 3 would be a very common practice.
Finally, we have the comparison of the most frequent fragment categories between our passwords and then the passwords of class 4. You can see here that number is as before the most common category. And it’s a lot higher in class 4 passwords than it was in all passwords although because the percentage of common numbers is a lot smaller than compared to all passwords, that could be a case of word classification by WordNet.
We see again that years and names are very popular. And what is very surprising is that we see that for special characters in all passwords they represent 2.33% of the total, whereas in class 4 passwords, they represent a 12% of the total, which means that it is a lot more common in stronger passwords to use special characters than it is in weaker passwords.
Finally, one other thing about class 4 that is noteworthy, is that compared to the average in Have I Been Pwned 2.1 fragments per password for class 4, we have 4.4 fragments per password, which is more than twice as many, which again suggests that a higher number of fragments elevates the complexity of the password.
So all of that information that we saw suggests that there is indeed context that can be found in passwords. And it also suggests that users tend to underestimate the predictability of using things like deeper words and they tend to overestimate the added bonus of adding, for example, a number at the end of the password, thinking that this will make it stronger.
We see that some of the most common factors we can use to assess context in passwords are demographic, for example it plays an important role in password selection, whether or not someone’s made a female, whether they’re English or non-English speaking.
For example, once I was told that 50% of Chinese users that study, use only numbers in their passwords and about 12% of them used birthdays and phone numbers, which is information that is very easily accessible and therefore would make your passwords weak to people that were privy to that information.
Another factor that plays an important role is the age range, where for example, professionals used longer passwords of nine characters and more, whereas students tended to use smaller and less [indistinct] passwords. Professionals of late, they’re always computer scientists choosing more difficult passwords than say people of other professions.
Another thing that we saw is that the use of personal information in passwords with people using male or female names, that could be the names of friends and family, birth dates, birth years, city names, pet names, or other information that is about their interests, like foods, swear words, sports and things like that.
So how could all that information be used in the context of a [indistinct] investigation? If a law enforcement officer is faced with a device that he needs access to, and time would be of the essence and an exhaustive search could not be performed, that was not a viable option, then they could focus on the information that they can get about the suspect from their local devices, like their computer or their smart words or information from their house, like the books they read, or the posters they have, their online presence, like who they talk to, who they are associated with and what they post about, what their interests are, and if they can get access to it, previous passwords, so the type of architecture they used and the common ways they construct their passwords.
All of that information can be used with mangling roles to create what we call the Smarter Dictionary List that is tailored to that suspect and can help recall the password more quickly.
So our preliminary analysis that we included in this paper, because Have I Been Pwned is big and diverse data set, we wanted to also look at one smaller data set that is from a specific community. And for that, we looked at mangatraders.com that contains 800,000 entries or 600,000 unique passwords.
And we looked at the top 100 passwords per category that represent the top 100 passwords in the data set that represents 4.76% of the total and out of this 1.79% were Manga related which was a very big percentage. And then in the top 100 base words, with a base word being a password that does not have any normal [indistinct] characters at the beginning or the end, we see that the top 100 base words represent 5% of the total amount of passwords and they also represent 3.29% of Manga related, which again is a very big percentage. And that also, again helps with the notion that the context of the website that the password is used for plays a role when the password is created.
So for our future work we have already started to work on the same work for evaluating password cracking wordlist quality, so that we can create our own custom dictionary lists that we can use for a specific target website. We can tweak the level of context and see what level of context would be best for a specific target. So thank you very much for your attention. And if you have any questions, I’ll be more than happy to answer them.
