Introduction
John the Ripper (JtR) is a powerful and popular open-source password cracking tool. One of its key strengths is its extensibility. While it supports a vast number of hash types out of the box, you may occasionally encounter a custom or obscure hashing scheme that JtR doesn't recognize. In such cases, you can create your own custom format to teach JtR how to crack it.
In this lab, you will learn the fundamental concepts behind creating a custom JtR format. This lab is conceptual and focuses on the structure and process, providing you with the foundational knowledge needed to develop fully functional formats. You will not be writing and compiling a fully working format, but you will understand all the necessary steps.
Understand John the Ripper Format Structure
In this step, you will learn about the fundamental structure of a John the Ripper format. Understanding this structure is the first and most crucial part of creating a custom format.
JtR formats are defined in C source files (.c) and are built around a core C structure named struct fmt_main. This structure acts as a blueprint, telling JtR everything it needs to know about the hash type.
The key fields within struct fmt_main include:
fmt_label: A short, unique string that you use to specify the format with the--format=command-line option (e.g.,md5raw).fmt_tag: A unique identifier used internally by JtR to distinguish the format in saved session files (e.g.,$dynamic_0$).algorithm_name: A string describing the hashing algorithm and its properties, used for informational purposes.plaintext_length: The maximum password length the format can handle.binary_size: The size of the raw binary hash. For MD5, this would be 16 bytes.salt_size: The size of the salt, if the hash type uses one.methods: This is a nested structure containing pointers to various functions that implement the format's logic. The most important functions are:valid(): Checks if a given hash string from an input file is valid for this format. It's the first gatekeeper.split(): If a hash string contains more than just the hash (like a username), this function separates the components.binary(): Converts the hexadecimal or base64 hash string into its raw binary representation.salt(): Extracts the salt from the hash string.crypt_all(): The core function. It takes a set of candidate passwords, hashes them, and prepares them for comparison.cmp_all(): Compares the newly computed hashes fromcrypt_all()against the target hash to see if there's a match.
This step is purely theoretical. You do not need to execute any commands. In the next step, we will examine a real format file to see this structure in action.
Identify Existing Format Definitions
In this step, you will locate the source code for existing formats within the John the Ripper source tree. Examining these files is the best way to learn how to build your own.
The setup script for this lab has already cloned the JtR source code into the ~/project/john directory.
First, navigate into the src directory where the source files are located.
cd ~/project/john/src
Now, list all the files that define a format. By convention, these files end with _fmt_plug.c.
ls *_fmt_plug.c
You will see a long list of files, each corresponding to a different hash type that JtR supports.
7z_fmt_plug.c des_fmt_plug.c lotus5_fmt_plug.c ...
afs_fmt_plug.c django_fmt_plug.c mdc2_fmt_plug.c
aix_smd5_fmt_plug.c dmg_fmt_plug.c md4_fmt_plug.c
aix_ssha_fmt_plug.c dominosec_fmt_plug.c md5_fmt_plug.c
...and many more...
Let's take a look at a relatively simple one, md5_fmt_plug.c, to see the struct fmt_main we discussed. We'll use cat combined with head to view just the top portion of the file.
cat md5_fmt_plug.c | head -n 50
In the output, you will be able to see the struct fmt_main fmt_MD5 definition, with its label, tag, algorithm name, and method pointers, just as we described in the previous step.
Create a Simple Custom Format (Conceptual)
In this step, we will conceptually create a new C source file for a simple custom format. This exercise will illustrate how you would structure a new format file.
Our goal is to create a format that recognizes hashes with a special prefix: labex_md5$. For example, a hash like labex_md5$87e4e494b2399b0921d44e03693518f9.
Ensure you are still in the ~/project/john/src directory. We will use the nano text editor to create our new file.
nano labex_md5_fmt_plug.c
Now, copy and paste the following C code into the nano editor. This is a simplified skeleton and is not a fully functional format, but it demonstrates the core components.
#if FMT_EXTERNS_H
extern struct fmt_main fmt_labex_md5;
#elif FMT_REGISTERS_H
john_register_one(&fmt_labex_md5);
#else
#include <string.h>
#include "arch.h"
#include "common.h"
#include "formats.h"
#define FORMAT_LABEL "labex-md5"
#define FORMAT_NAME "LabEx Custom MD5"
#define ALGORITHM_NAME "MD5 32/64"
#define PLAINTEXT_LENGTH 32
#define BINARY_SIZE 16
#define SALT_SIZE 0
#define TAG_PREFIX "labex_md5$"
#define TAG_LENGTH (sizeof(TAG_PREFIX) - 1)
// This function checks if a hash string is valid for our format
static int valid(char *ciphertext, struct fmt_main *self)
{
if (strncmp(ciphertext, TAG_PREFIX, TAG_LENGTH))
return 0;
char *p = ciphertext + TAG_LENGTH;
if (hexlenu(p, 0) != 32)
return 0;
return 1;
}
struct fmt_main fmt_labex_md5 = {
{
FORMAT_LABEL,
FORMAT_NAME,
ALGORITHM_NAME,
BENCHMARK_COMMENT,
BENCHMARK_LENGTH,
0,
PLAINTEXT_LENGTH,
BINARY_SIZE,
BINARY_ALIGN,
SALT_SIZE,
SALT_ALIGN,
MIN_KEYS_PER_CRYPT,
MAX_KEYS_PER_CRYPT,
FMT_CASE | FMT_8_BIT,
{ NULL },
{ TAG_PREFIX },
NULL
}, {
/* init */ init,
/* done */ done,
/* reset */ reset,
/* prepare */ prepare,
/* valid */ valid,
/* split */ split,
/* binary */ binary,
/* salt */ salt,
{ NULL },
/* source */ source,
{
/* get_hash* */ get_hash_0,
/* get_hash* */ get_hash_1,
/* get_hash* */ get_hash_2,
/* get_hash* */ get_hash_3,
/* get_hash* */ get_hash_4,
/* get_hash* */ get_hash_5,
/* get_hash* */ get_hash_6
},
/* cmp_all */ cmp_all,
/* cmp_one */ cmp_one,
/* cmp_exact */ cmp_exact
}
};
#endif
After pasting the code, save the file and exit nano by pressing Ctrl+X, then Y, and then Enter.
You have now created the source file for a new format. The most important part is the valid() function, which simply checks for our labex_md5$ prefix and ensures the following hash is 32 characters long.
Compile John the Ripper with New Format (Conceptual)
Now that you have a conceptual format file, this step explains how to integrate it into the John the Ripper build process and compile it.
First, ensure you are in the src directory.
cd ~/project/john/src
The first step in compiling JtR is to run the configure script. This script checks your system for required libraries and sets up the build environment.
./configure
After configuration is complete, you can compile the source code using the make command. We will use make clean to remove any previous builds and make -sj4 to run the compilation using 4 parallel jobs, which speeds up the process.
make -s clean && make -sj4
The compilation will take a minute or two.
Important Note: In a real-world scenario, simply creating the .c file is not enough. You would also need to edit a configuration file (like Makefile.in) to tell the build system to include your new labex_md5_fmt_plug.o object file when linking the final john executable. We are skipping that modification for simplicity in this lab. Therefore, the compilation will succeed, but our new format will not actually be included in the final binary. This illustrates a critical step in the development process.
Test Custom Format (Conceptual)
In this final step, we will discuss how you would test your custom format with the newly compiled John the Ripper executable.
The compiled john binary is located in the ~/project/john/run/ directory. Let's navigate there.
cd ~/project/john/run
You can list all the formats that your compiled version of JtR supports.
./john --list=formats
Scroll through the list. You will notice that our labex-md5 format is not present. This is expected, because as mentioned in the previous step, we did not modify the build files to include it.
Now, let's create a sample hash file and a wordlist to see how we would use the format if it were successfully compiled.
First, create a file named hashes.txt in your project's root directory containing a hash that matches our custom format. The hash 87e4e494b2399b0921d44e03693518f9 is the MD5 hash of the password "labex".
echo "labex_md5$87e4e494b2399b0921d44e03693518f9" > ~/project/hashes.txt
Next, create a simple wordlist containing the correct password.
echo "labex" > ~/project/wordlist.txt
Finally, this is the command you would run to crack the hash using the custom format.
./john --format=labex-md5 --wordlist=~/project/wordlist.txt ~/project/hashes.txt
When you run this command, JtR will report an error because the format is unknown, confirming our understanding of the compilation process.
Unknown format name: "labex-md5"
This completes our conceptual walkthrough of creating and testing a custom JtR format.
Summary
In this lab, you explored the conceptual process of creating custom formats for John the Ripper. You have gained a foundational understanding of the key steps involved in extending JtR's capabilities.
You learned about:
- The core
struct fmt_mainstructure that defines every JtR format. - How to locate and examine existing format definitions in the JtR source code.
- The process of creating a new format source file with a basic
valid()function. - The standard procedure for compiling JtR using
./configureandmake. - The critical (and intentionally omitted) step of modifying build files to include a new format.
- How to test a format using the
--formatflag and a sample hash file.
While this lab did not result in a fully functional custom format, it has provided you with the essential knowledge and a clear roadmap for developing your own formats to tackle unique password hashing schemes.
