Vim in the Future

There is something of a tradition of programmers writing posts about Vim. Many proselytize, and others are narratives of discovery. What I want to contribute to the conversation about Vim, in this late year of 2018, is what role it still plays in a world that is technologically rushing by. I’m going to tell you more about Vim, why I use it, and how I use it, so that I can say why it might still be worthwhile looking at—or not.

I have learned Vim as a programming-centric tool, but I use it for other tasks, too. This post assumes a reader isn’t necessarily a programmer but is curious about how tech things get done.

Vim in the Past

First, let me talk a little bit about what Vim grew from.

Vim is old. It’s a program that was originally written for the Amiga operating system, first available in 1991. The even older program it emulates, vi, began in 1976 and was available in 1979. Vim has become the most common implementation of vi currently in use today, as Vim has become compatible with the vast majority of computers in existence.

This long history implies a couple of important things. First, vi predates many ideas which are today so ubiquitous that we consider them integral to how computers work: graphical interfaces, word processing, instant visual editing, multitasking, and so on. Second, it means that vi (and by extension, as its modern counterpart, Vim) occupies a special and integral role in the history of computing itself and the standards upon which many server operating systems are built. A vi command compatible with its original functionality remains a part of the POSIX standard even to this day, as well as the Single UNIX Specification. Vim provides that today.

In other words, Vim is a relic of another time—one which is now over forty years gone. It can be obtuse, arcane, and occasionally downright wrong for today’s world as a result. The original author of vi used a terminal, not a computer directly: one slow enough that he had to issue instructions without seeing the feedback right away. This, among many other constraints, guided vi‘s design. Vim inherited these decisions in turn.

So it may surprise you that Vim has not only endured but flourished. The key lies in its flexibility within its historical constraints.

Vim in the Present

So what is Vim? It edits plain text, and it does so within a terminal emulator today.

Most operating systems offer a program called a terminal emulator which presents a boxy window of text and a prompt at which you can enter commands to execute programs, and the programs will run and usually output text in return as a result. This is also just called the command line or just a terminal. Nowadays, mine looks like the one below.

iTerm2 running on macOS Mojave 10.14.1. No command has been run—only a prompt is visible.
iTerm2 running on macOS Mojave 10.14.1

At the top left, there’s a little lightning bolt, which is what I’ve told my prompt to look like, and there’s a little rectangle, which indicates where my cursor is and where text will go if I type something. If I type “vim” and press Enter, the entire terminal changes to look like the image below.

Vim running in iTerm 2. There is no buffer being edited. The top and bottom of the Vim window have a highly customized appearance.
Vim running in iTerm 2

Vim has changed the terminal’s screen into a text editor. The cursor is still there, but now it’s on the first line of a nameless buffer (an area of memory for editing).

There is no text there in the buffer, so I can’t really do anything. My options are to open a file which already exists, or I can insert new text into this empty buffer. Either of these options will call on me to take an action which feels pretty unfamiliar to a modern computer user.

If I want to open something, I type a colon followed by a command, like “:edit,” followed by the name of a file. This is an “Ex” command, a type of interface which harkens back to the editor which predates even vi. Vim has hundreds of these.

If I want to insert text, however, I need to change into Insert mode, by pressing “i.”  Now I can type in text as if I were at a normal editor, like Notepad or TextEdit. However, I can no longer enter Ex commands, so I can leave Insert mode by pressing the Escape key. Doing so puts me back in Normal mode. In this mode, I can move the cursor around, enter Ex commands (by entering a colon), and use some other commands, but I can’t directly insert text. In Insert mode, I can insert text, but I can’t access the other commands. So I have to switch back and forth.

This switching process is one of the relics of the 1970s which remains in the editor today: modal editing. It is simultaneously a frustrating obstacle to new users and a profitable tool to those who learn to appreciate it.

How can it be useful to be unable to enter text in a text editor? Consider that, in Normal mode, your keys do not enter text. That means they’re freed up to do other things. Vim therefore overloads every character on the keyboard in Normal mode with a meaning, such as an action (called an operator) or a modifier (such as a motion).

For example, the “d” key means delete in Normal mode. Yet it does not delete on its own. It’s merely an operator. For it to achieve something, it needs to be paired with a motion—some amount of text over which to accomplish the action. Motions are usually over text objects like words, sentences, paragraphs, lines, and so on. So to delete a word, it’s possible in Vim to put your cursor over that word, press “dw” (the combination of the “d” operator followed by the “w” motion), and Vim will delete the word. The combination of an operator and a motion is a command.

Throw in numbers (such as “d3w” for “delete three words”), and Normal mode provides a fully composable set of primitives for building up a language of text editing expressions. There exist other operators for changing, copying (“yanking” in Vim parlance—vi predates the existence of a clipboard), pasting (“putting”), formatting, swapping case, filtering through another program, and so on.

Vim is meant to be used primarily in Normal mode, where this language can be used for navigating the text, rearranging it, copying things around, manipulating things precisely, and so on. The number of operators and motions are relatively few (perhaps several dozen each—they can each be a couple of characters long). Yet from these, thousands and thousands of expressions may be formed.

This may sound overwhelming to learn at first, but there’s always time to learn as you go. vi has functioned this way for over forty years. Vim continues to function exactly the same way. If you learned only one operator or motion every week for a year, you’d know hundreds of expressions by heart after that time.

Vim in the Future

Vim has already been around almost thirty years just in its current incarnation. Every time I learn something new about Vim, or every time I customize it a little more, I benefit from that indefinitely. For all its problems, idiosyncrasies, and hangups, I know that I will likely never have to learn another text editor again for the rest of my life. Vim is capable enough and flexible enough to carry me into the future and meet whatever need I have without changing substantially. I feel relatively sure of this because it’s already lasted this long. At the heart of how it endures are those parts of Vim that resist change.

There is a large core component of its functionality guaranteed to work a certain way (the vi compatibility). The maintainer’s vision for Vim’s future firmly roots the editor to its past and its place within the wider Unix ecosystem. For example, I run the editor within a terminal, and while this is limiting (it has no native interface of its own), I appreciate the constraints. There are fewer things to change over time; the customization is tied to that environment, which I also control; and it is more closely integrated with the other tools I know.

I have heavily edited Vim to meet my needs over time. My .vimrc file is only the beginning. The .vim directory I use contains a massive number of plugins, some written by me, and others written by a number of online authors.

I write, and my writing is contained in hundreds of plain text files (Markdown formatted) containing tens of thousands of words, all edited and formatted in Vim. These files’ formatting can be converted into whatever I need on the fly, but the editing remains as plain text so that I can use the same tool I’ve always known. I won’t have to worry about Vim replacing its toolbar with a “ribbon” next year or suddenly changing to a subscription model or dropping compatibility for my operating system.

Individually, those changes never bother me much, but other time, I noticed with most programs that I end up moving around from one thing to the next through the years as the platforms shift under me, as people learn how to do things differently. For example, I have switched web browsers several times over the years. With text, maybe we’ll learn some new tricks, but Vim is just a tool among many, and I imagine it will thrive as it has so far because it knows how to harmonize with a bigger ecosystem and learn to interoperate with new ideas without becoming something new.

In other words, I feel confident continuing with Vim and learning more about it over time because the core of how it works is stable, integral, and flexible enough to ensure it will work in the future.

Vim in Your Future

Do you want to use Vim after reading all that? The world is full of advice on how to do just this. Maybe you’ve already tried, and you came away frustrated.

I won’t say my advice is the only way to go. I think of Vim as a single cog in the larger clockwork of the terminal’s command-line interface. Some familiarity with the command line is helpful before enmeshing yourself in the teeth of Vim’s gears. (I hope to write about this wider someday soon.) But if you’re determined, I’ll give you a primer with the first steps. What you do after that is up to you.

Before you begin, you’ll want Vim on your computer, and if you’re using macOS or Linux, you’ve probably already got it. On some versions of Linux, you’ll need to install something like “vim” (Ubuntu) or “vim-enhanced” (Red Hat/CentOS/Fedora). For macOS users, the default version installed is fine, but a newer version through Homebrew is available if you want it.

A Gentle Ascent

If you’ve tried Vim before and it didn’t take, I’m willing to guess someone advised you to try Vim in a “pure” or strict way that makes no compromises on how you use it—as close to the original 1979 vision as possible. I don’t believe this makes sense in a modern world.

There’s another extreme available: pre-packaged configurations of Vim which come juiced to the gills with bells and whistles that make it seem polished, utterly flexible, and even magical. Possibly the most popular variant on this is called “spf13-vim.”

I suggest a compromise. You want to ease the transition in to Vim so you can be productive as soon as possible, but you also want to have your feet planted firmly on a solid bedrock so that you feel confident with each step you take. Below, I’ll try to guide you through how to do this by

  • first, smoothing the transition with some minimal settings;
  • next, providing a very minimal set of commands to do the most basic tasks;
  • then, showing how to run the tutor and suggesting a reasonable pace for it; and
  • finally pointing to the help system.

Baseline Settings

Now, remember how I said Vim is old? Yeah, it’s old. By default, it starts in something called “compatible” mode. This mode isn’t like the text editing modes I mentioned earlier. This means all its settings are modified to make it as much like the original vi as possible.

This default changed in version 7.4.2111, which happened back in 2016, so compatible mode might not affect you. If you have a newer version of Vim (8 or above), you probably have a “defaults.vim” somewhere which came with your version which sets a few things to more reasonable settings without your having to do anything.

However, let’s cover our bases and just lay the groundwork for a configuration file. This will accomplish a few things. First, the mere presence of a user configuration file will cause Vim to kick out of compatibility mode and behave more like you would expect, regardless of its version. Second, by having one, you will know where to stash new settings as you think to add them. Third, we’ll have a few more reasonable default settings—nothing too weird, just things that almost everyone universally ends up configuring almost right away.

The defaults.vim file is a fine place to start, but I’d suggest taking it a step farther. Tim Pope (a prolific Vim plugin writer) has written a baseline settings file he calls “sensible.vim.” You can copy this wholesale to your home directory as a file called .vimrc.

Let’s add one final thing to it, though. If you’re the sort who uses the mouse ever, you might appreciate these few lines.

if has('mouse')
  set mouse=a
endif

That little fragment can go on any line. Save that file to your home directory as .vimrc, and when you start vim, you’ll see something very minimal, but not entirely unreasonable. Here’s what you would see if you edited the sensible settings file itself.

iTerm2 running Vim with only sensible.vim settings, editing sensible.vim. The window has text in it with code, but no colors. A simple statusbar is at the bottom.
iTerm2 running Vim with only sensible.vim settings, editing sensible.vim

We could tinker with these settings, but let’s just move on for now. You can do all that later. You have the entire rest of your life.

Basic Tasks

You really only need to accomplish a very small handful of things to get the barest of functionality from Vim. Let’s cover each.

  1. Open Vim. You type “vim” on the command line. Easy, right? You can also type “vim” followed by a filename to open a file directly (e.g., “vim path/to/a/file.txt“). That’s one way to open a file.
  2. Open a file. If you’re already in Vim, though, you want to open files without closing the program entirely and then reopening it. Easiest way to do that is with the Ex command “:edit” followed by the filename, such as “:edit path/to/a/file.txt.” You can use “:e” as a shorthand for “:edit.”
  3. Change a file. You can move the cursor around with your arrow keys in both Insert mode and Normal mode. (The Vim purists will tell you to use one of “h,” “j,” “k,” or “l,” to move the cursor around, and you may do so, but this works only in Normal mode, and it may feel unnatural at first.) Use “i” to put Vim into insert mode. Change the text as you like. Then use the Escape key to leave Insert mode.
  4. Save a file. You can do this with an Ex command, “:write.” You can use “:w” as a shorthand for “:write.”
  5. Close Vim. You can use “:exit” to do this, another Ex command. If you want to, you can shorten this to “:xit” or just “:x.” This command happens to save any file you’re working on before quitting. If it can’t write a file, it will balk and tell you. If you merely want to bail out without saving, you can use “:quit!” to force the matter (this has a “:q!” counterpart as well).

That’s it. With that set of commands, strictly speaking, you don’t need anything else. You could use Vim the rest of your life, configured or not, old or new, and never worry about the rest. In fact, all those commands are vi-compatible, so you could go back to 1979 and be fine.

Of course, if you’ve configured the mouse, it’ll be nice being able to scroll and click things, but otherwise, you’re ready to start.

The Vim Tutorial

You might find your 1979 life very dull and slow, though. You’ll want to learn new things, and the place to start is with Vim’s own tutorial.

Many Vim users never actually look at the tutorial, which is a shame. The tutorial is actually just a text file. If you run the command “vimtutor” from the command line, Vim opens in a bare, minimally configured mode with a copy of this text file.

iTerm2 running the Vim tutor. It is a very simple display: just a box with text describing the Vim tutor.
iTerm2 running the Vim tutor

The tutorial is divided into multiple brief lessons, each on a different topic which builds on the last. The tutorial itself, being text, is designed to be edited so that you can interact with it and learn by example.

Though the tutorial says it will take about a half hour to complete, I do not recommend chewing through the whole thing in a single half hour and never returning. Instead, I suggest you take in the first few lessons and then skim the rest. After that, come back to it a few days later, take a few more lessons, and so on. It’s a very information-dense resource, and I believe it takes a few reads before it sinks in. I believe it’s reasonable to spend some weeks just absorbing it.

It never hurts to return to the tutorial even after you’re settled in and feeling more confident. There are always dusty, cobwebby corners of expertise which could use some refreshing, and the tutorial hits upon a broad swath of topics.

Finally, there are several suggestions from the tutorial which you need not incorporate into your life at all. Use your judgment on what is appropriate for you. It suggests moving away from the arrow keys, for example. I have never done this. I probably never will. It’s not appropriate for me because I have difficulty distinguishing right and left, and so mapping “j” and “k” (which are laterally adjacent) to up and down in my mind is extremely confusing. The spatial layout of the arrow keys on my keyboard helps me keep things straight. So from the tutorial, take what you need and leave what you don’t.

Further Learning

If you have mastered the tutorial, you stand head and shoulders above many Vim users already in sheer ability to manipulate text. A lot of people learn whatever commands they need piecemeal and never go out of their way to discover new ones until they see them in action or feel the need to look.

Yet you may feel many things are lacking. You might start to wonder how to accomplish the same task over and over again and wonder whether Vim has a way to do so (it does, using the repeat operator or recording). Or you might wonder what some of the settings in your .vimrc file mean. Or wonder how to switch between open files (buffers).

Here, the Vim manual can be very helpful. It is shockingly extensive. Almost every kind of functionality, topic, setting, or concept is documented by name in the manual somewhere. It’s a matter of typing “:help” followed by some word related to what you want to know.

Don’t know what word to search for? Vim will take parts of words and try to find the best match. For example, “:help reco” will teach you about the “:recover” command. Want to see what’s available? If you’ve used the sensible configuration from above, you can use tab-completion on the command line in Vim to see available help topics and cycle through them. For example, typing “:help reco” and then pressing Tab shows a range of possibilities in the statusline. The first one is filled in automatically. Press Tab again, and the second suggestion is filled in. You can press Tab to cycle through all the available options.

Sometimes you can’t find the answer within Vim’s help, and you need an Internet search. Very often, this will bring the suggested fix of using a plugin to provide a piece of functionality. As you learn Vim, I suggest you incorporate plugins slowly at first.

Being choosy and slow to adapt plugins will help you learn Vim’s core internals better instead of overriding them in a fragile or even nonsensical way. I have added many plugins to my configuration over time, but I’ve removed at least as many. Some have gone unused, but others actively interfered with Vim’s workings because the plugin was not designed to harmonize with the design of the editor itself.

Tim Pope has written dozens of plugins and is expert in knowing how to write ones which extend Vim in nearly invisible ways. For example, the vim-surround plugin adds operators and motions for wrapping or unwrapping quotes, tags, and blocks of text with parentheses, brackets, and so on. It operates so fluently within the context of Vim that it’s difficult to remember that it’s not part of the original program.

Less harmonious plugins attempt to do things like create kinds of UIs (such as for navigating filesystems) or override parts of Vim in fragile ways (such as faking multiple-cursor support by coloring parts of the buffer to look like cursors and then repeating the actual cursor’s actions at those regions). Sometimes these plugins work out. Sometimes they don’t.

My suggestion here is to use Vim as is for a little while, and if some process feels particularly painful or annoying for a few days, then seek out a better way—first through Vim itself then through a plugin if available.

If the idea of tweaking Vim to your liking (or learning some of the really interesting Vim commands) sounds appealing, a really great resource is a book called Practical Vim. It’s a fully modern approach to Vim with plenty of things that feel very empowering, and it’s arranged in a recipe-like fashion so that you don’t have to read from end to end (unlike the Vim tutorial).

Finally, I am always happy to answer questions if no other resource is helpful or available.

Editing Your Future

That’s the plan. Set up a basic configuration so that you have an easier start of it, read the tutorial, and know where to go when you need more info. Take things slow, but feel free to beat your own path. It’s your program, and you’re free to determine what works best for you. Don’t let anyone tell you what the “Vim way” is.

You’re not at all obligated to stick with Vim or even try it. My only goal is to tear down whatever obstacles may have been in your way to begin with—overly rigid guidance, misconceptions, mystification—if you were curious to begin with.

I fully admit Vim is imperfect and may not be the best text editor for this century altogether. However, for my own part, I’m certain I can look forward to a long future of return on my investment in Vim. For this reason alone, I think it’s worth a look.

A Gentle Primer on Reverse Engineering

Over the weekend at Women Who Hack I gave a short demonstration on reverse engineering. I wanted to show how “cracking” works, to give a better understanding of how programs work once they’re compiled. It also serves my abiding interest in processors and other low-level stuff from the 80s.

My goal was to write a program which accepts a password and outputs whether the password is correct or not. Then I would compile the program to binary form (the way in which most programs are distributed) and attempt to alter the compiled program to accept any password. I did the demonstration on OS X, but the entire process uses open source tools from beginning to end, so you can easily do this on Windows (in an environment like Cygwin) or on Linux. If you want to follow along at home, I’m assuming an audience familiar with programming, in some form or another, but not much else.

Building a Program

I opened a terminal window and fired up my text editor (Vim) to create a new file called program.c. I wanted to write something that would be easy to understand, edit, and yet still could be compiled, so C seemed like a fine choice. My program wasn’t doing anything that would’ve been strange in the year 1982.

First, I wrote a function for validating a password.

int is_valid(const char* password)
{
    if (strcmp(password, "poop") == 0) {
        return 1;
    } else {
        return 0;
    }
}

This function accepts a string and returns a 1 if the string is “poop” and 0 otherwise. I’ve chosen to call it is_valid to make it easier to find later. You’ll understand what I mean a few sections down.

Now we need a bit of code to accept a string as input and call is_valid on it.

int main()
{
    char* input = NULL;
    input = malloc(256);
    printf("Please input a word: ");
    scanf("%s", input);

    if (is_valid(input)) {
        printf("That's correct!\n");
    } else {
        printf("That's not correct!\n");
    }

    free(input);
    return 0;
}

This source code is likewise pretty standard. It prompts the user to type in a string and reads it in to a variable called input. Once that’s done, it calls is_valid with that string. Depending on the result, it either prints “That’s correct!” or “That’s not correct!” and exits, returning control to the operating system. With a couple of “include” directives at the top, this is a fully functioning program.

Let’s build it! I saved the file program.c and used the command gcc program.c -o program to build it.

This outputs a file in the current directory called program which can be executed directly. Let’s run our program by typing ./program. It’ll ask us to put in a word to check. We already know what to put in (“poop”), so let’s do that and make sure we see the result we expect.

Please input a word: poop
That's correct!

And if we run it again and type in the wrong word, we get the other possible result.

Please input a word: butts
That's not correct!

So far, so good.

A Deeper Look

There’s nothing special about this program that makes it different than your web browser or photo editor; it’s just a lot simpler. I can demonstrate this on my system with the file command. Trying it first on the program I just built, with the command file program, I see:

program: Mach-O 64-bit executable x86_64

This is the file format OS X uses to store programs. If this kind of file seems unfamiliar, the reason is that most applications are distributed as app bundles which are essentially folders holding the executable program itself and some ancillary resources. Again, with file, we can see this directly by running file /Applications/Safari.app/Contents/MacOS/Safari:

/Applications/Safari.app/Contents/MacOS/Safari: Mach-O 64-bit executable x86_64

Let’s learn a little more about the binary we just built. We can’t open it in a text editor, or else we get garbage. Using a program called hexdump we can see the raw binary information (translated to hexadecimal) contained in the file. Let’s get a glimpse with hexdump -C program | head -n 20.

00000000  cf fa ed fe 07 00 00 01  03 00 00 80 02 00 00 00  |................|
00000010  10 00 00 00 10 05 00 00  85 00 20 00 00 00 00 00  |.......... .....|
00000020  19 00 00 00 48 00 00 00  5f 5f 50 41 47 45 5a 45  |....H...__PAGEZE|
00000030  52 4f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |RO..............|
00000040  00 00 00 00 01 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  19 00 00 00 28 02 00 00  |............(...|
00000070  5f 5f 54 45 58 54 00 00  00 00 00 00 00 00 00 00  |__TEXT..........|
00000080  00 00 00 00 01 00 00 00  00 10 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
000000a0  07 00 00 00 05 00 00 00  06 00 00 00 00 00 00 00  |................|
000000b0  5f 5f 74 65 78 74 00 00  00 00 00 00 00 00 00 00  |__text..........|
000000c0  5f 5f 54 45 58 54 00 00  00 00 00 00 00 00 00 00  |__TEXT..........|
000000d0  10 0e 00 00 01 00 00 00  e7 00 00 00 00 00 00 00  |................|
000000e0  10 0e 00 00 04 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 04 00 80 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  5f 5f 73 74 75 62 73 00  00 00 00 00 00 00 00 00  |__stubs.........|
00000110  5f 5f 54 45 58 54 00 00  00 00 00 00 00 00 00 00  |__TEXT..........|
00000120  f8 0e 00 00 01 00 00 00  1e 00 00 00 00 00 00 00  |................|
00000130  f8 0e 00 00 01 00 00 00  00 00 00 00 00 00 00 00  |................|

The left column is the “offset,” in hexadecimal (like line numbering, it tells us how many bytes into the file we are on a particular line). The middle two columns are the actual contents of the file itself, again in hexadecimal. The right column shows an ASCII equivalent for the file’s contents, where possible. If you pipe the file’s contents to less you can scan through and see mostly a lot of garbage and also a few familiar strings. If you’re interested in knowing what pieces of text are embedded in a file, the program strings speeds this process up a great deal. In our case, it tells us:

poop
Please input a word:
That's correct!
That's not correct!

So clearly those strings are still floating around in the file. What’s the rest of this stuff? Volumes of documentation exist out there on the Mach-O file format, but I don’t want to bog down in the details. I have to level with you here—I honestly don’t actually know much about it. Analogizing from other executable formats I’ve seen before, I know there’s probably a header of some kind that helps the operating system know what kind of file this is and points out how the rest of the file is laid out. The rest of the file, incidentally, is made up of sections which may contain any of a number of things, including data (the strings in this case) built into the program; information on how to find code called from elsewhere in the system (imports, like our printf and strcmp functions, among others); and executable machine code.

Disassembling the Program

It’s the machine code we’re interested in now. This is the interesting part! Machine code is binary data, a long string of numbers which correspond to instructions the processor understands. When we run our program, the operating system looks at the file, lays it out in memory, finds the entry point, and starts feeding those instructions directly to the processor.

If you’re used to scripted programming languages, this concept might seem a little odd, but it bears on what we’re about to do to our binary. There’s no interpreter going over things, checking stuff, making sure it makes sense, throwing exceptions for errors and ensuring they get handled. These instructions go right into the processor, and being a physical machine, it has no choice but to accept them and execute each one. This knowledge is very empowering because we have the final say over what these instructions are.

As you may know, the compiler gcc translated my source code I wrote earlier into machine language (and packaged it nicely in an executable file). This allows the operating system to execute it directly, but as another important consequence of this process, we also no longer need the source code. Most of the programs you run likely came as binary executables without source code at all. Others may have source code available, but they’re distributed in binary form.

Whatever the case, let’s imagine I lost the source code to program up above and can’t remember it. Let’s also imagine I can’t even remember the password, and now my program holds hostage important secrets.

You might think I could run the binary through the strings utility, hoping the password gets printed out, and in this case, you’d be on the right track. Imagine if the program didn’t have a single password built in and only accepted passwords whose letters were in alphabetical order or added up (in binary) a specific way. Without the source code, I couldn’t scan to see which strings seem interesting, and I wouldn’t have a clue what to type in.

But we don’t need to lose heart because we already know that the program contains machine code, and since this machine code is meant to be fed directly to the processor, there’s no chance it’s been obfuscated or otherwise hidden. It’s there, and it can’t hide. If we knew how to read the machine code, there would be no need for the source code.

Machine code is hard for a human to read. There’s a nice GNU utility called objdump which helps enormously in this respect. We’ll use it to disassemble the binary. This process is called “disassembly” instead of “decompilation” because we can’t get back the original source code; instead we can recover the names of the instructions encoded in machine code. It’s not ideal, but we’ll have to do our best. (Many people use a debugger to do this job, and there’s a ton of benefits to doing so, like being able to watch instructions execute step by step, inspect values in memory, and so on, but a disassembly listing is simpler and less abstract.)

I looked up the documentation for gobjdump (as it’s called on my system) and picked out some options that made sense for my purposes. I ended up running gobjdump -S -l -C -F -t -w program | less to get the disassembly. This is probably more than we’d care to know about our program’s binary, much of it mysterious to me, but there’s some very useful information here too.

The Disassembly

I’ll share at least what I can make of the disassembly. At the top of the listing is some general information. This symbol table is interesting. We can see the names of the functions I defined. If I had truly never seen the source code, I would at this point take an especial amount of interest in a function called is_valid, wouldn’t I?

Immediately below this is a “Disassembly of section .text”. I happen to know from past experience that the “.text” bit is a bit misleading for historical reasons; a “.text” section actually contains machine code! The leftmost column contains offsets (the place in the file where each instruction begins). The next column is the binary instructions themselves, represented in hexadecimal. After that are the names and parameters of each instruction (sometimes with a helpful little annotation left by objdump).

Of course, the very first thing I see is the instructions of the is_valid function.

Disassembly of section .text:

0000000100000e10  (File Offset: 0xe10):
   100000e10:   55                      push   %rbp
   100000e11:   48 89 e5                mov    %rsp,%rbp
   100000e14:   48 83 ec 10             sub    $0x10,%rsp
   100000e18:   48 89 7d f0             mov    %rdi,-0x10(%rbp)
   100000e1c:   48 8b 7d f0             mov    -0x10(%rbp),%rdi
   100000e20:   48 8d 35 33 01 00 00    lea    0x133(%rip),%rsi        # 100000f5a <strcmp$stub+0x4a> (File Offset: 0xf5a)
   100000e27:   e8 e4 00 00 00          callq  100000f10 <strcmp$stub> (File Offset: 0xf10)
   100000e2c:   3d 00 00 00 00          cmp    $0x0,%eax
   100000e31:   0f 85 0c 00 00 00       jne    100000e43 <is_valid+0x33> (File Offset: 0xe43)
   100000e37:   c7 45 fc 01 00 00 00    movl   $0x1,-0x4(%rbp)
   100000e3e:   e9 07 00 00 00          jmpq   100000e4a <is_valid+0x3a> (File Offset: 0xe4a)
   100000e43:   c7 45 fc 00 00 00 00    movl   $0x0,-0x4(%rbp)
   100000e4a:   8b 45 fc                mov    -0x4(%rbp),%eax
   100000e4d:   48 83 c4 10             add    $0x10,%rsp
   100000e51:   5d                      pop    %rbp
   100000e52:   c3                      retq   
   100000e53:   66 66 66 66 2e 0f 1f 84 00 00 00 00 00  data16 data16 data16 nopw %cs:0x0(%rax,%rax,1)

This is super exciting because we’re about to read assembly language. There are lots of books and sites on this subject, and my own understanding of assembly language is a bit rusty from years of disuse, but I know enough to get the gist. Let’s break it down.

  • The first three instructions (the first three lines, starting with 100000e10) are a preamble that begin most functions in assembly language generated by a compiler. They’re not important for us. (It saves the old frame pointer, gets a new frame pointer, and clears space on the stack for locals.)
  • The next two instructions set up for our strcmp function. This looks a bit odd in assembly language compared to what we’re used to. The mov instructions are shifting data from a location in memory to a register and vice versa. Because registers are involved, the disassembly wasn’t able to hint very well what these values may be, but we can guess it’s moving the strings to compare into place. I know this because of the calling convention for the function call (basically, set up the data and then make the call, which will know where to find the data); because the %rbp is the base register, which usually points to data; and because -0x10(%rbp) is a way of saying “look sixteen bytes earlier in memory than the address in the %rbp register.”
  • The lea and callq instructions load and call the strcmp function using the parameters we just moved in place. That function lives elsewhere in the system, so some magic happens here to transfer control of our program to that function.
  • By the time we reach the cmp instruction, strcmp has done its thing and stored its result in the accumulator register %eax. By convention, return values usually live in %eax, so given that we’re using a cmp (“compare”), and it’s acting on %eax and $0x0 (a zero), it’s a safe bet we’re checking to make sure strcmp returned zero. This instruction has the side effect of setting a flag in the processor called ZF to either 1 or 0, depending on if the comparison is true or not.
  • The next instruction is jne which is short for “jump if not equal.” It checks the ZF flag, and if it’s zero, skips ahead twelve bytes (bypassing any instructions in the intervening space).
  • That’s followed by a movl and a jmpq. These instructions move a 1 into a location in memory and skip ahead another seven bytes. Look at the two-digit hexadecimal numbers to the left of these two instructions. They add up to twelve!
  • Likewise, after these instructions, one other instruction moves the value 0 into the same location of memory and continues ahead. This instruction is exactly seven bytes long. So these jumps accomplish one of either two things: either the memory location -0x4(%rbp) is going to hold a 1 or a 0 by the time we get to the final mov. This is how assembly language does an if—a very interesting detail we’ll return to.
  • That last mov puts the value at -0x4(%rbp) (we just saw it’s either a 1 or a 0) into %eax, which we know is going to be the return value.
  • Finally, the function undoes the work from the preamble and returns. (After that is some junk that’s never executed.)

That was a lengthy explanation, so to sum up, we learned that the binary executable has a function called is_valid, and this function calls strcmp with some values and returns either a 1 or a 0 based on its return value. That’s a pretty accurate picture based on what we know of the source code, so I’m pleased as punch!

Directly below the definition for this function is the main function. It’s longer, but it’s no more complex. It does the same basic tasks of moving values around, calling functions, inspecting the values, and branching based on this. Again, the values are difficult to get insight into because many registers are used, and there’s a bit more setup. For the sake of brevity, I’ll leave analyzing this function as an exercise for the reader (I promise it won’t be on the test).

Breaking the Program

Remember, we don’t have the slightest idea what the password is, and there’s no good indication from the disassembly what it might be. Now that we have a good understanding of how the program works, we stand a good chance of modifying the program so that it believes any password is correct, which is the next best thing.

We can’t modify this disassembly listing itself. It’s output from objdump meant to help us understand the machine code (the stuff in the second column). We have to modify the program file itself by finding and changing those hexadecimal numbers somewhere in the file.

After looking over how both is_valid and main work, there are lots of opportunities to change the flow of the program to get the result we want, but we have to stay within a few rules. Notice how a lot of the instructions specify where other parts of the program are in terms of relative counts of bytes? That means that we can’t change the number of bytes anywhere, or else we’d break all the symbol references, section locations, jumps, offsets, and so on. We also need to put in numbers which are valid processor instructions so that the program doesn’t crash.

If this were your first program, I’d be forced to assume you wouldn’t know what numbers mean what to the processor. Luckily, the disassembly gives us hints on how to attack it. Let’s confine our possibilities (such as changing jump logic or overwriting instructions with dummy instructions) to only those we can exploit by using looking at this disassembly itself. There isn’t a lot of variety here.

To me, one neat thing about is_valid stands out. Two of the lines are extremely similar: movl $0x0,-0x4(%rbp) and movl $0x1,-0x4(%rbp). They do complementing things with the same memory location, use the same number of bytes (seven), involve the same setup, are near one another, and directly set up the return value for is_valid. This says to me the machine code for each instruction would be interchangeable, and by changing one or the other, we can directly change the return value for is_valid to whatever we want. It’s a safe bet, with a function named that, we want it to return a 1, but if we weren’t sure, I could look ahead to the main function and see how its return value gets used later on.

In other words, we want to change movl $0x0,-0x4(%rbp) to be movl $0x1,-0x4(%rbp) so that no matter what, is_valid returns a one. The machine code for the instruction we have is c7 45 fc 00 00 00 00. Conveniently, the machine code for that precise instruction we want is just two lines above: c7 45 fc 01 00 00 00. The last challenge ahead is to find these bytes in the actual file and change them.

Where in the file are these bytes? Note that the listing says “File Offset: 0xe10” for the function is_valid. That’s actually the count of bytes into the file we’d find the first instruction for this function (3648 bytes, in decimal), and the offset in the left column for the first instruction is “100000e10”, so those offsets in the left column look like they tell where in the file each instruction’s machine code is. The instruction we care about is at “100000e43”, so it must be 3651 bytes into the file. We only need to change the fourth byte of the instruction, so we can add four to that count to get 3655 bytes.

Using hexdump -C program | less and scrolling ahead a bit, I find a line like this one:

00000e40  00 00 00 c7 45 fc 00 00  00 00 8b 45 fc 48 83 c4  |....E......E.H..|

Sure enough, there’s the instruction, and the seventh byte on this line is the one we want to change. Patching a binary file from the command line is sort of difficult, but this command should do the trick:

printf '\x01' | dd of=program bs=1 seek=3654 count=1 conv=notrunc

dd is writing to the file program (of=program), seeking by one byte at a time (bs=1), skipping ahead 3654 bytes past the first one to land on 3655 (seek=3654), changing only one byte (count=1), and not truncating the rest of the file (conv=notrunc).

Now I’ll run the program the same way we did before (./program) and see if this worked.

Please input a word: butts
That's correct!

Success!

Conclusions

That’s about it. It’s a contrived example, and I knew it would work out before the end, but this is a great way to start learning how programs are compiled, how processors work, and how software cracking happens. The concepts here also apply themselves to understanding how many security exploits work on a mechanistic level.

Why I Don’t “Code”

I don’t really like the terms code or coding used in place of program or programming respectively. I’ve felt this way for a while, and I’ve tried to prefer saying “programming” over “coding” wherever possible.

As a shortened form of source code, “code” and “coding” were probably inevitable. To me, though, I can’t help but hear them as synonyms for cipher. They feel like a subtle call-out to programming-as-esotericism, into which a small elite has been initiated, using its own arcane symbols and language.

It doesn’t help that phrases like “bro down and crush some code” have come into (and hopefully back out of) vogue.

Toward Test-Driven Writing

In programming, we use a methodology called test-driven development. For every piece of a program we write (often called a module), we also write an accompanying suite of test cases. Ideally, you won’t find a single piece of source code that goes untested. It sounds slower, but it actually improves quality so much that it has become (mostly) uncontroversially accepted as a best practice.

Testing a piece (a unit) of code is pretty simple. Each little test asserts an assumed behavior by following a similar pattern. It sets up the environment in which that behavior should happen, runs the relevant unit of the program, and asserts the expected result. You’ll find dozens, hundreds, or thousands of these little tests for a given project, and importantly, no one may change how the project works without making a test case for their change and proving it works, independent of the other parts.

Better yet, to an extent, tests can replace explanatory comments. By encapsulating the expectations for your program, I can better describe a program’s design by illustration. Unit tests therefore serve as documentation, sometimes even the only way to understand a program.

I’m beginning to see how it might be useful to apply the test-driven model to writing. I want my writing to make a strong, believable case, against which readers can make unquestionable assertions. I’ve heard the advice to “show, don’t tell,” and I always thought it meant something like including supporting detail. Maybe that’s not quite right. The advice is not to tell at all, instead of simply backing it up. It tells me both what to write and what not to write. Just like with a unit test, I don’t need to stop and explain.

For example, suppose I want to describe a character who loves another. I might say, “Alice loves Beth.” But that’s clumsy. It sticks out of the story, rather than reading as part of it; I’ve stopped telling the story to impart this premise. Worse yet, “love” is a slippery word—it means something different for each of us. What kind of love is this? Family love? Romantic love?

What if my idea of love falls outside the reader’s experience? That is to say, what do I have to say about love itself? “Alice loves Beth” squanders the opportunity and remains silent on the subject.

In fact, let’s not say “love” in the first place. That word carries a lot of baggage. Maybe I’m not even describing something the reader would ever qualify as love. I might find I’m just bluffing, and the reader only has to call my bluff for the assertion of love to fail.

Suppose instead I write a passage like this.

Taking it down from its hanger at the back of the closet, Alice held Beth’s shirt close and inhaled slowly, afraid that she’d exhaust the source of Beth’s lingering scent. Like a flashbulb, it revealed fleeting memories, which rushed into Alice’s mind’s eye barely long enough to register before slipping away again.

I didn’t say, “Alice loved Beth,” nor did I say, “Alice missed Beth,” but who could question that those statements are true? And there’s more. We know Beth is gone, in some form that allowed her to leave behind a shirt but not much else. A faded scent demarcates a rough idea of the time passed. Something irrevocable happened. A bad breakup? A death? Possibly Alice is moving, or maybe packing up Beth’s belongings. I didn’t even indicate whose closet Alice was in. We do definitely get a sense of some romantic love that could have involved cohabitation, so that distinction may be immaterial.

My point is, it’s not just about how you feel after you read the passage above. It’s more about stating what’s immediate and undeniable, making a case, while leaving out flimsy or even misleading commentary. Within this context, every action in the story fits and drives the narrative forward. The idea of test-driven writing—even though there are no actual tests involved, beyond the reader—helps me better understand what to write, and what not to write.

Self-evaluation

Since reviewing my self-evaluation with my immediate manager the other day, I’ve gone through a lot of thoughts and feelings over it that I would not have expected.

It went really well! That’s the first surprise. This is my first job in the tech field, and so I haven’t been programming for a living very long. I’ve had my current job for perhaps a year and a half, so this was my first full evaluation.

This began as a 360 review that I filled out for myself, and I put down “meets expectations” in every aspect of my a career as a middle- of-the-road response. I genuinely did not think I could justify anything more, and I felt sure I was doing well, so I didn’t feel like I needed particular improvement in any area, either.

My manager put that I “exceeded expectations” in most of the fields and talked with me about it. I fought hard to hold back the urge to disagree with him, but internally, I did keep thinking that somehow he was a bit biased or skewed in favoring me. I wasn’t sure where he got that impression, but I was sure that my evaluation was going to lead to disappointment (either during or after).

The second surprise, however, was how much he won me over to his point of view instead of the other way around. I had had the idea that I was one of the least productive people on the team, but not only did he point out how productive I was, he had pulled the figures to support his view directly from our issue-tracking system. He compared me not only to the rest of his team but to all the engineers in my department, showing me the average time I spent on a task (about half the overall average) and the sheer number of tasks I had turned over in the last year.

I definitely told him how useful it was to see that; otherwise, I just plainly would not have believed him.

Thirdly, the language he used to describe my role in the team surprised me. This is the first time I heard myself described plainly as a “lead” for the rest of the team. I knew that I had had taken some leadership-like roles on (especially training), but it’s only starting to sink in for the first time to what extent that’s happened and how my manager has nurtured that. He’s sought out my opinion on almost every single member of my immediate team to the point where I’ve practically been able to handpick all of them (at the very least, I’ve had veto power, but that isn’t too unusual in tech environments). I’ve been doing a lot of training, pairing, and guiding on tasks our team handles, to the point where sometimes I feel stretched thin.

Of course, I knew about all these things, but I hadn’t internalized myself as anything resembling a leader, and that led to his only negative (but oh, so constructive) feedback for me. I’ve always been a really candid person about my feelings, and he pointed out the extent to which those feelings are influential to the rest of the team when I vent about something negatively.

I suppose I just assume people will disagree with me or that I’ll hold a minority opinion (and often, I do), but the idea that as a leader (whether de facto or de jure) I hold some influence over the opinions of others on my team is a novel one to me.

I’ve always had trouble being professional in past jobs I’ve had, and it’s becoming clearer to me why that’s held me back, and I’ve begun to examine what I say and how I say it in that light.

It’s been a huge shift in how I think about myself, and this review process has generally given me a huge step in internalizing a different idea of myself and my career. I also have to give huge props to my boss, who is maybe right about more things than I gave him credit for in the past.