Writing My First Vim Plugin

Screenshot of Vim editing a file, with a search for the term "fibonacci" in progress. The plugin I wrote has added "3 matches of /\vfibonacci/" to the statusline.
Screenshot of Vim editing a file, with a search for the term “fibonacci” in progress. The plugin I wrote has added “3 matches of /\vfibonacci/” to the statusline.

Last night, I put my first full Vim plugin up on GitHub. It’s called match-count-statusline. It’s intended for anyone who uses Vim to edit text files. It adds a small item to the statusline (Vim’s version of a status bar) describing how many times the current search occurs in the current buffer and, if there’s enough room, what the current search pattern is.

This began as something I needed because I kept opening files and needing to count how many times a term or pattern occurred in that file. I wrote a small script to do it, and I tacked on more things to the script over the next few days and weeks.

Eventually I felt confident enough to share it. I’d also like to share about the process of writing it and preparing it for use by others. If you’re interested, read on.

Who Should Read This?

This guide is meant for advanced Vim users who wish to understand more about the process of writing a plugin through a first-hand narrative.

I won’t be explaining how to use or configure Vim itself, nor the basics of how to write in Vim’s scripting language. I’ll assume familiarity with both. If you don’t know Vim’s scripting language (which I am going to call “VimL” for the rest of this post), I cannot recommend enough Steve Losh’s Learn Vimscript the Hard Way. It is freely available online, and if you find it useful, I encourage you to buy a copy as well.

That book will set you on your way, but it can never include everything you need to know. Vim is always changing, and its own documentation is deep and voluminous. It is for very good reason that Steve sends his readers to Vim’s own documentation at the end of every section—he could not ever hope to include everything in a single book.

I’ll be referring to my plugin as it exists at the time of this writing by commit hash. It is liable to change in the future as I learn how to improve it. The newest version should always be found here, which will incorporate fixes and improvements.

The Problem

I tend to like to open log files in Vim and search around in them to find patterns while I’m investigating a problem at work. I’ve developed over time a lot of tools for doing so. Some of them are crude, but when I find myself doing the same crude thing over and over, I tend to want to polish it.

One common thing I do is discover how often a search pattern occurs in a log. This indicates to me whether something is signal or noise. I’ll place my cursor over something like an ID, press *, and then count how many times it occurs and write that down in another file.

I could do this from the command line using something like grep | wc -l, but I get a lot of value from seeing the whole context and being able to skip between the occurrences of my search, finding out what happened before and after, and so on. I’m very visual. I like to see them in the file.

So to count them, I’d been doing a crude thing. I would run :g//d and watch how many lines got deleted. (Without a pattern specified, that command will delete whatever the current search pattern is.) This has a few drawbacks. First, I had to undelete the lines after I was done. In large files, it was somewhat slow. And finally, it only told me the count of lines, not the count of occurrences.

I figured what I really wanted was simple, and it should be built into Vim.

The Solution

It is not really built into Vim, though. I figured I’d never find it in Vim’s vast documentation, so I searched online and found a Vim Tips article called, “Count number of matches of a pattern.” The very first thing offered in the article was the answer I needed.

:%s/pattern//gn

I knew that when the pattern was already searched for, it could be implied, so this simplified down even more.

:%s///gn

That did it. I ran that after searching for something, and a message printed to the bottom of the screen resembling the following.

2 matches on 2 lines

I could have stopped there.

However, I tend to forget obscure looking commands, and I thought it would be nice if this happened automatically anytime I searched for something. I knew graphical text editors tended to automatically surface information like this in their status bar, and I was inspired to make the same thing happen.

The Solution According to Vim

I probably should not have given up on Vim’s documentation so easily. I didn’t know that the substitution command (:s) was the place to start, but my use of the global command (:g) before might have been a hint.

What I didn’t know is that Vim already internally documents how to count items, and it’s filed under “count-items,” logically enough. You can find it mentioned under the documentation for the substitution section (:help :s, or more specifically, :help :s_flags and see the “n” flag) as well. To find out how this works, run :help count-items. You’ll see it suggests ways to count characters, words, and so on. Generally, it boils down to a very similar command.

:%s//&/gn

Again, this assumes an existing pattern has been searched for. The one difference is the ampersand. The ampersand has a special meaning in the substitution, referring to the thing being replaced. In this case, it would effectively replace a thing with itself, causing nothing to happen. But since the “n” flag is there, no replacement will happen regardless, so there’s effectively no difference if it’s included or not. I have included it in subsequent examples as it is safer (in case the “n” is forgotten).

The Script

So that single, small command forms the core of the functionality I later expanded.

To display it, I learned how to insert an item into vim-airline‘s statusline. That wasn’t too difficult once I found an example online, and you can see how I accomplish that here. Using that, I began a small script in my personal configuration files which added the count when I was searching for a pattern, and otherwise displayed nothing.

Over the next several days, I added on improvements because I found it was slow on larger files. This included caching its results momentarily rather than updating constantly. I also found that each time it updated the search, it would also move the mouse cursor, so I needed to fix that.

The resulting script grew in size and became safer and better performing. Eventually, I figured I’d put enough work into it that it would be safe enough to share with others without breaking their Vim installs.

The Plugin

To create a plugin, I added some documentation, reworked the script to make it configurable, and “licensed” it into the public domain. Then I put it all in a Git repository and hosted it on Github.

I want to spend the rest of this post talking about all the funny little things I did in the core of the script contained in the plugin to build it up from that one substitute hack into more fully fledged functionality.

Below, I’ll be referencing parts of the script itself as contained in the plugin. Each section covers a different topic I applied as I elaborated on the design and improved the plugin so it could be used by more people.

Opening Stanza

The script begins as many plugins do, with an opening stanza ensuring it’s safe to run.

" require 7.4.1658 for v:vim_did_enter
if &compatible
      \ || (v:version <= 704 && !has('patch1658'))
      \ || exists('g:loaded_match_count_statusline')
  finish
endif
let g:loaded_match_count_statusline = v:true

This bit of code makes three checks. First, it checks if &compatible is set. If it is, then there’s no need to check anything else—the script won’t work, and we can exit right here.

Next, it checks for a version of Vim greater than or equal to 7.4.1658 (using Vim’s peculiar syntax for doing so, the predefined variable v:version). That patch, 1658, is the one in which v:vim_did_enter was introduced. (I discovered this by searching the vim_dev mailing list.) That patch was introduced two and a half years ago, so I felt comfortable with that baseline.

Finally, I check to see if my plugin is already loaded using a global variable I will set directly after this condition. If the variable is already set, it means I’ve reached that line before, and I should exit before reaching it again. This prevents loading the script more than once.

The “finish” command ends the script if any of these conditions are true.

The Global Flag

Notice how the above examples of the substitute command all included the “g” flag. That tells the substitution to apply to all matches in a line, not just the first one.

That probably seems like a funny default: only doing one substitution per line and then moving along. It’s just the way regular expressions work from the olden days, and Vi (the editor that Vim is based on) picked up the behavior. Vim never breaks compatibility, so its substitution also needs that “g” flag to apply  substitutions globally.

For that reason, Vim introduced a setting called &gdefault (see :help gdefault). It has the effect of implying a “g” flag on every substitution. Unfortunately, it also means that if you add the “g” flag to a substitution while that setting is on, it undoes the effect. That means you never know what the “g” flag will do unless you know what the setting &gdefault is. That’s not confusing, right?

All of that means that setting &gdefault would effectively break counting if I did not check it. That’s what these lines are all about.

if &gdefault
  let s:match_command = '%s//&/ne'
else
  let s:match_command = '%s//&/gne'
endif

Here, a command to run to search for matches is saved to a variable based on whether &gdefault is set.

There is one minor drawback to doing this here as opposed to dynamically setting it every time the match is calculated. If the user changes &gdefault after loading Vim, the counts will be incorrect when a pattern occurs more than once on a line. This could be a bug! The fix would be to set the match command each time we run a match.

Settings and Defaults

When I decided to release this as a plugin, I realized that I had baked in several decisions about the script that I knew others would disagree with, and I wanted to give them the freedom to configure the script the way they wanted. The next several lines check for global variables which the user may set and then either use their values or use my defaults. Here’s an example of a couple of such settings.

let s:start =
      \ get(g:, 'match_count_start', '')

let s:end =
      \ get(g:, 'match_count_end', '')

To use a variable in Vim, while falling back to a default if it’s undefined, an old way was to use the exists() function. My syntax uses a stupid Vim trick involving variable scoping.

Notice how all the variables I use begin with a letter followed by a colon. These are known in VimL as “internal variables” (see :help internal-variables), and each kind begins with a letter denoting its scope, a colon, and then the name of the variable. (Vim also refers to these scopes as “name spaces” in its documentation.)

Each scope has a special meaning and is treated by Vim in a special way. For example, variables scoped to b: only exist for the one buffer you’re in, and no other buffer can see that variable.

More importantly for our purposes, though, every scope is also considered by Vim to be a Dictionary unto itself, and you can treat all the variables in it as keys containing values. Weird? Yes, weird. But it lets you do things like, “:echo has_key(g:, 'match_count_end'),” which will tell you if that variable is defined.

So in the case of each global setting, I use the get() function, which takes a Dictionary, a key, and optionally also a default value to use in case the key is missing. This allows me to set variables scoped to my script (in s:) based on whether variables in the global scope (in g:) are defined or not. Then the rest of the script uses those script-scoped variables.

This unfortunately means that no changes to the global variables can get picked up while Vim is running. To fix this, I could set all these inside the match counting function I defined, if I chose.

Caching

Caching is one of the earlier things I implemented, and it’s gone through a few revisions. Here’s how it works today.

The basic idea is, the match counting function gets called repeatedly every time the statusline needs to get redrawn, and this happens far more often than the match count itself will change. If the script remembers the match count for each buffer, the match counting function (MatchCountStatusline()) won’t have to recalculate until it’s really necessary, saving itself some work.

So caching is one among many ways MatchCountStatusline() attempts to defer doing any hard work. How does MatchCountStatusline() find out when to update its cache? A couple of ways.

The cache is a simple Dictionary made anew for each buffer. It remembers the last time the cached values were referenced or updated, the number of changes to the current buffer, the current search pattern, and the number of matches.

let s:unused_cache_values = {
      \   'pattern':     -1,
      \   'changedtick': -1,
      \   'match_count': -1,
      \   'last_run':    -1
      \ }

The default values for the cache are ones that could never occur normally, so I call them “sentinel values.” Sentinel values are generally used to represent invalid values, so in this case I can recognize when the cache contains values which have not yet been set by MatchCountStatusline().

When MatchCountStatusline() is first run, if the buffer doesn’t already have a cache, a new one is created with the default sentinel values.

let b:count_cache = 
    \ get(b:, 'count_cache', copy(s:unused_cache_values))

I know that because the cache is scoped to the buffer, each buffer gets a new set of values. On subsequent calls to MatchCountStatusline() for that buffer, it reuses the cache which already exists. (Notice also that I had to copy() the values. If I didn’t, modifications to the buffer’s count cache would modify the script’s count cache defaults and would affect all the other buffers as well. See :help copy.)

A few lines later, MatchCountStatusline() checks to see if the values in the cache match the state of the buffer. If there is a match, there is no need to continue, and the function ends here. Otherwise, we need to set the cache’s values with new ones farther down.

if b:count_cache.pattern == @/
    \ && b:count_cache.changedtick == b:changedtick
  return s:PrintMatchCount(b:count_cache)
endif

It’s a simple check. All I look at here is whether the pattern has changed or the buffer has changed. I don’t have to look at the buffer itself to determine whether it’s changed. A predefined Vim variable increments for every change, called b:changedtick. If it’s changed, I know the buffer has changed. See :help b:changedtick. Those two things let me know if a match count needs to be recalculated for the current buffer.

Notice also that the current search pattern is stored in a register called @/. All registers can be referred to by a name which starts with the @ sigil. There are a bunch of generic registers with single letter names, “a” through “z,” and then there are special-use registers such as @/. It’s easy to see where it got its name, with a little thought—the / key begins a search in Vim, so the @/ register holds the search pattern. Like any register, it may be echoed, compared to, or even assigned to. Any time in my script I want to see what’s being searched for, I inspect @/. For more information, see :help @/ and :help registers. The kinds of registers available will definitely surprise you!


You probably noticed the cache holds two other things besides the number of changes to the current buffer and the current search pattern. The match_count value is the thing we’re caching, so we never care what its value is until we’re printing it out. But last_run deserves an explanation.

I decided early on that I wanted to keep and print out the same cached values for a short period of time, even if the buffer has changed. That’s what last_run is for. It lets me know how long the cache has been holding onto those values so I can know when they do need updating.

Why would I want to keep old values around, even if they don’t reflect reality anymore? It’s not always necessary to reflect reality the very instant it changes. Humans don’t need updates right away for everything—a delay of a tenth or a quarter of a second is tolerable sometimes.

When someone is actively typing, or if they’re entering a new search (and &incsearch is enabled), every single keystroke would invalidate the cache and trigger a new calculation. If we instead wait for a fraction of a second, we can allow the user to type a few characters before updating. By the time they look down, we’ve probably already updated, but we won’t have wasted time needlessly updating for every keystroke.

The function responsible for determining if enough time has passed before even checking the cache is called s:IsCacheStale(). (Notice that it’s script-scoped, so that it doesn’t collide with other scripts’ functions.) It comes before the cache check in the program, so it “short-circuits” that logic, precluding it when it’s not needed. The function s:IsCacheStale() has a very simple mandate—it receives the cache and checks how much time has gone by. If enough time has passed, it returns a true value. Otherwise, it evaluates to false. How it implements this is a little complicated.

I won’t copy in the whole body of the function here, but go glance over it if you want. There are a couple of weird things going on, and I’ll touch on them here.

First, instead of using ones and zeros to represent true and false, I’m using Vim’s predefined values of v:true and v:false. Vim introduced these in 7.4.1154 to help with JSON parsing. They evaluate to 1 and 0 respectively, so I’m using them to make the script more readable.

Next, I put in a little stanza at the beginning to optimize when loading a new buffer.

if a:count_cache == s:unused_cache_values
  if has('reltime')
    let a:count_cache.last_run = reltime()
  else
    let a:count_cache.last_run = localtime()
  endif

  return v:false
endif

The very first time that MatchCountStatusline() gets called for a buffer, recall that it puts in the sentinel values for the buffer’s count cache. Here, I detect whether or not that’s the case. If it is, I merely return false (meaning that the cache is not stale), but I also importantly update the last_run cached value with the current time, so that this condition never evaluates to true in the future for this buffer.

What this accomplishes is that the first time a buffer is loaded, the cache always appears to be stale (even though it really only contains invalid values). This serves to cause no match counting to occur for the first cache timeout period (by default, a tenth of a second) after a buffer gets loaded. After that, the first match count occurs. This lets the buffer get up on its feet and lets other plugins and automatic commands do things before we start.

Finally, both here and further down, I test whether the “reltime” feature exists in Vim and use it only if it does. Otherwise, I fall back to using the localtime() function. There are a handful of other places I’ve checked for functionality before using it. This is one of the ways I’ve tried to make the plugin safer for a wider audience. I imagine most people will have a Vim with “reltime” available, but I couldn’t tell how widely available it would be.

The cache check algorithm is pretty simple otherwise. It checks the current time against the last_run time. If it’s greater than or equal to the cache timeout, last_run gets updated, and the function returns true (meaning the cache is stale). Otherwise it returns false, leaving last_run alone so that the next time it’s called, it will be clear how much time has passed since the last time the cache was stale.

Both by giving the cache a little time before it’s updated, and by checking its values, MatchCountStatusline() avoids updating the cache (and running the expensive match-counting calculating) whenever possible.

Another Optimization: v:vim_did_enter

One of the more recent Vim features I took advantage of was the v:vim_did_enter predefined variable (see :help v:vim_did_enter). It allows MatchCountStatusline() to know if it’s being called before Vim has even loaded all the way, and so it’s the very first check. This means that the cache isn’t even warmed up before Vim has properly started up, so the cache grace period kicks in after. This allows Vim to start up faster.

" don't bother executing until Vim has fully loaded
if v:vim_did_enter == v:false
  return ''
endif

File Size Checking

I discovered early on that match counting works really poorly for large files. I tried throwing in a couple of optimizations for the counting itself (I’ll mention those below), but the counting still dragged, and so I assumed it would never be fast enough to work transparently and interactively.

Then I considered whether I could do the counting asynchronously. I eventually ruled this out for now because it means shelling outside of the Vim process, as near as I can tell. I wanted my script to be purely VimL and self-contained.

So I settled on checking for whether the size of the file exceeds a certain size and stopping there. The function responsible for checking is called s:IsLargeFile().

function! s:IsLargeFile(force)
  if a:force
    return v:false
  else
    if getfsize(expand(@%)) >= s:max_file_size_in_bytes
      return v:true
    else
      return v:false
    endif
  endif
endfunction

Notice that it takes an argument called force. This allows MatchCountStatusline() to tell it to skip checking the file size and just report that the file is safe in all cases. Otherwise, this one is very straightforward.

Toggling Match Counting

I realized that sometimes, even for large files, I still wanted a count, so I implemented a command which would manipulate two buffer variables to force match counting to occur (or would disable match counting altogether). The logic here is a bit weird because of the file size checking. I wanted to be able to force things on if they’re turned off due to the file size, and I wanted to be able to toggle things normally otherwise.

You can review the function which implements the toggling, but I won’t cover it in more detail here.

Optimizing Match Counting Itself

We’ve covered several preliminary checks that MatchCountStatusline() makes, and we’re down to the nitty gritty—updating the cache and then calling s:PrintMatchCount() to render the statusline.

When MatchCountStatusline() sees there’s no pattern for the current buffer, its job is easy—it clears the cached values, and that’s it.

" don't count matches that aren't being searched for
if @/ == ''
  let b:count_cache.pattern     = ''
  let b:count_cache.match_count = 0
  let b:count_cache.changedtick = b:changedtick
[...]

Otherwise, we have work to do.

(Incidentally, there’s a small optimization I forgot to make here. See it? We’re updating the cache. We can set last_run here and keep these cached values for longer. We should do this here and anywhere below where we update the cache.)


Since we’re now in the part of the script involves the actual calculation, and since it has the chance of raising exceptions, I begin a try/catch/finally block here. It’s important to begin it here, when I start to change the state of the buffer, so that I know my changes will get fixed up by the finally block below. The worst case scenario would be attempting to count the matches, having an error occur, and leaving the user’s editor in a broken state.

Now we’re free to begin mucking about. First thing I do is save the current view.

" freeze the view in place
let l:view = winsaveview()

In the finally block, the last thing I do is restore this view.

call winrestview(l:view)

This ensures that nothing visibly changes while I do what I’m about to do. (See :help views-sessions, :help winsaveview, and :help winrestview.) One problem I had was that the core match count command (s//&/gne) would cause the cursor to jump to the beginning of the line. Saving and restoring the view fixes that problem.

Next, I disable automatic commands and highlight search. Likewise, I re-enable those later on. Because these are compile-time features, I check that I can do so before I do. I save off their values as local variables so that I can restore them as they were before I touched them.

By turning these features off, Vim won’t attempt to fire off events during the fake substitution it’s about to run, which can save a little time. Turning off &hlsearch in particular is one of Vim’s own suggestions in :help count-items.

Match Counting

Finally, I perform the actual count.

" this trick counts the matches (see :help count-items)
redir => l:match_output
silent! execute s:match_command
redir END

I added a comment to help others understand how this worked because it’s probably the least obvious part of the program.

The redir here sends all the output from that command to the l:match_output local variable. The actual command uses echom to display a message with the results, like I showed above, something like “2 matches on 2 lines“. If no matches are found, the output is empty due to our use of the n flag.

Then the string is parsed, and the cache is updated with the new match count and other values. The finally block restores things how they were, as I mentioned earlier. And at the very end of the MatchCountStatusline() function, the cache is rendered into a string by s:PrintMatchCount().

What Else?

There are several things I didn’t cover here, like how I implemented the global customizations. I might cover those in a future post if there’s interest, but they’re straightforward enough if you followed the above.

Making it appear in the statusline was interesting, but it’s more of interest to vim-airline users only, probably. Feel free to look at the stanza which attempts to guess how to do that, though.

Do also check out the documentation for my plugin, which consumed most of the time it took to get it ready for wider distribution.

I hope that you found what you were looking for in this post and took away at least one useful thing!

Pandora’s Checkbox

The Information Age brought with it a cliché—that unread agreement you dismiss to get to the software you need to use. There’s no way you’re going to read it. For example, macOS High Sierra comes with a software license agreement totaling 535 pages in PDF form, which contain (by my count) 280,599 words of intensely detailed yet maddeningly vague legal language. On that operating system, Apple Music has another license, and the App Store has yet another, and so on.

It would take thousands of dollars in consulting fees with a lawyer to make a fully informed decision, or you can proceed regardless. So you proceed. You always have. Each little app, website, or gizmo peppers you with a new set of terms and conditions. Each upgrade gets a few extra clauses thrown in, and you agree again.

You’re not a fool. You assume you’re signing away rights and control you want. It comes in the bargain. You try to skim the terms and conditions, and this deal feels a bit more Faustian all the time—mandatory binding arbitration, data collection, disclaimers of liability, and so on.

None of this is really news to you if you’ve dug into it. You’re not really in possession of your software; you’ve merely licensed the use of it. You can’t really hold them responsible for flaws; you agreed to accept the software as is. You can’t really control what information they collect about you; you hand that over and get a free or discounted product in return.

However, where things get slippery is that a company with whom you’ve entered into a transaction has also signed agreements with yet other companies. Worked into those overwrought terms and conditions you clicked through, with their vague-yet-precise language, are ways of ensuring that you’ve already agreed to these subsequent proxy agreements as well.

What the T&C often allow is for your data to commingle at some broker whose name you’ve never heard of. A common situation in which this happens is when any entity responsible for handling money.

Say that you learn about a subscription service called Company A. You find them in your web browser or your mobile app, and you sign up, agreeing to their T&C. Then you ask to subscribe to a new e-mail about scarves every day, or whatever Company A does. They in turn ask for your credit card info, your billing address, and maybe a few other demographic details about you.

Company A turns to Company B to determine how risky you are. To do this, they ship off some information about you. If you used a mobile app, they’re possibly reading off what Wi-Fi networks are nearby, what Bluetooth devices are nearby, what apps are installed on your phone, what IP addresses you’re using, what fonts you have installed, and a wealth of other information. If you’ve used a browser, the information is similar but more limited. You’re being geographically located in either case. The headers from your browser are sent. The last website you were at before visiting Company A is probably sent.

Company B collects this information and compares it to all the other data it has on millions of other requests it’s collected from other companies. It has no real duty to sequester Company A’s data from Company Z (neither of which know anything about one another), and by putting it all together, it can detect patterns better. For example, it may have the ability to know where you are, even if you are behind a proxy. It may be able to track your traffic across the Internet as you move from Company A to Company Z and so on—because the number of details it gets are enough usually to uniquely identify you. It needs no cookies or other storage on your end for this.

This means that Company B has the role of an invisible data broker whose job it is to assess fraud risk on behalf of companies. The more clients it has feeding it data, the stronger its signals become, so Company B is incentivized to gather as many sources of data as possible, and it wants those data to be as rich and as frequently updated as possible.

Company A gets back something like a score from Company B indicating how much risk you pose—whether or not you’re likely to try to scam them out of free services (or if you’re even a human or not). Assuming you’re fine, then Company A sends your info off to Company C, a credit card processor who is the one actually responsible for charging you money and giving it back to Company A.

Company C is collecting data as well because they stand the greatest risk during this transaction. They collect data themselves, and they’re almost certainly using a data broker of some kind as well—either Company B or more likely something else, a Company D.

These interactions happen quite quickly and, usually, smoothly. In a few seconds, enough info about you to identify your browsing patterns and correlate you with your purchase of Scarf Facts has now been aggregated by one or two data brokers.

These brokers sell their services to companies hoping to prevent fraud, and they make money because they are able to draw from ever larger sources of traffic and gain a clearer picture of the Internet. You agreed to this, but I doubt it was clear to you that entities other than you and Company A were involved.

If you’re wondering whether or not this is really happening, this sort of collection has become increasingly common as businesses have tried to compete with one another by reducing friction around their sign-up processes. Simple CAPTCHAs have not been enough to hold back the tide of automated and human attempts to overwhelm large and small businesses attempting to sell services and goods online, and they have turned to data-based solutions to fight back. We can’t wind back the clock to a simpler time.

Unfortunately, most people are uninvolved and have become bycatch in the vast nets we’ve spun. It is likely, as time goes on, that the brokers who collect and analyze the data collected this way will try to sell them, or analyses of them, to profit in other ways. The value of these data increases as they become more representative of the traffic of the Internet as a whole.

I’m not asking you to stop and read the T&C on the next website you sign up for. That’s ever going to be practical. But now you know about another piece of your soul you’re possibly chipping off in return for clicking “Accept.”

Math, She Rote

My friends often have different educational backgrounds than mine. Some of them are younger, but even if they aren’t, they’re often from urban areas that had moved to more modern educational curricula before my school system had. The way I learned basic arithmetic remained unchanged from how it was taught from the early 1980s by the time I learned it in the late 1980s and early 1990s because that’s when our books dated from.

I learned during an interesting period in mathematical education history. It represented a kind of educational interbellum—a bit after the “New Math” of the 1960s and 1970s but before the “math wars,” instigated by the 1989 Curriculum and Evaluation Standards for School Mathematics. The latter 1989 publication has been called “reform mathematics,” which emphasizes processes and concepts over correctness and manual thinking. In other words, the educators promoting reform mathematics began to believe that the path students took toward the answer mattered more than whether they got the answer right. Many states’ standards and federally funded textbooks followed reform mathematics in the 1990s and beyond.

Reform mathematics emphasized constructivist teaching methods. Under this approach, instead of prescribing to students the best way how to solve a problem, teachers pose a problem and allow the student to surmount it by building on their own knowledge, experiences, perspective, and agency. The teacher provides tools and guidance to help the student along the way. Constructivist approaches involve experiments, discussions, trips, films, and hands-on experiences.

One example of a constructivist-influenced math curriculum, used in elementary school to teach basic arithmetic, was known as Investigations in Numbers, Data, and Space. It came with a heavy emphasis on learning devices called manipulatives, which are tactile objects which the student can physically see, touch, and move, to solve problems. These are items like cubes, spinners, tapes, rulers, weights, and so on.

As another example, someone I know recently described a system they learned in elementary school called TouchMath for adding one-digit numbers, which makes the experience more visual or tactile (analogous to manipulatives). They explained that for each computation, they counted the “TouchPoints” in the operands to arrive at the result.

I had never heard of TouchMath. In fact, I never solved problems using manipulatives, nor any analogue of them. I had little experience with this form of math education. We were given explicit instructions on traditional ways to solve problems (carrying, long division, and so on). Accompanying drawings or diagrams rarely became more elaborate than number lines, grids, or arrangements of abstract groupings of shapes which could be counted. They served only as tools to allow students to internalize the lesson, not to draw their own independent methods or conclusions.

I contrasted my friend’s experience with TouchMath to my experience. To add or subtract one-digit numbers, we merely counted. We were given worksheets full of these to do, and since counting for each problem would have been tedious and impractical, memorization for each combination of numbers would become inevitable. Given the expectations and time constraints, I’m certain rote memorization was the goal.

In a couple of years, we were multiplying and dividing, and we were adding and subtracting two- or three-digit numbers using carrying—processing the numbers digit-wise. At the same time, we were asked to commit the multiplication tables to memory. These expectations came in third grade, and it would be nearly impossible to make it out of fourth grade without committing the multiplication table and all single-digit addition and subtraction to memory (the age of ten for me).


Our teachers did not bother to force us to memorize any two-digit arithmetic operations. But I have some recollection a lot of years ago of my grandma telling me she had most two-digit additions and subtractions still memorized. It was just an offhand remark—maybe something she said as I was reaching for a calculator for something she had already figured out. Maybe we were playing Scrabble.

For context, she would have gone to school in rural Georgia in the 1940s and 1950s, and she graduated high school. (In that time and place, it was commonplace for many who intended to do manual, trade, or agricultural work not to continue through secondary school.)

I remember feeling incredulous at the time about the number of possible two-digit arithmetic operations that would imply memorizing. Of course, many would be trivial (anything plus or minus ten or one, or anything minus itself); others would be commonplace enough to easily memorize, while still others would be rare enough to ignore. But that still leaves several thousand figures to remember.

The more I thought about it, the more I saw that, in her world, it would make better sense to memorize literally thousands of things rather than work them out over and over. She had no way of knowing that affordable, handheld calculators would exist in a few decades after she graduated from school, after all. Each time she memorized a two-digit addition or subtraction, she saved herself from working out the problem from scratch over and over again for the rest of her life. This saved her effort and time every time she

  • balanced her checkbook,
  • filled out a deposit slip at the bank,
  • calculated the tax or tip on something,
  • tallied up the score for her card game,
  • totaled up a balance sheet for her business,
  • made change for a customer, or
  • checked that the change she was being given was correct,

to say nothing of all the hundred little things I can’t think of. She married young and has run small businesses for supplemental income all her life, so managing the purse strings fell squarely into her traditional gender role. Numbers were part of her daily life.

So for the first half of her life, none of this could be automated. There were no portable machines to do the job, and even the non-portable ones were expensive, loud, slow, and needed to be double-checked by hand.

I don’t believe she remembered these all at once for a test, the way I learned the multiplication tables in third grade. It seems likely she memorized them over time. It’s possible that expectations in her school forced a lot of memorization that I didn’t experience when I went many decades later, but maybe she was just extra studious.


I recall, as I went through school, having to rely more on a calculator as I approached advanced subjects. Before calculators became available to students, appendices of lookup tables contained pre-calculated values for many logarithms, trigonometric functions, radicals, and so on. Students relied on these to solve many problems. Anything else—even if it were just the square root of a number—came from a pen-and-paper calculation. (Many of my early math books did not acknowledge calculators yet, but this changed by the 1990s.)

Charles Babbage reported that he was inspired to attempt to mechanize computation when he observed the fallibility of making tables of logarithms by hand. He began in the 1820s. After a hundred and fifty years, arithmetic computation would become handheld and affordable, fomenting new tension around what rote memorization plays in both learning and in daily life.

Today, we’re still trying to resolve that tension. Memorization may feel like it has a diminished role in a post-reform education environment, but it’s by no means dead. Current U.S. Common Core State Standards include expectations that students “[b]y end of Grade 2, know from memory all sums of two one-digit numbers,” and, “[b]y the end of Grade 3, know from memory all products of two one-digit numbers.” That sounds exactly like the pre-reform expectations I had to meet.

All this means is that there has been neither a steady march away from rote memorization nor a retreat back to it. Research is still unclear about what facts are best memorized, when, or how, and so there’s no obvious curriculum that fits all students at all ages. For example, the Common Core Standards cite contributing research from a paper which reports on findings from California, concluding that students are counting more than memorizing when pushed to memorize arithmetic facts earlier. The paper reasons this is probably due to deficiencies in the particulars of the curriculum at the time of the research (2008).


I’m not an expert, and I don’t have easy answers, but my instinct is that rote memorization will always play an inextricable role in math education.

Having learned about the different directions in which the traditional and reform movements of math education have tugged the standards over the years, I tend to lean more traditional, but I attribute this to two things. One is that I was educated with what I remember to be a more traditional-math background, and though I didn’t like it, it seems serviceable to me in retrospect.

The other reason is that, for me, memorization has always come easily. I don’t really know why this is. It’s just some automatic way I experience the world. Having this point of view, though, I can easily see how beneficial it is to have answers to a set of frequent problems ready at hand. It’s efficient, and its benefits never cease giving over time. The earlier you remember something, the more it helps you, and the better you internalize it. Even for those who can’t remember things as easily, the returns on doing so are just as useful.

I do completely agree with the underlying rationale of the constructivist approach. Its underpinnings are based on Piaget’s model of cognitive development, which is incredibly insightful. It seems useful to learn early to accommodate the discomfort of adapting your internal mental model to new information by taking an active role in learning new ideas in order to surmount new problems.

I don’t necessarily believe that a constructivist learning approach is intrinsically at odds with rote memorization—that is to say, that memorization necessarily requires passive acquisition. In fact, the experience of active experimentation and active role may help form stronger memories. It’s more likely they compete in curricula for time. It takes longer to mathematically derive a formula for area or volume by independent invention, for example, than to have it given to you.

In fact, constructivist learning works better when the student has a broader reservoir of knowledge in the first place from which to draw to begin with when trying to find novel solutions to problems. In other words, rote memorization aids constructivist learning, which then in turn aids remembering new information.

My feeling is that math will always require a traditional approach at its very heart to set in place a broad foundation of facts, at least at first, before other learning approaches can have success. Though the idea of critical periods in language acquisition has detractors and heavy criticism, there is a kernel of truth to the idea that younger minds undergo a period of innate and intense linguistic fecundity. Maybe as time goes by, we can learn more about math acquisition and find out which kinds of math learning children are more receptive to at which ages. Until then, I feel like we’re figuring out the best way to teach ourselves a second language.

I am grateful to Rachel Kelly for her feedback on a draft of this post.

Privacy Policy Updates: Data Storage

I updated WordPress today to version 4.9.6. I noticed this version comes with support for implementing privacy policies throughout the site. I seem to have been ahead of the curve in implementing my own, but when the GDPR in the EU comes into effect this month, it will clarify and simplify data privacy for much of Europe. This implies enforcement will become a more direct matter as well. Any web service accessible to Europe and which does business in Europe now has updated their privacy policies to ensure it complies with the GDPR—which is why everyone has gotten a raft of privacy policy updates.

Most of these privacy policy updates pertain to what rights customers or users have to their own data. Often, they grant new rights or clarify existing rights. This week’s new version of WordPress is yet another GDPR accommodation.

Today, I have to announce my own GDPR update. Yes, I’m just a tiny website no one reads, and I provide no actual services. But having already committed to a privacy policy, which I promised to keep up to date (and announce those changes), I’m here to make another update.

One nice thing that came with the the WordPress update is a raft of suggestions on a good privacy policy (and in what ways WordPress and its plugins may cause privacy concerns). I found that I had covered most of them, but one thing I needed to revisit was a piece of functionality in Wordfence.

I use Wordfence for security: It monitors malware probes and uses some static blacklists of known bad actors. It also, by default, sends cookies to browsers in order to track which users are recurring ones or which users are automated clients. The tracking consisted only of an anonymous, unique token which distinguished visitors from one another. Unfortunately, this functionality had no opt-out and did not respect Do Not Track.

Although my tracking was only for security purposes—not for advertising—and although did not store any personal information, nor did I share with anyone else, I realized I would have to disable it.

I had made explicit mention of this tracking in my previous revision of my privacy policy:

I run an extra plugin for security which tracks visits in the database for the website, but these are, again, stored locally, and no one has access to these.

This is unfortunately more vague than it should have been, since it doesn’t mention cookies. It also provides no provision for consent. It merely states the consequences of visiting my site.

The GDPR makes it clear that that all tracking techniques (and specifically cookies) require prior consent. Again, I’m not a company, and I don’t provide any service. I’m not even hosted in the EU’s jurisdiction. My goal, though, is to exist as harmoniously with my visitors as possible, whomever they may be, and have the lightest possible touch.

So I’ve disabled Wordfence’s cookie tracking. I’ve added a couple of points to my privacy policy which clarify more precisely which data is logged and under which circumstances cookies may be sent to the browser.

This interferes my analytics, unfortunately—it’s no longer possible to be sure which visitors are humans anymore. I think it’s worth it, regardless.

I also made a couple of other changes based on WordPress’s suggestions. I moved a few bullet points around to put some points closer together which feel more logically grouped. I also added a point which specifies which URL my site uses (meaning the policy would be void if viewed in an archived format, within a frame, or copied elsewhere).

Adding a Privacy Policy

I’ve decided to give my website a privacy policy. It’s maybe more of a privacy promise.

It might sound strange to make a privacy policy for a website with which I don’t intend users to interact, but I’ve realized that even browsing news websites or social media has privacy implications for users who visit them. So I wanted to state what assurances users can have when visiting my website—and set a standard for myself to meet when I make modifications to my website.

Most of the points in it boil down to one thing—if you visit my site, that fact remains between you and my site. No one else will know—not Google, not Facebook, not your ISP, not the airplane WiFi you’re using, not some ad network.

I went to some trouble to make these assurances. For example, I had to create a WordPress child theme which prevents loading stylesheets associated with Google Fonts used by default. Then—since I still wanted to use some of those fonts—I needed to check the licensing on them, download them, convert them to a form I could host locally, and incorporate them into a stylesheet on my own server.

I also needed to audit the source code for all the WordPress plugins I use to see what requests they make, if any, to other parties (and I’ll have to repeat this process if I ever add a new plugin). This was more challenging than I realized.

I needed to ensure I had no malware present and that my website remain free of malware. I began with WordPress’s hardening guide. I found a very thorough plugin for comparing file versions against known-good versions (WordFence, which I found recommended in the hardening guide). I also made additional checks of file permissions, excised unused plugins, made sure all server software was up to date, and incorporated additional protections into the web server configuration to limit my attack surface.

Finally, I had to browse my website for a while using my local developer tools built into my browser, both to see if any requests went to a domain other than my own and to inspect what cookies, local storage, and session storage data were created. This turned up a plugin that brought in icons from a third party site, which I had to replace.

After all that, I feel sure I can make the assurances my privacy policy makes.

Our Shitposts Only Feed the Beast

We’re drowning each other out with shitposts, and I’m starting to suspect Twitter encourages it.


I literally have no idea what’s going on in my friends’ lives anymore because there are so many posts to wade through. Twitter gave up on the firehose approach of showing us everything, and now it tries to curate for us, but its algorithms have narrowed my age down to somewhere between thirteen and fifty-four, and it thinks I’m interested in—not kidding—dads.

To get back my firehose, I use Tweetbot. I just took a quick estimate of my extensive mute list (which I personally curate), and it weighs in at over eight hundred mutes at this point. The vast proliferation of image posts, a workaround for the strict character limit Twitter imposes, has made these mutes almost worthless, so I’ve had to mute entire people. Occasionally Tweetbot freezes when I mute a person who’s particularly prolific.

What am I muting?

  • Laborious, overwrought, played out jokes. (But usually these are spread in images, so I have to mute people. Sometimes they are blessedly hashtags.)
  • Conferences I’m not attending due to health reasons. (But often the conference has no official hashtag, and—in the case of Google I/O this year—I muted something like five hashtags, three people, and Google itself.)
  • People who repeatedly retraumatize me by putting violence, threats, horrific news and images, extensive and voluminous exegeses of injustice and hate, and soul-rending reminders of hatred (much of which is aimed at me) each and every day.
    • There is little I can do but mute these people entirely. Though they often need dozens or hundreds of tweets to spread their message, and though Twitter is itself a centralized and proprietary platform, they do not use any long-form, self-owned medium to promulgate their message. Why?
    • This is an extremely delicate and controversial point, I know. The anger and sheer revulsion at our world right now come in involuntary, peristaltic waves sometimes. And it’s hard to know who’s reaching whom with what message. Twitter itself bears a lot of blame for giving no one the tools for finer filtering of content. Rarely does this stuff come in hashtagged formats that I can selectively mute. There’s no way to exclude a single tweet or a thread for exclusion.
  • One-off news stories or other events.
    • In 2016, each celebrity death garnered a mute. I share in the psychic pain each caused, but each person’s reaction flared it anew, and it’s not that each person had one reaction, but some repeatedly brought it up for days.
    • In 2017, each news story echoes for hours over dozens or hundreds of tweets, despite every mute on the subject matter I can put up. Much of it is speculations or jokes.
    • Movie releases, sports events, galas and parties, press events, and a million other things I am literally not healthy enough to properly participate in, enjoy, or motivate myself to find interest in.
  • Downright awful, hateful stuff that my friends ought to know better than to share but just don’t.
    • “Drumpf” jokes, fat jokes, or intelligence jokes about Trump. Of all the fertile material (his malice, his incompetence, his apathy) to dredge up, why choose to band with the people whom we should be resisting? Why pile on his typos, ridicule his body, mock his lack of social grace, point out his unsophisticated food choices, or ride his faux pas? When you make fun of him for something, I assume you have forgotten people like myself who are suffering under his administration. Never forget what he is and what he has done.
    • Transphobic shit and people who are on my shit list for it: Erika Moen, Margaret Cho, RuPaul’s Drag Race, Tyra Banks, etc.
  • Shit I just can’t handle (e.g., horrific prison conditions), or other specific situations and people: “triggers”. Nothing anyone can do about this. I mute it the best I can.

I recognize that this problem can be read as mine rather than Twitter’s. My thought is, fewer tweets altogether comes out to higher value for each tweet. So I try to restrain myself a bit, though I’m not always successful.

But here’s the whole damnable hitch: the more restraint I show, the more likely whatever fewer tweets I do emit get lost in the noise. Or, alternatively: the more I value a tweet, the fewer people who will probably see it. And it’s cyclical. Someone who only tweets when it really counts for them might get fewer followers in the first place and will be more likely to have their tweet drown in the ocean.


I recently wrote close to seven thousand words about my astrophotography hobby. Then I shared it in a tweet since a blog post is a rather dormant thing on its own. I’ve been sharing about astrophotography for a little while, so I usually share on Twitter. It did get a decent amount of engagement, but I discovered something strange happening afterwards. I noticed after a while there were still swaths of friends who had never seen any of my tweets about the subject at all.

Haven’t they seen any of my tweets over the last year about it? Any of the photos? Any of the posts I’d written on this site and then shared? No. None of that, they would say. They didn’t even know I had a website.

If this had happened just once or twice, I might’ve dismissed it. But this has happened repeatedly. These tweets just get lost somehow. If it’s not a marathon thread, or tweeted at the precise right moment, or retweeted by the right person, or some other magical thing I haven’t found, then it gets mislaid, I guess. I’m not sure what’s going on.


Or, maybe I do. Twitter turned off their firehose last year. Facebook did years ago. This game went pay-to-play. Either you’re already a person who drives a lot of engagement, who gets visibility, or you pay for the same.

Except, it would be super clumsy to literally have people pay to get their tweets seen. That’s just an ad, and it’s going to look like an ad, and nobody wants to click an ad, right?

But, like, right now, some people are living as ads. They drive particular kinds of traffic, specific kinds of engagement. They don’t look like ads. They target niches with surgical precision. They do this by churning out bulks of, more or less, “pulp” tweets. Each drives more engagement and synergistically works with the others.

It doesn’t particularly matter what they post. Could be they shitpost a very specific thing that a very specific set of weird Twitter just really likes. Maybe @dril is an ad. Maybe you’ve seen more of @dril retweets than mine.

And once there’s a massive, captive audience, there’s potential for…something. Analyzing those people’s interests or behaviors? Subtly linking a video that happens to have an ad? “Yvan Eht Nioj”? I don’t know.


If this whole profit motive part of my post seems vague, it’s because I’m speculating on the mechanism. I’ve veered off into a conspiracy theory. I have friends who assure me I’m wrong, that I’m attributing to malice what is really staggering incompetence. Nevertheless, it’s likely Twitter will soon learn to capitalize upon making some people more visible than others. This is a thing Facebook has already done.

And indeed, this fact is beside my actual point. More to that point, our intemperate shitposting has abetted this imbalance of visibility and enables the profit potential. It justifies the algorithmic curation, and the rest—pay-for-views, filter bubbles, propaganda, outright abuse—follows from there.

I see no easy way to turn it back. It is what it is.

Minimizing Your Trust Footprint

I originally published the following last year as an article in The Recompiler, a magazine focused on technology through the lens of feminism. I present it here with as few modifications from its original publication as possible.


For everyone who chooses to engage with the Internet, it poses a conflict between convenience and control of our identities and of our data. However trivially we interact with online services—playing games, finding movies or music, connecting to others on social media—we leave identifying information behind, intentionally or not. In addition, we relinquish some or even all rights to our own creations when we offer our content to share with others, such as whenever we write on Medium.

Most of us give this incongruity some cursory thought—even if we don’t frame it as a conflict—such as by when we set our privacy settings on Facebook. With major data breaches (of identifying, health, financial, or personal info) and revelations of widespread, indiscriminate government surveillance in the news over the last few years, probably more of us are thinking about it these days. In some way or another, we all must face up to the issue.

At one extreme, it’s possible to embrace convenience completely. Doing so means handing over information about ourselves without regard for how it will be used or by whom. At the other extreme, there’s a Unabomber-like strategy of complete disconnection. This form of non-participation comes along with considerable economic and social disenfranchisement.

The rest of us stride a line between, maybe hewing nearer to one extreme or another as our circumstances allow. This includes me—and as time passes, I am usually trying to exert more control over my online life, but I still trade off for convenience or access. I use an idea I call my trust footprint to make this decision on a case-by-case basis.

For example, I realized I began to distrust Google because the core of their business model is based on advertising. I wrote a short post on my personal website about my motives and process, but to sum up, I didn’t want to be beholden to a collection of services that made no promises about my privacy or their functionality or availability in the future. I felt powerless using Google, and I knew this wouldn’t change because they have built their empire on advertising, a business model which puts the customers’ privacy and autonomy at odds with their success.

Before I began to distrust Google, I didn’t give my online privacy or autonomy as much thought as I do today. When I began getting rid of my Google account and trying to find ways to replace its functionality, I had to examine my motives, in order to clarify the intangible problem Google posed for me.

I concluded that companies which derive their income from advertising necessarily pit themselves adversarially against their customers in a zero-sum game to control those customers’ personal information. So I try to avoid companies whose success is based on selling the customer instead of a product.

Facebook, as another example, needs to learn more about their users and the connections between them in order to charge advertisers more and, in turn, increase revenue. To do so, they encourage staying in their ecosystem with games and attempt to increase connections among users with suggestions and groups. As noted in this story about Facebook by The Consumerist last year:

Targeted ads are about being able to charge a premium to advertisers who want to know exactly who they’re reaching. Unfortunately, in order to do so, Facebook has to compromise the privacy of its hundreds of millions of users.

Most social networks engage in similar practices, like Twitter.

Consequently, my first consideration when gauging my trust footprint is to ask who benefits from my business: What motivates them to engage with users, and what will motivate them in the future? This includes thinking about the business model under which online services I choose operate—to the extent this information is available and accurate, of course.

Of course, this information often isn’t clear, up front, available, or permanent, so it’s really a lot of guessing. The “trust” part is quite literal—I don’t actually know what’s going to happen or if my information will eventually be leaked, abused, or sold. Some reading and research can inform my guesses, but they remain guesses. I don’t trust blindly, but it is still something of an act of faith.

It’s for that reason my goal isn’t to completely avoid online services or only use those who are fully and radically transparent. I only want to minimize the risk I take with my information, to reduce the scale of the information I provide, and to limit my exposure to events I can’t control.

The second consideration I make in keeping my trust footprint in check is to question whether a decision I make actually enlarges it. For instance, when I needed a new calendaring service after leaving Google, I realized that I could use iCloud to house and sync my information because I had already exposed personal information to iCloud. I didn’t have to sign up for a new account anywhere, so my trust footprint wasn’t affected.

The tricky part about that last consideration is that online services have tendrils that themselves creep into yet more services. In the case of Dropbox, which provides file storage and synchronization, they essentially resell Amazon’s Simple Storage Service (AWS S3), so if you don’t trust Amazon or otherwise wish to boycott them, then avoiding Dropbox comes along in the bargain. The same goes for a raft of other services, like Netflix and Reddit, who all use Amazon Web Services to drive their technology.

That means it’s not just home users who are storing their backups and music on servers they don’t control. Whether you call it software-as-a-service or just the “cloud,” services have become interconnected in increasingly techological and political ways.

It doesn’t end with only outsourcing the services themselves. All these online activities generate vast amounts of data which must be refined into information—for which there is copious value, even for things as innocuous as who’s watching what on TV. Nielsen’s business model of asking what customers are watching has already become outdated. Nowadays, the media companies know what you watch; the box you used to get the content has dutifully reported it back, and in turn, they’ve handed that data over to another company altogether to mine it for useful information. This sort of media analytics has become an industry in its own right.

As time passes, it will become harder to avoid interacting with unknown services. Economies of scale have caused tech stacks to trend more and more toward centralization. It makes sense for companies because, if Amazon controls all their storage, as an example, then storage becomes wholly Amazon’s problem, and they can offer it even more cheaply than companies which go out and build their own reliable storage.

Centralization doesn’t have to be bad, of course. It’s enabled companies to spring up which may not have been viable in the past. For example, Simple is an online bank which started from the realization that to get started with an entirely new online bank, “pretty much all you need is a license from the Fed and a few computers.”

The upshot is that the process of managing your online life to be entirely within your control becomes increasingly fraught as centralization proceeds. When you back up to “the cloud,” try to imagine whether your information is sitting on a hard disk drive in northern Virginia, or maybe a high-density tape in the Oregon countryside.

It’s not even necessary to go online yourself to interact with these business-to-business services. Small businesses have always relied upon vendors for components of their business they simply can’t provide on their own, and those vendors have learned they can resell other bulk services in turn. The next time you see the doctor, ask yourself, into which CRM system did your doctor just input your health information? Where did the CRM store that information? Maybe in some cosmic coincidence, it’s sitting alongside your backups on the same disk somewhere in a warehouse. Probably not, but it could happen.

My trust footprint, just like my carbon footprint, is a fuzzy but useful idea for me, which acknowledges that participation in the online world carries inevitable risk—or at least an inevitable cost. It helps me gauge whether I’m closer or further away from my ideal privacy goals. And just the same way that we can’t all become carbon neutral overnight without destroying the global economy, it’s not practical to run around telling everyone to unplug or boycott all online services.

Next time you’re filling out yet another form online, opening yet another service, trying out one more new thing, remember that you’re also relinquishing a little control over what you create and even a small part of who you are. And if this thought at all gives you pause, see if there’s anything you can do to reduce your trust footprint a little. Maybe you can look into hosting your own blog for your writing, getting network-attached storage for your home instead of using a cloud service, limiting what you disclose on social media, or investing in technology that takes privacy seriously.

Beginning with Regular Expressions

I originally published the following last year as an article in The Recompiler, a magazine focused on technology through the lens of feminism. It began as a primer on picking up regular expressions for a friend who was learning to program at the time. I regarded it as an exercise in making a complex topic as accessible as possible.

It assumes an audience familiar with general computer concepts (such as editing text), but it does not necessarily assume a programming background. I present it here with as few modifications from its original publication as possible.


Regular expressions are short pieces of text (often I’ll call a single piece of text a “string,” interchangeably) which describe patterns in text. These patterns can be used to identify parts of a larger text which conform to them. When this happens, the identified part is said to match the pattern. In this way, unknown text can be scanned for patterns, ranging from very simple (a letter or number) to quite complex (URLs, e-mail addresses, phone numbers, and so on).

The patterns shine in situations where you’re not precisely sure what you’re looking for or where to find it. For this reason, regular expressions are a feature common to many technical programs which focus on using lots of text. Most programming languages also incorporate them as a feature.

One common application of regular expressions is to move through a body of text to the first part which matches a pattern—in other words, to find something. It’s then possible to build on this search capability then to replace a pattern automatically. Another use is to validate text, determining whether it conforms to a pattern and acting accordingly. Finally, you (or your program) may only care about text which matches a pattern, and all other text is irrelevant noise. With regular expressions, you can cull a large text down to something easier to use, more meaningful, or suitable to further manipulation.

A Simple First Regular Expression

A regular expression, like I said, is itself a short piece of text. Often, it’s written in a special way to set it apart as a regular expression as opposed to normal text, usually by surrounding it with slashes. Whenever I write a regular expression in this post, I will also surround it with slashes on both sides. For example, /a/ is a valid regular expression which matches the string a. That particular expression could be used to find the first occurrence of the letter a in a longer string of text, such as, Where is the cat?. If the pattern /a/ were applied against that sentence, it would match the a in the middle of cat.

There’s a clear benefit to using regular expressions to do pattern matching in text. They let you ask for what you want rather than specifying how to find it. To be technical, we’d say that regular expressions are a kind of declarative syntax; contrast that with an imperative method of asking for the same thing. In this case, to do this in an imperative way, you’d have to write instructions to loop through each letter in the text, comparing it to the letter a. In the case of regular expressions, the how isn’t our problem. We’re left simply stating the pattern and letting the computer figure it out.

Regular expressions are rather rigid and will only do what you say, sometimes with surprising results. For example, /a/ only matches a single occurrence of a, never A, nor à, and will only match the first one. If it were applied to the phrase “At the car wash”, it would match against the first a in car. It would skip over the A at the beginning, and it would stop looking before even seeing the word wash.

As rigid as regular expressions are, they have an elaborate syntax which can describe vast varieties of patterns. It’s possible to create patterns which can look for entire words, multiple occurrences of words, words which only happen in certain places, optional words, and so on. It’s a question of learning the syntax.

While I intend to touch on the various features which allow flexible and useful patterns, I won’t exhaust all the options here, and I recommend consulting a syntax reference once the idea feels solid. (Before getting into some of the common features of regular expression syntax, it’s important to note that regular expressions vary from implementation to implementation. The idea has been around a long time and has been incorporated into countless programs, each in slightly different ways, and there have been multiple attempts to standardize them. Despite the confusion, though, there is a lot of middle ground. I’m going to try to stay firmly on this middle ground.)

Metacharacters

Let’s elaborate a bit on our first pattern. Suppose we’re not sure what we’re looking for, only that we know it begins with a c and ends with a t. Let’s think about what kinds of words we might want to match, so we can talk intelligently about what patterns exist in those words. We know that /a/ matches cat. What if we want to match cut instead? We could just use /u/, but we know this also matches unrelated strings, like bun or ambiguous.

Now, /cat/ is a perfectly reasonable pattern, and so is /cut/, but we’d probably have an easier go if we create a single pattern that says we expect the letter c, some other letter we don’t care about, and then the letter t. Regular expressions let us use metacharacters to describe the kinds of letters, numbers, or other symbols we might expect to find without naming them directly. (“Character” is a useful word to encompass letters, numbers, spaces, punctuation, and other symbols—anything that makes up part of a string—so “metacharacter” is a character describing other characters.) In this case, we’ll use a .—a simple dot. In regular expression patterns, a dot metacharacter matches any individual character whatsoever. Our regular expression now looks like /c.t/ and matches cat, cut, and cot, among other things.

In fact, we might describe metacharacters as being any character which does not have its literal meaning, and so regular expressions may contain either characters and metacharacters. Occasionally, it can be confusing to know which is which. Mostly, it will be necessary to consult a reference for regular expressions which best suits your situation. Sometimes, even more confusingly, we want to use a metacharacter as a character, or vice versa. In that situation, we need to escape the character.

Escaping

We can see in the above example that a dot has a special meaning in a regular expression. Sometimes, though, we might wish to describe a literal dot in a pattern. For this reason, we need a way to describe literal characters which don’t carry their ordinary meaning, as well as employ ordinary characters for new meanings. In a regular expression pattern (as in many other programming languages), a backslash (\\) does this job. Specifically, it means that the character directly after it should not be interpreted as usual.

Most often, it can be used to define a pattern containing a special character as an ordinary one. In this context, the backslash is said to be an escape character, which lets us write a character while escaping its usual meaning.

For example, suppose we cared about situations where a sentence ends in the letter t. The easiest pattern to describe that situation might be the letter, followed by a period and a space, but we can’t type a literal dot for that period, or else we’d match words like to. Therefore, our pattern must escape the dot. The pattern we want is written as /t\. /.

Quantifiers

Metacharacters may do more than stand in for another kind of character. They may modify the meaning of characters after it (as we’ve already seen with the escape metacharacter) or those before it. They may also stand in for more abstract concepts, such as word boundaries.

Let’s first consider a new situation, using a metacharacter to modify the preceding character. Think back to earlier, when we said we know we want something that begins with a c and ends with a t. Using the pattern /c.t/, we already know that we can match words like cut and cat.

We need a few more special metacharacters, though, before our expression meets our requirements. /c.t/ won’t match, for example, carrot, but it will match concatenate and subcutaneous.

First of all, we need to be able to describe a pattern that basically leaves the number of characters in the middle flexible. Quantifiers allow us to describe how many occurrences of the preceding character we may match. We can say if we expect zero or more, one or more, or even a very particular count of a character or larger expression.

Such patterns become far more versatile in practice. Take, for example, the quantifier +. It lets us specify that the character just before it may occur one or more times, but it doesn’t name an upper limit.

Remember the pattern we wrote to match sentences ending in t? What if we wanted to make sure we matched all the spaces which may come after the sentence? Some writers like to space twice between sentences, after all. In that case, our pattern could look like /t\. +/. This pattern describes a situation in which the letter t is followed by a literal dot and then any number of spaces.

Quantifiers may also modify metacharacters, which make them truly powerful and very useful. Using the + again, let’s insert it into our /c.t/ pattern to modify the dot metacharacter, giving us /c.+t/. Now we can match “carrot”! In fact, this pattern matches a c followed by any number of any character at all, as long as a t occurs sometime later on.

There are a few other quantifiers needed to cover all the bases. The following three quantifiers cover the vast majority of circumstances, in which you’re not particularly sure what number of characters you intend to match:

  • * matches zero or more times
  • + matches one or more times
  • ? matches exactly once or zero times

On the other hand, you may have a better idea about the minimum or maximum number of times you need to match, and the following expressions can be used as quantifiers as well.

  • {n} matches exactly n times
  • {n,} matches at least n or more times
  • {n,m} matches at least n but not more than m times

Anchors

We still have “concatenate” and “subcutaneous” to deal with, though. /c.+t/ matches those because it doesn’t care about what comes before or after the match. One strategy we can use is to anchor the beginning or end of the pattern to stipulate we want the text to begin or end there. This is a case where a metacharacter matches a more abstract concept.

Anchors, in this case, let us match the concept of the beginning or the end of a string. (Anchors really refer to the beginning and ends of lines, most of the time, but it comes to the same thing in this case. See a reference guide for more information on this point.) The ^ anchor, which may only begin a pattern, matches the beginning of a string. Likewise, a $ at the end means the text being matched must end there. Using both of these, our pattern becomes /^c.+t$/.

To break this pattern down, we’re matching a string which begins with a c, followed by some indeterminate number of characters, and finally ends with a t. As ^ and $ represent the very beginning and end of the string, we know that we won’t match any string containing anything at all on the line other than the pattern.

Character Classes

Using anchors, though, may not be the best solution. It assumes the string we’re searching within may only contain the pattern we’re looking for, and so often, this is not the case.

The dot is a very powerful metacharacter. Its biggest flaw is that it is too flexible. For example, /^c.+t$/ would match a string such as cat butt. Patterns try to match as much as possible. Some regular expression implementations allow you to specify a non-greedy pattern (which I won’t cover here—see a reference), but a better approach is to revisit our requirements and reword them slightly to be more explicit.

We want to match a single word (some combination of letters, unbroken by anything that’s not a letter) which begins with c and ends with t. Considering this in terms of the kinds of characters which may come before, during, and after the match, we want to match something which contains not-alphabetical characters before it, followed by the letter c, then some other alphabetical letters, then the letter t, and then something else that’s not alphabetical.

In the /^c.+t$/ pattern, we need to replace both of the anchors and the middle metacharacter .. Assuming words come surrounded by spaces, we can replace each anchor with just a space. Our pattern now looks like / c.+t /.

Now, as for the dot, we can use a character class instead. Character classes begin and end with a bracket. Anything between is treated as a list of possibilities for the character it may match. For example, /[abc]/ matches a single character which may be either a, b, or c. Ranges are also acceptable. /[0-9]/ matches any single-digit number.

We can use a range which captures the whole alphabet, and luckily, a character class is considered a single character in the context of a pattern, so the quantifier after refers to any character in the class. Putting all this together, we end up with the pattern / c[a-z]+t /.

If we want to mix up upper- and lower-case letters, character classes help in this situation, too: / [Cc][a-z]+t /. Now we can match on names like Curt.

Our assumption that words will be surrounded by spaces is a fragile one. It falls apart if the word we want to match is at the very beginning or end, or if it’s surrounded by quotation marks or other punctuation. Luckily, character classes may also list what they do not include by beginning the list with a ^. When ^ comes within brackets, instead of at the beginning of a pattern, instead of serving as an anchor, it inverts the meaning of the character class.

If we consider a word to be a grouping of alphabetical characters, then anything that’s around the word would be anything that’s not alphabetical. Let’s adjust our pattern accordingly: /[^A-Za-z0-9][Cc][a-z]+t[^A-Za-z0-9]/. We’re using the same pattern as before, but the beginning and ending space have become [^A-Za-z0-9].

Escape Sequences

If our pattern is starting to look cumbersome and odd to you, you’re not alone in thinking that. There’s absolutely nothing wrong with the pattern we just wrote, but it has gotten a bit long-winded. This makes it difficult to read, write, and later update.

In fact, many character classes get used so often (and can otherwise be so annoying to write repeatedly) that they’re usually also available as backslashed sequences, such as \b or \w. (This is escaping, again, as I mentioned before, but instead of escaping a special character’s meaning, we’re escaping these letters’ literal meaning. In other words, we’re imbuing them with a new meaning.)

The availability and specific meaning of these escape sequences vary a bit from situation to situation, so it’s important to consult a reference. That said, in our case, we only need a couple which tend to be very common to find.

One of the very most common such escape sequences is the \w which stands in for any “word” character. For our purposes, it matches any alphanumeric character. This is good enough for the inside of a word, so we can revisit our pattern and turn it into /[^\w][Cc]\w+t[^\w]/. Our pattern reads a little more logically now: We’re searching for one not-word character (like punctuation or whitespace) followed by an upper- or lower-case c, some indefinite count of word characters, the letter t, and then finally one not-word character.

Notice how I used the escape sequence inside the character classes at the beginning and end of the word. This is perfectly valid and sometimes desirable. For example, it would allow us to combine escape sequences for which there’s no single suitable one.

It also lets us invert their meaning, as you saw in the most recent example, but many escape sequences can be modified in the same way by capitalizing them, such as \W. As a mnemonic to remember this trick, think of it as shifting the escape sequence (using shift to type it). In cases where a character class may be inverted in meaning, often a capitalized counterpart exists.

Using \W, now we can pare down the pattern back to something a little more readable: /\W[Cc]\w+t\W/.

More Reading

For today, I’m satisfied with our pattern. In a string like I would like some carrot cake., it matches carrot with no trouble, but it doesn’t match cake or even subcutaneous tissue.

There are many more ways to improve it, though. We’ve only laid the groundwork for understanding more of the advanced concepts of regular expressions, many of which could help us make our expression even more powerful and readable, such as pattern qualifiers and zero-width assertions.

Concepts like grouping allow you to break up and manipulate matches in fine-grained ways. Backtracking and extended patterns allow patterns to make decisions based on what they’ve already seen or will see. Some programmers have even written entire programs based on regular expressions, only using patterns!

In short, regular expressions are a deep and powerful topic that very few programmers completely master every corner of. Don’t be afraid to keep a reference close at hand—hopefully it will now empower you instead of daunt you, now that you have a grasp of how to get started composing patterns.

In the Back of the House

I got my first job at fifteen, going on sixteen. I worked for my hometown newspaper as an inserter, and as time passed, I began filling in occasionally as a “pressman.” Inserters were a collective bunch of old ladies (and me) who made spare money assembling the newspaper sections and stuffing in the ad inserts. When I got to help with the actual printing, it took the form of developing, treating, and bending the lithographic plates in preparation for printing. More often, I caught the papers as they rolled off the press to bundle them up for distribution. I also cleaned up, sweeping and trash takeout and the like, but I wasn’t good at it. I liked to take breaks to play my guitar at the back of the shop, so I think the editor-in-chief who ran things probably was annoyed as piss at me half the time.

There was no question I worked in the bowels of the operation. The real fun (and to the extent a small, rural paper could afford it, the real money) happened at the front of the building where the editor-in-chief and reporters worked. I passed through to gather up trash a few times a week. As I went, I admired the editor-in-chief’s ancient typewriter collection in his office. I enjoyed talking to the lead reporter, who loved Star Trek. The layout team’s work fascinated me, especially as they transitioned to digital layout from cutting and splicing pieces of paper together.

After my tour, I returned to the back, and I only heard from the front when it was time to go to press or when we had to stop the presses. We weren’t a separate world by any means, but we had a job to do, and that job was entirely a pragmatic one, keeping the machinery running and enabling the actual enterprise which paid us. Inasmuch as I felt like an important part of the whole, it was in a sense of responsibility toward the final product.

About a decade later, I stumbled across my current programming thing. Now I find myself at the back of the house again. The work echoes my first job sometimes—working on the machinery, keeping things running, along with other programmers and operations folks. This time the job comes with a dose of values dissonance for me. It feels like a wildly inverted amount of prestige goes to us, to the people running the machines, instead of the others who are closer to the actual creation (and the customers using it).

I’m not sure our perceived value is unwarranted—programming is hard. I’m more concerned about the relationship between the front and back of the house. It could be that we, as programmers and tech people, undervalue the people making the content and interacting with the customers. I see the skewed relationship when I look at inflated tech salaries. It makes itself evident in startups made up of all or mostly engineers. I felt it most acutely when I considered becoming a tech writer, only to be reminded it could derail my career and cost me monetarily.

I don’t think my observation comes with a cogent point. Maybe only that tech can’t be just about the engineering, no more than a newspaper can be only a printing press.

Functional Programming for Everyone Else

Functional programming has become a hot topic in the last few years for programmers, but non-programmers might not understand what we’re talking about. I can imagine them wondering, “Did programs not work before? Why are they suddenly ‘functional’ now?”

Earlier today, I was tweeting a bit about the challenge of explaining to family or primary school students what the big deal is all about. Even we programmers take some time to cotton on to the notion. I know I’d have to pause for a moment if someone asked me what Haskell offers over Python.

If you’re not a programmer and have been wondering what the big deal is, here’s my attempt to lay it out.


First, consider the time just before computers existed. By the 1920s, a kind of math problem called a decision problem led various people to learn how to investigate the problem solving process itself. To do this, we had to invent an idea we call computability, meaning to automate problem solving using small, reusable pieces. A couple of mathematicians tackled this idea in two different ways (though they each came to the same conclusion), and today we have two ways to think about computability as a result.

I’m partial to Alan Turing’s approach because it’s very practical. He envisioned a process that’s quite mechanical. In fact, we now call it a Turing machine, even though his machine never actually existed. It was more of a mental exercise than something he intended to build.

To solve a problem with a Turing machine, he would break a problem into a sequence of steps which would pass through the machine on an endless tape. The machine itself knew how to understand the steps on the tape, and it knew how to store information in its memory. As the steps on the tape passed through, one at a time, the machine would consult the tape and its own memory to figure out what to do. This usually meant modifying something in its memory, which in turn could affect the following step, over and over until the steps ran out. By choosing the right set of steps, when you were done, the machine’s memory would end up with the answer you needed.

Since that time, most computers and programs are based on this concept of stringing together instructions which modify values in memory to arrive at a result. Learning to program means learning a vast number of details, but much of it boils down to understanding how to break a problem into instructions to accomplish the same thing. Programs made this way would not be considered “functional.”

At the same time, another mathematician, Alonzo Church, came up with another approach called lambda calculus. At its heart, it has a lot in common with Turing’s approach: lambda calculus breaks up a problem into small parts called functions. Instead of modifying things in memory, though, the key proposition of a function is that it takes input and calculates a result—nothing more. To solve a problem this way, little functions are written to calculate small parts of the problem, which are in turn fed to other functions which do something else, and so on until you get an answer.

Lambda calculus takes a much more abstract approach, so it took longer to work out how to make programs with it. When we did, we called these programs “functional programs” because functions were so fundamental to how they worked.


Putting all this together, I think of functional programs as ones which do their jobs without stopping to take notes along the way. As a practical consequence, this implies a few odd things. The little niceties that come first nature to procedural programs—like storing values, printing out text, or doing more than one thing at once—don’t come easy to functional programs. On the other hand, functional programs allow for understanding better what a program will do, since it will do the same thing every time if its input doesn’t change.

I think both approaches have something to offer, and in fact, most programs are made with a combination of these ideas. Turing proved neither approach was better than the other. They’re just two ways of ending up at the same result. Programmers each have to decide for themselves which approach suits best—and that decision problem can’t be solved by a program yet.