Flash Fiction: Wish Fulfillment

“I don’t want much,” I said. “I don’t want power. I don’t want to rule over others. I don’t want love. I don’t want anyone dead. I don’t even want to hurt anyone.”

The genie smiled blandly.

“All I want is a healthy retirement fund,” I said. “And I want it without any tricks.”

“Without tricks!” The genie rose to his full height, taken aback. “I am no magician! Tricks indeed!”

I said, “Well, you know what I mean. Without…you know, strings attached.”

“Without consequences, you mean? There can be no such thing.” The genie smiled broadly without using his eyes, and he once again settled into the armchair. “There are simply natural outcomes you must expect from any good fortune.”

All right, I figured—so long as no one gets hurt. I get to decide the terms. That much the genie had made clear.

The genie asked, “Have you decided?”

“Yes,” I said, “all right. I’ve decided.”

The genie fixed me with his eyes and waited.

“I want a million dollars…”

The genie smiled.

“…in United States dollars, tax-free…”

The genie’s smile widened.

“…legally…”

The genie’s smile widened more, his eyebrows lifting in expectation.

“…which will actually be available to me, and to me alone…”

“Yes?” the genie crooned.

“…without anyone having to lose anything for me to gain it…”

The genie’s smile somehow grew still more.

“…and without anyone getting hurt.”

I began to wonder whether anything I said could possibly disrupt the genie’s unflappable sangfroid. None of my conditions had so far seemed to ruffle him in the slightest.

He asked, “Will that be all?”

I mulled it over, trying to think if there were some possible way this could backfire, and I could not think of anything. “Yes, that’s right. That’s what I want.”

“Very well,” the genie said, seeming to suppress glee. “Nothing could be simpler.”

I waited expectantly. I wondered whether I was supposed to close my eyes or something. After looking around, I asked, “Well? What now?”

“I’ve granted the wish, just as you asked,” he said.

“Really? How? Where?”

“It’s being deposited into your checking account. I thought you might find that convenient!”

I picked up my phone and opened my banking app. I didn’t know why I was so tremulous. Maybe I was overwhelmed with the unreality of what I expected to find. I pulled up my balance, and it looked unchanged.

I sighed and said, “I don’t understand. There’s nothing here.”

“Of course there is! It’s being deposited as we speak.”

Being deposited…?” I looked at the transaction list, and I noticed a handful of new credits—each for only one cent.

“Yes,” the genie said, “do you see?”

“But these are for one cent each,” I said.

“Precisely! You never asked for the amount now. Indeed, you mentioned a ‘retirement’ fund. I merely took it on my own initiative to amortize the funds over the rest of your life, and that is precisely what you will receive: one cent every twelve-point-oh-two-five-six-eight-three seconds. Over the span of the rest of your life, it will total exactly one million dollars, in U.S. currency, tax-free! Quite as you asked.”

The genie laughed in a dry, inward way.

At that moment, a new deposit arrived. My phone’s banking app showed that it was another cent. So far I had six cents out of my million. He was right—the genie had granted my wish in the most frustrating, useless, and maddening way possible.

Then my initial frustration fell away, as if through a trap door into a bottomless terror, as I realized what the genie had actually given me. This wasn’t just a million dollars. Each cent was like a grain of sand falling through the hourglass timer of my newly circumscribed life. A simple multiplication would reveal the exact moment of my death.

I looked down at my phone as another cent rolled in. When I looked up, I saw that I was alone.

“Oh, no.”

Writing My First Vim Plugin

Screenshot of Vim editing a file, with a search for the term "fibonacci" in progress. The plugin I wrote has added "3 matches of /\vfibonacci/" to the statusline.
Screenshot of Vim editing a file, with a search for the term “fibonacci” in progress. The plugin I wrote has added “3 matches of /\vfibonacci/” to the statusline.

Last night, I put my first full Vim plugin up on GitHub. It’s called match-count-statusline. It’s intended for anyone who uses Vim to edit text files. It adds a small item to the statusline (Vim’s version of a status bar) describing how many times the current search occurs in the current buffer and, if there’s enough room, what the current search pattern is.

This began as something I needed because I kept opening files and needing to count how many times a term or pattern occurred in that file. I wrote a small script to do it, and I tacked on more things to the script over the next few days and weeks.

Eventually I felt confident enough to share it. I’d also like to share about the process of writing it and preparing it for use by others. If you’re interested, read on.

Who Should Read This?

This guide is meant for advanced Vim users who wish to understand more about the process of writing a plugin through a first-hand narrative.

I won’t be explaining how to use or configure Vim itself, nor the basics of how to write in Vim’s scripting language. I’ll assume familiarity with both. If you don’t know Vim’s scripting language (which I am going to call “VimL” for the rest of this post1), I cannot recommend enough Steve Losh’s Learn Vimscript the Hard Way. It is freely available online, and if you find it useful, I encourage you to buy a copy as well.

That book will set you on your way, but it can never include everything you need to know. Vim is always changing, and its own documentation is deep and voluminous. It is for very good reason that Steve sends his readers to Vim’s own documentation at the end of every section—he could not ever hope to include everything in a single book.

I’ll be referring to my plugin as it exists at the time of this writing by commit hash. It is liable to change in the future as I learn how to improve it. The newest version should always be found here, which will incorporate fixes and improvements.

The Problem

I tend to like to open log files in Vim and search around in them to find patterns while I’m investigating a problem at work. I’ve developed over time a lot of tools for doing so. Some of them are crude, but when I find myself doing the same crude thing over and over, I tend to want to polish it.

One common thing I do is discover how often a search pattern occurs in a log. This indicates to me whether something is signal or noise. I’ll place my cursor over something like an ID, press *, and then count how many times it occurs and write that down in another file.

I could do this from the command line using something like grep | wc -l, but I get a lot of value from seeing the whole context and being able to skip between the occurrences of my search, finding out what happened before and after, and so on. I’m very visual. I like to see them in the file.

So to count them, I’d been doing a crude thing. I would run :g//d and watch how many lines got deleted. (Without a pattern specified, that command will delete whatever the current search pattern is.) This has a few drawbacks. First, I had to undelete the lines after I was done. In large files, it was somewhat slow. And finally, it only told me the count of lines, not the count of occurrences.

I figured what I really wanted was simple, and it should be built into Vim.

The Solution

It is not really built into Vim, though. I figured I’d never find it in Vim’s vast documentation, so I searched online and found a Vim Tips article called, “Count number of matches of a pattern.” The very first thing offered in the article was the answer I needed.

:%s/pattern//gn

I knew that when the pattern was already searched for, it could be implied, so this simplified down even more.

:%s///gn

That did it. I ran that after searching for something, and a message printed to the bottom of the screen resembling the following.

2 matches on 2 lines

I could have stopped there.

However, I tend to forget obscure looking commands, and I thought it would be nice if this happened automatically anytime I searched for something. I knew graphical text editors tended to automatically surface information like this in their status bar, and I was inspired to make the same thing happen.

The Solution According to Vim

I probably should not have given up on Vim’s documentation so easily. I didn’t know that the substitution command (:s) was the place to start, but my use of the global command (:g) before might have been a hint.

What I didn’t know is that Vim already internally documents how to count items, and it’s filed under “count-items,” logically enough. You can find it mentioned under the documentation for the substitution section (:help :s, or more specifically, :help :s_flags and see the “n” flag) as well. To find out how this works, run :help count-items. You’ll see it suggests ways to count characters, words, and so on. Generally, it boils down to a very similar command.

:%s//&/gn

Again, this assumes an existing pattern has been searched for. The one difference is the ampersand. The ampersand has a special meaning in the substitution, referring to the thing being replaced. In this case, it would effectively replace a thing with itself, causing nothing to happen. But since the “n” flag is there, no replacement will happen regardless, so there’s effectively no difference if it’s included or not. I have included it in subsequent examples as it is safer (in case the “n” is forgotten).

The Script

So that single, small command forms the core of the functionality I later expanded.

To display it, I learned how to insert an item into vim-airline‘s statusline. That wasn’t too difficult once I found an example online, and you can see how I accomplish that here. Using that, I began a small script in my personal configuration files which added the count when I was searching for a pattern, and otherwise displayed nothing.

Over the next several days, I added on improvements because I found it was slow on larger files. This included caching its results momentarily rather than updating constantly. I also found that each time it updated the search, it would also move the mouse cursor, so I needed to fix that.

The resulting script grew in size and became safer and better performing. Eventually, I figured I’d put enough work into it that it would be safe enough to share with others without breaking their Vim installs.

The Plugin

To create a plugin, I added some documentation, reworked the script to make it configurable, and “licensed” it into the public domain. Then I put it all in a Git repository and hosted it on Github.

I want to spend the rest of this post talking about all the funny little things I did in the core of the script contained in the plugin to build it up from that one substitute hack into more fully fledged functionality.

Below, I’ll be referencing parts of the script itself as contained in the plugin. Each section covers a different topic I applied as I elaborated on the design and improved the plugin so it could be used by more people.

Opening Stanza

The script begins as many plugins do, with an opening stanza ensuring it’s safe to run.

" require 7.4.1658 for v:vim_did_enter
if &compatible
      \ || (v:version <= 704 && !has('patch1658'))
      \ || exists('g:loaded_match_count_statusline')
  finish
endif
let g:loaded_match_count_statusline = v:true

This bit of code makes three checks. First, it checks if &compatible is set. If it is, then there’s no need to check anything else—the script won’t work, and we can exit right here.

Next, it checks for a version of Vim greater than or equal to 7.4.1658 (using Vim’s peculiar syntax for doing so, the predefined variable v:version). That patch, 1658, is the one in which v:vim_did_enter was introduced. (I discovered this by searching the vim_dev mailing list.) That patch was introduced two and a half years ago, so I felt comfortable with that baseline.

Finally, I check to see if my plugin is already loaded using a global variable I will set directly after this condition. If the variable is already set, it means I’ve reached that line before, and I should exit before reaching it again. This prevents loading the script more than once.

The “finish” command ends the script if any of these conditions are true.

The Global Flag

Notice how the above examples of the substitute command all included the “g” flag. That tells the substitution to apply to all matches in a line, not just the first one.

That probably seems like a funny default: only doing one substitution per line and then moving along. It’s just the way regular expressions work from the olden days, and Vi (the editor that Vim is based on) picked up the behavior. Vim never breaks compatibility, so its substitution also needs that “g” flag to apply  substitutions globally.

For that reason, Vim introduced a setting called &gdefault (see :help gdefault). It has the effect of implying a “g” flag on every substitution. Unfortunately, it also means that if you add the “g” flag to a substitution while that setting is on, it undoes the effect. That means you never know what the “g” flag will do unless you know what the setting &gdefault is. That’s not confusing, right?

All of that means that setting &gdefault would effectively break counting if I did not check it. That’s what these lines are all about.

if &gdefault
  let s:match_command = '%s//&/ne'
else
  let s:match_command = '%s//&/gne'
endif

Here, a command to run to search for matches is saved to a variable based on whether &gdefault is set.

There is one minor drawback to doing this here as opposed to dynamically setting it every time the match is calculated. If the user changes &gdefault after loading Vim, the counts will be incorrect when a pattern occurs more than once on a line. This could be a bug! The fix would be to set the match command each time we run a match.

Settings and Defaults

When I decided to release this as a plugin, I realized that I had baked in several decisions about the script that I knew others would disagree with, and I wanted to give them the freedom to configure the script the way they wanted. The next several lines check for global variables which the user may set and then either use their values or use my defaults. Here’s an example of a couple of such settings.

let s:start =
      \ get(g:, 'match_count_start', '')

let s:end =
      \ get(g:, 'match_count_end', '')

To use a variable in Vim, while falling back to a default if it’s undefined, an old way was to use the exists() function. My syntax uses a stupid Vim trick involving variable scoping.

Notice how all the variables I use begin with a letter followed by a colon. These are known in VimL as “internal variables” (see :help internal-variables), and each kind begins with a letter denoting its scope, a colon, and then the name of the variable. (Vim also refers to these scopes as “name spaces” in its documentation.)

Each scope has a special meaning and is treated by Vim in a special way. For example, variables scoped to b: only exist for the one buffer you’re in, and no other buffer can see that variable.

More importantly for our purposes, though, every scope is also considered by Vim to be a Dictionary unto itself, and you can treat all the variables in it as keys containing values. Weird? Yes, weird. But it lets you do things like, “:echo has_key(g:, 'match_count_end'),” which will tell you if that variable is defined.

So in the case of each global setting, I use the get() function, which takes a Dictionary, a key, and optionally also a default value to use in case the key is missing. This allows me to set variables scoped to my script (in s:) based on whether variables in the global scope (in g:) are defined or not. Then the rest of the script uses those script-scoped variables.

This unfortunately means that no changes to the global variables can get picked up while Vim is running. To fix this, I could set all these inside the match counting function I defined, if I chose.

Caching

Caching is one of the earlier things I implemented, and it’s gone through a few revisions. Here’s how it works today.

The basic idea is, the match counting function gets called repeatedly every time the statusline needs to get redrawn, and this happens far more often than the match count itself will change. If the script remembers the match count for each buffer, the match counting function (MatchCountStatusline()) won’t have to recalculate until it’s really necessary, saving itself some work.

So caching is one among many ways MatchCountStatusline() attempts to defer doing any hard work. How does MatchCountStatusline() find out when to update its cache? A couple of ways.

The cache is a simple Dictionary made anew for each buffer. It remembers the last time the cached values were referenced or updated, the number of changes to the current buffer, the current search pattern, and the number of matches.

let s:unused_cache_values = {
      \   'pattern':     -1,
      \   'changedtick': -1,
      \   'match_count': -1,
      \   'last_run':    -1
      \ }

The default values for the cache are ones that could never occur normally, so I call them “sentinel values.” Sentinel values are generally used to represent invalid values, so in this case I can recognize when the cache contains values which have not yet been set by MatchCountStatusline().

When MatchCountStatusline() is first run, if the buffer doesn’t already have a cache, a new one is created with the default sentinel values.

let b:count_cache = 
    \ get(b:, 'count_cache', copy(s:unused_cache_values))

I know that because the cache is scoped to the buffer, each buffer gets a new set of values. On subsequent calls to MatchCountStatusline() for that buffer, it reuses the cache which already exists. (Notice also that I had to copy() the values. If I didn’t, modifications to the buffer’s count cache would modify the script’s count cache defaults and would affect all the other buffers as well. See :help copy.)

A few lines later, MatchCountStatusline() checks to see if the values in the cache match the state of the buffer. If there is a match, there is no need to continue, and the function ends here. Otherwise, we need to set the cache’s values with new ones farther down.

if b:count_cache.pattern == @/
    \ && b:count_cache.changedtick == b:changedtick
  return s:PrintMatchCount(b:count_cache)
endif

It’s a simple check. All I look at here is whether the pattern has changed or the buffer has changed. I don’t have to look at the buffer itself to determine whether it’s changed. A predefined Vim variable increments for every change, called b:changedtick. If it’s changed, I know the buffer has changed. See :help b:changedtick. Those two things let me know if a match count needs to be recalculated for the current buffer.

Notice also that the current search pattern is stored in a register called @/. All registers can be referred to by a name which starts with the @ sigil. There are a bunch of generic registers with single letter names, “a” through “z,” and then there are special-use registers such as @/. It’s easy to see where it got its name, with a little thought—the / key begins a search in Vim, so the @/ register holds the search pattern. Like any register, it may be echoed, compared to, or even assigned to. Any time in my script I want to see what’s being searched for, I inspect @/. For more information, see :help @/ and :help registers. The kinds of registers available will definitely surprise you!


You probably noticed the cache holds two other things besides the number of changes to the current buffer and the current search pattern. The match_count value is the thing we’re caching, so we never care what its value is until we’re printing it out. But last_run deserves an explanation.

I decided early on that I wanted to keep and print out the same cached values for a short period of time, even if the buffer has changed. That’s what last_run is for. It lets me know how long the cache has been holding onto those values so I can know when they do need updating.

Why would I want to keep old values around, even if they don’t reflect reality anymore? It’s not always necessary to reflect reality the very instant it changes. Humans don’t need updates right away for everything—a delay of a tenth or a quarter of a second is tolerable sometimes.

When someone is actively typing, or if they’re entering a new search (and &incsearch is enabled), every single keystroke would invalidate the cache and trigger a new calculation. If we instead wait for a fraction of a second, we can allow the user to type a few characters before updating. By the time they look down, we’ve probably already updated, but we won’t have wasted time needlessly updating for every keystroke.

The function responsible for determining if enough time has passed before even checking the cache is called s:IsCacheStale(). (Notice that it’s script-scoped, so that it doesn’t collide with other scripts’ functions.) It comes before the cache check in the program, so it “short-circuits” that logic, precluding it when it’s not needed. The function s:IsCacheStale() has a very simple mandate—it receives the cache and checks how much time has gone by. If enough time has passed, it returns a true value. Otherwise, it evaluates to false. How it implements this is a little complicated.

I won’t copy in the whole body of the function here, but go glance over it if you want. There are a couple of weird things going on, and I’ll touch on them here.

First, instead of using ones and zeros to represent true and false, I’m using Vim’s predefined values of v:true and v:false. Vim introduced these in 7.4.1154 to help with JSON parsing. They evaluate to 1 and 0 respectively, so I’m using them to make the script more readable.

Next, I put in a little stanza at the beginning to optimize when loading a new buffer.

if a:count_cache == s:unused_cache_values
  if has('reltime')
    let a:count_cache.last_run = reltime()
  else
    let a:count_cache.last_run = localtime()
  endif

  return v:false
endif

The very first time that MatchCountStatusline() gets called for a buffer, recall that it puts in the sentinel values for the buffer’s count cache. Here, I detect whether or not that’s the case. If it is, I merely return false (meaning that the cache is not stale), but I also importantly update the last_run cached value with the current time, so that this condition never evaluates to true in the future for this buffer.

What this accomplishes is that the first time a buffer is loaded, the cache always appears to be stale (even though it really only contains invalid values). This serves to cause no match counting to occur for the first cache timeout period (by default, a tenth of a second) after a buffer gets loaded. After that, the first match count occurs. This lets the buffer get up on its feet and lets other plugins and automatic commands do things before we start.

Finally, both here and further down, I test whether the “reltime” feature exists in Vim and use it only if it does. Otherwise, I fall back to using the localtime() function. There are a handful of other places I’ve checked for functionality before using it. This is one of the ways I’ve tried to make the plugin safer for a wider audience. I imagine most people will have a Vim with “reltime” available, but I couldn’t tell how widely available it would be.

The cache check algorithm is pretty simple otherwise. It checks the current time against the last_run time. If it’s greater than or equal to the cache timeout, last_run gets updated, and the function returns true (meaning the cache is stale). Otherwise it returns false, leaving last_run alone so that the next time it’s called, it will be clear how much time has passed since the last time the cache was stale.

Both by giving the cache a little time before it’s updated, and by checking its values, MatchCountStatusline() avoids updating the cache (and running the expensive match-counting calculating) whenever possible.

Another Optimization: v:vim_did_enter

One of the more recent Vim features I took advantage of was the v:vim_did_enter predefined variable (see :help v:vim_did_enter). It allows MatchCountStatusline() to know if it’s being called before Vim has even loaded all the way, and so it’s the very first check. This means that the cache isn’t even warmed up before Vim has properly started up, so the cache grace period kicks in after. This allows Vim to start up faster.

" don't bother executing until Vim has fully loaded
if v:vim_did_enter == v:false
  return ''
endif

File Size Checking

I discovered early on that match counting works really poorly for large files. I tried throwing in a couple of optimizations for the counting itself (I’ll mention those below), but the counting still dragged, and so I assumed it would never be fast enough to work transparently and interactively.

Then I considered whether I could do the counting asynchronously. I eventually ruled this out for now because it means shelling outside of the Vim process, as near as I can tell. I wanted my script to be purely VimL and self-contained.

So I settled on checking for whether the size of the file exceeds a certain size and stopping there. The function responsible for checking is called s:IsLargeFile().

function! s:IsLargeFile(force)
  if a:force
    return v:false
  else
    if getfsize(expand(@%)) >= s:max_file_size_in_bytes
      return v:true
    else
      return v:false
    endif
  endif
endfunction

Notice that it takes an argument called force. This allows MatchCountStatusline() to tell it to skip checking the file size and just report that the file is safe in all cases. Otherwise, this one is very straightforward.

Toggling Match Counting

I realized that sometimes, even for large files, I still wanted a count, so I implemented a command which would manipulate two buffer variables to force match counting to occur (or would disable match counting altogether). The logic here is a bit weird because of the file size checking. I wanted to be able to force things on if they’re turned off due to the file size, and I wanted to be able to toggle things normally otherwise.

You can review the function which implements the toggling, but I won’t cover it in more detail here.

Optimizing Match Counting Itself

We’ve covered several preliminary checks that MatchCountStatusline() makes, and we’re down to the nitty gritty—updating the cache and then calling s:PrintMatchCount() to render the statusline.

When MatchCountStatusline() sees there’s no pattern for the current buffer, its job is easy—it clears the cached values, and that’s it.

" don't count matches that aren't being searched for
if @/ == ''
  let b:count_cache.pattern     = ''
  let b:count_cache.match_count = 0
  let b:count_cache.changedtick = b:changedtick
[...]

Otherwise, we have work to do.

(Incidentally, there’s a small optimization I forgot to make here. See it? We’re updating the cache. We can set last_run here and keep these cached values for longer. We should do this here and anywhere below where we update the cache.)


Since we’re now in the part of the script involves the actual calculation, and since it has the chance of raising exceptions, I begin a try/catch/finally block here. It’s important to begin it here, when I start to change the state of the buffer, so that I know my changes will get fixed up by the finally block below. The worst case scenario would be attempting to count the matches, having an error occur, and leaving the user’s editor in a broken state.

Now we’re free to begin mucking about. First thing I do is save the current view.

" freeze the view in place
let l:view = winsaveview()

In the finally block, the last thing I do is restore this view.

call winrestview(l:view)

This ensures that nothing visibly changes while I do what I’m about to do. (See :help views-sessions, :help winsaveview, and :help winrestview.) One problem I had was that the core match count command (s//&/gne) would cause the cursor to jump to the beginning of the line. Saving and restoring the view fixes that problem.

Next, I disable automatic commands and highlight search. Likewise, I re-enable those later on. Because these are compile-time features, I check that I can do so before I do. I save off their values as local variables so that I can restore them as they were before I touched them.

By turning these features off, Vim won’t attempt to fire off events during the fake substitution it’s about to run, which can save a little time. Turning off &hlsearch in particular is one of Vim’s own suggestions in :help count-items.

Match Counting

Finally, I perform the actual count.

" this trick counts the matches (see :help count-items)
redir => l:match_output
silent! execute s:match_command
redir END

I added a comment to help others understand how this worked because it’s probably the least obvious part of the program.

The redir here sends all the output from that command to the l:match_output local variable. The actual command uses echom to display a message with the results, like I showed above, something like “2 matches on 2 lines“. If no matches are found, the output is empty due to our use of the n flag.

Then the string is parsed, and the cache is updated with the new match count and other values. The finally block restores things how they were, as I mentioned earlier. And at the very end of the MatchCountStatusline() function, the cache is rendered into a string by s:PrintMatchCount().

What Else?

There are several things I didn’t cover here, like how I implemented the global customizations. I might cover those in a future post if there’s interest, but they’re straightforward enough if you followed the above.

Making it appear in the statusline was interesting, but it’s more of interest to vim-airline users only, probably. Feel free to look at the stanza which attempts to guess how to do that, though.

Do also check out the documentation for my plugin, which consumed most of the time it took to get it ready for wider distribution.

I hope that you found what you were looking for in this post and took away at least one useful thing!

Beginning Astrophotography: The Deeper Sky

 

Screenshot of PixInsight during ImageIntegration processing of six Omega Nebula exposures
Screenshot of PixInsight during ImageIntegration processing of six Omega Nebula exposures

Of my previous wide-field photos of the night sky, none have been more than single long exposures of thirty seconds or less. Recently I’ve taken my first steps into experimenting with stacking these non-planetary photos. Below, I show the process and results from my first attempts to stack both an in-telescope photo and a wide-field photo.

Stacking is, as I’ve mentioned in the past, a way of combining separate photos into a single, longer exposure. With highly detailed, small objects like planets, stacking can be used to get more detail and clarity through lucky imaging and the shift-and-add technique.2 With a wider-field photo, the goal changes a bit. Certainly, more detail and clarity result, but you also gather more light and reduce camera sensor noise.

Noise, Noise, Noise!

I have been limited by camera sensor noise in all the individual astronomical photos I have ever made. To make a relatively decent exposure of the night sky, it’s necessary to boost the ISO to at least 1600, which increases the sensor gain. On its own, this usually isn’t a grave concern, but it limits how much I can subsequently push the photo to bring out its details.

Small section of a Milky Way photo from 14 July 2018 showing abundant chrominance and luminance noise
Small section of a Milky Way photo from 14 July 2018 showing abundant chrominance and luminance noise

Inside of a single photo, there’s no real way to overcome this noise without manipulating the photo aggressively, such as using a powerful noise reduction algorithm. I typically avoid doing so because it’s difficult for such an algorithm to distinguish noise from fainter stars, and even the brighter details lose much of their finer qualities (dust lanes in the Milky Way core, for example).

Instead of eliminating the noise, I usually just leave it in. I limit the amount I push a photo so that the noise remains relatively unapparent when seen in context, and generally the noise does not mask the most important parts of the photo.

Yet, that noise limits my light. I can’t turn up the light without turning up the noise—both in the camera (I must keep the ISO low) and in the computer (I must avoid pushing the photo too far). What can I do? Stacking! Taking many photos and averaging them together means not only do I combine the light from them to make that light brighter, but the noise (which is largely random) gets canceled out because it varies between each photo.

New Techniques, New Tools

Stacking deep-sky and wide-field photos is a different process than stacking planetary photos. The exposures are much longer (several seconds instead of small fractions of a second), and often you have fewer of them.

In many ways, it is a more advanced technique. I have not yet tapped a lot of the tools available to me, and I won’t be discussing them today. I have proceeded by taking tiny steps, seeing what happens, and observing the result. Each time, I figure out what changed, what limitations I’ve hit, and what new techniques I can draw on. I will mention a few avenues of improvement I’ve passed up, though.

For example, for stacking photos of dim subjects (the Milky Way, nebulae, and so on), it is common for astrophotographers to prepare ahead of time a series of preliminary photos used to calibrate the process. These are known as darksflats, and bias frames. These aren’t pictures of the sky but instead of (essentially) nothingness, allowing you to photograph your camera’s inherent sensor variations. For example, dark frames are photos taken with the lens cap on.

All digital cameras have inherent variations in the sensor. When you stack photos taken with your camera, you’re also stacking up these variations and exaggerating them as well. By taking these extra frames ahead of time and incorporating them into the process, it’s possible to subtract the sensor variations and come out with a smoother photo which can be more freely manipulated.

I did not, of course, prepare any darks, flats, or biases. All I had were lights, which is to say, photos of the actual subject. This is because I was only experimenting and hadn’t planned ahead. I had never done this before, and I was using photos from either months or a year ago.

I also knew I needed to use a new tool. The stacking programs (like AutoStakkert!3) I had been using were more designed for planetary objects or the Moon. These existing processes and tools might have worked okay, but they are quite rigid, and I wanted something more advanced.

For example, in wider-field photos, aligning different sections of the sky means actually conforming the photos somewhat to a single projection. This is necessary because the sky is a large, three-dimensional dome, and each photo projects a piece of that dome onto a two-dimensional image. Any movement in the camera causes that projection to change somewhat, so alignment of the photos together requires a projectional transformation—which looks like a slight warping. (This sort of warping may be easier to imagine if you considered what would happen if you photographed the entire sky all at once and then attempted to stitch it together into a panorama. The panorama would show the horizon on all sides, and the sky would be a circle in the middle. Each photo would have to be bent to complete parts of this circle.)

Instead, I used a much more advanced tool called PixInsight. It is not free software in any sense of the word, unfortunately, but it’s extraordinarily powerful and flexible. This is the only tool I used (aside from Apple Photos), and it’s what I’ll discuss below.

Omega Nebula

Last year, the night before the 2017 eclipse, I took some photos of the Omega Nebula. I got perhaps eight or so that night, trying different settings. None of them were great, but they showed the nebula for what it was—some glowing gas in the sky.

The Omega Nebula, taken via Celestron eleven-inch telescope and Sony α6300 camera
Before stacking: The Omega Nebula, taken as a single exposure via a Celestron eleven-inch telescope and Sony α6300 camera

Totally an accidental thing—I had been aimlessly roaming with my tracking motor and just happened to see a blob. I couldn’t quite make it out with my eye, so I used the camera to photograph it more clearly. I decided I’d use the photos to identify it later, which I did. It took a lot of work to get it to show up nicely in an image.

A couple of days ago, on the anniversary of the eclipse, I decided to revisit those photos. I figured, well, I had maybe eight photos of the thing, so maybe I could do something with that. I read some wickedly complicated PixInsight tutorials (including this one), skipped around in them, and sort of scrummaged together a workflow. It’s not perfect, but I’ll share it.

My PixInsight Process for the Omega Nebula

With PixInsight open, first, I went to the “Process” menu, and under “ColorSpaces,” I chose the “Debayer” process. This is a little hard to explain, but essentially it’s a way to deconstruct a limitation of the camera sensor. The images I began with were the RAW images (dumps of the raw sensor data from when I photographed). The sensor’s pixels each have no ability to differentiate color, only light intensity, so a color filter array is placed over each pixel sensor to allow each to see one of red, green, or blue. That then must be debayered or demosaiced to construct the color image accurately. To know which kind of mosaic pattern, I searched the Internet for the one applicable to my camera, and it seemed like “RGGB” was the way to go.

Screenshot of the PixInsight Debayering process dialog, set to the RGGB mosaic pattern and ready to receive pictures to debayer
Screenshot of the PixInsight Debayering process dialog, set to the RGGB mosaic pattern and ready to receive pictures to debayer

I added images to the “Debayer” process and let it run, and it output a series of files which had been debayered, which had been renamed with a “_d” at the end and were in PixInsight’s own file format, XISF.

The next step was to align the images. PixInsight calls this “registration,” and it has multiple forms. Under the “Process” menu, I went to “ImageRegistration” and found “StarAlignment.”

Screenshot of PixInsight's StarAlignment process, primed with a reference and the images to align which have already been debayered
Screenshot of PixInsight’s StarAlignment process, primed with a reference and the images to align which have already been debayered

In it, I chose one of the images from my set as a “reference,” meaning it would be the image against all the others would be aligned. For this, I would use the output of the debayering, so I used the XISF files output from the last step. I also told it to output “drizzle” data, which is used to reconstruct undersampled images. It can add resolution that’s missing using interpolation. It’s possible to configure the star matching and star detection parameters, but I found I did not need to do so.

The output from this step was similar to the previous one, but the resulting files now ended in “_d_r.xisf”. These images had been moved around and warped such that when laid over top of one another, they would match perfectly. Not all the photos could be aligned, and only six survived the process. I proceeded with these.

There was one more step I did before the final stacking, and that was a normalization. Under “Process” I went to “ImageCalibration” and then “LocalNormalization.” This allowed me to create new files (not image files but metadata files) containing normalization data. These data allow reducing noise and cleaning up the signal even further. I learned about it from this extensive tutorial which explains better than I can (which is the source I used to piece together much of this workflow).

Screenshot of PixInsight's LocalNormalization process, showing it primed with a reference photo with the other debayered and registered photos ready to normalize
Screenshot of PixInsight’s LocalNormalization process, showing it primed with a reference photo with the other debayered and registered photos ready to normalize

After it ran, I finally had all the data I needed for the final stack. PixInsight calls this “ImageIntegration,” which is under the “Process” menu and “ImageIntegration” submenu.

Screenshot of the PixInsight ImageIntegration process, showing six images ready for integration using the median combination algorithm
Screenshot of the PixInsight ImageIntegration process, showing six images ready for integration using the median combination algorithm

I chose the six images which I had debayered, registered (aligned), normalized, and drizzled. I added them to the process. I added the normalization files and the drizzle files which had been output. I chose the average combination algorithm, which is the default. Switched normalization to “Local normalization,” but I left other parameters alone. Then I ran it.

The result was three views, two of which contained rejected pixels and one of which contained the integration itself. (A view, in PixInsight, is like an unsaved file—something you can see but which doesn’t necessarily exist on disk yet.)

Screenshot of the result of the six-image median integration of the Omega Nebula images, before any further processing
Screenshot of the result of the six-image median integration of the Omega Nebula images, before any further processing

It still appeared dim and indistinct, but I knew this was a raw product, ready to be manipulated. The rejection views were blank in this case, so I discarded them.

I figured that I would use PixInsight to stretch the image, and so under “Process” and “IntensityTransformations,” I first tried “AdaptiveStretch,” but I found this to be too aggressive. With its default parameters, the image was washed out by noise, and I couldn’t tame its parameters enough for a more natural result.

Screenshot of a preview of an aggressive AdaptiveStretch in PixInsight, showing noise as a green glow, dithering, and vignetting which masks the nebula almost entirely
Screenshot of a preview of an aggressive AdaptiveStretch in PixInsight, showing noise as a green glow, dithering, and vignetting which masks the nebula almost entirely

It’s possible in that screenshot to see the artifacts of the alignment process as well (the neat lines where the noise increases near the bottom and right). This is because the images didn’t cover precisely the same area, so after stacking, the places where they don’t overlap are visible. The intense green color is probably either contributed by my camera’s noise or from skyglow I picked up. In either case, it’s not what I want. I threw it away.

I then hit upon trying an “AutoHistogram” in the same submenu, and this was much gentler and more helpful. I bumped up its parameters a bit.

Screenshot of PixInsight's AutoHistogram process dialog, showing a stretch method of "Rational Interpolation (MTF)" and a parameterized value of 0.35 for all channels
Screenshot of PixInsight’s AutoHistogram process dialog, showing a stretch method of “Rational Interpolation (MTF)” and a parameterized value of 0.35 for all channels

Now this truly got me somewhere.

Screenshot of the Omega Nebula median integration after applying the AutoHistogram process, revealing more color and structure
Screenshot of the Omega Nebula median integration after applying the AutoHistogram process, revealing more color and structure

A lot of additional color and structure leapt out. Notice down on the bottom and the right, the places where the alignment didn’t quite overlap, there’s some color distortion? This is an interesting outcome of the process—a kind of color correction.

This result definitely seemed much closer to what I wanted, but it’s still quite washed out. I could continue in PixInsight, but I really wanted it only for the stacking part. I’m a little more used to editing photos in Apple Photos, as crude as it can be, so I decided to save this file and move it over (as a 32-bit TIFF).

Finishing Omega Nebula in Apple Photos

I first flipped the photo vertically (to undo the flip introduced by the telescope) and cropped away the parts of the nebula which didn’t align fully.

Then I maxed out the saturation so that I could easily see any tint and color temperature adjustments I would need to make. I changed the photo’s warmth to 4800K and did my utmost with the tint to reduce any green cast. After that, I bumped the saturation way back down.

My next goal was to reduce the washed out appearance of the background sky without losing details of the nebula, so I used a curves adjustment. Apple Photos allows using a targeting tool to set points on the curve based on points on the photo, so I tend to do that. (It also allows setting a black point, but I usually find that too aggressive for astrophotography.) A gentle S-shaped curve of all channels often helps. I try not to be too aggressive with the curves adjustment because I can also use a levels adjustment to even out the histogram even more.

Screenshot of the Omega Nebula integration in Apple Photos after white balance, curves, and levels adjustments
Screenshot of the Omega Nebula integration in Apple Photos after white balance, curves, and levels adjustments

Using the “Selective Color” adjustment, I can pick out the color of the nebula and raise its luminance, which will boost the visibility of some of its dimmer portions.

After this, I make some grosser adjustments, using black level, contrast, highlights, shadows, and exposure.

The focus is very, very soft, but I usually don’t apply any sharpening or added definition because it will more often than not exaggerate distortions and noise without adding any new information. The reason for the soft-looking focus is down to a few reasons. First, I didn’t have perfect tracking on the telescope when I made these photos because I didn’t expect to photograph a nebula. Second, the exposures were long enough that the seeing (the ordinary twinkling of the sky) allowed the objects (like stars and other fine points) to smear into larger discs. Third, I hadn’t spent any time getting the focus tack-sharp because I was in a hurry. Fourth, this is a zoomed in section of a combination of several photos, which already tends to blend together some finer details (despite the drizzle data).

The Omega Nebula After Stacking

For what it’s worth, I think it turned out fine for a completely unexpected outing with just a few photos taken over a few minutes. After the entire process of stacking, which took a couple of hours, I came up with this.

The Omega Nebula, composited from six individual exposures taken via a Celestron eleven-inch telescope and Sony α6300 camera on the night of 20 August 2018
The Omega Nebula, composited from six individual exposures taken via a Celestron eleven-inch telescope and Sony α6300 camera on the night of 20 August 2018

Here are the before and after photos side-by-side so you can compare.

The latter image has more structure, more detail, more color, and all with less noise. All this, even with imperfect, brief photos and with an imperfect, incomplete process.

The Milky Way

I decided to see if I could apply the same process to some of the Milky Way photos I had from earlier in July. I had taken several toward the core, including ones which used my portrait lens. I thought the results were middling, and I was frustrated by all the noise in them.

I’m not going to step through the entire process of the stacking because it’s largely the same as the one I applied for the Omega Nebula. I have tried different kinds of parameters here and there (such as comparing average versus median image integration), but in the end, I used largely the same method.

One interesting wrinkle was that my Milky Way photos included trees along the bottom. Because the stars moved slightly between each shot, the registration process left the trees moving slightly between each. This caused a severe glitch after the PixInsight processing.

Photo of the core of the Milky Way composited from twelve individual exposures, showing a glitched tree-covered horizon at the bottom
Photo of the core of the Milky Way composited from twelve individual exposures, showing a glitched tree-covered horizon at the bottom

It’s likely I could have used a rejection algorithm, a mask, or tweaked the combination algorithm not to cause this, but I haven’t learned how to do that yet, so I let PixInsight do what it did.

Before I did any further processing, I needed to hide the glitch, and I decided cropping would be awkward. So I took the trees from another photo and laid them over top as best as I could. It looks sort of crude when you understand what happened, but unless you squint, it works well enough.

Photo of the core of the Milky Way composited from twelve individual exposures, with the glitches at the bottom covered with a crudely pasted in tree line
Photo of the core of the Milky Way composited from twelve individual exposures, with the glitches at the bottom covered with a crudely pasted in tree line

It covers a lot of the photo, unfortunately, and it looks really weird when you look closely at it, but hopefully the attention is drawn to the sky.

The Milky Way doesn’t look all that much improved over versions I’ve shown in the past, but it took a lot less work to get it there, and the noise and fine details are significantly improved.

Small section of the composited Milky Way photo from 14 July 2018 showing reduced noise and finer details
Small section of the composited Milky Way photo from 14 July 2018 showing reduced noise and finer details

The photo above shows a similar section of the sky as the noisy patch I showed earlier. (They’re not exactly the same section but very close; the same bright star is seen in both.) Here, there’s much less noise, and it’s possible to see indistinct tendrils of dust among the glowing sections of the Milky Way. The stars are easier to distinguish from the background. Below, I’ll place the two side by side for comparison.

That’s the difference—the photo has more underlying signal, so I can eke more  detail from it. The overall photo ends up looking better defined as a result, even if it doesn’t appear, superficially, all that much more improved.

Next

What’s missing?

I need those calibration shots, for sure: the darks, flats, and biases. I can do those without a night sky, though. I just need to get around to it.

I also have a better idea of what kinds of photos align and stack better than others, so I should leave the glitchy trees at home next time. When I’m using the telescope, I should re-examine my focus; use consistent exposure settings; take many, many photos so that I have some to discard; and track as well as I can manage.

After that, I can elaborate on my process and show better photos than ever before.

Beginning Astrophotography: Milky Way on 14 July 2018

Milky Way core, photographed at 22:58 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.
Milky Way core, photographed at 22:58 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.

On the night of the 14th, I got to take my camera out to a friend’s farm—the same one I visited last year—and try more photos of the Milky Way. None of them came out particularly special, but I thought I’d share a few here in one place.

My favorite of the evening might’ve been while I was waiting for dusk, watching the last rays of the sun over the countryside.

Sunset seen over the Oregon farmland, photographed at 20:37 on the evening of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/8 and exposed for 1/160 seconds at 400 ISO.
Sunset seen over the Oregon farmland, photographed at 20:37 on the evening of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/8 and exposed for 1/160 seconds at 400 ISO.

I ended up using my Zeiss Touit lens more than usual this time. It has considerable aberrations and some vignetting, as I’ve pointed out in the past, but its longer focal length let me frame the core of the Milky Way more tightly. It’s a 32mm lens, meaning that on my camera’s APS-C sensor, it is the equivalent of a 48mm lens on a full frame sensor. It’s ideal for things like portraiture, not really for landscapes or astrophotography, but I wanted to give it a try.

I took several photos dead into the Milky Way core with it. I haven’t yet reached the point where I’m taking longer exposures to combine them for more detail. I’ve been instead experimenting with seeing how much detail I can get from individual photos using different settings.

The photo I pushed the most used an ISO of 3200.

Core of the Milky Way, photographed at 22:55 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.
Milky Way core, photographed at 22:55 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.

A lot of the brightness comes from aggressive processing after the fact, though. With another photo from the set, taken with identical settings and nearly identical framing, I used more subdued processing.

Milky Way core, photographed at 22:58 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.
Milky Way core, photographed at 22:58 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 8 seconds at 3200 ISO.

I also turned the camera up to the zenith to catch Vega, Lyra, some of Cygnus, and a bit of the North American Nebula.

Zenith, including constellation Lyra and North American Nebula, photographed at 23:19 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 13 seconds at 1600 ISO.
Zenith, including constellation Lyra and North American Nebula, photographed at 23:19 on the night of 14 July 18 with my Sony α6300 using a Zeiss Touit 32mm lens stopped to 𝑓/1.8 and exposed for 13 seconds at 1600 ISO.

By the time I got out the lens I normally use for night sky wide-field photos, the Rokinon, a few clouds had drifted into view and began to spoil the shots in the direction of the core. So I got nothing so wonderful as last year, but still some nice and expansive shots. My friend suggested portrait aspect, and I definitely got the most out of that.

Milky Way core partially obscured by foreground clouds, photographed at 00:20 on the morning of 15 July 18 with my Sony α6300 using a Rokinon 12mm lens stopped to 𝑓/2.2 and exposed for 20 seconds at 2500 ISO.
Milky Way core partially obscured by foreground clouds, photographed at 00:20 on the morning of 15 July 18 with my Sony α6300 using a Rokinon 12mm lens stopped to 𝑓/2.2 and exposed for 20 seconds at 2500 ISO.

I took photos facing both toward and away from the center of the galaxy, though the latter required some additional processing to reduce the distorted colors from light pollution. There’s a small glimpse of the Andromeda Galaxy as a small blur in the lower right, but not much definition is there—I’d need a zoom lens and many exposures to get more.

View looking toward trailing end of Milky Way, with Andromeda Galaxy and Cassiopeia, photographed at 00:25 on the morning of 15 July 18 with my Sony α6300 using a Rokinon 12mm lens stopped to 𝑓/2.2 and exposed for 15 seconds at 3200 ISO.
View looking toward trailing end of Milky Way, with Andromeda Galaxy and Cassiopeia, photographed at 00:25 on the morning of 15 July 18 with my Sony α6300 using a Rokinon 12mm lens stopped to 𝑓/2.2 and exposed for 15 seconds at 3200 ISO.

Pandora’s Checkbox

The Information Age brought with it a cliché—that unread agreement you dismiss to get to the software you need to use. There’s no way you’re going to read it. For example, macOS High Sierra comes with a software license agreement totaling 535 pages in PDF form, which contain (by my count) 280,599 words of intensely detailed yet maddeningly vague legal language. On that operating system, Apple Music has another license, and the App Store has yet another, and so on.

It would take thousands of dollars in consulting fees with a lawyer to make a fully informed decision, or you can proceed regardless. So you proceed. You always have. Each little app, website, or gizmo peppers you with a new set of terms and conditions. Each upgrade gets a few extra clauses thrown in, and you agree again.

You’re not a fool. You assume you’re signing away rights and control you want. It comes in the bargain. You try to skim the terms and conditions, and this deal feels a bit more Faustian all the time—mandatory binding arbitration, data collection, disclaimers of liability, and so on.

None of this is really news to you if you’ve dug into it. You’re not really in possession of your software; you’ve merely licensed the use of it. You can’t really hold them responsible for flaws; you agreed to accept the software as is. You can’t really control what information they collect about you; you hand that over and get a free or discounted product in return.

However, where things get slippery is that a company with whom you’ve entered into a transaction has also signed agreements with yet other companies. Worked into those overwrought terms and conditions you clicked through, with their vague-yet-precise language, are ways of ensuring that you’ve already agreed to these subsequent proxy agreements as well.

What the T&C often allow is for your data to commingle at some broker whose name you’ve never heard of. A common situation in which this happens is when any entity responsible for handling money.

Say that you learn about a subscription service called Company A. You find them in your web browser or your mobile app, and you sign up, agreeing to their T&C. Then you ask to subscribe to a new e-mail about scarves every day, or whatever Company A does. They in turn ask for your credit card info, your billing address, and maybe a few other demographic details about you.

Company A turns to Company B to determine how risky you are. To do this, they ship off some information about you. If you used a mobile app, they’re possibly reading off what Wi-Fi networks are nearby, what Bluetooth devices are nearby, what apps are installed on your phone, what IP addresses you’re using, what fonts you have installed, and a wealth of other information. If you’ve used a browser, the information is similar but more limited. You’re being geographically located in either case. The headers from your browser are sent. The last website you were at before visiting Company A is probably sent.

Company B collects this information and compares it to all the other data it has on millions of other requests it’s collected from other companies. It has no real duty to sequester Company A’s data from Company Z (neither of which know anything about one another), and by putting it all together, it can detect patterns better. For example, it may have the ability to know where you are, even if you are behind a proxy. It may be able to track your traffic across the Internet as you move from Company A to Company Z and so on—because the number of details it gets are enough usually to uniquely identify you. It needs no cookies or other storage on your end for this.

This means that Company B has the role of an invisible data broker whose job it is to assess fraud risk on behalf of companies. The more clients it has feeding it data, the stronger its signals become, so Company B is incentivized to gather as many sources of data as possible, and it wants those data to be as rich and as frequently updated as possible.

Company A gets back something like a score from Company B indicating how much risk you pose—whether or not you’re likely to try to scam them out of free services (or if you’re even a human or not). Assuming you’re fine, then Company A sends your info off to Company C, a credit card processor who is the one actually responsible for charging you money and giving it back to Company A.

Company C is collecting data as well because they stand the greatest risk during this transaction. They collect data themselves, and they’re almost certainly using a data broker of some kind as well—either Company B or more likely something else, a Company D.

These interactions happen quite quickly and, usually, smoothly. In a few seconds, enough info about you to identify your browsing patterns and correlate you with your purchase of Scarf Facts has now been aggregated by one or two data brokers.

These brokers sell their services to companies hoping to prevent fraud, and they make money because they are able to draw from ever larger sources of traffic and gain a clearer picture of the Internet. You agreed to this, but I doubt it was clear to you that entities other than you and Company A were involved.

If you’re wondering whether or not this is really happening, this sort of collection has become increasingly common as businesses have tried to compete with one another by reducing friction around their sign-up processes. Simple CAPTCHAs have not been enough to hold back the tide of automated and human attempts to overwhelm large and small businesses attempting to sell services and goods online, and they have turned to data-based solutions to fight back. We can’t wind back the clock to a simpler time.

Unfortunately, most people are uninvolved and have become bycatch in the vast nets we’ve spun. It is likely, as time goes on, that the brokers who collect and analyze the data collected this way will try to sell them, or analyses of them, to profit in other ways. The value of these data increases as they become more representative of the traffic of the Internet as a whole.

I’m not asking you to stop and read the T&C on the next website you sign up for. That’s ever going to be practical. But now you know about another piece of your soul you’re possibly chipping off in return for clicking “Accept.”

A Taxonomy of Disagreements

I share my world with people with whom I disagree. The question is how and when to act upon it.

Not every disagreement deserves the same reaction. It’s not strictly necessary that I find common ground in every disagreement, and not every disagreement requires my engagement. Even among the cross product of these categories, I can respond in different ways.

I view disagreements along two axes which I’ll call triviality and consensus. By triviality I mean that the subject matter has little impact on at least one party’s life. Consensus means that agreement must be reached; this is not an agree-to-disagree situation.

I’ll lay out what each combination means.

  • Trivial, non-consensus disagreements—disagreements about an unimportant subject which doesn’t strongly impact all parties, or does so unequally. Food preferences are a perfect example. If one person likes mayo, another likes Miracle Whip, and yet another thinks they’re both kind of unpleasant, this is a trivial disagreement. It’s also pretty irrelevant to disagree because nobody has to change their lives too much over this disagreement. Live and let live.
  • Trivial, consensus disagreements—disagreements about an unimportant subject which impacts all parties and for which a single decision needs to be made. This is common in families and offices, like setting the thermostat or choosing where to go for dinner. Contention over shared resources, or picking common tools or workflows at work, can lead to a lot of nitpicking, but the problem is solvable, sometimes even with a coin-toss.
  • Nontrivial, non-consensus disagreements—disagreements about a subject which impacts all parties strongly but for which consensus is not needed, or is even impossible. The most salient example is any question of faith. Faith doesn’t respond to reason and occupies maybe the most important part of some people’s self-identity and self-determination, but agreement over the details of faith or religion are impossible to bring into accord. It’s unrealistic to try. Yet we have to try to find some way to live with people of different faiths. The very intimate, personal nature of their beliefs makes them immutable—non-consensus, as I’m calling it—since we can’t all share a singular faith and probably wouldn’t want to.
  • Nontrivial, consensus disagreements—disagreements which impact all parties strongly and which require agreement. This is the really hard stuff: fundamental human rights, ethics, land-use rights, traffic laws, and so on. For these disagreements, I permit no quarter for non-consensus because I believe that aspects of human rights are both of paramount importance and cannot be yielded to, appeased, or ignored. To do so—to say “live and let live,” “agree to disagree,” to fundamental questions of humanity, dignity, life and death—gives those viewpoints with which I disagree a place to dwell, a platform from which to speak, and an implicit permission for action. The crossover between non-consensus and consensus for nontrivial disagreements begins at the threshold for potential harm.

Within the triviality axis, the consensus degree of freedom actually can be a bit blurry. Taking the trivial disagreements to start with, it’s easy to see where certain topics that should have been non-consensus have blended into consensus in people’s lives—like food preferences, which culture has buried with spades of shame and influence in order to make people eat the same things in the same ways. I work in tech, where similar things have happened for decades, such as the Editor Wars: who edits what and how on their own computer should be an agree-to-disagree situation, but it became a holy war.

Unfortunately, at the other triviality extreme, the same kinds of confusion take place. Nontrivial disagreements which should be non-consensus (which should look like agree-to-disagree) have become literal holy wars. Worse yet, disagreements about basic human dignity and rights have begun to look like agree-to-disagree situations.

I believe we all have a similar taxonomy in our heads, that we believe we’re “entitled to our opinions,” regarding certain questions of faith and politics. In some matters, we are. We’re entitled to our opinions regarding how much funding the Federal Highway Administration should get. Whatever my beliefs about interstate highways, I could break bread with a person who believes in gutting their funding.

However, the idea that we’re “entitled to our opinions” leads to a simplified taxonomy that doesn’t take into account which opinions—which disagreements—are over harmless questions and which are over potentially harmful, dehumanizing, or traumatizing ones.

More complicatedly yet, matters of faith—a place within many of us untouchable by consensus or persuasion—have enabled some people to spread the non-consensus umbrella over many other areas of their worldview, seeing them all as speciously linked by faith and therefore unimpeachable. As such, their political opinions about personhood, their ethical behaviors, their votes—no matter what their source, they are all placed into a category beyond rational discussion.

I have found myself exhorted to meet these people in the middle, to attempt to understand them, to “agree to disagree” with them, or to attempt to include them in wider political efforts to advance my own political will. These efforts often come from centrist-liberal sources.

What I’m here to tell you is that if your politics touches a human, if it has the potential to visit harm and suffering, if it detains a person, I have no place for you at my table, in my home, or in my life. If you use the idea of free expression to shirk the responsibility of examining your own ideas, you have abrogated your duty as a citizen under the guise of entitlement.

Truth, Light, and Statistics

Today, an article I wrote called “Truth, Light, and Statistics” got published as an online extra for The Recompiler, a local feminist hacker magazine edited by my friend Audrey.

I’m thrilled about finally getting it edited and published. I put a lot of care into it, the same way I’ve put a lot of care into improving my astrophotography over the years. The sense of the article is to contrast reality versus perception, signal versus noise—to show how photo-manipulation can sometimes paradoxically get us a little closer to the truth rather than take us farther away.

The best part about image stacking is how the very randomness of the sky’s turbulence provides the key to seeing through its own distortions, kind of like a mathematical judo. Read through if you want to find out how.

Ordinary Synesthesia

For the last fourteen years or so, I have privately described some of my sensory experiences as a phenomenon called synesthesia. I don’t talk about it much because I am not sure whether synesthesia is an accurate description. Over the years, though, I find that term still feels appropriate in many ways. Maybe it fits something you experience, too.

To talk meaningfully about what synesthesia is, I’m drawing from the first paper I read on the subject, one called “Synesthesia: Phenomenology and Neuropsychology” by a man named Richard E. Cytowic.3 Later research has appeared since 1995, so I’ve looked some of that up as well—much of that by Cytowic as well.

What is synesthesia? It’s an inextricable linking of distinct senses, such as sight and sound. Cytowic says this more with more academic language: “[T]he stimulation of one sensory modality reliably causes a perception in one or more different senses.” It’s not symbolic or metaphorical. It’s a literal, sensory experience that happens reliably.

Importantly, though, it’s not a mental disorder. It cannot be diagnosed using the ICD or DSM. There’s no hallucinatory aspect to synesthesia, and it does not impair those affected by it.

What do I mean by “linking of distinct senses”? There are numerous forms of synesthesia, but as an example, consider the form that associates sounds with visual experiences (like color). When a person who experiences synesthesia (a synesthete) hears a sound which triggers an association, that sound itself is perceived both as the sound and as the visual event (such as the color green). This isn’t to say that the synesthete has a hallucination in which the color green appears literally and visually before their eyes—a phenomenon that would only be described as a hallucination. What I mean instead is that the sound is itself the color green in their brain. By hearing it, they have experienced the color green, with all its appertaining associations.

There is a certain ineffable quality to that mixture of sensory experiences. Consider it for a moment. How would I know, as an unaware synesthete, that the color green is the correct association? I haven’t seen the color green in any literally visual sense.

I might make sense of this by working backwards from the associations green has in my mind—each tied both to the sound and to the color. Or else, I might find the color linked rather directly to the sound, working backwards from what associations the sound has in my mind. Stranger still, I might find associations between sounds and colors I haven’t even seen in reality.

Synesthesia seems to glom things together until the experiences occur not only simultaneously but literally as a unified sensory experience. To experience the trigger is to experience its association.

I believe this causes synesthesia to go under-observed and misunderstood. Many of us experience synesthesia without understanding it for what it is or how common it is, how subtle and integrated into our sensory experience. I don’t believe it’s universal, but I believe it’s possibly a widespread feature that exists on a spectrum.

I believe synesthesia-like phenomena underlie certain kinds of universal sound symbolism, such as the bouba/kiki effect, which has been found across different ages and cultures across time. Ramachandran and Hubbard did some influential experiments in this area.4

So as for me? I experience compelling visual sensations brought on by specific auditory experiences—in particular, music at certain frequencies. I didn’t have much breadth of exposure to music growing up (only hearing country music on radios around me until I was a teenager), so I didn’t really understand much about myself and music until I was nearly an adult.

I began to put it together when I was in a college class for music (with a powerful sound system), and I found myself instinctively blinking and averting my eyes while listening to some baroque music, and for the first time I realized how forcefully visual the music became for me. I started reading more about synesthesia and thought maybe this was a reality for me. Since then, I’ve learned some of the details of how music affects me.

My experiences have some color components, but I struggle to describe what those colors are, beyond cool or warm. They often have textural or spatial components, disjointed in space nearby.

Percussive sounds cause white or otherwise desaturated interruptions in the visual experience. They are like visual noise—snow, static, walls. I tend to seek out music which avoids or minimizes percussion.

Vocal accompaniment causes almost no visual sensation whatsoever. I tend to ignore vocals in music or seek out purely instrumental music. Highly distorted, distinctly stylistic, or highly polyphonic vocals are an exception.

Higher pitched sounds tend to have stronger associations, but I get fuller, more textured experiences from richer musical arrangements. These can be classical, electronic, guitar bands, or whatever.

Sounds of different pitches or timbres tend to make themselves more or less visually salient. Usually higher pitches layer over or through lower ones and have more compact visual representations, warmer colors. The progressions of melodies and overall chord progressions tend to lead to eddies and swirls.

Chromaticism from modernist compositions cause some of the most interesting visuals. “Clair de lune” starts with such rich, variegated lavenders, which yield then to legato scintillations of all colors, covered with lots of warm notes, like stars embedded in a cool sky. The Tristan chord from Tristan und Isolde felt like a greenish-yellowish blight melting into a veil billowing in the wind as the prelude carried into further dissonances—while the final “Liebestod” glowed like a hot, clean pink for me.5 “Aquarium” from Le carnaval des animaux by Camille Saint-Saëns (you probably know it as “that music that plays in cartoons when someone is underwater”) has all these piano glissandos riding over top which cause indescribable motes of light to flit away.

I don’t believe I’d call synesthesia (if that’s what this is) a blessing or a curse. They simply shape the way I enjoy music. I find them vivid, memorable, and affecting—they add a substance. I’m glad it’s there, but I don’t really have any explanation for it, and I enjoy plenty of things without it. I’ve found it gives me a better sensory recollection for things that happen while I’m listening to music, but that might be the only benefit.

I don’t really talk about synesthesia. (I searched my Twitter account for mentions, and I see I’ve only ever mentioned the word once before today.) It’s an extremely personal, subjective experience, and part of it is ineffable. It’s like describing a dream—no one really cares but you.

Since there’s no way to convey the affect portion of the experience, it’s hard to communicate your intentions. It sounds like an attempt to make yourself seem special or gifted in some way. Synesthesia has been associated with artists and art since the 1800s, especially musical composers. It became faddish enough for a time that it was even popular to fake aspects of it.

I want to emphasize again that I believe there is a universal quality to sensory crossover. My personal belief is that synesthesia-like experiences exist on a spectrum in many people—some more than others. The more we talk about it for what it is and how it actually is experienced, the more readily others will recognize the experience in themselves and normalize it.

For this reason, I don’t want to state definitively I have synesthesia. I’m not saying that. I will say that I have experiences that feel could be appropriately described by the term, so I wouldn’t rule it out. I imagine that many people feel like I do or have some similar quality to their sensorium. I just want to open us up to the possibility of synesthesia being ordinary.

Thematic Rewriting

I have been revisiting On Thematic Storytelling in my thoughts lately. Part of it is because I’ve been helping a friend story-doctor their writing a little. It’s also because I’ve been dwelling on my own story notes and refining them.

This has led me to questions I had not considered before. First of all, why do we write in symbolic, allegorical ways in the first place? Secondly, how do these themes end up in our stories at all, ready to develop, even if we don’t set out to use those themes at first? I think the answers to these questions are linked.

People have a long history of telling fables and parables to relate messages to one another, using figurative, allusive language. I believe this works because humans are designed to learn by example. We have wiring which internalizes vicarious experiences and reifies them as personal ones.6 Allegorical stories, like fables, adapt our ability to learn through example by employing our imagination.

We respond to fables well because of their indirection. On the one hand, it may be easier just to state the moral of a story outright and not even bother with telling “The Grasshopper and the Ant” or “The Tortoise and the Hare.” However, a moral without its fable is only a command. By telling the stories, the storyteller guides the listener through the journey. Figurative language and characters who represent ideas help to involve the listener and keep them engaged. The moral comes through naturally by the end as a result of the listener following the internal logic of the narrative, so the storyteller’s intended meaning does not need to be inculcated extrinsically.

In this way, indirect stories use symbolic language to draw in listeners. We, listening to the story, relate to the figures and characters because they allow us to involve ourselves. In turn, because we get invested, we take parts of the story and make them about ourselves. We care about what happens and empathize with the characters because we care about ourselves. This is what I actually mean when I say fables work well because of their indirection. We’re not actually interested in grasshoppers and ants, tortoises and hares, but we are interested in representations of our own values, our setbacks, and our triumphs. We put parts of ourselves into the story and get something else back.

And this is why I believe fables and parables have such staying power. Mythologies endure for similar reasons: their pantheons, even if filled with otherworldly gods or spirits, explain the ordinary—the sun, the night, the ocean, the sky—and embody our own better and worser natures—love, anger, and so on. In these myths we see ourselves, our friends, our enemies, and our histories.

So on the one hand, highly figurative language involves the audience in the way that literal language does not. How do we write like this in the first place?

Fables, parables, myths, allegories—they all use symbols that have endured over the centuries and been recapitulated in various ways, but in one way or another, they’re told using a long-lived cultural language. When we tell stories, when we write, we use the basic words of our native language, and with those come the building blocks of our cultural language as well. It is as difficult to avoid using these as it is to write a novel without the letter “e.”7

We may not often think about the kinds of cultural language we use because we’re unaware of where it comes from. This is one of the primary goals of studying literature, to learn about the breadth of prior influences so we can study our “cultural language” (I am not sure what a better word for this is, but I am sure there is one). Even when we don’t intend to dwell on influences and allusions, we write with the language of symbols that surrounds us.

What’s interesting to consider is what we’re saying without always thinking about it. Just as we grew up with stories that drew us in using powerful symbolic language, we imbue our original stories with ourselves, using similar symbols.

I’ve realized that different writers tend to perseverate on different kinds of questions and beliefs as their experiences allow, and these emerge as common themes in their writing, the same way certain stock characters persist in an author’s repertoire. If, for example, I find myself primarily concerned with questions of faith, my stories may spontaneously concenter themselves around themes of faith, through no real intentional process. In the process, I might even embed symbols which convey the theme without meaning it (for example, religious trappings such as lambs, crosses, or even clergy).

I have come to identify themes and symbols which are either inherent to the story itself or accidentally embedded by my execution as part of the planning and editing process in my writing. Once I understand them, then decide whether to keep those and how to refine and harmonize them. For example, if I do have religious symbols within a story, there are unavoidable allusions this implies, and I have to work through how to harmonize those with my story or cut them out. As another example, if I have a character who is alone on a desert island, the themes of isolation and survival are both inherent parts of the story structure which cannot be avoided and will be addressed in some way or another. If I write about political conflict, then cooperation-versus-competition is lurking behind nearly every character’s motivation.

In a practical sense, how do I develop themes and work in symbols? Generally, editing first occurs at a larger scale and then moves to a smaller scale, so I tend to think in similar terms with themes. I identify whether broader themes already exist and ask myself if they carry through the entire narrative. If there is an inchoate message buried in the subtext that I didn’t intend to put there, I should decide if it belongs or not. If I want to keep it, then I need to clarify what it is and how it works as a theme.

I examine the narrative through the point of view of this theme and see which elements fit and which don’t. I see how I can adapt it better to fit the theme—a process I actually love because it often burnishes rough narrative ideas.

To give an idea of what I mean, I’ve been writing a story whose central theme has to do with disintegrity of mind and body—the feeling of being not only under a microscope but flayed open at the same time. I began with a premise that didn’t really involve this theme, but I synthesized ideas from elsewhere that really pushed the story in that direction. When I began considering reworking the style and ending, I realized I needed more narrative disorientation and ambiguity to convey the protagonist’s feeling of disintegrity. The changes I had to make involved researching certain plays and removing dialogue tags to make it uncertain who’s speaking (implying that the protagonist could be speaking when she believes her interrogator to be speaking).

Before I go on, I also ask myself what the theme means for me. If I were making a statement about the theme, what would I want to say? More to the point, what does the story have to say? Sometimes, there are even multiple themes in conflict—hints at determinism here, others at fatalism there—so that the overall picture gets confused. The most important thing is that the narrative contains an internal logic that works at every scale, from the broadest thematic sense to the word-by-word meaning. I consider—at least at some point, even if it’s only after a draft is done—what the overall story is saying and then ensure no smaller element of the story contradicts itself.

After I’ve made some large-scale changes, there may be smaller narrative gaps to fill, and I find I can also add certain ornamentations to settings or characters based on theme. This is where I can use the language of symbolism. I try to be somewhat coy. Like I said, indirect, allegorical language allows for stories that are more interesting because they’re more relatable and let the reader insert themselves. The illusion falls apart if the allegory is naked and obvious.

I don’t mean that I necessarily want to make symbols which are obscure allusions, either. I personally like symbols which have a logic within the narrative. I believe it’s possible both ways. The Lord of the Flies is an example of an highly allegorical novel which uses symbols this way. The conch shell symbolizes civilization because it’s used as a rallying point for the boys who remain faithful to the rules. Golding embeds its symbolism completely within the narrative logic—expressed in terms of the story—and the idea it represents erodes as the physical item is neglected and then destroyed.

Sometimes I’m not working with a story but just a premise, and it’s one to which many themes could attach. I could choose a theme, and that choice would influence the direction in which I take the premise. A lot of the ideas I decide to develop end up being conglomerations of ideas, and I’m never quite sure which ones should go together. Themes can sometimes be links which join disparate ideas into a framework, allowing me to decide what to synthesize and how. This way, a premise and a theme determines how a story grows and what characters, settings, and events I place into it.

It may seem like a lot of effort to run through this exercise for a story which is purely fanciful entertainment, which sets out not to say anything in the first place. Not everyone sets out to write an allegory. However, like I said, I think to some extent it’s not possible to avoid planting some subtextual themes because we all speak with a shared cultural language. My goal is to consider what I say between the lines and harmonize that thematic content. Hopefully, I end up with a story with a wider meaning running through it, giving it some backbone. I never set out to make a moral—maybe at most a statement?—but I do at least try to structure my narrative ideas to make the most impact.

I am extraordinarily grateful to Zuzu O. for the time and care she put into editing this post.

Removing: PGP Key and Content Licensing

I have removed two pages from my website—my PGP key and my content license.

PGP Key

I removed my public PGP key because I no longer intend to use it for signing messages. My same key remains on Keybase and other public key servers, but I no longer sign outgoing mail, nor do I intend to use my key regularly in any way.

I don’t feel that my key has been compromised. However, it does me little good to keep using it, and most encryption in actual use in my daily life doesn’t involve PGP. In the wake of a mostly minor vulnerability called E-Fail earlier this month—which didn’t impact me—I found myself persuaded of the ultimate futility of keeping up with PGP by an op-ed on Ars Technica from a couple of years ago.

Content Licensing

I removed my CC BY-NC 4.0 license notice page from my site. This post hereby serves as notice that from this date forward, I no longer license my existing or new content (writing, photos, videos, or audio) under a CC license, and so all that content falls back to the default copyright of the relevant jurisdiction.

Any works that have already been used under the CC license have been done so irrevocably, and so I have no ability to revoke those licenses. They may be licensed until their rights lapse under the copyright laws of those jurisdictions.

If anyone wants to use some of mine, they are certainly welcome. The removal of the license only implies one practical change—you must ask permission. That’s all.