Why is this thus? And what is the reason for this thusness?
— Artemus Ward, Moses, the Sassy
And if you gaze long into a navel, the navel also gazes into you.
— Friedrich Wilhelm Nietzsche, Beyond Good and Evil (paraphrase)
quid·di·ty (/ˈkwɪdɪti/; Latin: quidditas) n.: the particular qualities shared by all things within a category.
haec·cei·ty (/hɛkˈsiːɪti, hiːk-/; Latin: haecceitas) n.: the particular qualities which distinguish a thing from others within a category.
In the High Middle Ages, European medieval scholars recovered the Greek classics, thanks to their Muslim friends who had preserved the works of Aristotle, Plato, Euclid, and the rest. The Europeans took a very peculiar point of view on these works. They assumed the ancients had already figured out the universe. The only task remaining to them, the medieval scholars, was to figure out what the ancient writing meant.
They did this through arguing—either in person or through commentaries they handed down to one another through the years. This intellectual slap-fighting gets the fancy name of “dialectical reasoning.” They found a lot to argue about. Sometimes, the scholars would even invite the general public to ask questions in an ask-me-anything format they’d call quodlibetical questions. Making the ancient explanations of the world make sense in the face of both Christianity and how the world actually worked involved a lot of rhetoric.
This environment served as a fecund ground for lengthy discourses on how existence itself worked. The hard work of trying to reconcile all the things which medieval scholars required to be true about the world ended up being called scholasticism. I’m going to level with you here—most of it is of little relevance today. What, you never heard of such towering intellectual figures as William of Auvergne or Henry of Ghent?
You must’ve heard of Duns Scotus. He’s considered one of the “three great” philosophers of scholasticism, along with Thomas Aquinas (a guy you probably did hear about in school) and William of Ockham (a guy you probably remember from his razor commercials). Just in case you haven’t heard of Duns Scotus, a few pretty interesting ideas came out of this guy, like quiddity and haecceity.
He asked, what features of a thing made it belong to a category (quiddity)? What features of a thing distinguished it from other members in a category (haecceity)? This goes deeper than merely identifying what something is or telling apart two things.
Asking about a chair’s quiddity, for example, was akin to asking what features made it a chair at all. What changes could you make to a chair until it would no longer be a chair? (Do you recognize echoes of Platonism?) This is a little like a Ship of Theseus question: If I began to shave off one millimeter of a chair with a giant blade, with which cut does it cease to be a chair? What property did I break? What’s more, does the quiddity of chairs still exist even if I break every chair in the world? When the first chair was built, did that quiddity pop into existence then?
Haecceity, on the other hand, went beyond distinguishing among many objects. It was about identity itself. What made a thing one thing and not another? Why are you you and not someone else? Why do you remain you? What gave rise to plurality in the first place? This is to say, why is quiddity even distinct from haecceity? Why don’t we live in a universe where there is a single one of everything, where the idea of a thing is itself the singular instantiation of it?
And here, I’ve only considered a concrete object. In a pluralistic, materialistic universe, it’s trivial to think about individual instances of a category of thing—about something’s haecceity—until you start to apply it to abstract concepts, which aren’t material. What about colors? Can you have multiple individual instances of the concept of the color red? What does that mean? If there aren’t individual instances, does that mean that, out there, somewhere, is there some universal, singular instance of red? Do abstract ideas really exist in any real way in a materialistic universe due to the concept of haecceity?
Duns Scotus probably would’ve enjoyed programming languages, if only because they reify these questions as design decisions. I’m imagining him learning Common Lisp and discovering keyword symbols. What is the haecceity of an interned keyword symbol, like :ordnatio? There’s only one in the entire Lisp interpreter. It’s a unity of being.
Other things are universal in their being, too, within the interpreter, like packages and functions. Yet they have a real existence—a singular universal hosted within the interpreter forever (or at least until garbage collection). There are even abstract concepts which are built into Lisp itself—like the taxonomy of the empty list, or nil (the only false-like concept Lisp acknowledges), versus not-empty/not-nil/true (all the same when taken as an expression).
Since I’m certain you all know Common Lisp inside and out already, maybe looking at Ruby would be interesting. Scotus would probably notice first how Ruby “borrowed” keyword symbols as its own symbol feature. Same idea—once the symbol exists (a kind of Platonic ideal floating in the ethereal stack of the Ruby interpreter), it’s lodged there for good and reused whenever the same symbol is referenced. It’s a symbol the way the color “red” is a symbol for an idea in reality, a proxy for an abstraction.
Ruby is strenuously object-oriented, though. Object-oriented programming is thought to let programmers model the real world more easily. Scotus might have agreed. Classes map pretty well to quiddities; objects, to haecceities. Before object-oriented programming, most things lived singularly and eternally (except for, say, automatic or dynamic variables): functions, structures, literal values, and so on.
A quiddity can be described as a class, a conceptual proxy cookie-cutter from which individual objects can be instantiated. Haecceity is realized as an #object_id method which distinguishes each instance and helps Ruby keep track of which object is which in a durable way.
But haecceity can be more than just distinguishing things from one another or even maintaining identity. It includes the nature of things as individual. Many object-oriented programming languages include an idea that an object may take on a life of its own, but Ruby centers the object over the class in multiple ways.
You can create objects entirely without classes at all and imbue them with attributes and methods by calling Object#new.
scrooge = Object.new
puts 'life is like a hurricane'
scrooge.lyric # prints 'life is like a hurricane'
What is the quiddity of scrooge? It is something like an Object (a base class in Ruby), but it has taken on a new life, and it has a quality distinct from all other Objects. Does it still belong to the class of Object?
Ruby makes a distinction here which I think would make Duns Scotus look like Dunce Scotus. You’re welcome to ask Ruby what class that scrooge belongs to, using the #class method, and you’d get the answer of Object, but you can also ask for its #singleton_class, and you get something faintly different: #<Class:#<Object:0x00007f95017ed2e8>> (or similar). Ruby is communicating that scrooge has a singleton class which started as an Object but is now its own thing. Its quiddities have their own haecceities: the categories of things are not just immutable, singular universals but individuable concepts with pluralistic existences which you create at will anytime you instantiate an object. Ruby gives every object its own singleton class because it centers the object over the class, and in doing so, it acknowledges that each object has its own destiny which can be described as having its own qualities.
For what it’s worth, I am pretty sure that examining the idea of Ruby’s singleton class via medieval scholastic philosophy cannot be significantly more convoluted than any other explanation of the idea I’ve read online. Considering that I haven’t done any proofreading, research, or fact-checking, it’s bound to be at least fifty-percent correct, but you should check your local Ruby install for more information about singleton classes, Duns Scotus, or scholastic philosophy.
Gravitational-wave detection began just a few short years ago, and only starting in 2017, scientists began to get the first really interesting observations. These most recent events have enabled us to understand fundamental physics better by ruling out or circumscribing a few grand theories of everything, and they have allowed us to see the universe in ways we never could before.
Yet the events themselves are mere “inspiral chirps” which happen in less than a second and usually at a remove of a billion parsecs or more. How can something so brief and so distant, be so meaningful? That’s what today’s explainer is all about.
To get the big deal about gravitational waves, let’s review what we know about gravity.
What Are Gravitational Waves?
Gravity happens because mass warps space and time. All mass does this, and the warping effect stretches out across the universe infinitely. Even you, as you sit reading this, cause a very tiny change in the space in your vicinity and in the passage of time, and that extends beyond you out into space forever, however faintly.
Two things are important to note, though. First, the degree to which any object warps space around it weakens very sharply as you move away from it. This is because the farther you get from it, the amount of space there is to warp grows exponentially, so the warpage falls off in turn. (This is a kind of conservation law, known as the inverse-square law.) Second, that warpage—that gravity—doesn’t travel instantly. It’s bound by the same speed limit as everything else, just as special relativity requires.
So given these facts, we know that gravity spreads out in much the same way light does. How does it form waves? Waves result from any cyclical phenomenon which traverses a distance. Think of a cork bobbing in a pond. The cork, stationary, merely moves up and down, but the ripples move out in waves. Waves made of gravity can therefore ripple outward anytime a source of mass changes in a repeating way.
This is what gravitational waves are—cyclical changes in gravity. The ones we can detect result from vastly large masses spiraling in toward one another extremely rapidly and colliding. Their circling motion causes the gravity from them to ripple out in a pattern of repetitive change—in waves—as the masses revolve around each other. This spiraling pattern causes the masses to alternate positions quickly, sometimes lining up or sometimes sitting side-by-side (from our point of view). As they revolve, they also draw closer and closer to one another. Finally, when the masses collide, the wave source stops in a sudden “inspiral chirp”—so called because the gravitational wave is so rapid and stops so suddenly, it sounds like a chirping sound when played as audio.
How Do We Detect Them?
We detect gravitational waves with a lasers, of course. Actually, it’s a bit more complicated. There are multiple lasers. And we bounce them down tunnels (called “arms”) over four kilometers long—long enough that the Earth curves downward by a meter over their length—and back.
When a gravitational wave ripples through a LIGO facility, the phenomenon literally causes those arms to change shape and size according to general relativity. The facility is a vast instrument called an interferometer which causes the lasers to interfere with one another in a very specific and measurable way.
The idea is that a pair of lasers are fired over a very great distance (through the arms) and bounced back to the detector, which is a specialized kind of digital camera. When they bounce back, they’re meant to interfere with each other in a very precise way because of how they overlap when they hit the detector. However, minute vibrations upset the delicate interference pattern, and the detector can see that.
The lasers have to be so long because they’re directly measuring very tiny warpages in the shape of spacetime itself. Gravitational waves ripple out from violent but brief, distant events, and so these instruments must be extraordinarily sensitive. LIGO reports that at its most sensitive, it can detect a change in distance ten-thousandth the width of a single proton. The facility in Hanford, Washington, detects vibrations so sensitive that it can pick up ocean waves crashing on the beach several hundred miles away.
Using multiple facilities located in different locations, it’s possible to detect gravitational waves very quickly using advanced, purpose-made software (used to separate the data from the noise) and roughly locate the source in the sky.
What Do We Do With This Information?
Gravitational-wave detection is one of the newest and most profound breakthroughs in recent observational cosmology. Even merely detecting a gravitational wave is a feat not to be understated—it signifies that we have directly measured a ripple in the fabric of spacetime itself and further cemented the theory of general relativity. It took nearly a century after their first theoretical prediction to achieve a direct detection.
Gravitational-wave astronomy gives us our first look at the universe beyond electromagnetic radiation (light, infrared, x-rays, and so on). We are finally able to see the ripples of the pond in which we all live, not just the specks of light. Gravity behaves differently than EM radiation in several important ways, so it promises new insights into massive phenomena like neutron stars, supermassive black holes, and the like—all at incredible distances difficult to observe otherwise. The promise of revelations into the formation of galaxies, exotic phenomena, dark matter, or even the creation of the universe all await.
Already, though, we’ve seen the birth of a new form of astronomy altogether called multi-messenger astronomy which combines both gravitational wave observations along with traditional radio or optical telescopic observations of the same event. Until now, humanity has only ever been able to see the light from the stars and make educated guesses about distance, mass, and so on. What’s more, we still have more questions than answers about how EM radiation and gravity relate to one another. The most fundamental explanations of all of creation, from the subatomic level to the cosmic level, depend on answers to these questions.
The first event observed via both gravity and light was called GW170817. Gravitational waves from this event was detected by three detector facilities in real time, and a corresponding gamma-ray burst (the most violent kind of explosion in the universe) was found at the same location in the sky by dozens of observatories. This event, which is thought to be two neutron stars colliding, has already taught us new things and begun to constrain models of fundamental physics.
For example, since it was observed via both light and gravity, we can compare the time it took for both to reach us and see what differences may exist. Some grand unifying theories of everything thought that perhaps gravity would take longer to cross the distance to us because it had to travel differently (through hidden, “compactified” spacetime dimensions, for example). Since that didn’t happen, those theoretical physicists will have to go back to the drawing board.
Gravity travels unattenuated by dust and unscattered over vast distances. Events like GW170817 travel over distances only affected by other masses, allowing us to “see” the universe in a different and maybe clearer way. Some scientists hope that we may even find primordial gravitational waves leftover from the earliest epochs of the universe, before even light could emerge because matter was too dense. Gravitational waves may let us pierce the wall of creation’s primordial fire and look beyond into nearly the very earliest moments of the universe itself.
Gravitational-wave astronomy and multi-messenger astronomy are extraordinarily young sciences. The data from the events we’ve observed are still being pored over by scientists as they attempt to make or break new theories and find new signals in the noise.
In the future, we may be able to put extraordinarily large interferometers into space which extend over massive distances and which would not be subject to earthly vibrations such as trucks, oceans, footfalls, or earthquakes. One such planned project is called LISA. We would be able to observe many more sources of waves with such a detector, even ones within our own galaxy. Perhaps we will even find sources of gravitational waves we never even expected. We’re standing at the verge of a whole new universe.
An old story relates that Newton figured out gravity when an apple fell on his head. Newton himself doesn’t mention the apple falling on his head—this appears to be a later embellishment—but he does mention the apple anecdote a couple of times in his dotage. John Conduitt remembered,
In the year 1666 [Newton] retired again from Cambridge to his mother in Lincolnshire. Whilst he was pensively meandering in a garden it came into his thought that the power of gravity (which brought an apple from a tree to the ground) was not limited to a certain distance from Earth, but that this power must extend much further than was usually thought.
Why not as high as the Moon said he to himself & if so, that must influence her motion & perhaps retain her orbit, whereupon he fell a calculating what would be the effect of that supposition.
This anecdote describes a key quality of gravity as understood then: its nature as an occult force—something working mysteriously and unseen across space.
Before Newton, it was known that the planets moved according to well known laws (Kepler’s laws) which allowed their motions to be predictable. It was not understood, however, why they should move in that way. Kepler’s laws merely came from generalizations after many observations.
Philosophers at the time were troubled that the planets appeared to have no reason to move as they did. Aristotelian thought required that something must drive the planets in their motions. If concentric spheres of quintessence did not, what could this be? For a while, we believed space might be full of a kind of fluid which moved in vortices which propelled the planets like clockwork. This explanation was unexpectedly successful for decades precisely because it did not require belief in occult forces—which is to say, it didn’t require something invisible to reach magically over distances and cause a thing to happen without touching it. It pushed instead of pulled.
Newton had looked at the apple and realized nothing had pushed it to the ground. It seemed to have fallen of its own accord. Newton then extrapolated this idea out beyond the garden into the stars. Once he did, a compact set of laws allowed him to explain all the motions of the heavens very tidily. His explanation, eventually known as the Principia, laid the groundwork for fundamental physics for centuries to come. It was a feat on par with Euclid’s Elements and fully completed the Scientific Revolution which Galileo had inaugurated.
From Hypotheses to Theories
In the second edition of the Principia, Newton tacked on some notes by popular demand. In this General Scholium, he explained that he was in no position to explain what gravity could tangibly be. Famously, he said, “Hypotheses non fingo” (“I do not feign hypotheses [of what gravity could be]”). He described nature as he found it, and the explanation worked. That’s how the matter lay for centuries.
One problem is that, over time, we observed that Newton’s explanations were not perfect after all. There were subtle but galling errors which cropped up in very rare circumstances (such as predicting where Mercury would be over time). Another problem was more metaphysical—Newton’s laws only explained how gravity worked, not what it was.
Einstein solved both problems in a single stroke with general relativity. His theory of general relativity followed in the decade after special relativity as a consequence of the latter. The general theory extended the special one to more situations and provided a more fundamental explanation of universal phenomena, particularly gravity.
Equivalence All the Way Down
If you’ve made it this far, you’ve read how energy is an impetus to change over time. Motion can be a form of energy because it can impart motion on another object, accelerating it. Energy is also equivalent to mass, and mass to energy—even at rest. Finally, you’ve seen how motion itself changes energy, space, and time relative to someone observing the motion.
Now we add a new equivalence—one so incredible in its implications that Einstein called it his “happiest thought.” It’s now simply known as the equivalence principle, special enough to stand alone by that name. It states that it’s impossible to distinguish between acceleration and gravity in any real, physical way.
That is to say, if you were trapped in some enclosed box and unable to see outside, you could not devise any instrument which would be able to tell you whether that box were accelerating in some direction steadily (and therefore drawing you toward the floor) or within a gravitational field (which would accomplish the same effect). Therefore, experiencing acceleration is equivalent to experiencing a gravitational field.
Einstein realized this in November 1907. From that point, he realized that energy, mass, space, time, and gravity were all inseparably linked, and he spent the next several years feverishly working toward a general theory of relativity to explain how it all works. The explanation he came up with in 1915 works so well that its predictive power overturned Newton and has held up even to this day.
Motion in a Bottle
As a result of special relativity, we saw that motion warps space and time. We also know that motion relative to an observer represents kinetic energy, which is equivalent to any other form of energy. Finally, we know that energy is equivalent to mass and vice versa. The final piece of the puzzle to put into place here is that, since motion—and therefore energy—warps time and space, so does mass.
Think of mass as bottled motion. Mass–energy equivalence lets us treat mass as energy which has congealed, more or less, into one place. As I said in the last essay, it’s not enough to think of mass and energy as distinct things sharing some properties—they are a single substance. Therefore, all the same properties and consequences which apply to one form also apply to the other. That means that all the warping effects which apply to energy—to motion—also apply to mass.
So mass warps time and space, but what does this actually mean in reality? The result is gravity! Gravity is an emergent consequence of how mass warps time and space, exactly the same way motion warps time and space due to special relativity. Gravity is in fact not a force reaching mysteriously across distances but instead a bending of space and time which changes the paths of objects traveling through that space and time, leading them inexorably closer to one another.
The Conservative Appeal of Gravity
Let’s dispense with the tired bowling-ball-on-a-rubber-sheet imagery and talk about what that last paragraph actually means. We can begin with the classic assumptions about how objects behave. Newton’s laws state that objects in motion tend to stay in motion, or at rest, unless acted on. They also state that there’s always an opposite and equal reaction for every action.
These are, at their heart, conservation laws. For things to behave otherwise would mean creating or destroying energy. An action must impart an opposite and equal reaction, or energy would go missing. An object at rest must stay at rest, or energy would spontaneously appear. An object in motion must stay in motion, or energy would vanish.
In flat space, therefore, moving objects tend to stay the course in order to conserve energy. You can trace the line of how the object moves geometrically as a straight line. Now if we introduce a mass nearby, space and time contract and stretch, respectively, in the vicinity of that mass. The object’s path still needs to conserve energy, and in order to do so, the line we trace now curves closer to the mass. It appears as if the object “falls” inwards toward the mass—exactly as you’d expect from a gravitational field.
Occult Forces and Fictitious Forces
We no longer need an “occult force” to explain the mechanism of gravity. General relativity—which geometrically describes space and time as it bends under the influence of mass and energy—provides the complete picture.
As it turns out, gravity is not a force at all in the ordinary sense. It only appears to exert a force in the way that a merry-go-round in motion appears to make a ball curve through the air when you throw it from one side to the other. Gravity plays a similar trick on us: we’re constantly on a path through time and space which, were it not for the gigantic rock beneath us, would cause us to curve inexorably toward the center of the Earth. Since the Earth itself interrupts our course, we press against it, and it against us, which imparts the force we’re familiar with.
By uniting conservation laws and a handful of postulates, we can fully explain the substance and behavior of gravity. When we combine this knowledge with the speed limit of the universe, we see that even gravity takes time to travel, which means that changes in gravity take time to travel. This allows gravity to ripple across space and time. We’ll now be prepared to look at these waves in the next explainer.
The most recognizable equation of the twentieth century equates mass to energy.
\(E = mc^2\)
Specifically, this equation relates a very small quantity of mass to a huge quantity of energy. Why should that be true? What does it imply?
It follows as a consequence of special relativity—one which emerged only after Einstein and his friends worked out the initial theory when they considered how energy is conserved.
Energy: Always Transforming Yet Never Changing
The conservation of energy means that it is never created or destroyed; it only changes form. This cosmic bookkeeping of energy suggests that its different forms are in fact the same underlying phenomenon which is conserved in quantity through each transformation. We should look at this idea more closely, though.
What is energy? This is actually a hard question to answer in an univocal way, but we should adopt a definition useful for our purposes. I will describe energy as that quantitative property of an object which provides an impetus to change over time.
For example, kinetic energy is capable of accelerating an object. Chemical energy is capable of inducing a chemical reaction and changing one substance into another. Nuclear energy is capable of driving a nuclear reaction, thereby disassembling an atomic nucleus. All these forms of energy are equivalent in their respective quantities.
Being equivalent, they may endlessly transform from one form to another. Friction is an example of a phenomenon by which kinetic energy becomes heat energy. We can turn heat back into motion (assuming we could capture it with perfect efficiency, though we can’t) by using the heat to drive a turbine. We can turn the rotational motion of a turbine into electrical energy using induction, and we can transform that into light, sound, motion, magnetism, or heat all over again.
Everyday life relies on hundreds of examples of energy transformations. Even if we can’t capture and reuse energy with total efficiency, we always can always measure it and account for it. Since energy is always perfectly conserved, it makes sense to think of energy as a single phenomenon which changes form endlessly.
While the train is in motion, it has energy—kinetic energy—relative to an observer standing on the ground watching it pass. The faster the train goes, the more velocity it has, the more kinetic energy it acquires, and so on.
However, our observer standing on the ground has already noticed odd effects due to the cosmic bookkeeping which makes special relativity work. That train is getting shorter, and the time on the train is getting slower. Our thought experiment has significantly exaggerated the effects of special relativity because we’ve lowered the speed limit of the universe to 100 km/h. Otherwise, everything we see obeys the actual laws of physics.
Kinetic energy is only dependent on the mass and velocity of an object: as both increase, so does the kinetic energy. This fact remains true whether you consider special relativity or not. However, instead of being half the product of the mass and the square of the speed, as in classical mechanics, the kinetic energy instead tends toward infinity as we approach the speed limit in relativistic physics. As motion begins to warp time and space the closer we come to the speed limit, it must make similar adjustments to kinetic energy.
Points of View in Collision
If kinetic energy didn’t consider the speed limit of the universe, energy would not be conserved properly. Stranger yet, these consequences affect not only energy but mass. We can show this with an example.
Imagine watching from the train as someone throws a ball at 100 km/h to us standing still on the ground watching the train pass at 30 km/h. The person who threw it—who is moving along at the same speed with the ball—doesn’t see anything out of the ordinary with the ball’s motion, energy, or momentum. It moves at 100 km/h relative to them because that’s the speed at which they threw it.
From our vantage point on the ground, we also see the ball arrive at 100 km/h because that’s the speed limit of the universe. The train has been moving at 30 km/h, and so the train imparted some kinetic energy onto the ball, even if the train could not in fact make the ball go any faster than the speed limit. Although the ball is stuck moving at the speed limit, it took some kinetic energy from the train regardless. We know this because the ball imparted an opposite and equal reaction to the train as it was thrown. This means the train lost some energy, and that energy has to go somewhere—so it went into the ball.
We catch the ball at 100 km/h, but the ball somehow has more kinetic energy—and more momentum—than it should because it was thrown from a 30 km/h train. We feel that additional energy in the impact when we catch it. It makes a louder thud in our catcher’s mitt, too. Yet it’s not going any faster than a 100 km/h ball thrown from the ground.
What could be happening here? Here’s more cosmic bookkeeping: since we know the ball cannot move any faster than 100 km/h in our thought experiment, some other quantity has to increase to make up the difference. We also know that kinetic energy relates velocity and mass to one another. The only two things which impart more energy to an impact is adding a heavier object or making it go faster. Therefore, if velocity must stay constant, then mass must increase as a result. We are forced to conclude that the energy imparted onto the ball has added to the mass content of the ball instead of the velocity.
The amount of mass added isn’t much, to be sure—just enough to make up the difference between one ball thrown at 100 km/h and another ball thrown at 130 km/h. Remember also that we’ve lowered the speed limit of our imaginary universe, which exaggerates all the effects. In reality, the speed limit is actually about 1,079,252,848.8 km/h, so differences in speed impart vanishingly tiny bits of mass because ordinary, everyday speeds are tiny in proportion to the universal speed limit, \(c\). The difference in mass to “make up” the missing velocity is usually quite small.
Velocity isn’t the only quantity which “turns into” mass, due to the way energy transforms. All forms of energy are equivalent, so they all represent some amount of mass which can be quantified and calculated.
Once we take this idea to its logical conclusion, we hit upon the unavoidable consequence that the relationship works in reverse as well—that all forms of mass also are equivalent to energy and are quantifiable as such. Even mass at rest has some energy content, the amount of which grows as the mass is set into motion. Motion merely increases the mass–energy.
The Implications of Mass–energy Equivalence
As we just worked out, the math works out such that any tiny bit of mass adds up to an enormous amount of energy, thanks to the fact that the speed of light is so fast. For this reason, it took us a very long time to notice or test this phenomenon.
For example, one kilogram is equivalent to almost ninety quadrillion joules of energy. That’s the same energy output as a twenty-one megaton bomb, or four-fifths the energy output of the 2004 Indian Ocean earthquake and tsunami. In the other direction, the output of a sixty-watt incandescent bulb over an hour—both its light and heat—weighs only 2.4 nanograms, or about the mass of thirty red blood cells.
Special relativity implies that mass and energy are in actuality a single underlying phenomenon, called mass–energy, which we encounter in two familiar forms. In other words, they’re not just similar on some level—they literally are the same thing. Consider, for example, that the Earth weighs approximately 2.38 billion metric tons more due just to the rotational energy of spinning than it would if we changed nothing at all except to cause it not to spin. To stop the world from spinning would be the equivalent of shedding over thirteen million blue whales of mass.
From the seemingly contradictory postulates of the principle of relativity and the invariance of the speed of light, we have been able to learn new things about the very substance of the universe. If we add in one more principle, we generalize special relativity into a much broader and much more powerful theory which overturned Newton’s theory of gravity. I’ll cover that in the next five-minute explainer.
Note: The section “A Constant Speed of Light” was revised on 28 Jan 19. Some slight inaccuracies were corrected regarding the relativistic treatment of time, and some vague wording was clarified regarding the same.
This is going to be the the briefest history of time ever. When I’m done, my goal is for you to understand not only that motion changes time and space but how and why.
Einstein created the theory of special relativity to answer these questions, and it does so in a very satisfying and complete way which physicists still haven’t improved on. It may surprise you to know that the paper in which he originally described it was only thirty-one pages long.
Before we start, we need just the slightest background here on what Einstein had to work with when he came up with special relativity. Namely, he used two assumptions.
The laws of physics are the same for any point of view which seems stationary to the observer. This is known as the principle of relativity. It means that it’s physically impossible to distinguish between whether I’m moving or the world is moving if I were to jump up and down. Both are true at the same time. You can take whichever point of view you like. That stationary point of view is called an inertial frame of reference.
The speed of light is always the same in all inertial frames of reference, regardless of the speed of whatever is emitting the light.
There are some additional assumptions you have to bake in, like conservation laws and so on, but none of those would strike you as terribly strange, and they’re subtle enough points that they don’t bear discussing here.
From those two assumptions, which are today called postulates, the rest of the theory emerges. Remember that carefully—every strange part of special relativity is an emergent consequence of those two postulates. No alternative to the theory could work without some change to the postulates, and at the time Einstein was working, he was almost certain they were true. Einstein merely carried the assumptions to their logical conclusion: special relativity.
The Principle of Relativity
Einstein loved to visualize things, and he used trains to illustrate his theory originally because of the train station in Bern, Switzerland. We’ll use trains, too. To make things even easier to visualize, we’ll not deal with light directly but instead use a thought experiment involving a thrown ball.
Imagine a person standing still on the ground can throw a ball at 100 kilometers per hour. The ball crosses a distance of 27.78 meters in 1 second. The thrower can always throw at that speed, so over the course of 1 second, the distance traversed is always 27.78 meters.
Now imagine this same person is standing on a train car. The train is moving at 30 kilometers per hour. They throw the ball in the direction of travel at 100 kilometers per hour. How far does the ball travel?
For an ordinary ball, the distance traveled depends on where you stand. The person throwing the ball on the train sees nothing out of the ordinary because they too are moving along. They see the ball travel at 100 kilometers per hour, and so they likewise see the ball cross 27.78 meters in 1 second. This accords with the first postulate, the principle of relativity. No matter that the train is moving. To the person on the train, it might as well be still while the rest of the world moves backward.
However, a person standing stationary on the ground next to the train would see the ball thrown at 130 kilometers per hour because the train’s motion adds onto the ball’s motion. Therefore, the ball travels 36.11 meters over 1 second. The velocities add together.
A Constant Speed of Light
So far, I have described the ordinary, intuitive behavior you would expect in this situation. Now let’s change things up: The ball can never travel at any other speed than 100 kilometers per hour in any direction, regardless of who is standing where or who is in motion compared to whom. This is similar to how light behaves, according to the second postulate.
The person on the train throws the ball, and it travels at 100 kilometers per hour relative to them, crossing 27.78 meters in 1 second. So far, so good! But the person on the ground also sees the ball travel 100 kilometers per hour, despite the velocity of the train being 30 kilometers per hour. The velocity of the train no longer adds onto the velocity of the ball, yet the ball is still in motion when it is thrown.
How far does the ball travel now?
Here, the universe encounters some very awkward bookkeeping problems. If the person on the ground also saw it travel 27.78 meters in 1 second at 100 kilometers per hour, that ball would land somewhere other than where the person on the train sees it land. This train is in motion, along with the ball. We expect it to cover a distance in 1 second equivalent to the motion of the train in addition to the 27.78 meters which the person on the train sees. That’s 36.11 meters. Yet it cannot cross that distance in 1 second because the ball cannot go faster than 100 kilometers per hour.
For the ball to go different places for different people violates causality itself—it would mean cause and effect were broken. Einstein assumed cause and effect would work out because without them, science wouldn’t do us much good for describing the universe. At the same time, however, he had this very annoying issue of how to solve this problem of reconciling time and distance under these conditions. He thought about it until he understood that some of the assumptions he held about what the universe would keep constant were not, in fact, fixed.
The way to resolve the problem above is that the ball does travel only 27.78 meters. How can that be? Because the train itself must get shorter. The person standing stationary on the ground, looking up at the train, would see the train (and everything on it) squished in the direction of travel. The universe solves the bookkeeping problem around the speed of light by altering space itself! In point of fact, those 27.78 meters over which the ball travels would look different to a person standing on the ground compared to someone on the train because they would be shorter meters.
It is the way the universe must work because of the finite and invariant speed of light—or, in our thought experiment, the invariant speed of the ball thrown. It simply cannot be any other way without breaking causality or changing one of the postulates.
While we’ve fixed up the distance problem, though, we’ve created a new problem. Imagine that the person on the train who threw the ball walks over and picks it up after throwing it. They’re moving through a shorter train (at less than the speed limit). If we only contract the length through which they walk, they would cover the distance to arrive to where the ball landed too quickly. This outcome is equally as bad as before, when the distances didn’t work out. How do we solve this?
This problem really bothered Einstein until he let go of the assumption that the universe kept an absolute time clock somewhere. In other words, he discarded the idea that there’s one real, absolute time. This allowed the universe bend time in order to make the math come out right. This means that the solution to the problem is, everything on the train must slow down when the train is in motion relative to an observer.
The sum total of these effects leads to the stationary person on the ground seeing the everything on the train squished in the direction of travel while moving more slowly at the same time. The faster the train goes, the more pronounced these effects become. For the person on the train, they see everything happening there normally, but due to the principle of relativity, they see things on the ground also squished and in slow motion.
There can be no distortion in space without a matching distortion in time, and it appears that space and time are so inextricably bound that it’s easier to deal with them as a single thing called spacetime.
Cause and Effect
By changing the rules the ball followed in our thought experiment, some very unintuitive consequences emerged. As it happens, light really does behave the way the ball does. Because time and space warp for light, they warp for anything moving at any speed, though the speed limit is so high that the effects aren’t obvious in ordinary life.
It was only necessary to change the behavior of the ball—and hold everything else equal—to see the effect of special relativity on time and space. It caused the everything in motion to squish (length contraction) and begin moving in slow motion (time dilation) when seen by the person on the ground next to the train.
Why should this be? Especially where vaccination or global warming are concerned, the stakes ought to be too high to allow vagaries of ignorance to win out. Yet facts don’t cut it. You risk entrenching the other side, and you come away even more convinced of their wrongness. Everyone goes home angry.
There are strategies to take for winning over people with different viewpoints: the best one being to find common ground. This tack has nothing to do with the facts at issue, but it’s the best way forward.
How did we get in this situation where it’s possible to disagree with facts themselves? I propose that it’s not so much that we find ourselves arguing with a reasoned point of view but with an identity.
Meet Alice and Bob
Consider Alice, who believes that human-caused global warming is changing the Earth’s climate (hereafter “climate change”). She’s trying to convince Bob, who just doesn’t believe Alice. Everything he’s heard leads him to believe that there’s just too much doubt to know for sure if the Earth is really warming, and if it is, there’s no way that humans could be the cause.
No matter what Alice says, Bob believes Alice is wrong. What’s Bob’s deal? Fundamentally, this is not a discussion about whose facts win out over whose. Instead the question is about who is arguing from the more meaningful authority.
Some of you might be wondering, well, gosh, Alice isn’t arguing from authority, is she? She has facts and figures and charts and scientific consensus to back up her side. Here’s the problem: science itself has been turned into an authority over the years—in Bob’s mind and even in Alice’s mind.
Let’s leave them to their intractable argument and visit this idea of battling authorities.
The Conflict Thesis
It’s probably a vast oversimplification to consider Alice and Bob above as proxies for science and religion. However, they likely carry feelings that science and religion conflict irresolvably, and elements of that conflict almost certainly underlie several of their attitudes. Where did these feelings come from? What do they mean?
The conflict thesis is not so much a description of the reality of science as it is a historiographical approach to the history of science itself. It’s a belief that religion is inherently and attitudinally adverse to science and vice versa. It permeates Western science education and many current Western religious doctrines.
For most people raised in the United States, the idea of the conflict thesis will feel very familiar—it may conjure up images of Galileo’s house arrest or the Scopes Trial. Many contemporary popular scientists, speakers, and writers have promulgated elements of the conflict model, such as Neil deGrasse Tyson or Isaac Asimov. Quoting Stephen Hawking, who stated the conflict very forthrightly near his latter years,
There is a fundamental difference between religion, which is based on authority, [and] science, which is based on observation and reason. Science will win because it works.
The conflict model is not especially useful, though, for understanding the relationship between religion and science. Historians have mostly moved to more nuanced models for describing the history of science. The evidence available doesn’t support conflict.
Primarily, two episodes in history foment the supposed conflict between religion and science (or more particularly, Christianity and science): the Galileo affair and Darwin’s theory of evolution. In point of fact, before the modern period, religion was such a dominant force in society that scientific thought was not seen as in conflict with religion so much as aiding it by discovering God’s plan. This view is literally ancient: Saint Augustine of Hippo considered God’s word, as written, fallible because of the imperfection of language (PDF download). Therefore, where natural knowledge and science contradicted the Bible, God’s former scripture—creation itself—wins out.
In the case of the Galileo affair, Galileo’s persecution had less to do with the Church’s disagreement about heliocentrism or of Galileo’s supposed heresies than about Galileo running afoul politically of the pope. Heliocentrism as a mathematical model predates Galileo by nearly a century, and (even despite having some contemporary detractors and competing models, such as the complicated Tychonian model), the Church had no problem with its use. The pope also approved the publication of the Dialogue Concerning the Two Chief World Systems before subsequently banning it (possibly because some statements the pope had made within Galileo’s hearing which he had then placed in the mouth of a character he named Simplicio).
This is a five-minute explainer, so I won’t go into other historical examples in detail. Suffice it to say, many episodes of putative conflict in the past (such as the Galileo affair or the Scopes Trial) had political and personal motivations and issues at play as well as, or rather than, pure conflict between religion and science.
The Problem of Authority
So we come back to Alice and Bob. How does the conflict thesis shape their argument?
Recall that I mentioned that Alice and Bob are both fundamentally arguing from authority. This may seem like a sharp tilt on what’s happening, but here’s what I see. The conflict thesis has become common enough to make it into textbooks, popular writing, TV shows, and even public policy. People who believe science and religion can coexist (let alone build on one another) find themselves in a minority.
One of religion’s main functions as an institution is providing a foundation for community via shared mores and beliefs. From this position, religion becomes an authority, and in that capacity, the Christian Church served as a powerful authority for many centuries. The conflict thesis emerges naturally as a way of supplanting that authority in order to center a scientific model of reality.
The problem is that we have exchanged one authority for another. This happened as a natural outgrowth of the process of deconstructing religion as authority. However, science is not designed to be an authority, and it doesn’t function best that way.
Yet it’s taught that way. Consider how scientific news flows to the public—channeled through layers of intermediate journalism: the actual scientific publications, next scientific journalism, then mainstream journalism. Along the way, what remains are names of lofty institutions, their impenetrable facts, and their avatars of a new faith. Worse yet, consider what’s emphasized in school: not processes, not approaches, but perfected facts and great minds beyond impeaching.
Missing is the human element: the struggle with ambiguity, the charting of unknown territory, the failures and blind alleys. Science can contain narratives which empower people, if only we can burnish its anti-authoritarian stance: question everything and everyone. Finding the right questions is within anyone’s power, and science is more about questions than answers.
Resolution for Alice and Bob
Turning back to Alice and Bob again—isn’t that actually what Bob is doing? Questioning Alice? Questioning the science behind climate change? Unfortunately, he’s not really questioning them. This is mere denial because he’s not examining them in any rigorous, critical way. In his mind, however, he may have an anti-authoritarian stance, believing Alice has bought into climate change without critical thought on her part.
Meanwhile, Alice hasn’t taken Bob’s point of view seriously at all. She hasn’t sought to understand it, and therefore she has no idea how to engage critically with it. Maybe his point of view isn’t consistent, rational, or even coherent, but it’s where they have to begin if he’s going to join Alice.
Both sides are merely dismissing the other, and that’s why nobody is making headway. They both fervently believe in what they have learned, and they have bound up their own identities in those beliefs. Unfortunately for both of them, the conflict thesis has interwoven tightly into their beliefs as well, leading them to a place where their systems of thought are thoroughly immiscible.
Even if they can’t bridge the gap between belief systems, though, the hope is that Alice can reach Bob in a way that doesn’t force him to abandon his beliefs, his authority, or even his identity in order to incorporate new knowledge. For this reason, finding common ground is key.
Winning hearts and minds is about overturning the root of conflict through which we see science and religion. It allows Bob (and Alice!) to entertain multiple ways of seeing the world simultaneously. With the conflict resolved, Alice can move from changing Bob’s mind to adding to what Bob knows instead, and Bob can move from losing foundational beliefs to incorporating new ideas into those foundations.
There is something of a tradition of programmers writing posts about Vim. Many proselytize, and others are narratives of discovery. What I want to contribute to the conversation about Vim, in this late year of 2018, is what role it still plays in a world that is technologically rushing by. I’m going to tell you more about Vim, why I use it, and how I use it, so that I can say why it might still be worthwhile looking at—or not.
I have learned Vim as a programming-centric tool, but I use it for other tasks, too. This post assumes a reader isn’t necessarily a programmer but is curious about how tech things get done.
Vim in the Past
First, let me talk a little bit about what Vim grew from.
Vim is old. It’s a program that was originally written for the Amiga operating system, first available in 1991. The even older program it emulates, vi, began in 1976 and was available in 1979. Vim has become the most common implementation of vi currently in use today, as Vim has become compatible with the vast majority of computers in existence.
This long history implies a couple of important things. First, vi predates many ideas which are today so ubiquitous that we consider them integral to how computers work: graphical interfaces, word processing, instant visual editing, multitasking, and so on. Second, it means that vi (and by extension, as its modern counterpart, Vim) occupies a special and integral role in the history of computing itself and the standards upon which many server operating systems are built. A vi command compatible with its original functionality remains a part of the POSIX standard even to this day, as well as the Single UNIX Specification. Vim provides that today.
In other words, Vim is a relic of another time—one which is now over forty years gone. It can be obtuse, arcane, and occasionally downright wrong for today’s world as a result. The original author of vi used a terminal, not a computer directly: one slow enough that he had to issue instructions without seeing the feedback right away. This, among many other constraints, guided vi‘s design. Vim inherited these decisions in turn.
So it may surprise you that Vim has not only endured but flourished. The key lies in its flexibility within its historical constraints.
Vim in the Present
So what is Vim? It edits plain text, and it does so within a terminal emulator today.
Most operating systems offer a program called a terminal emulator which presents a boxy window of text and a prompt at which you can enter commands to execute programs, and the programs will run and usually output text in return as a result. This is also just called the command line or just a terminal. Nowadays, mine looks like the one below.
At the top left, there’s a little lightning bolt, which is what I’ve told my prompt to look like, and there’s a little rectangle, which indicates where my cursor is and where text will go if I type something. If I type “vim” and press Enter, the entire terminal changes to look like the image below.
Vim has changed the terminal’s screen into a text editor. The cursor is still there, but now it’s on the first line of a nameless buffer (an area of memory for editing).
There is no text there in the buffer, so I can’t really do anything. My options are to open a file which already exists, or I can insert new text into this empty buffer. Either of these options will call on me to take an action which feels pretty unfamiliar to a modern computer user.
If I want to open something, I type a colon followed by a command, like “:edit,” followed by the name of a file. This is an “Ex” command, a type of interface which harkens back to the editor which predates even vi. Vim has hundreds of these.
If I want to insert text, however, I need to change into Insert mode, by pressing “i.” Now I can type in text as if I were at a normal editor, like Notepad or TextEdit. However, I can no longer enter Ex commands, so I can leave Insert mode by pressing the Escape key. Doing so puts me back in Normal mode. In this mode, I can move the cursor around, enter Ex commands (by entering a colon), and use some other commands, but I can’t directly insert text. In Insert mode, I can insert text, but I can’t access the other commands. So I have to switch back and forth.
This switching process is one of the relics of the 1970s which remains in the editor today: modal editing. It is simultaneously a frustrating obstacle to new users and a profitable tool to those who learn to appreciate it.
How can it be useful to be unable to enter text in a text editor? Consider that, in Normal mode, your keys do not enter text. That means they’re freed up to do other things. Vim therefore overloads every character on the keyboard in Normal mode with a meaning, such as an action (called an operator) or a modifier (such as a motion).
For example, the “d” key means delete in Normal mode. Yet it does not delete on its own. It’s merely an operator. For it to achieve something, it needs to be paired with a motion—some amount of text over which to accomplish the action. Motions are usually over text objects like words, sentences, paragraphs, lines, and so on. So to delete a word, it’s possible in Vim to put your cursor over that word, press “dw” (the combination of the “d” operator followed by the “w” motion), and Vim will delete the word. The combination of an operator and a motion is a command.
Throw in numbers (such as “d3w” for “delete three words”), and Normal mode provides a fully composable set of primitives for building up a language of text editing expressions. There exist other operators for changing, copying (“yanking” in Vim parlance—vi predates the existence of a clipboard), pasting (“putting”), formatting, swapping case, filtering through another program, and so on.
Vim is meant to be used primarily in Normal mode, where this language can be used for navigating the text, rearranging it, copying things around, manipulating things precisely, and so on. The number of operators and motions are relatively few (perhaps several dozen each—they can each be a couple of characters long). Yet from these, thousands and thousands of expressions may be formed.
This may sound overwhelming to learn at first, but there’s always time to learn as you go. vi has functioned this way for over forty years. Vim continues to function exactly the same way. If you learned only one operator or motion every week for a year, you’d know hundreds of expressions by heart after that time.
Vim in the Future
Vim has already been around almost thirty years just in its current incarnation. Every time I learn something new about Vim, or every time I customize it a little more, I benefit from that indefinitely. For all its problems, idiosyncrasies, and hangups, I know that I will likely never have to learn another text editor again for the rest of my life. Vim is capable enough and flexible enough to carry me into the future and meet whatever need I have without changing substantially. I feel relatively sure of this because it’s already lasted this long. At the heart of how it endures are those parts of Vim that resist change.
There is a large core component of its functionality guaranteed to work a certain way (the vi compatibility). The maintainer’s vision for Vim’s future firmly roots the editor to its past and its place within the wider Unix ecosystem. For example, I run the editor within a terminal, and while this is limiting (it has no native interface of its own), I appreciate the constraints. There are fewer things to change over time; the customization is tied to that environment, which I also control; and it is more closely integrated with the other tools I know.
I have heavily edited Vim to meet my needs over time. My .vimrc file is only the beginning. The .vim directory I use contains a massive number of plugins, some written by me, and others written by a number of online authors.
I write, and my writing is contained in hundreds of plain text files (Markdown formatted) containing tens of thousands of words, all edited and formatted in Vim. These files’ formatting can be converted into whatever I need on the fly, but the editing remains as plain text so that I can use the same tool I’ve always known. I won’t have to worry about Vim replacing its toolbar with a “ribbon” next year or suddenly changing to a subscription model or dropping compatibility for my operating system.
Individually, those changes never bother me much, but other time, I noticed with most programs that I end up moving around from one thing to the next through the years as the platforms shift under me, as people learn how to do things differently. For example, I have switched web browsers several times over the years. With text, maybe we’ll learn some new tricks, but Vim is just a tool among many, and I imagine it will thrive as it has so far because it knows how to harmonize with a bigger ecosystem and learn to interoperate with new ideas without becoming something new.
In other words, I feel confident continuing with Vim and learning more about it over time because the core of how it works is stable, integral, and flexible enough to ensure it will work in the future.
Vim in Your Future
Do you want to use Vim after reading all that? The world is full of advice on how to do just this. Maybe you’ve already tried, and you came away frustrated.
I won’t say my advice is the only way to go. I think of Vim as a single cog in the larger clockwork of the terminal’s command-line interface. Some familiarity with the command line is helpful before enmeshing yourself in the teeth of Vim’s gears. (I hope to write about this wider someday soon.) But if you’re determined, I’ll give you a primer with the first steps. What you do after that is up to you.
Before you begin, you’ll want Vim on your computer, and if you’re using macOS or Linux, you’ve probably already got it. On some versions of Linux, you’ll need to install something like “vim” (Ubuntu) or “vim-enhanced” (Red Hat/CentOS/Fedora). For macOS users, the default version installed is fine, but a newer version through Homebrew is available if you want it.
A Gentle Ascent
If you’ve tried Vim before and it didn’t take, I’m willing to guess someone advised you to try Vim in a “pure” or strict way that makes no compromises on how you use it—as close to the original 1979 vision as possible. I don’t believe this makes sense in a modern world.
There’s another extreme available: pre-packaged configurations of Vim which come juiced to the gills with bells and whistles that make it seem polished, utterly flexible, and even magical. Possibly the most popular variant on this is called “spf13-vim.”
I suggest a compromise. You want to ease the transition in to Vim so you can be productive as soon as possible, but you also want to have your feet planted firmly on a solid bedrock so that you feel confident with each step you take. Below, I’ll try to guide you through how to do this by
first, smoothing the transition with some minimal settings;
next, providing a very minimal set of commands to do the most basic tasks;
then, showing how to run the tutor and suggesting a reasonable pace for it; and
finally pointing to the help system.
Now, remember how I said Vim is old? Yeah, it’s old. By default, it starts in something called “compatible” mode. This mode isn’t like the text editing modes I mentioned earlier. This means all its settings are modified to make it as much like the original vi as possible.
This default changed in version 7.4.2111, which happened back in 2016, so compatible mode might not affect you. If you have a newer version of Vim (8 or above), you probably have a “defaults.vim” somewhere which came with your version which sets a few things to more reasonable settings without your having to do anything.
However, let’s cover our bases and just lay the groundwork for a configuration file. This will accomplish a few things. First, the mere presence of a user configuration file will cause Vim to kick out of compatibility mode and behave more like you would expect, regardless of its version. Second, by having one, you will know where to stash new settings as you think to add them. Third, we’ll have a few more reasonable default settings—nothing too weird, just things that almost everyone universally ends up configuring almost right away.
Let’s add one final thing to it, though. If you’re the sort who uses the mouse ever, you might appreciate these few lines.
That little fragment can go on any line. Save that file to your home directory as .vimrc, and when you start vim, you’ll see something very minimal, but not entirely unreasonable. Here’s what you would see if you edited the sensible settings file itself.
We could tinker with these settings, but let’s just move on for now. You can do all that later. You have the entire rest of your life.
You really only need to accomplish a very small handful of things to get the barest of functionality from Vim. Let’s cover each.
Open Vim. You type “vim” on the command line. Easy, right? You can also type “vim” followed by a filename to open a file directly (e.g., “vim path/to/a/file.txt“). That’s one way to open a file.
Open a file. If you’re already in Vim, though, you want to open files without closing the program entirely and then reopening it. Easiest way to do that is with the Ex command “:edit” followed by the filename, such as “:edit path/to/a/file.txt.” You can use “:e” as a shorthand for “:edit.”
Change a file. You can move the cursor around with your arrow keys in both Insert mode and Normal mode. (The Vim purists will tell you to use one of “h,” “j,” “k,” or “l,” to move the cursor around, and you may do so, but this works only in Normal mode, and it may feel unnatural at first.) Use “i” to put Vim into insert mode. Change the text as you like. Then use the Escape key to leave Insert mode.
Save a file. You can do this with an Ex command, “:write.” You can use “:w” as a shorthand for “:write.”
Close Vim. You can use “:exit” to do this, another Ex command. If you want to, you can shorten this to “:xit” or just “:x.” This command happens to save any file you’re working on before quitting. If it can’t write a file, it will balk and tell you. If you merely want to bail out without saving, you can use “:quit!” to force the matter (this has a “:q!” counterpart as well).
That’s it. With that set of commands, strictly speaking, you don’t need anything else. You could use Vim the rest of your life, configured or not, old or new, and never worry about the rest. In fact, all those commands are vi-compatible, so you could go back to 1979 and be fine.
Of course, if you’ve configured the mouse, it’ll be nice being able to scroll and click things, but otherwise, you’re ready to start.
The Vim Tutorial
You might find your 1979 life very dull and slow, though. You’ll want to learn new things, and the place to start is with Vim’s own tutorial.
Many Vim users never actually look at the tutorial, which is a shame. The tutorial is actually just a text file. If you run the command “vimtutor” from the command line, Vim opens in a bare, minimally configured mode with a copy of this text file.
The tutorial is divided into multiple brief lessons, each on a different topic which builds on the last. The tutorial itself, being text, is designed to be edited so that you can interact with it and learn by example.
Though the tutorial says it will take about a half hour to complete, I do not recommend chewing through the whole thing in a single half hour and never returning. Instead, I suggest you take in the first few lessons and then skim the rest. After that, come back to it a few days later, take a few more lessons, and so on. It’s a very information-dense resource, and I believe it takes a few reads before it sinks in. I believe it’s reasonable to spend some weeks just absorbing it.
It never hurts to return to the tutorial even after you’re settled in and feeling more confident. There are always dusty, cobwebby corners of expertise which could use some refreshing, and the tutorial hits upon a broad swath of topics.
Finally, there are several suggestions from the tutorial which you need not incorporate into your life at all. Use your judgment on what is appropriate for you. It suggests moving away from the arrow keys, for example. I have never done this. I probably never will. It’s not appropriate for me because I have difficulty distinguishing right and left, and so mapping “j” and “k” (which are laterally adjacent) to up and down in my mind is extremely confusing. The spatial layout of the arrow keys on my keyboard helps me keep things straight. So from the tutorial, take what you need and leave what you don’t.
If you have mastered the tutorial, you stand head and shoulders above many Vim users already in sheer ability to manipulate text. A lot of people learn whatever commands they need piecemeal and never go out of their way to discover new ones until they see them in action or feel the need to look.
Yet you may feel many things are lacking. You might start to wonder how to accomplish the same task over and over again and wonder whether Vim has a way to do so (it does, using the repeat operator or recording). Or you might wonder what some of the settings in your .vimrc file mean. Or wonder how to switch between open files (buffers).
Here, the Vim manual can be very helpful. It is shockingly extensive. Almost every kind of functionality, topic, setting, or concept is documented by name in the manual somewhere. It’s a matter of typing “:help” followed by some word related to what you want to know.
Don’t know what word to search for? Vim will take parts of words and try to find the best match. For example, “:help reco” will teach you about the “:recover” command. Want to see what’s available? If you’ve used the sensible configuration from above, you can use tab-completion on the command line in Vim to see available help topics and cycle through them. For example, typing “:help reco” and then pressing Tab shows a range of possibilities in the statusline. The first one is filled in automatically. Press Tab again, and the second suggestion is filled in. You can press Tab to cycle through all the available options.
Sometimes you can’t find the answer within Vim’s help, and you need an Internet search. Very often, this will bring the suggested fix of using a plugin to provide a piece of functionality. As you learn Vim, I suggest you incorporate plugins slowly at first.
Being choosy and slow to adapt plugins will help you learn Vim’s core internals better instead of overriding them in a fragile or even nonsensical way. I have added many plugins to my configuration over time, but I’ve removed at least as many. Some have gone unused, but others actively interfered with Vim’s workings because the plugin was not designed to harmonize with the design of the editor itself.
Tim Pope has written dozens of plugins and is expert in knowing how to write ones which extend Vim in nearly invisible ways. For example, the vim-surround plugin adds operators and motions for wrapping or unwrapping quotes, tags, and blocks of text with parentheses, brackets, and so on. It operates so fluently within the context of Vim that it’s difficult to remember that it’s not part of the original program.
Less harmonious plugins attempt to do things like create kinds of UIs (such as for navigating filesystems) or override parts of Vim in fragile ways (such as faking multiple-cursor support by coloring parts of the buffer to look like cursors and then repeating the actual cursor’s actions at those regions). Sometimes these plugins work out. Sometimes they don’t.
My suggestion here is to use Vim as is for a little while, and if some process feels particularly painful or annoying for a few days, then seek out a better way—first through Vim itself then through a plugin if available.
If the idea of tweaking Vim to your liking (or learning some of the really interesting Vim commands) sounds appealing, a really great resource is a book called Practical Vim. It’s a fully modern approach to Vim with plenty of things that feel very empowering, and it’s arranged in a recipe-like fashion so that you don’t have to read from end to end (unlike the Vim tutorial).
Finally, I am always happy to answer questions if no other resource is helpful or available.
Editing Your Future
That’s the plan. Set up a basic configuration so that you have an easier start of it, read the tutorial, and know where to go when you need more info. Take things slow, but feel free to beat your own path. It’s your program, and you’re free to determine what works best for you. Don’t let anyone tell you what the “Vim way” is.
You’re not at all obligated to stick with Vim or even try it. My only goal is to tear down whatever obstacles may have been in your way to begin with—overly rigid guidance, misconceptions, mystification—if you were curious to begin with.
I fully admit Vim is imperfect and may not be the best text editor for this century altogether. However, for my own part, I’m certain I can look forward to a long future of return on my investment in Vim. For this reason alone, I think it’s worth a look.
“I don’t want much,” I said. “I don’t want power. I don’t want to rule over others. I don’t want love. I don’t want anyone dead. I don’t even want to hurt anyone.”
The genie smiled blandly.
“All I want is a healthy retirement fund,” I said. “And I want it without any tricks.”
“Without tricks!” The genie rose to his full height, taken aback. “I am no magician! Tricks indeed!”
I said, “Well, you know what I mean. Without…you know, strings attached.”
“Without consequences, you mean? There can be no such thing.” The genie smiled broadly without using his eyes, and he once again settled into the armchair. “There are simply natural outcomes you must expect from any good fortune.”
All right, I figured—so long as no one gets hurt. I get to decide the terms. That much the genie had made clear.
The genie asked, “Have you decided?”
“Yes,” I said, “all right. I’ve decided.”
The genie fixed me with his eyes and waited.
“I want a million dollars…”
The genie smiled.
“…in United States dollars, tax-free…”
The genie’s smile widened.
The genie’s smile widened more, his eyebrows lifting in expectation.
“…which will actually be available to me, and to me alone…”
“Yes?” the genie crooned.
“…without anyone having to lose anything for me to gain it…”
The genie’s smile somehow grew still more.
“…and without anyone getting hurt.”
I began to wonder whether anything I said could possibly disrupt the genie’s unflappable sangfroid. None of my conditions had so far seemed to ruffle him in the slightest.
He asked, “Will that be all?”
I mulled it over, trying to think if there were some possible way this could backfire, and I could not think of anything. “Yes, that’s right. That’s what I want.”
“Very well,” the genie said, seeming to suppress glee. “Nothing could be simpler.”
I waited expectantly. I wondered whether I was supposed to close my eyes or something. After looking around, I asked, “Well? What now?”
“I’ve granted the wish, just as you asked,” he said.
“Really? How? Where?”
“It’s being deposited into your checking account. I thought you might find that convenient!”
I picked up my phone and opened my banking app. I didn’t know why I was so tremulous. Maybe I was overwhelmed with the unreality of what I expected to find. I pulled up my balance, and it looked unchanged.
I sighed and said, “I don’t understand. There’s nothing here.”
“Of course there is! It’s being deposited as we speak.”
“Being deposited…?” I looked at the transaction list, and I noticed a handful of new credits—each for only one cent.
“Yes,” the genie said, “do you see?”
“But these are for one cent each,” I said.
“Precisely! You never asked for the amount now. Indeed, you mentioned a ‘retirement’ fund. I merely took it on my own initiative to amortize the funds over the rest of your life, and that is precisely what you will receive: one cent every twelve-point-oh-two-five-six-eight-three seconds. Over the span of the rest of your life, it will total exactly one million dollars, in U.S. currency, tax-free! Quite as you asked.”
The genie laughed in a dry, inward way.
At that moment, a new deposit arrived. My phone’s banking app showed that it was another cent. So far I had six cents out of my million. He was right—the genie had granted my wish in the most frustrating, useless, and maddening way possible.
Then my initial frustration fell away, as if through a trap door into a bottomless terror, as I realized what the genie had actually given me. This wasn’t just a million dollars. Each cent was like a grain of sand falling through the hourglass timer of my newly circumscribed life. A simple multiplication would reveal the exact moment of my death.
I looked down at my phone as another cent rolled in. When I looked up, I saw that I was alone.
Last night, I put my first full Vim plugin up on GitHub. It’s called match-count-statusline. It’s intended for anyone who uses Vim to edit text files. It adds a small item to the statusline (Vim’s version of a status bar) describing how many times the current search occurs in the current buffer and, if there’s enough room, what the current search pattern is.
This began as something I needed because I kept opening files and needing to count how many times a term or pattern occurred in that file. I wrote a small script to do it, and I tacked on more things to the script over the next few days and weeks.
Eventually I felt confident enough to share it. I’d also like to share about the process of writing it and preparing it for use by others. If you’re interested, read on.
Who Should Read This?
This guide is meant for advanced Vim users who wish to understand more about the process of writing a plugin through a first-hand narrative.
I won’t be explaining how to use or configure Vim itself, nor the basics of how to write in Vim’s scripting language. I’ll assume familiarity with both. If you don’t know Vim’s scripting language (which I am going to call “VimL” for the rest of this post), I cannot recommend enough Steve Losh’s Learn Vimscript the Hard Way. It is freely available online, and if you find it useful, I encourage you to buy a copy as well.
That book will set you on your way, but it can never include everything you need to know. Vim is always changing, and its own documentation is deep and voluminous. It is for very good reason that Steve sends his readers to Vim’s own documentation at the end of every section—he could not ever hope to include everything in a single book.
I tend to like to open log files in Vim and search around in them to find patterns while I’m investigating a problem at work. I’ve developed over time a lot of tools for doing so. Some of them are crude, but when I find myself doing the same crude thing over and over, I tend to want to polish it.
One common thing I do is discover how often a search pattern occurs in a log. This indicates to me whether something is signal or noise. I’ll place my cursor over something like an ID, press *, and then count how many times it occurs and write that down in another file.
I could do this from the command line using something like grep | wc -l, but I get a lot of value from seeing the whole context and being able to skip between the occurrences of my search, finding out what happened before and after, and so on. I’m very visual. I like to see them in the file.
So to count them, I’d been doing a crude thing. I would run :g//d and watch how many lines got deleted. (Without a pattern specified, that command will delete whatever the current search pattern is.) This has a few drawbacks. First, I had to undelete the lines after I was done. In large files, it was somewhat slow. And finally, it only told me the count of lines, not the count of occurrences.
I figured what I really wanted was simple, and it should be built into Vim.
I knew that when the pattern was already searched for, it could be implied, so this simplified down even more.
That did it. I ran that after searching for something, and a message printed to the bottom of the screen resembling the following.
2 matches on 2 lines
I could have stopped there.
However, I tend to forget obscure looking commands, and I thought it would be nice if this happened automatically anytime I searched for something. I knew graphical text editors tended to automatically surface information like this in their status bar, and I was inspired to make the same thing happen.
The Solution According to Vim
I probably should not have given up on Vim’s documentation so easily. I didn’t know that the substitution command (:s) was the place to start, but my use of the global command (:g) before might have been a hint.
What I didn’t know is that Vim already internally documents how to count items, and it’s filed under “count-items,” logically enough. You can find it mentioned under the documentation for the substitution section (:help :s, or more specifically, :help :s_flags and see the “n” flag) as well. To find out how this works, run :help count-items. You’ll see it suggests ways to count characters, words, and so on. Generally, it boils down to a very similar command.
Again, this assumes an existing pattern has been searched for. The one difference is the ampersand. The ampersand has a special meaning in the substitution, referring to the thing being replaced. In this case, it would effectively replace a thing with itself, causing nothing to happen. But since the “n” flag is there, no replacement will happen regardless, so there’s effectively no difference if it’s included or not. I have included it in subsequent examples as it is safer (in case the “n” is forgotten).
So that single, small command forms the core of the functionality I later expanded.
To display it, I learned how to insert an item into vim-airline‘s statusline. That wasn’t too difficult once I found an example online, and you can see how I accomplish that here. Using that, I began a small script in my personal configuration files which added the count when I was searching for a pattern, and otherwise displayed nothing.
Over the next several days, I added on improvements because I found it was slow on larger files. This included caching its results momentarily rather than updating constantly. I also found that each time it updated the search, it would also move the mouse cursor, so I needed to fix that.
The resulting script grew in size and became safer and better performing. Eventually, I figured I’d put enough work into it that it would be safe enough to share with others without breaking their Vim installs.
I want to spend the rest of this post talking about all the funny little things I did in the core of the script contained in the plugin to build it up from that one substitute hack into more fully fledged functionality.
" require 7.4.1658 for v:vim_did_enter
\ || (v:version <= 704 && !has('patch1658'))
\ || exists('g:loaded_match_count_statusline')
let g:loaded_match_count_statusline = v:true
This bit of code makes three checks. First, it checks if &compatible is set. If it is, then there’s no need to check anything else—the script won’t work, and we can exit right here.
Next, it checks for a version of Vim greater than or equal to 7.4.1658 (using Vim’s peculiar syntax for doing so, the predefined variable v:version). That patch, 1658, is the one in which v:vim_did_enter was introduced. (I discovered this by searching the vim_dev mailing list.) That patch was introduced two and a half years ago, so I felt comfortable with that baseline.
Finally, I check to see if my plugin is already loaded using a global variable I will set directly after this condition. If the variable is already set, it means I’ve reached that line before, and I should exit before reaching it again. This prevents loading the script more than once.
The “finish” command ends the script if any of these conditions are true.
The Global Flag
Notice how the above examples of the substitute command all included the “g” flag. That tells the substitution to apply to all matches in a line, not just the first one.
That probably seems like a funny default: only doing one substitution per line and then moving along. It’s just the way regular expressions work from the olden days, and Vi (the editor that Vim is based on) picked up the behavior. Vim never breaks compatibility, so its substitution also needs that “g” flag to apply substitutions globally.
For that reason, Vim introduced a setting called &gdefault (see :help gdefault). It has the effect of implying a “g” flag on every substitution. Unfortunately, it also means that if you add the “g” flag to a substitution while that setting is on, it undoes the effect. That means you never know what the “g” flag will do unless you know what the setting &gdefault is. That’s not confusing, right?
All of that means that setting &gdefault would effectively break counting if I did not check it. That’s what these lines are all about.
let s:match_command = '%s//&/ne'
let s:match_command = '%s//&/gne'
Here, a command to run to search for matches is saved to a variable based on whether &gdefault is set.
There is one minor drawback to doing this here as opposed to dynamically setting it every time the match is calculated. If the user changes &gdefault after loading Vim, the counts will be incorrect when a pattern occurs more than once on a line. This could be a bug! The fix would be to set the match command each time we run a match.
let s:start =
\ get(g:, 'match_count_start', '')
let s:end =
\ get(g:, 'match_count_end', '')
To use a variable in Vim, while falling back to a default if it’s undefined, an old way was to use the exists() function. My syntax uses a stupid Vim trick involving variable scoping.
Notice how all the variables I use begin with a letter followed by a colon. These are known in VimL as “internal variables” (see :help internal-variables), and each kind begins with a letter denoting its scope, a colon, and then the name of the variable. (Vim also refers to these scopes as “name spaces” in its documentation.)
Each scope has a special meaning and is treated by Vim in a special way. For example, variables scoped to b: only exist for the one buffer you’re in, and no other buffer can see that variable.
More importantly for our purposes, though, every scope is also considered by Vim to be a Dictionary unto itself, and you can treat all the variables in it as keys containing values. Weird? Yes, weird. But it lets you do things like, “:echo has_key(g:, 'match_count_end'),” which will tell you if that variable is defined.
So in the case of each global setting, I use the get() function, which takes a Dictionary, a key, and optionally also a default value to use in case the key is missing. This allows me to set variables scoped to my script (in s:) based on whether variables in the global scope (in g:) are defined or not. Then the rest of the script uses those script-scoped variables.
This unfortunately means that no changes to the global variables can get picked up while Vim is running. To fix this, I could set all these inside the match counting function I defined, if I chose.
Caching is one of the earlier things I implemented, and it’s gone through a few revisions. Here’s how it works today.
The basic idea is, the match counting function gets called repeatedly every time the statusline needs to get redrawn, and this happens far more often than the match count itself will change. If the script remembers the match count for each buffer, the match counting function (MatchCountStatusline()) won’t have to recalculate until it’s really necessary, saving itself some work.
So caching is one among many ways MatchCountStatusline() attempts to defer doing any hard work. How does MatchCountStatusline() find out when to update its cache? A couple of ways.
The cache is a simple Dictionary made anew for each buffer. It remembers the last time the cached values were referenced or updated, the number of changes to the current buffer, the current search pattern, and the number of matches.
The default values for the cache are ones that could never occur normally, so I call them “sentinel values.” Sentinel values are generally used to represent invalid values, so in this case I can recognize when the cache contains values which have not yet been set by MatchCountStatusline().
let b:count_cache =
\ get(b:, 'count_cache', copy(s:unused_cache_values))
I know that because the cache is scoped to the buffer, each buffer gets a new set of values. On subsequent calls to MatchCountStatusline() for that buffer, it reuses the cache which already exists. (Notice also that I had to copy() the values. If I didn’t, modifications to the buffer’s count cache would modify the script’s count cache defaults and would affect all the other buffers as well. See :help copy.)
It’s a simple check. All I look at here is whether the pattern has changed or the buffer has changed. I don’t have to look at the buffer itself to determine whether it’s changed. A predefined Vim variable increments for every change, called b:changedtick. If it’s changed, I know the buffer has changed. See :help b:changedtick. Those two things let me know if a match count needs to be recalculated for the current buffer.
Notice also that the current search pattern is stored in a register called @/. All registers can be referred to by a name which starts with the @ sigil. There are a bunch of generic registers with single letter names, “a” through “z,” and then there are special-use registers such as @/. It’s easy to see where it got its name, with a little thought—the / key begins a search in Vim, so the @/ register holds the search pattern. Like any register, it may be echoed, compared to, or even assigned to. Any time in my script I want to see what’s being searched for, I inspect @/. For more information, see :help @/ and :help registers. The kinds of registers available will definitely surprise you!
You probably noticed the cache holds two other things besides the number of changes to the current buffer and the current search pattern. The match_count value is the thing we’re caching, so we never care what its value is until we’re printing it out. But last_run deserves an explanation.
I decided early on that I wanted to keep and print out the same cached values for a short period of time, even if the buffer has changed. That’s what last_run is for. It lets me know how long the cache has been holding onto those values so I can know when they do need updating.
Why would I want to keep old values around, even if they don’t reflect reality anymore? It’s not always necessary to reflect reality the very instant it changes. Humans don’t need updates right away for everything—a delay of a tenth or a quarter of a second is tolerable sometimes.
When someone is actively typing, or if they’re entering a new search (and &incsearch is enabled), every single keystroke would invalidate the cache and trigger a new calculation. If we instead wait for a fraction of a second, we can allow the user to type a few characters before updating. By the time they look down, we’ve probably already updated, but we won’t have wasted time needlessly updating for every keystroke.
The function responsible for determining if enough time has passed before even checking the cache is called s:IsCacheStale(). (Notice that it’s script-scoped, so that it doesn’t collide with other scripts’ functions.) It comes before the cache check in the program, so it “short-circuits” that logic, precluding it when it’s not needed. The function s:IsCacheStale() has a very simple mandate—it receives the cache and checks how much time has gone by. If enough time has passed, it returns a true value. Otherwise, it evaluates to false. How it implements this is a little complicated.
I won’t copy in the whole body of the function here, but go glance over it if you want. There are a couple of weird things going on, and I’ll touch on them here.
First, instead of using ones and zeros to represent true and false, I’m using Vim’s predefined values of v:true and v:false. Vim introduced these in 7.4.1154 to help with JSON parsing. They evaluate to 1 and 0 respectively, so I’m using them to make the script more readable.
Next, I put in a little stanza at the beginning to optimize when loading a new buffer.
if a:count_cache == s:unused_cache_values
let a:count_cache.last_run = reltime()
let a:count_cache.last_run = localtime()
The very first time that MatchCountStatusline() gets called for a buffer, recall that it puts in the sentinel values for the buffer’s count cache. Here, I detect whether or not that’s the case. If it is, I merely return false (meaning that the cache is not stale), but I also importantly update the last_run cached value with the current time, so that this condition never evaluates to true in the future for this buffer.
What this accomplishes is that the first time a buffer is loaded, the cache always appears to be stale (even though it really only contains invalid values). This serves to cause no match counting to occur for the first cache timeout period (by default, a tenth of a second) after a buffer gets loaded. After that, the first match count occurs. This lets the buffer get up on its feet and lets other plugins and automatic commands do things before we start.
Finally, both here and further down, I test whether the “reltime” feature exists in Vim and use it only if it does. Otherwise, I fall back to using the localtime() function. There are a handful of other places I’ve checked for functionality before using it. This is one of the ways I’ve tried to make the plugin safer for a wider audience. I imagine most people will have a Vim with “reltime” available, but I couldn’t tell how widely available it would be.
The cache check algorithm is pretty simple otherwise. It checks the current time against the last_run time. If it’s greater than or equal to the cache timeout, last_run gets updated, and the function returns true (meaning the cache is stale). Otherwise it returns false, leaving last_run alone so that the next time it’s called, it will be clear how much time has passed since the last time the cache was stale.
Both by giving the cache a little time before it’s updated, and by checking its values, MatchCountStatusline() avoids updating the cache (and running the expensive match-counting calculating) whenever possible.
Another Optimization: v:vim_did_enter
One of the more recent Vim features I took advantage of was the v:vim_did_enter predefined variable (see :help v:vim_did_enter). It allows MatchCountStatusline() to know if it’s being called before Vim has even loaded all the way, and so it’s the very first check. This means that the cache isn’t even warmed up before Vim has properly started up, so the cache grace period kicks in after. This allows Vim to start up faster.
" don't bother executing until Vim has fully loaded
if v:vim_did_enter == v:false
File Size Checking
I discovered early on that match counting works really poorly for large files. I tried throwing in a couple of optimizations for the counting itself (I’ll mention those below), but the counting still dragged, and so I assumed it would never be fast enough to work transparently and interactively.
Then I considered whether I could do the counting asynchronously. I eventually ruled this out for now because it means shelling outside of the Vim process, as near as I can tell. I wanted my script to be purely VimL and self-contained.
if getfsize(expand(@%)) >= s:max_file_size_in_bytes
Notice that it takes an argument called force. This allows MatchCountStatusline() to tell it to skip checking the file size and just report that the file is safe in all cases. Otherwise, this one is very straightforward.
Toggling Match Counting
I realized that sometimes, even for large files, I still wanted a count, so I implemented a command which would manipulate two buffer variables to force match counting to occur (or would disable match counting altogether). The logic here is a bit weird because of the file size checking. I wanted to be able to force things on if they’re turned off due to the file size, and I wanted to be able to toggle things normally otherwise.
" don't count matches that aren't being searched for
if @/ == ''
let b:count_cache.pattern = ''
let b:count_cache.match_count = 0
let b:count_cache.changedtick = b:changedtick
Otherwise, we have work to do.
(Incidentally, there’s a small optimization I forgot to make here. See it? We’re updating the cache. We can set last_run here and keep these cached values for longer. We should do this here and anywhere below where we update the cache.)
Since we’re now in the part of the script involves the actual calculation, and since it has the chance of raising exceptions, I begin a try/catch/finally block here. It’s important to begin it here, when I start to change the state of the buffer, so that I know my changes will get fixed up by the finally block below. The worst case scenario would be attempting to count the matches, having an error occur, and leaving the user’s editor in a broken state.
Now we’re free to begin mucking about. First thing I do is save the current view.
" freeze the view in place
let l:view = winsaveview()
In the finally block, the last thing I do is restore this view.
This ensures that nothing visibly changes while I do what I’m about to do. (See :help views-sessions, :help winsaveview, and :help winrestview.) One problem I had was that the core match count command (s//&/gne) would cause the cursor to jump to the beginning of the line. Saving and restoring the view fixes that problem.
By turning these features off, Vim won’t attempt to fire off events during the fake substitution it’s about to run, which can save a little time. Turning off &hlsearch in particular is one of Vim’s own suggestions in :help count-items.
" this trick counts the matches (see :help count-items)
redir => l:match_output
silent! execute s:match_command
I added a comment to help others understand how this worked because it’s probably the least obvious part of the program.
The redir here sends all the output from that command to the l:match_output local variable. The actual command uses echom to display a message with the results, like I showed above, something like “2 matches on 2 lines“. If no matches are found, the output is empty due to our use of the n flag.
There are several things I didn’t cover here, like how I implemented the global customizations. I might cover those in a future post if there’s interest, but they’re straightforward enough if you followed the above.
Of my previous wide-field photos of the night sky, none have been more than single long exposures of thirty seconds or less. Recently I’ve taken my first steps into experimenting with stacking these non-planetary photos. Below, I show the process and results from my first attempts to stack both an in-telescope photo and a wide-field photo.
Stacking is, as I’ve mentioned in the past, a way of combining separate photos into a single, longer exposure. With highly detailed, small objects like planets, stacking can be used to get more detail and clarity through lucky imaging and the shift-and-add technique. With a wider-field photo, the goal changes a bit. Certainly, more detail and clarity result, but you also gather more light and reduce camera sensor noise.
Noise, Noise, Noise!
I have been limited by camera sensor noise in all the individual astronomical photos I have ever made. To make a relatively decent exposure of the night sky, it’s necessary to boost the ISO to at least 1600, which increases the sensor gain. On its own, this usually isn’t a grave concern, but it limits how much I can subsequently push the photo to bring out its details.
Inside of a single photo, there’s no real way to overcome this noise without manipulating the photo aggressively, such as using a powerful noise reduction algorithm. I typically avoid doing so because it’s difficult for such an algorithm to distinguish noise from fainter stars, and even the brighter details lose much of their finer qualities (dust lanes in the Milky Way core, for example).
Instead of eliminating the noise, I usually just leave it in. I limit the amount I push a photo so that the noise remains relatively unapparent when seen in context, and generally the noise does not mask the most important parts of the photo.
Yet, that noise limits my light. I can’t turn up the light without turning up the noise—both in the camera (I must keep the ISO low) and in the computer (I must avoid pushing the photo too far). What can I do? Stacking! Taking many photos and averaging them together means not only do I combine the light from them to make that light brighter, but the noise (which is largely random) gets canceled out because it varies between each photo.
New Techniques, New Tools
Stacking deep-sky and wide-field photos is a different process than stacking planetary photos. The exposures are much longer (several seconds instead of small fractions of a second), and often you have fewer of them.
In many ways, it is a more advanced technique. I have not yet tapped a lot of the tools available to me, and I won’t be discussing them today. I have proceeded by taking tiny steps, seeing what happens, and observing the result. Each time, I figure out what changed, what limitations I’ve hit, and what new techniques I can draw on. I will mention a few avenues of improvement I’ve passed up, though.
For example, for stacking photos of dim subjects (the Milky Way, nebulae, and so on), it is common for astrophotographers to prepare ahead of time a series of preliminary photos used to calibrate the process. These are known as darks, flats, and bias frames. These aren’t pictures of the sky but instead of (essentially) nothingness, allowing you to photograph your camera’s inherent sensor variations. For example, dark frames are photos taken with the lens cap on.
All digital cameras have inherent variations in the sensor. When you stack photos taken with your camera, you’re also stacking up these variations and exaggerating them as well. By taking these extra frames ahead of time and incorporating them into the process, it’s possible to subtract the sensor variations and come out with a smoother photo which can be more freely manipulated.
I did not, of course, prepare any darks, flats, or biases. All I had were lights, which is to say, photos of the actual subject. This is because I was only experimenting and hadn’t planned ahead. I had never done this before, and I was using photos from either months or a year ago.
I also knew I needed to use a new tool. The stacking programs (like AutoStakkert!3) I had been using were more designed for planetary objects or the Moon. These existing processes and tools might have worked okay, but they are quite rigid, and I wanted something more advanced.
For example, in wider-field photos, aligning different sections of the sky means actually conforming the photos somewhat to a single projection. This is necessary because the sky is a large, three-dimensional dome, and each photo projects a piece of that dome onto a two-dimensional image. Any movement in the camera causes that projection to change somewhat, so alignment of the photos together requires a projectional transformation—which looks like a slight warping. (This sort of warping may be easier to imagine if you considered what would happen if you photographed the entire sky all at once and then attempted to stitch it together into a panorama. The panorama would show the horizon on all sides, and the sky would be a circle in the middle. Each photo would have to be bent to complete parts of this circle.)
Instead, I used a much more advanced tool called PixInsight. It is not free software in any sense of the word, unfortunately, but it’s extraordinarily powerful and flexible. This is the only tool I used (aside from Apple Photos), and it’s what I’ll discuss below.
Totally an accidental thing—I had been aimlessly roaming with my tracking motor and just happened to see a blob. I couldn’t quite make it out with my eye, so I used the camera to photograph it more clearly. I decided I’d use the photos to identify it later, which I did. It took a lot of work to get it to show up nicely in an image.
A couple of days ago, on the anniversary of the eclipse, I decided to revisit those photos. I figured, well, I had maybe eight photos of the thing, so maybe I could do something with that. I read some wickedly complicated PixInsight tutorials (including this one), skipped around in them, and sort of scrummaged together a workflow. It’s not perfect, but I’ll share it.
My PixInsight Process for the Omega Nebula
With PixInsight open, first, I went to the “Process” menu, and under “ColorSpaces,” I chose the “Debayer” process. This is a little hard to explain, but essentially it’s a way to deconstruct a limitation of the camera sensor. The images I began with were the RAW images (dumps of the raw sensor data from when I photographed). The sensor’s pixels each have no ability to differentiate color, only light intensity, so a color filter array is placed over each pixel sensor to allow each to see one of red, green, or blue. That then must be debayered or demosaiced to construct the color image accurately. To know which kind of mosaic pattern, I searched the Internet for the one applicable to my camera, and it seemed like “RGGB” was the way to go.
I added images to the “Debayer” process and let it run, and it output a series of files which had been debayered, which had been renamed with a “_d” at the end and were in PixInsight’s own file format, XISF.
The next step was to align the images. PixInsight calls this “registration,” and it has multiple forms. Under the “Process” menu, I went to “ImageRegistration” and found “StarAlignment.”
In it, I chose one of the images from my set as a “reference,” meaning it would be the image against all the others would be aligned. For this, I would use the output of the debayering, so I used the XISF files output from the last step. I also told it to output “drizzle” data, which is used to reconstruct undersampled images. It can add resolution that’s missing using interpolation. It’s possible to configure the star matching and star detection parameters, but I found I did not need to do so.
The output from this step was similar to the previous one, but the resulting files now ended in “_d_r.xisf”. These images had been moved around and warped such that when laid over top of one another, they would match perfectly. Not all the photos could be aligned, and only six survived the process. I proceeded with these.
There was one more step I did before the final stacking, and that was a normalization. Under “Process” I went to “ImageCalibration” and then “LocalNormalization.” This allowed me to create new files (not image files but metadata files) containing normalization data. These data allow reducing noise and cleaning up the signal even further. I learned about it from this extensive tutorial which explains better than I can (which is the source I used to piece together much of this workflow).
After it ran, I finally had all the data I needed for the final stack. PixInsight calls this “ImageIntegration,” which is under the “Process” menu and “ImageIntegration” submenu.
I chose the six images which I had debayered, registered (aligned), normalized, and drizzled. I added them to the process. I added the normalization files and the drizzle files which had been output. I chose the average combination algorithm, which is the default. Switched normalization to “Local normalization,” but I left other parameters alone. Then I ran it.
The result was three views, two of which contained rejected pixels and one of which contained the integration itself. (A view, in PixInsight, is like an unsaved file—something you can see but which doesn’t necessarily exist on disk yet.)
It still appeared dim and indistinct, but I knew this was a raw product, ready to be manipulated. The rejection views were blank in this case, so I discarded them.
I figured that I would use PixInsight to stretch the image, and so under “Process” and “IntensityTransformations,” I first tried “AdaptiveStretch,” but I found this to be too aggressive. With its default parameters, the image was washed out by noise, and I couldn’t tame its parameters enough for a more natural result.
It’s possible in that screenshot to see the artifacts of the alignment process as well (the neat lines where the noise increases near the bottom and right). This is because the images didn’t cover precisely the same area, so after stacking, the places where they don’t overlap are visible. The intense green color is probably either contributed by my camera’s noise or from skyglow I picked up. In either case, it’s not what I want. I threw it away.
I then hit upon trying an “AutoHistogram” in the same submenu, and this was much gentler and more helpful. I bumped up its parameters a bit.
Now this truly got me somewhere.
A lot of additional color and structure leapt out. Notice down on the bottom and the right, the places where the alignment didn’t quite overlap, there’s some color distortion? This is an interesting outcome of the process—a kind of color correction.
This result definitely seemed much closer to what I wanted, but it’s still quite washed out. I could continue in PixInsight, but I really wanted it only for the stacking part. I’m a little more used to editing photos in Apple Photos, as crude as it can be, so I decided to save this file and move it over (as a 32-bit TIFF).
Finishing Omega Nebula in Apple Photos
I first flipped the photo vertically (to undo the flip introduced by the telescope) and cropped away the parts of the nebula which didn’t align fully.
Then I maxed out the saturation so that I could easily see any tint and color temperature adjustments I would need to make. I changed the photo’s warmth to 4800K and did my utmost with the tint to reduce any green cast. After that, I bumped the saturation way back down.
My next goal was to reduce the washed out appearance of the background sky without losing details of the nebula, so I used a curves adjustment. Apple Photos allows using a targeting tool to set points on the curve based on points on the photo, so I tend to do that. (It also allows setting a black point, but I usually find that too aggressive for astrophotography.) A gentle S-shaped curve of all channels often helps. I try not to be too aggressive with the curves adjustment because I can also use a levels adjustment to even out the histogram even more.
Using the “Selective Color” adjustment, I can pick out the color of the nebula and raise its luminance, which will boost the visibility of some of its dimmer portions.
After this, I make some grosser adjustments, using black level, contrast, highlights, shadows, and exposure.
The focus is very, very soft, but I usually don’t apply any sharpening or added definition because it will more often than not exaggerate distortions and noise without adding any new information. The reason for the soft-looking focus is down to a few reasons. First, I didn’t have perfect tracking on the telescope when I made these photos because I didn’t expect to photograph a nebula. Second, the exposures were long enough that the seeing (the ordinary twinkling of the sky) allowed the objects (like stars and other fine points) to smear into larger discs. Third, I hadn’t spent any time getting the focus tack-sharp because I was in a hurry. Fourth, this is a zoomed in section of a combination of several photos, which already tends to blend together some finer details (despite the drizzle data).
The Omega Nebula After Stacking
For what it’s worth, I think it turned out fine for a completely unexpected outing with just a few photos taken over a few minutes. After the entire process of stacking, which took a couple of hours, I came up with this.
Here are the before and after photos side-by-side so you can compare.
The latter image has more structure, more detail, more color, and all with less noise. All this, even with imperfect, brief photos and with an imperfect, incomplete process.
The Milky Way
I decided to see if I could apply the same process to some of the Milky Way photos I had from earlier in July. I had taken several toward the core, including ones which used my portrait lens. I thought the results were middling, and I was frustrated by all the noise in them.
I’m not going to step through the entire process of the stacking because it’s largely the same as the one I applied for the Omega Nebula. I have tried different kinds of parameters here and there (such as comparing average versus median image integration), but in the end, I used largely the same method.
One interesting wrinkle was that my Milky Way photos included trees along the bottom. Because the stars moved slightly between each shot, the registration process left the trees moving slightly between each. This caused a severe glitch after the PixInsight processing.
It’s likely I could have used a rejection algorithm, a mask, or tweaked the combination algorithm not to cause this, but I haven’t learned how to do that yet, so I let PixInsight do what it did.
Before I did any further processing, I needed to hide the glitch, and I decided cropping would be awkward. So I took the trees from another photo and laid them over top as best as I could. It looks sort of crude when you understand what happened, but unless you squint, it works well enough.
It covers a lot of the photo, unfortunately, and it looks really weird when you look closely at it, but hopefully the attention is drawn to the sky.
The Milky Way doesn’t look all that much improved over versions I’ve shown in the past, but it took a lot less work to get it there, and the noise and fine details are significantly improved.
The photo above shows a similar section of the sky as the noisy patch I showed earlier. (They’re not exactly the same section but very close; the same bright star is seen in both.) Here, there’s much less noise, and it’s possible to see indistinct tendrils of dust among the glowing sections of the Milky Way. The stars are easier to distinguish from the background. Below, I’ll place the two side by side for comparison.
That’s the difference—the photo has more underlying signal, so I can eke more detail from it. The overall photo ends up looking better defined as a result, even if it doesn’t appear, superficially, all that much more improved.
I need those calibration shots, for sure: the darks, flats, and biases. I can do those without a night sky, though. I just need to get around to it.
I also have a better idea of what kinds of photos align and stack better than others, so I should leave the glitchy trees at home next time. When I’m using the telescope, I should re-examine my focus; use consistent exposure settings; take many, many photos so that I have some to discard; and track as well as I can manage.
After that, I can elaborate on my process and show better photos than ever before.