Luke Plant's home page (Posts about Emacs)https://lukeplant.me.uk/blog/categories/emacs.xml2023-11-11T20:15:33ZLuke PlantNikolaEverything is an Xhttps://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/2020-11-11T16:00:00Z2020-11-11T16:00:00ZLuke Plant<p>Analysis and examples of the popular high-level pattern of making all the things in your system conform to a common interface.</p><p>Translations of this post (I can't vouch for their accuracy):</p>
<ul class="simple">
<li><p><a class="reference external" href="https://www.chenqing.work/?p=3430">Chinese</a></p></li>
</ul>
<hr class="docutils">
<p>“Everything is an X” is a very high level pattern that you see applied in the
design of lots of systems, including programming languages and user interfaces.
It has a lot of advantages, and some disadvantages. I'll discuss some of these,
then look at some examples, which will be necessary to understand what I'm
really talking about.</p>
<p>When we say “everything” here, we’re talking very loosely – many things in the
system are not, in fact, instances of “X”, but are something lower level, or
just completely different. “Everything is an X” really means “a surprising
number of things in this system are an X”.</p>
<section id="advantages">
<h2>Advantages</h2>
<ul>
<li><p>Simplicity of implementation for the implementer. They don't have to design
and implement a new interface for every new thing in the system. Instead they
implement a single interface that they can re-use everywhere and refine as
necessary.</p></li>
<li><p>Simplicity for the user. They only have to learn one thing, and knowledge
gained in one area is immediately applicable to many others.</p></li>
<li><p>Meta-powers. Not only can the user apply knowledge or skills from one instance
in another area, very often the “X” interface can be applied at a higher
level, and this gives the user super-powers – they can go from manipulating X
to manipulating meta-X with very little extra knowledge needed.</p>
<p>When there are multiple meta-levels that don't seem to stop, another name for
this pattern is “X all the way down“.</p>
</li>
<li><p>Elegance and beauty. Beyond utility, systems designed like this have an
aesthetic value that I think is motivating for both the implementer and the
user.</p></li>
</ul>
</section>
<section id="disadvantages">
<h2>Disadvantages</h2>
<ul class="simple">
<li><p>For some of the things in the system, making them behave as “X” can be a very
bad and awkward fit. This could have the effect of:</p>
<ul>
<li><p>producing an under-powered interface for some things, which has to be
patched up in some way.</p></li>
<li><p>giving an over-powered, bloated interface for other things.</p></li>
<li><p>making “X” too broad to be useful.</p></li>
</ul>
</li>
<li><p>Giving the user a higher-level, meta interface could bring challenges, and
limit how you restructure or optimise internals because you've given the user
too much.</p></li>
<li><p>There are also issues when you have to break the “everything is an X” pattern
for some pragmatic reason – you end up with an ugly corner of the design.</p></li>
</ul>
</section>
<section id="examples">
<h2>Examples</h2>
<section id="programming">
<h3>Programming</h3>
<ul>
<li><p>Java: everything is a class (sort of).</p>
<p>This gives a certain uniformity in how APIs work that tends to reduce the
amount of work you have to do when learning a new library.</p>
<p>In terms of meta-levels, Java doesn't have so much. It has some reflection
abilities that allow classes to be represented as runtime objects, although it
is not very deep in terms of what it offers (contrast Python below).</p>
<p>The limited power for users is undoubtedly useful for compiler writers,
however – you can get excellent performance with Java due to the more limited
nature of the language (again, contrast Python below).</p>
<p>Java breaks its own pattern of basing everything on classes with “primitive
types”, which means you have also have to deal with boxing, auto-boxing etc.
Javascript has similar issues with <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number">Number</a>.</p>
</li>
<li><p>Python: everything is an object.</p>
<p>Note that this is very different from Java's version of “everything is a
class/object” – far more things are actually runtime objects in Python,
including functions, methods, types/classes and modules, as well as instances
of classes. This makes <a class="reference external" href="https://www.toptal.com/python/python-parameterized-design-patterns">parameterisation</a> an
extremely powerful and general pattern in Python.</p>
<p>You also get super powers that come from how meta-classes work:</p>
<ul class="simple">
<li><p>instances are objects</p></li>
<li><p>classes (the things that produce instances) are objects</p></li>
<li><p>meta-classes (the things that produce classes) are objects</p></li>
</ul>
<p>So meta-class programming is just normal programming. It can be done at a
REPL, or with any other technique that you have already learned. Using
meta-classes of course requires learning new things, but things still work in
very familiar ways. Python feels like it's made out of Python all the way
down.</p>
<p>We also see the disadvantages. For example, Python integers are full Python
objects, which is extremely wasteful of memory compared to compact
representations (like C would use, or numpy uses for example). Plus, they are
immutable (for good reasons), so doing simple addition can cause memory
allocations. This is pretty horrifying from an efficiency point of view!</p>
<p>[Correction - in depends on implementation; CPython caches integers between -5
and 255 and has some <a class="reference external" href="https://www.laurentluce.com/posts/python-integer-objects-implementation/">other optimisations</a>,
but still has a lot of overhead in terms of the size of the object]</p>
</li>
<li><p>ML family languages, like Haskell/OCaml and relatives: everything is an
expression.</p>
<p>So, for example, there is no <code class="docutils literal">if</code> statement, there is instead an <code class="docutils literal">if</code>
expression. Assignments, too, are actually expressions – for example a <code class="docutils literal">let</code>
statement is instead an expression that defines some local values and returns
the value of the ‘body’.</p>
<p>Exceptions: Top level assignments, and type signatures. At least in Haskell,
the type system feels quite separate. Due to this, AFAICS the meta level is
kind of missing in Haskell. Haskell “super-powers” come from a bunch of
different features, such as <code class="docutils literal">deriving</code> and TemplateHaskell, and programming
at type level, especially with various language extensions. These are all cool
features, but they have to be learned separately, and I think this is what
gives Haskell the feeling of being a big and intimidating language.</p>
</li>
<li><p>Lisp: everything is an s-expression.</p>
<p>In contrast to Haskell, this goes much further and includes top level
statements. This uniformity of syntax is hugely helpful in writing macros,
probably the biggest meta-power that Lisp boasts.</p>
</li>
</ul>
</section>
<section id="user-interfaces">
<h3>User interfaces</h3>
<ul>
<li><p>Emacs: everything is a buffer.</p>
<p>This is one of Emacs' big ideas – it has very few different UI elements,
because almost everything appears in a buffer. As well as the text of a file
to edit, you also have:</p>
<ul class="simple">
<li><p>help manuals</p></li>
<li><p>auto-generated help for the buffer you are using (such as listing all
keyboard shortcuts).</p></li>
<li><p>search results (this is an essential part of my workflow for recursive
searching that remembers context)</p></li>
<li><p>many more things.</p></li>
</ul>
<p>Once you have learned how to navigate and use buffers, you know how to
navigate these things, <a class="reference external" href="https://www.masteringemacs.org/article/why-emacs-has-buffers">and much more</a>.</p>
<p>Of course, there are other UI elements, such as the toolbar and menubar, to
make Emacs slightly less terrifying to new users. But these tend to be turned
off by experienced users – they're not very useful because…they're not
buffers!</p>
<p>You also gain meta-powers: the list of all buffers is of course a buffer, and
you can manipulate the buffer-buffer using normal mechanisms (e.g. delete line
to kill a buffer).</p>
</li>
<li><p>RDMSs: everything is a table/relation, and can be queried as such.</p>
<p>With some implementations such as PostgreSQL, as well as the normal relations
you have created, internal structures are also presented as relations,
including the lists of tables/relations/indexes, and runtime information.</p>
<p>So you can get a table of all tables:</p>
<div class="code"><pre class="code sql"><a id="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-1" name="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-1" href="https://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/#rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-1"></a><span class="k">SELECT</span><span class="w"> </span><span class="k">table_name</span><span class="w"></span>
<a id="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-2" name="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-2" href="https://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/#rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-2"></a><span class="k">FROM</span><span class="w"> </span><span class="n">information_schema</span><span class="p">.</span><span class="n">tables</span><span class="w"></span>
<a id="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-3" name="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-3" href="https://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/#rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-3"></a><span class="k">WHERE</span><span class="w"> </span><span class="n">table_schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'public'</span><span class="w"></span>
<a id="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-4" name="rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-4" href="https://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/#rest_code_a3ad6230fc49408fbdb4a22b5151b9eb-4"></a><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="k">table_name</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>And query the currently running queries:</p>
<div class="code"><pre class="code sql"><a id="rest_code_3f03ff2ad2c84f65a50cd79d5aced85d-1" name="rest_code_3f03ff2ad2c84f65a50cd79d5aced85d-1" href="https://lukeplant.me.uk/blog/posts/everything-is-an-x-pattern/#rest_code_3f03ff2ad2c84f65a50cd79d5aced85d-1"></a><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_stat_activity</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>And <a class="reference external" href="https://pgstats.dev/">many other things</a>. These things benefit from all
the usual powers of <code class="docutils literal">SELECT</code>, such as joins, filtering, ordering etc.</p>
<p>If you extended the meta-powers, you could create/delete tables by inserting
into/deleting from the “tables table”, and similarly add/remove/alter columns
by manipulating the “columns table”. Your schema then becomes data, and your
power has become a super-power. I don't know if any DB has implemented this,
or if the advantages would be really compelling over normal DDL.</p>
</li>
<li><p><a class="reference external" href="https://www.visidata.org/">Visidata</a>: everything is a sheet.</p>
<p>At points this seems to go further than PostgreSQL in terms of giving you
meta-powers and sticking to the pattern. In addition to the normal data
sheets:</p>
<ul class="simple">
<li><p>rows are sheets (or can be viewed as such)</p></li>
<li><p>column meta-data is a sheet (and not just read-only - you can <a class="reference external" href="https://jsvine.github.io/intro-to-visidata/basics/understanding-columns/#manipulating-columns-from-the-columns-sheet">manipulate it</a>)</p></li>
<li><p>the <a class="reference external" href="https://jsvine.github.io/intro-to-visidata/basics/understanding-sheets/#how-to-use-the-sheets-sheet">list of sheets</a> is a sheet</p></li>
<li><p><a class="reference external" href="https://jsvine.github.io/intro-to-visidata/advanced/configuring-visidata/#the-global-options-sheet">configuration</a>
is a sheet</p></li>
</ul>
<p>…and more.</p>
<p>I'm loving the power of Visidata, which is basically “Excel as a text user
interface”, and I've realised that a lot of its power comes from its <a class="reference external" href="https://lukeplant.me.uk/blog/posts/less-powerful-languages/">lack of
power</a>.</p>
<p>In Excel, you can have multiple tables in a sheet, and put them wherever you
like. You can have headers, or not, you can have a different format or data
type for every cell etc. In Visidata, you have exactly one table per sheet,
you must have headers, each column has exactly one data type etc. The enforced
uniformity of "Everything is a sheet (and a sheet has a tightly defined
structure)” is actually crucial to making Visidata so much nicer than Excel
for many tasks – without it, most of the interface shortcuts and meta-powers
would fail.</p>
</li>
<li><p><a class="reference external" href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a>:
everything is a resource.</p>
<p>(More technically, I’m talking about <a class="reference external" href="https://en.wikipedia.org/wiki/Richardson_Maturity_Model#Level_2_:_HTTP_verbs">Pseudo-REST level 1 and 2</a>,
as opposed to <a class="reference external" href="https://htmx.org/essays/how-did-rest-come-to-mean-the-opposite-of-rest/">actual REST which is almost, but not quite, entirely different</a>).</p>
<p>The idea of everything in your system being a “resource” which you can
manipulate with a few well-understood verbs (GET, PUT, DELETE etc.) is a
really attractive part of REST.</p>
<p>It comes with some meta-powers - with things like <a class="reference external" href="https://www.openapis.org/">OpenAPI</a>, the list of resources is typically also served
as a resource.</p>
<p>Relatively frequently with JSON APIs designed like this, I feel I hit the
“awkward fit” disadvantage. Modelling some actions as “manipulating a
resource” is very bizarre, and what you really want is unembarrassed RPC.
Examples of this are “clone/copy/move” actions where there are hidden
properties that you also want to be handled.</p>
</li>
<li><p>Unix: <a class="reference external" href="https://en.wikipedia.org/wiki/Everything_is_a_file">everything is a file</a>.</p>
<p>This is a pretty successful abstraction, but it does leak quite a bit – for
example, stdout and stdin are “files”, except when they are terminals, which
you very often want to special case because they are not really very much like
files when they have humans sitting in front of them. Also, files turn out to
be <a class="reference external" href="https://danluu.com/deconstruct-files/">much harder</a> than you think, even
before you start treating things that aren't files as files. And in some cases
you might want to deal with files as if they are not files but memory anyway
(mmap).</p>
<p>A lot of people would argue that it’s been taken too far when applied to other
things, like USB interfaces etc., producing some <a class="reference external" href="https://www.youtube.com/watch?v=9-IWMbJXoLM">pretty nasty interfaces</a>.</p>
</li>
</ul>
</section>
</section>
<section id="questions">
<h2>Questions</h2>
<p>I'm sure there are many, many more examples of this pattern, and I'm only
scratching the surface. <a class="reference external" href="https://wiki.c2.com/?EverythingIsa">EverythingIsa on c2.com</a> has a longer list, for example, but
without much analysis. What are your favourite examples? Are there examples
where it works very well – or very badly? Are there other
advantages/disadvantages that I've missed?</p>
</section>
<section id="links">
<h2>Links</h2>
<ul class="simple">
<li><p><a class="reference external" href="https://www.reddit.com/r/programming/comments/jsnffk/everything_is_an_x_lukeplantmeuk/">Discussion of this post on Reddit</a></p></li>
<li><p><a class="reference external" href="https://twitter.com/search?q=https%3A%2F%2Flukeplant.me.uk%2Fblog%2Fposts%2Feverything-is-an-x-pattern%2F&src=typed_query">Discussion of this post on Twitter</a></p></li>
<li><p><a class="reference external" href="https://lobste.rs/s/g7c661">Discussion of this post on Lobsters</a></p></li>
</ul>
</section>
<section id="notes">
<h2>Notes</h2>
<ul class="simple">
<li><p>2023-07: For “everything is a file”, added paragraph and link to “What UNIX Cost Us” video.</p></li>
</ul>
</section>