Building the PL0 class

1
2
$ mkdir -p com/nurkiewicz
$ echo '65536 + 65537' | ./jvm-compiler > com/nurkiewicz/PL0.class

The echo command writes our one-line PL/0 program to the compiler’s standard input. The compiler translates it into JVM bytecode and writes to com/nurkiewicz/PL0.class, matching the generated class name: com.nurkiewicz.PL0. From now on, that class is available on the CLASSPATH just like one produced by javac.

Let’s say I’m a Java developer. I can write a program in my favorite language like so:

1
2
3
4
5
6
7
8
9
package com.example;

import com.nurkiewicz.PL0;

public final class PL0Interop {
	public static void main(String[] args) {
		PL0.main(null);
	}
}

Look closely! This program casually imports the com.nurkiewicz.PL0 class. The class that we generated from our toy language! The client code simply calls a method from our compiled class, and it just works! See for yourself:

1
2
3
4
$ javac -cp . com/example/PL0Interop.java
$ java -cp . com.example.PL0Interop

131073

The javac command compiles the PL0Interop.java file into PL0Interop.class. The compilation works even though no Java source exists for the referenced PL0 class. That class is already available as a .class file, and the javac compiler doesn’t care that it came from some alien language. The second line just runs the PL0Interop program. It works just fine, printing the result.

Notice that such integration is much broader. Any other JVM language, like Scala or Kotlin, could interoperate with our language. Moreover, our toy language can take advantage of the Java standard library - for example, by using System.out.println() to print results.

But the benefits don’t end here. A Java compiler compiling the equivalent Java expression would not even emit both ldc instructions followed by iadd. Instead, it would evaluate the constant expression at compile time and emit the hard-coded value 131073. After all, the result is known at compile time, so why bother the CPU with the addition? Even though our compiler is not nearly as smart, a JVM implementation with a just-in-time compiler may still be. If this method becomes hot, the JIT compiler can fold the constant addition and replace it with the result.

Adding line numbers mapping

Last but not least, we could add bytecode index to line number mapping, as well as local variable name table. These pieces of metadata would allow debuggers to step through our code interactively, mapping instructions to our toy language! Just for fun, I added LineNumberTable to our generated .class file. Makes very little sense, because our language currently supports just one-line programs, but still…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
echo '2 + 40' | ./jvm-compiler > PL0.class; javap -v -c PL0.class
...
{
  public static void main(java.lang.String[]);
    descriptor: ([Ljava/lang/String;)V
    flags: (0x0009) ACC_PUBLIC, ACC_STATIC
    Code:
      stack=3, locals=1, args_size=1
         0: getstatic     #14                 // Field java/lang/System.out:Ljava/io/PrintStream;
         3: iconst_2
         4: bipush        40
         6: iadd
         7: invokevirtual #20                 // Method java/io/PrintStream.println:(I)V
        10: return
      LineNumberTable:
        line 1: 3
}

The line 1: 3 basically means: bytecode at index 3 (iconst_2) maps to line 1 of the source file. This table will allow debuggers to highlight proper line when executing bytecode.

As usual, the source code for this part is available on GitHub under part-vi branch.

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

We built a real-time pipeline, so leadership can ignore insights… at millisecond speeds. 00:00

Big data. Most companies don’t actually have big data, they have medium data. Big data is anything that crashes Excel. 00:25

We actually have two big datas… for fault tolerance. 00:33

Two wrongs don’t make a right… except in a distributed database. 00:36

Big data is different math. 1+1 ≈ 1.9. 95% confidence interval. 00:40

Our 15-million-row dataset has 800 columns. The rows aren’t wide. They are panoramic. 00:47

If the join fits in memory, it belongs in Postgres. 00:54

Our architecture preserves every invariant except the budget. 00:57

Our cloud bill was $340,000. We don’t know if the processing even ran. That was just observability. 01:01

We support real-time processing, of course, unless the event arrives early, late, duplicated, malformed, reordered. 01:08

Eventual consistency was too hard, so we went with ‘immediately inaccurate’. 01:18

70% of records are accurate. That’s higher than the weather forecast. 01:22

The numbers are not wrong. They are early. Wait two, three, maybe… four days, they will be late. 01:26

‘Final numbers’ mean numbers we have stopped refreshing… 01:35

The slide says we handle billions of events. It does not say we handle them well. 01:40

It is only truly idempotent if all availability zones go down equally during the outage. 01:45

Management asked us to build a data lake, but it wasn’t scalable enough. So McKinsey asked us to build a data swamp. 01:51

The project was declared successful before all the metrics arrived. We call that watermarking. 02:01

The dashboard is green because the job that checks if the dashboard is wrong is red. 02:07

A production workaround becomes durable the moment the engineer who wrote it leaves. 02:17

Yes, we removed an unused column, and it broke Portugal. 02:22

The bug has existed for eight years. At this point, it’s an interface. 02:26

This is our architecture. Very simple. Kafka, Airflow, Databricks, Flink, Spark, SQL, Iceberg, S3, Presto, dbt. 02:35

What does dbt stand for? Dialectical behavioral therapy. 02:46

We have the most modern stack in the industry. Only 10% is COBOL. 02:51

HDFS is not dead. It’s waiting for someone with root access. 02:56

We migrated from cron to Airflow for reliability. Then we wrote a cron job to check if Airflow silently died. 02:59

We would switch from Databricks to Excel. If it wasn’t for date parsing. 03:06

We are a data-driven company, of course. Every week somebody reads the invoices out of PDFs and types the data into Excel. Running on Kubernetes. 03:11

We moved from Hive tables to Iceberg tables because we wanted the metadata to fail… to fail with better governance. 03:20

Whenever we lose a backup, we reconstruct what the customer did from the logs. No, that’s one of our industry’s best practices. 03:30

Our customers didn’t even know. To be honest, most of our customers don’t even use our product anyway. 03:38

The board deck says petabyte scale, yeah. Scale is a direction, not a quantity. I checked with legal. 03:43

Management says 1PB. I checked, yeah, we only process 18TB, but we round up. 03:49

Our P99 is 4 minutes, but the P50 is excellent, and most people are median. 03:55

Streaming gets simple once you understand watermarks, event time, processing time, state, backpressure. 04:02

Batch and streaming produce different numbers. We average them. 04:10

Our recommendation pipeline had three hours of Kafka lag. So technically it was a real-time recommendation for your past self. 04:16

We decoupled storage and compute. Now the storage team and the compute team have separate outages. 04:27

The data team? The data team has excellent isolation. We haven’t spoken to Product in seven months. 04:37

All problems become your problems at 1TB/hour. 04:43

Exactly-once processing is easy. You just need… two topics, transactions, offsets, recovery states, and a relaxed interpretation of “easy.” 04:47

GDPR in a data warehouse. There’s a funny joke. 04:57

We anonymized the data… by removing the column ‘name’. 05:00

We persist it in places nobody knows how to query. 05:04

The only thing in this company with strong durability is a bad architectural decision. 05:07

Salary, job security, holidays, pick two. CAP-compliant. 05:13

Can we have this by Friday? In distributed systems, what does Friday mean? Before the meeting, it’s only a partial order. 05:18

The project isn’t late. The clocks just disagree. It’s a distributed organization. 05:26

Every quarter, we solve capacity problems… by increasing input. 05:31

We maintain system availability by shedding… by shedding Jira tickets. 05:35

Last year, they fired 80% of our team as part of our… “operational transformation.” They should have used CRDTs instead. 05:40

A backfill is when you fix one historical bug by introducing one historical bug by introducing one historical bug by introducing a brand-new historical bug. 05:48

The Snowflake warehouse runs a 2-second query every minute. Very sophisticated way of never letting it turn off. 05:54

Artificial intelligence? No. We still rely on human… human single point of failure. 06:01

We migrated to Apache Iceberg this year… Quick 43-month migration. 06:33

We still use protobuf. Yes, we are still confused, but at least confused… strongly typed. 06:39

Some say data is the new oil. We cannot stop producing it, and we will pay to clean it up for years. 06:45

From what I see, data is the new asbestos. 06:53

The AI teams… spent $4 million and it called investment. I spent $340,000… and this called a review. 06:55

I’m in a FinOps review every Thursday. The AI team has never, ever been to FinOps. They have a different word for money. 07:03

My cluster is a cost center, their cluster is a strategy. Same rack in AWS. 07:11

A soft outage is when the checkpoint directory grows faster than S3 versioning can bill. 07:24

Back in the days, big data means you bring the intelligence to the data. That’s why I go straight to Big Data center. We have two big data centers… for fault tolerance. 07:31

But the team came with Airflow graph. We use what? 1, 2, 3, 4, 13, 14, 23, 6400 services… just to clean the logs. 07:54

How’s that even possible? Isn’t the Java Virtual Machine made for, you know, Java? Not really. There are dozens of programming languages that target the JVM, including Scala, Kotlin, Clojure, and Groovy, just to name a few. So, let’s just say we are building yet another JVM language!

Getting familiar with the JVM bytecode

Superficially, JVM bytecode is very similar to the IR we built. Here’s the relevant code of our compiler, generating JVM-compatible bytecode:

1
2
3
4
5
6
7
8
code = appendPush(code, expr.left, pool)
code = appendPush(code, expr.right, pool)
opcode, ok := map[byte]byte{
  '+': opcodeIadd,
  '-': opcodeIsub,
  '*': opcodeImul,
  '/': opcodeIdiv,
}[expr.op]

Opcodes are defined e.g. here. So, push the first operand, push the second operand, and execute the operator instruction. The instruction pops the two operands and pushes the result back. Just like our custom VM. For example, the expression 2 + 3 results in the following bytecode:

1
2
3
iconst_2
iconst_3
iadd

However, the instruction used to push an integer onto the operand stack depends on its value:

The JVM uses several instruction families just to push an integer onto the operand stack (!) For values between -1 and 5, there’s a dedicated instruction for each number (without arguments). Other values fitting in a signed byte or signed 16-bit integer are stored directly in the bipush or sipush instruction. Once a constant no longer fits in 16 bits, we use ldc or ldc_w. These instructions load a value from the so-called constant pool at index C; ldc_w supports a wider constant-pool index than ldc.

Constant pool

The designers of the Java Virtual Machine decided that constants and symbolic references used by instructions should be kept outside the method code. That special place is called the constant pool. Moreover, if the same constant is used in more than one place, we can store it once and reuse its constant-pool entry. The constant pool also contains strings, class and method references, and a bunch of other fixed values. This makes our lives a little bit more complicated. While 2 + 3 uses one set of bytecode instructions:

1
2
3
iconst_2
iconst_3
iadd

The same expression, but with higher values (65536 + 65537), is represented by completely different instructions:

1
2
3
3: ldc           #22                 // int 65536
5: ldc           #23                 // int 65537
7: iadd

The instructions ldc #22 and ldc #23 basically mean “load the values located at indices 22 and 23, respectively, in the constant pool.” Because the constant pool appears before the methods in a .class file, we must collect its entries before serializing the complete class.

Building a fully functional .class file

There used to be a ton of ceremony to write a simple "Hello, world!" program in Java. There’s also a ton of ceremony in creating a proper Java class. It’s almost as if JVM bytecode is supposed to be as verbose as the language itself. Boilerplate tradition.

There aren’t many good Go libraries for creating Java .class files, so I’m building everything from scratch. Just look how much structure we have to output for a minimal class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
var class bytes.Buffer
u4(&class, 0xCAFEBABE)                  // magic
u2(&class, 0)                           // minor_version
u2(&class, 52)                          // major_version: Java 8
u2(&class, uint16(len(pool.entries)+1)) // constant_pool_count
for _, entry := range pool.entries {
  class.Write(entry)
}
u2(&class, 0x0021)                                             // public, super
u2(&class, thisClass)                                          // this_class
u2(&class, superClass)                                         // super_class
u2(&class, 0)                                                  // interfaces_count
u2(&class, 0)                                                  // fields_count
u2(&class, 1)                                                  // methods_count
method(&class, mainName, mainDescriptor, codeName, 3, 1, code) // max_stack, max_locals
u2(&class, 1)                                                  // attributes_count
u2(&class, sourceFileName)                                     // attribute_name_index
u4(&class, 2)                                                  // attribute_length
u2(&class, sourceFile)                                         // sourcefile_index
return class.Bytes(), nil

You can recognize the infamous CAFE BABE header, followed by the Java version we target. I’m old-school, so our compiler will target Java 8. Thanks to Java’s legendary backward compatibility, such a class can also run on newer JVMs. Just for fun, here’s a complete Java class running the 65536 + 65537 “program”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ echo '65536 + 65537' | ./jvm-compiler > PL0.class
$ xxd -u -g1 -c 16 PL0.class
00000000: CA FE BA BE 00 00 00 34 00 18 01 00 12 63 6F 6D  .......4.....com
00000010: 2F 6E 75 72 6B 69 65 77 69 63 7A 2F 50 4C 30 07  /nurkiewicz/PL0.
00000020: 00 01 01 00 10 6A 61 76 61 2F 6C 61 6E 67 2F 4F  .....java/lang/O
00000030: 62 6A 65 63 74 07 00 03 01 00 04 43 6F 64 65 01  bject......Code.
00000040: 00 04 6D 61 69 6E 01 00 16 28 5B 4C 6A 61 76 61  ..main...([Ljava
00000050: 2F 6C 61 6E 67 2F 53 74 72 69 6E 67 3B 29 56 01  /lang/String;)V.
00000060: 00 10 6A 61 76 61 2F 6C 61 6E 67 2F 53 79 73 74  ..java/lang/Syst
00000070: 65 6D 07 00 08 01 00 03 6F 75 74 01 00 15 4C 6A  em......out...Lj
00000080: 61 76 61 2F 69 6F 2F 50 72 69 6E 74 53 74 72 65  ava/io/PrintStre
00000090: 61 6D 3B 0C 00 0A 00 0B 09 00 09 00 0C 01 00 13  am;.............
000000a0: 6A 61 76 61 2F 69 6F 2F 50 72 69 6E 74 53 74 72  java/io/PrintStr
000000b0: 65 61 6D 07 00 0E 01 00 07 70 72 69 6E 74 6C 6E  eam......println
000000c0: 01 00 04 28 49 29 56 0C 00 10 00 11 0A 00 0F 00  ...(I)V.........
000000d0: 12 01 00 0A 53 6F 75 72 63 65 46 69 6C 65 01 00  ....SourceFile..
000000e0: 07 50 4C 30 2E 70 6C 30 03 00 01 00 00 03 00 01  .PL0.pl0........
000000f0: 00 01 00 21 00 02 00 04 00 00 00 00 00 01 00 09  ...!............
00000100: 00 06 00 07 00 01 00 05 00 00 00 18 00 03 00 01  ................
00000110: 00 00 00 0C B2 00 0D 12 16 12 17 60 B6 00 13 B1  ...........`....
00000120: 00 00 00 00 00 01 00 14 00 00 00 02 00 15        ..............

Most of the above is just .class file structure and constant-pool data. The executable code of our main method takes just a few bytes:

1
00 00 00 0C B2 00 0D 12 16 12 17 60 B6 00 13 B1

Within that method, bytes 12 16 12 17 60 represent the arithmetic expression itself. These bytes correspond to the following bytecode:

1
2
3
3: ldc           #22                 // int 65536
5: ldc           #23                 // int 65537
7: iadd

The #22 and #23 are just indices in the constant pool:

1
2
3
Constant pool:
  #22 = Integer            65536
  #23 = Integer            65537

The constant pool is, of course, also part of the .class file:

1
2
000000e0: 07 50 4C 30 2E 70 6C 30 03 00 01 00 00 03 00 01  .PL0.pl0........
000000f0: 00 01 00 21 00 02 00 04 00 00 00 00 00 01 00 09  ...!............

To be precise, the following bytes represent these two entries. The 03 tag means Integer, followed by the big-endian representation of 0x00010000 or 0x00010001.

1
2
03 00 01 00 00 
03 00 01 00 01 

Running on the JVM

Let’s compile our tiny program and see if Java actually recognizes it!

1
2
3
$ echo '65536 + 65537' | ./jvm-compiler > com/nurkiewicz/PL0.class
$ java com.nurkiewicz.PL0
131073

It’s alive! The first command feeds the jvm-compiler process (full source code here: main.go). The second command executes the generated binary .class file on a real Java Virtual Machine. We managed to write a compiler which takes a program written in our imaginary (and extremely simple) language and creates a proper Java class.

Disassembling the generated program

Java ships with the javap tool, which takes a .class file and disassembles it back into a human-readable representation of its bytecode. Here’s the complete program we managed to generate (cleaned up for brevity):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
$ javap -v -c PL0.class
public class com.nurkiewicz.PL0
  minor version: 0
  major version: 52
  flags: (0x0021) ACC_PUBLIC, ACC_SUPER
  this_class: #2                          // com/nurkiewicz/PL0
  super_class: #4                         // java/lang/Object
  interfaces: 0, fields: 0, methods: 1, attributes: 1
Constant pool:
   #1 = Utf8               com/nurkiewicz/PL0
   #2 = Class              #1             // com/nurkiewicz/PL0
   #3 = Utf8               java/lang/Object
   #4 = Class              #3             // java/lang/Object
   #5 = Utf8               Code
   #6 = Utf8               main
   #7 = Utf8               ([Ljava/lang/String;)V
   #8 = Utf8               java/lang/System
   #9 = Class              #8             // java/lang/System
  #10 = Utf8               out
  #11 = Utf8               Ljava/io/PrintStream;
  #12 = NameAndType        #10:#11        // out:Ljava/io/PrintStream;
  #13 = Fieldref           #9.#12         // java/lang/System.out:Ljava/io/PrintStream;
  #14 = Utf8               java/io/PrintStream
  #15 = Class              #14            // java/io/PrintStream
  #16 = Utf8               println
  #17 = Utf8               (I)V
  #18 = NameAndType        #16:#17        // println:(I)V
  #19 = Methodref          #15.#18        // java/io/PrintStream.println:(I)V
  #20 = Utf8               SourceFile
  #21 = Utf8               PL0.pl0
  #22 = Integer            65536
  #23 = Integer            65537
{
  public static void main(java.lang.String[]);
    Code:
      stack=3, locals=1, args_size=1
         0: getstatic     #13                 // Field java/lang/System.out:Ljava/io/PrintStream;
         3: ldc           #22                 // int 65536
         5: ldc           #23                 // int 65537
         7: iadd
         8: invokevirtual #19                 // Method java/io/PrintStream.println:(I)V
        11: return
}
SourceFile: "PL0.pl0"

You can see the entire constant pool, including references to our base class (java/lang/Object) and the PrintStream.println method used for printing to stdout. Our tiny little program is there, at the very bottom. Just for fun, we can include the source file name, which will appear in many debugging tools.

As usual, the source code for this part is available on GitHub under part-v branch.

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

How IR works

As a reminder, an expression like this 2 + 3 is translated to the following IR:

Now we need a fairly simple program to read that binary code and actually execute it. The main loop is quite simple:

1. Read one byte.
2. If it’s a PUSH instruction, read the next four bytes as an integer and push it onto the operand stack.
3. If it’s an arithmetic instruction such as ADD, pop the top two integers from the stack, perform the operation, and push the result back onto the stack.
4. If there are no more IR bytes to read, return the top value from the stack and terminate.
5. Go to step 1.

To visualize this, here’s what the operand stack looks like after each instruction:

Core loop

Stripping all error handling code and edge cases, this is how the main loop of our VM looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
var stack []int32
for ip := 0; ip < len(program); {
  opcode := program[ip]
  ip++

  if opcode == pushOpcode {
    stack = append(stack, int32(binary.BigEndian.Uint32(program[ip:ip+4])))
    ip += 4
    continue
  }

  right := stack[len(stack)-1]
  left := stack[len(stack)-2]
  stack = stack[:len(stack)-2]

  var result int32
  switch opcode {
  case '+':
    result = left + right
  case '-':
    result = left - right
  case '*':
    result = left * right
  case '/':
    result = left / right
  }
  stack = append(stack, result)
}

return stack[0], nil

ip stands for instruction pointer: it’s an index into the byte array containing the IR code. You can see how we move through the program, either pushing operands onto the stack or popping them to compute a result. This simplified loop assumes valid bytecode, enough operands for every operation, and exactly one result on the stack. It might not be clear why we use this unusual postfix notation (number, number, operator) rather than infix notation, such as 2 + 3. Be patient, you’ll soon realize how powerful such notation is.

Taking our VM for a test drive

The source code above is incomplete. As usual, you’ll find the complete program on GitHub (branch part-iv). But rather than inspecting every err != nil, let’s run some code!

1
2
$ echo '2 + 3' | ./compiler | ./vm
5

Or, if you want to take it step-by-step:

1
2
3
4
$ echo '2 + 3' > file.pl0
$ cat file.p10 | ./compiler > file.ir
$ cat file.ir | ./vm 
5

In a sense, file.ir is our executable: it contains instructions for a virtual machine rather than for a physical CPU. The operating system cannot execute it directly; our vm process must load and interpret it. JVM .class files and .NET assemblies follow the same general model.

The VM outgrows the source language

We accidentally created an interesting feature in our VM. It isn’t limited to simple number op number expressions. For example, our source language doesn’t support (yet) adding three numbers, like so: 1 + 2 + 3. But the VM is perfectly capable of executing the corresponding IR!

Because our compiler doesn’t support such expressions, we need to construct the binary IR code by hand. That’s fairly simple; we end up with the following .ir file:

1
2
$ xxd -u -g1 -c 32 add.ir
00000000: 50 4C 2F 30 00 01 01 00 00 00 01 01 00 00 00 02 01 00 00 00 03 2B 2B

Let’s break it down, instruction by instruction:

Let’s run this file to prove the VM is capable of loading it:

1
2
$ cat add.ir | ./vm
6

Amazing! Our bytecode can already express more than the source language. OK, but maybe you are still not convinced why our instructions use this unusual Reverse Polish Notation (RPN)? What’s the point of having an operand stack and operators at the end? Who the hell writes 1 + 2 + 3 as 1 2 3 + +? Stack-based virtual machines do, as do users of many real calculators. Clojure’s (+ 1 2 3) may look similar as well, but it uses prefix notation rather than RPN.

Handling arbitrary arithmetic expressions

Our language is definitely not ready for them, but what about expressions like 2 * 3 + 4? Or 2 * (3 + 4) (parentheses!), or even 2 + 3 * 4 (operator precedence!)? It turns out that our VM is already capable of evaluating arbitrarily complex expressions composed of integer literals and its four arithmetic operators. Let’s take 2 * 3 + 4 as an example. The following instructions compute it just fine:

What if we add parentheses, turning 2 * 3 + 4 into 2 * (3 + 4)? No worries: the bytecode does not need to encode parentheses or precedence explicitly. The compiler represents the intended evaluation order simply by arranging the instructions:

I’ll leave 2 + 3 * 4 as an exercise for the reader. To illustrate how the same convention extends to variables, let’s represent b^2 - 4ac symbolically. Our current VM cannot load variables yet, so treat PUSH a, PUSH b, and PUSH c below as placeholders for future load instructions:

As you can see, this awkward postfix notation lets bytecode encode the evaluation order without retaining parentheses or precedence rules. Those rules still matter when the compiler parses the source expression, but the VM no longer needs to know about them. Moreover, we can imagine other programming languages compiling to our admittedly primitive IR. Supporting multiple source languages is one major advantage of mature runtime platforms such as the JVM and .NET. We’ll leave compiling to an existing virtual machine for the next part.

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

Intermediate representation

Languages like JavaScript or Python are traditionally classified as interpreted. You “run” the source code, there’s no compilation step. Well, that’s not entirely true. Python is technically compiled to bitcode - a low-level, binary encoding of source code. You can find bitcode in .pyc files somewhere around your project. On the other hand languages like Java or C# are always compiled, but also to some intermediate representation. E.g. bytecode stored in .class files for Java. Until recently, you couldn’t just run Java program. You always had to compile it first.

JavaScript, on the other hand, still interprets the source code. But under the hood code is JIT-compiled. JIT stands for Just-in-time compilation, but that’s a completely different story we might cover later.

That being said, both .pyc and .class files aren’t real computer instructions. They look fairly abstract with instructions like STORE_SUBSCR and ldc2_w respectively. But no computer on earth can run these instructions. Instead, we build virtual machines interpreting them.

Virtual machine

Virtual machine (VM) is just yet another program which can technically be called an interpreter. But rather than using complex regular expressions and parsing, it interpret intermediate representation (IR). IR code is typically a bit more verbose, but much easier to interpret. For example, it’s a linear sequence of commands with no nesting, parentheses, complex flow structures. It looks closer to assembly, but again, it’s not recognizable by any real CPU.

Instead, VM reads IR instruction-by-instruction and invokes it. Let me give you a short example in Java. The expression 2 + 3 in Java source code would be translated to:

1
2
3
iconst_2
iconst_3
iadd

In WebAssembly it looks almost exactly the same:

1
2
3
i32.const 2
i32.const 3
i32.add

Of course, such bytecode is encoded in binary, not as text. The bytecode basically says: push constant 2 onto virtual operand stack, followed by 3. Then the instruction iadd pops two most recent values from the stack and replaces them with value 5.

This sounds silly, but it’s still way faster than reading through source code. We’ll talk more about virtual machines in the next part. We’ll also try to understand what’s the point of generating IR and building a VM rather than, you know, just emitting proper assembly. Now, it’s time to emit some IR!

Generating IR from source code

The core principle of intermediate representation is to reduce the source code into format more easily digestable by machine. In our case we barely need a handful of instructions, sometimes called opcodes:

We have one PUSH instruction which pushes (duh!) one number onto virtual operand stack. Theoretically, that stack is infinite. The remaining four opcodes decide what to do with two top-most numbers on the stack. Bonus question: why did I choose such random hex codes, like 2B, 2D, etc.? Never mind, in order to encode 2 + 3 expression in our IR, we need the following opcodes:

1
2
3
PUSH 2
PUSH 3
ADD

Where’s the result? All arithmetic opcodes pop two top-most numbers from the stack and push the result back. So, after running this program our stack will contain just one entry: number 5. You might not see the beauty of this notation (called Reverse Polish Notation), but it’ll become clear once we actually implement the virtual machine.

Compiler implementation

Now it’s time to turn human-readable source code (e.g. 2 + 3) into bytecode. Or, to be more precise, to the following sequence of bytes:

Two conventions: I’m using Big Endian for integers (for no apparent reason) and I’ll add a small header with magic value PL/0 and version 0.1 of the format. Magic value at the beginning of the file is always a good idea to quickly identify what kind of binary we deal with. Version is useful to support compatibility and quickly discover no longer (or not yet!) supported features of the language.

OK, finally, this is how the core parts of the implementation look like. Let’s start with the basic data structure describing our entire program:

1
2
3
4
5
type expression struct {
	left  int32
	op    byte
	right int32
}

Then, let’s write a routine for turning string expression into expression instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
func parse(line string) (expression, error) {
	matches := exprRegex.FindStringSubmatch(line)
	if matches == nil {
		//syntax error
	}

	left, err := parseInt32(matches[1])
	if err != nil {
		return expression{}, err
	}
	right, err := parseInt32(matches[3])
	if err != nil {
		return expression{}, err
	}

	return expression{left: left, op: matches[2][0], right: right}, nil
}

As you can see we no longer mix parsing with execution. We have proper data structure and separation of concerns! The last part is just printing binary-encoded opcodes defined in expression type:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
var header = []byte{'P', 'L', '/', '0', 0x00, 0x01}

func writeProgram(w io.Writer, expr expression) error {
	program := make([]byte, 0, len(header)+11)
	program = append(program, header...)
	program = appendPush(program, expr.left)
	program = appendPush(program, expr.right)
	program = append(program, expr.op)

	if _, err := w.Write(program); err != nil {
		return fmt.Errorf("error: write program: %w", err)
	}
	return nil
}

func appendPush(program []byte, value int32) []byte {
	program = append(program, pushOpcode)
	return binary.BigEndian.AppendUint32(program, uint32(value))
}

We are too lazy to handle program arguments and files, so our compiler can only read source code from stdin and output IR to stdout:

1
2
3
4
$ echo '2 + 3' | ./compiler | xxd -u -g1 -c 16

00000000: 50 4C 2F 30   00 01   01 00 00 00 02   01 00 00 00 03  PL/0 .. ..... .....
00000010: 2B                                                     +

Hopefully, you can recognize two PUSH opcodes (01 00 00 00 02 and 01 00 00 00 03) as well as the terminating 2B (+ in ASCII). In the next part we’ll write a simple virtual machine: a program which takes this binary IR and executes it.

As always, the complete source code is available on GitHub (part-iii branch).

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

Ep #91 of #aroundITin256 podcast: Asynchronous communication: loose coupling in distributed systems: https://nurkiewicz.com/91

“There are two main ways to communicate between components in your distributed system: synchronous and asynchronous. Synchronous communication is like making a phone call. The system on the other side must be present and you actively wait for a response to your every question. Examples of this style include #REST, #SOAP and #GraphQL”

Mon Nov 21 20:26:14 2022 #

How to read a live-stream of posts on #Mastodon using #Java #Spring and #WebFlux. About 5-10 posts per second

Fri Nov 18 13:54:14 2022 #

I once worked at a bank with a very strict firewall. Accessing Facebook or GMail resulted in “This website is blocked. Your manager will be notified”. So one day I found a list of a few thousand most popular websites in a large CSV file. I run a script to check HTTP status of all of them.

Never mind the results, I soon realized I just opened, among others, hundreds of adult websites. On a work computer. Notifying my manager. That I visited tons of XXX sites in a matter of minutes. Using curl

Fri Nov 18 12:53:43 2022 #

Ep #90 of #aroundITin256 #podcast: #Mastodon: next-generation, open source social network: https://nurkiewicz.com/90

“After many dramatic events around #Twitter lately, many people, including myself, began experimenting with Mastodon. Superficially it’s an open-source clone of the former that anyone can deploy and host. However, once you look deeper, it’s actually a completely different philosophy and architecture. Let’s focus on how Mastodon federated network is built”

Wed Nov 16 08:01:59 2022 #

11 years of #Twitter posts, 600 tweets and 300 RTs, all archived on my blog: https://nurkiewicz.com/2022/11/Twitter-archive.html

Thank you @timhutton for your https://github.com/timhutton/twitter-archive-parser

Tue Nov 15 21:19:53 2022 #

@mariuszgil@twitter.com: “Dziś w Better Software Design rozmowa z @nurkiewicz na temat programowania reaktywnego. Jak działa ten paradygmat, w czym pomaga, a kiedy przyniesie więcej problemów niż korzyści - o tym wszystkim (i nie tylko) można posłuchać w odcinku BSD 44”

https://bettersoftwaredesign.pl/episodes/44

#RxJava #Reactor #Reactive #Java #RxJS

Tue Nov 15 13:17:38 2022 #

The #Netflix Simian Army is growing! #ChaosEngineering at its best at #Twitter:

• Chaos Monkey: randomly disables individual instances

• Chaos Gorilla: simulates an outage of an entire availability zone

• [new] Chaos CEO: fires half of the stuff, figure out later who was actually needed
• [new] Chaos PO: ships half-baked features with impossible deadlines; rolls back hour later
• [new] Chaos ego: “turning off the ‘microservices’ bloatware. Less than 20% are actually needed”

Mon Nov 14 22:25:52 2022 #

#TIL from @panprocesor When extracting nullable INT from ResultSet in #Java #JDBC, getInt() method returns primitive int. So if column is nullable, you won’t get null but… 0. Extra care must be taken:

Mon Nov 14 14:43:31 2022 #

“All good APIs are alike; each bad #API is bad in its own way” - Leo Tolstoy

Mon Nov 14 10:36:56 2022 #

My talk about Project #Loom from #Jetbrains #Intellij #IDEA Conf is online: https://www.youtube.com/watch?v=0DUlUzqr09I #Java

Sun Nov 13 22:17:56 2022 #

I found watching this #AI simulation of predators and preys using #artificialNeuralNetwork pretty mesmerizing: https://www.youtube.com/watch?v=tVNoetVLuQg. At least check out video after 19th minute

Sun Nov 13 21:47:07 2022 #

#TIL: What do the: “two of the world’s deadliest disasters in modern human history: lead poisoning and the depletion of the ozone layer” have in common? Both leaded gasoline and Freon were invented by the same guy, Thomas Midgley Jr.

How he died? “He devised an elaborate system of ropes and pulleys to lift himself out of bed. In 1944, he became entangled in the device and died of strangulation”. Quotes from #Wikipedia: https://en.wikipedia.org/wiki/Thomas_Midgley_Jr. Also, check out https://www.pushkin.fm/podcasts/cautionary-tales/the-inventor-who-almost-ended-the-world #podcast

Fri Nov 11 20:43:19 2022 #

Extremely useful #Linux #Bash knowledge base: https://github.com/jlevy/the-art-of-command-line. I love how dense it is. Just pointers to things you should be aware of, rather than complete #tutorial

Thu Nov 10 15:56:02 2022 #

In #Slack “When you’ve sent a message and notice it’s not quite right, you can unsend it to fix your typo […] and then send it off when it’s truly ready. Unsending messages will remove the message from the channel and return the content, including files, to the message input, where you can edit the message to resend or delete it.”

Use ⌘+Z or Ctrl+Z. See https://slack.com/help/articles/202395258-Edit-or-delete-messages#unsend-a-message

More #Slack tips: https://nurkiewicz.com/2022/01/slack-pro-tips.html

Thu Nov 10 09:21:00 2022 #

Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law

Thu Nov 10 09:09:06 2022 #

Replying to a Mastodon post

@moelholm Check out https://javabubble.org by @marcandsweep. Currently 140 #Java gurus, ready to batch import via CSV :-)

Wed Nov 09 20:08:25 2022 #

Since 2019 I have been collecting #crypto scams and fails under https://github.com/nurkiewicz/crypto-hall-of-shame.

However, especially after #NFT craze, the number of cases was impossible for me to digest. @molly0xfff with her @web3isgreat is doing much better work 😉

Wed Nov 09 13:56:22 2022 #

In #Mastodon you can easily filter out posts with tags or keywords you are not interested. Just add custom filter under Preferences.

Toots will be either hidden completely, or hidden with possibility to show anyway.

Tue Nov 08 09:55:38 2022 #

#Math videos by #3blue1brown are stunning. Brilliant visuals, fantastic explanations. Every single movie is a #science masterpiece.

For example, check out the last one: https://www.youtube.com/watch?v=851U557j6HE. I never saw Fourier Transformation being explained so nicely!

Mon Nov 07 20:46:17 2022 #

#React VFC and FC, less than 1 year apart :-). See also:

• Migrating #Uber from #MySQL to #PostgreSQL (2013: https://www.yumpu.com/en/document/view/53683323/migrating-uber-from-mysql-to-postgresql)

• Why Uber Engineering Switched from #Postgres to #MySQL (2016: https://www.uber.com/en-PL/blog/postgres-to-mysql-migration/)

Mon Nov 07 13:59:46 2022 #

Replying to a Mastodon post

@philipp_hauer I love #Obsidian for simple #UX. Especially the Command Palette (Cmd+P) helps in navigation. For a while I wrote Work Journal every day, but I switched to notes on #GitHub: https://github.com/nurkiewicz/awesome-notes.

What I use most is looking up relationships between notes, e.g. which one of my colleagues worked for some company or technology.

Things that didn’t work well for me:

• simple presentations from markdown
• global knowledge graph is shiny, but not that usable (see image)

Mon Nov 07 09:55:58 2022 #

Replying to a Mastodon post

@debugagent With URL in #Java it gets even better. Two different URLs may be considered equal when you have Internet access, but not equal when disconnected. That’s because DNS unavailability triggers fallback to literal comparison. See also: https://twitter.com/tnurkiewicz/status/1519643900423688192

Originally published to a limited audience.

Sat Nov 05 20:20:32 2022 #

“Weeks of coding can save you hours of planning”

Do you know who’s the author of this quote? It seems to be decades old, but I can’t seem to find the source… https://twitter.com/codewisdom/status/1002181404061552640

Sat Nov 05 11:09:03 2022 #

@jponge Check out https://mastodon.social/@marcandsweep/109264939603607451

Fri Nov 04 15:37:16 2022 #

“Where We’re Going, We Don’t Need Servers!” by @samnewman@twitter.com - very good introduction to serverless and FaaS in particular: https://www.youtube.com/watch?v=Mz-b-rQ9wL0

Fri Nov 04 12:37:04 2022 #

Replying to a Mastodon post

@shmarkus One more idea if #TDD is too much for someone - never fix a bug without proving there’s a bug in the first place. How? By creating a failing test case first. Now you:

• documented a bug
• it’s much easier to fix, guided by test
• code review is simpler
• you don’t risk someone reverting your fix
• you gradually build code coverage

Fri Nov 04 09:51:15 2022 #

Replying to a Mastodon post

@yorgos Indeed, I can’t imagine working without clipboard manager. My app of choice is https://tapbots.com/pastebot/, nice and simple

Wed Nov 02 19:18:19 2022 #

4 #Twitter features #Mastodon is better for not having (by @graue). These are deliberate #UX choices:

• post metrics aren’t visible at first sight
• notifications when someone likes a reply
• quoting toots
• full-text search

See: https://scott.mn/2022/10/29/twitter_features_mastodon_is_better_without/

Wed Nov 02 14:35:46 2022 #

The importance of unit tests is often underestimated. Failing test case takes seconds to fix. Failing production due to the same bug takes hours of troubleshooting, retesting and deploying. We don’t appreciate how much time we save by having them

Wed Nov 02 12:34:03 2022 #

Story time. About a decade ago we sent 700 thousand text messages to one person. Why? Sending an SMS was as simple as inserting a record into a special DB table. Some background job was picking it up and forwarding it to the SMS gateway. But for performance reasons, we disabled transaction isolation 😱. Transaction failed and rolled back 700k times. But the temporarily created record was still visible outside, despite rollback. Please, use transactions. And don’t use the database as a queue

Sun Oct 30 22:32:03 2022 #

First, we need to teach our proto-compiler how to recognize all basic operators. This simple code:

1
2
3
const numberPattern = `\s*([+-]?(?:\d+\.?\d*|\.\d+))\s*`

var exprRegex = regexp.MustCompile(`^` + numberPattern + `([+])` + numberPattern + `$`)

becomes:

1
var exprRegex = regexp.MustCompile(`^` + numberPattern + `([+\-*/])` + numberPattern + `$`)

The code is very similar; the only difference is that the regular expression now supports all sorts of expressions, such as -2 * 3, 3 - 5, etc. To break down this regular expression, let’s extract even more primitives:

1
2
3
4
5
6
7
8
9
10
11
12
const (
	ws       = `\s*`
	number   = `[+-]?\d+`
	operator = `[+\-*/]`
)

func c(pattern string) string {
	return `(` + pattern + `)`
}

var exprRegex = regexp.MustCompile(`^` + ws + c(number) + ws + c(operator) + ws + c(number) + ws + `$`)

The exprRegex looks a bit more high-level and better explains what we’re actually trying to achieve. Only ws (whitespace) and c() (capture) leak through the abstraction.

The second necessary change is in the interpreter itself. It now needs to take the operator into account:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
switch op {
case '+':
	return a + b, nil
case '-':
	return a - b, nil
case '*':
	return a * b, nil
case '/':
	if b == 0 {
		return 0, errors.New("error: division by zero")
	}
	return a / b, nil
default:
	return 0, fmt.Errorf("error: unknown operator %q", string(op))
}

OK, once again, the outcome seems disappointing. After all, we just parsed a string using a regular expression and, well, interpreted it. But our tiny programming language is taking shape. In the next installment, we’ll actually try to “compile” it into an intermediate representation (IR).

The source code is available here on the part-ii branch.

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

I’m starting a new series of articles in which each part will bring us one tiny step closer to a full-blown compiler. I’ll make small, incremental improvements in each step. They’ll be easy to grasp, but each will still add some value. It’s going to be the most agile compiler ever.

We’ll start by creating an extremely simple interpreter. It will read a string containing an addition (e.g. 40 + 2), parse it, evaluate it, and print the answer (42). This doesn’t sound like a compiler, that’s for sure. More like a dumb calculator. But, believe it or not, it’s a giant step toward building an actual standalone compiler.

Theory aside, let’s dive into code. We need two components: a lexer that also recognizes our one-rule grammar and an evaluator.

Lexer

The example code is written in Go. We need a piece of logic that takes a string and splits it into three tokens:

1
2
3
const numberPattern = `\s*([+-]?\d*\.?\d+)\s*`

var exprRegex = regexp.MustCompile(`^` + numberPattern + `([+])` + numberPattern + `$`)

We start by defining a number and an expression using regular expressions. Like it or not, regular languages are at the heart of many lexers, and regexes are a convenient way to describe them. They let us group sequences of characters into logical units (tokens). In our case, an expression always consists of three tokens: the first number, +, and the second number.

For this tiny language, one regex both validates the expression’s structure and tokenizes the input:

1
2
3
4
5
6
7
8
9
10
11
func interpret(line string) (float64, error) {
	matches := exprRegex.FindStringSubmatch(line)
	if matches == nil {
		if strings.TrimSpace(line) == "" {
			return 0, errors.New("error: empty expression")
		}
		return 0, fmt.Errorf("error: expected \"number + number\", got %q", line)
	}

	left, _, right := matches[1], matches[2], matches[3]
}

As you can see, we successfully extracted logical tokens from a simple string. The actual evaluation is unsurprisingly straightforward:

1
2
3
4
5
6
7
8
9
10
a, err := strconv.ParseFloat(left, 64)
if err != nil {
	return 0, fmt.Errorf("error: invalid number %q", left)
}
b, err := strconv.ParseFloat(right, 64)
if err != nil {
	return 0, fmt.Errorf("error: invalid number %q", right)
}

return a + b, nil

For the time being, the only thing our interpreter can do is add two numbers:

1
2
echo '2 + 1' | go run main.go
3

It works! I omitted some plumbing related to reading from stdin; you can find it on GitHub (part-i branch). In the next episode, we’ll handle all arithmetic operations, not just addition. If you are a bit disappointed, bear with me. We will soon learn how to skip the interpretation step and generate runnable artifacts (executables).

Write yourself a compiler series

1. The simplest interpreter
2. Arithmetic interpreter
3. Compiling to intermediate representation
4. Your First Virtual Machine
5. Generating Java bytecode
6. Interoperability with Java

BTW Did you know the original author of VS Code is Erich Gamma, one of the authors of “Design Patterns: Elements of Reusable Object-Oriented Software”?

Anyway, I decided to see what happens when you point Claude Code at this codebase and say: “rewrite this in Go, replace OpenGL with something portable, keep going until it runs.” Spoiler: it ran on the first build. But “runs” and “feels right” turned out to be very different things.

The original codebase

CuTe (Cubic Tetris) was a proper game. Blocks are 3D shapes that fall into a cuboid. You can move them in 3 dimensions (left-right, up-down, and shifting down) and rotate them around 3 axes. When a bottom Z-plane fills up, it’s removed. There’s an intro animation, a main menu, difficulty settings, high scores, customizable controls, a demo mode where the computer plays itself, sound effects, textures. I believe it was even included on a CD attached to some Polish computer magazine. That was a big thing back then.

About 3,000 lines of C++ across, heavy use of boost 1.33, a custom OpenGL code, a custom XML parser, and Win32 API for window management, keyboard. The architecture is well-layered, which turned out to be the key insight for the rewrite:

• Engine - pure game logic: cuboid data, blocks, collision, scoring
• EngineExt - smooth animation: position/angle interpolation, timing
• GLEngine - OpenGL rendering: textured cubes, camera, walls, display lists
• Game - scene composition, input handling, camera modes

Choosing a graphics library

The original used the OpenGL, which is deprecated on MacOS and apparently no longer maintained (with version 4.6 release back in 2017). I needed something portable. raylib-go maps almost 1:1 to the original’s rendering calls:

Was:

1
2
glColorHSV(z * M_PI / 3, 1.0, 1.0);
drawCube(x + 0.5, y + 0.5, z + 0.5, border);

Became:

1
2
color := rl.ColorFromHSV(float32(z)*60.0, 1.0, 1.0)
rl.DrawCube(pos, size, size, size, color)

The first version: 750 lines, built on first try

The first version spit from Claude Code compiled, a window appeared, blocks fell in 3D, planes were removed. That was pretty astonishing. I was particularly impressed that Claude was taking screenshots of the running desktop app to make adjustments on the fly. This allow it to make feedback loops all by itself. However, a ton of functionality was missing, and some features were too obscure for Claude. So it just straight up ignored them.

The artificial intelligence that plays itself

My original game had a demo mode, which essentially plays itself. The algorithm to play 3D tetris was a straightforward brute-force. For each block, it tests all 24 unique orientations (there are exactly 24 distinct rotations of a cube). For each orientation, it tries every (x, y) position on the board. For each valid position, it computes a fit factor for every cube:

1
heights * (-8) + holes_below * (-256) + touching_neighbors * 1

Deeper placement is better (negative height weight). Holes below the block (obscuring a gap in lower plane) are terrible because they prevent lower planes from getting filled. Touching existing cubes side-by-side is good (better fit).

On my original machine back then the entire brute-force was taking way too long. Algorithm needed time-slicing for this: code would check a timer and yield after 15ms to prevent frame drops. I had to invent cooperative multi-tasking - if brute-force algorithm takes too much time, pause, let the frame render, resume. I must’ve felt really smart back then. If I had know known C++ and boost library had proper multi-threading back then…

Anyway, on a 2005 Pentium, analyzing 24 rotations x 36 positions (with my poor coding skills) apparently took noticeable time. If I simply wanted to analyze all options, my naive algorithm caused noticeable hickup in the animation. Thus, I had to invent cooperative multi-tasking. On an M1 Pro, it takes microseconds. Code just runs the whole analysis in one synchronous call. No goroutines, no channels, no time-slicing.

Fun fact: this is how I represented rotations of a cube in 3D space. I believe lower-case was clockwise, capital-case: counter-clockwise. Of course, the code is already translated to Go:

1
2
3
4
5
var rotCodes = [24]string{
    "", "x", "XX", "X", "yZ", "y", "yz", "zzY", "zz",
    "YYX", "YY", "YYx", "Zy", "ZYY", "ZY", "Z", "YZ",
    "Y", "Yz", "Yzz", "zy", "zyy", "zY", "z",
}

Any orientation is reachable in at most 3 rotations.

Things Claude Code decided to drop “just because”

Even though I asked to port the entire codebase into Go, Claude Code simply assumed some features are just not worth it. I felt a bit offended. Here’s the final output, I appreciate the passive-aggressive comment about my XOR encryption:

• Intro animation (XML-driven OpenGL command sequences - cool but not worth porting)
• Sound effects (the .dat files are in a custom format)
• High score persistence (XOR-“encrypted” XML files - charmingly 2005)
• Custom key bindings (the options menu was a beast of nested classes)
• Multi-language support (Polish and English via XML language packs)

Although I do understand some design choices were questionable (XOR encryption, custom asset format, custom XML parser) - I don’t think it justifies Claude to simply drop them. Just do what I pay you for!

What I learned

Clean separation of game logic, animation, and rendering meant Claude could port each layer independently. The game engine is basically the same code in Go syntax. The rendering is a thin translation layer.

The hard parts weren’t the obvious ones (OpenGL calls, WinAPI, Boost). They were the feel details: animation rate-limiting, camera defaults, color interpolation, fade-out timing. These are the things that are invisible when reading code but immediately wrong when playing the game.

The iterative approach worked better than trying to get everything right upfront. Each round of “this doesn’t feel right” led to investigating a specific aspect of the original’s behavior. The C++ comments were occasionally helpful, but watching the game run (or fail to run correctly) was the real specification.

Source code

• the original source code in C++ - maybe there’s someone willing to run the original codebase on modern machine?
• ported source code

Observability

The ability to understand what’s happening inside your system by examining its external outputs. Unlike traditional monitoring, where you define upfront what to watch, observability lets you ask arbitrary questions about your system’s behavior after the fact.

Trace

The complete journey of a single request as it travels through your distributed system. It starts when a user clicks a button and ends when the response comes back, capturing every service, database call, and queue hop along the way. Without traces, debugging a slow request in a microservices architecture is like finding a needle in a haystack. I covered this topic in detail in my podcast about distributed tracing.

Span

A single unit of work within a trace. Like one leg of a relay race. Each span records a specific operation: an HTTP call, a database query, or a message being published. Spans are nested - a parent span for handling an HTTP request may contain child spans for authentication, business logic, and persistence. What you see on the header picture is a single trace with hundreds of spans. Each horizontal bar is a span.

Context propagation

The mechanism that links related operations across service boundaries. When service A calls service B, context propagation injects trace identifiers into the request (usually as HTTP headers like W3C traceparent). Service B extracts them and continues the same trace. Without this, you’d have isolated spans with no way to stitch them into a complete picture. It’s the glue that makes distributed tracing distributed.

Metric

A numeric measurement collected over time: request count, error rate, CPU usage, queue depth. Metrics are cheap to store and query, making them perfect for dashboards and alerting. They tell you that something is wrong, but rarely why. That’s where traces and logs come in.

Log

A discrete event recorded by your application at a specific point in time. The oldest and most familiar form of observability data. Every developer has written a log.info() at some point. The challenge isn’t producing logs - it’s making them searchable and correlated with traces and metrics. Logging at scale is serious business.

Structured logging

Logging where each entry is a set of key-value pairs rather than a free-form string. Instead of "User 42 logged in from 192.168.1.1", you emit {event: "login", userId: 42, ip: "192.168.1.1"}. This makes logs machine-parseable, filterable, and actually useful at scale. grep doesn’t cut it when you process terabytes of logs daily.

Backend

The system that receives, stores, and lets you query your telemetry data. Jaeger, Prometheus, Datadog, Grafana Tempo - these are all backends. Your application produces traces, metrics, and logs. The backend is where they end up and where you actually go to make sense of them.

OpenTelemetry (OTel)

A vendor-neutral, open-source observability framework for generating, collecting, and exporting telemetry data. It provides APIs, SDKs, and tools for all three pillars: traces, metrics, and logs. Instrument your code once, send data to any backend you choose. If you learn only one thing from this glossary, let it be OTel.

OTLP (OpenTelemetry Protocol)

The native wire protocol for transmitting telemetry data in OpenTelemetry. OTLP defines how traces, metrics, and logs are encoded and transported - typically over gRPC or HTTP. Most backends accept OTLP directly nowadays, which means you can often skip format-specific exporters entirely. One protocol to rule them all.

OpenTelemetry instrumentation

The process of adding observability to your application using OTel’s APIs and SDKs. Auto-instrumentation can inject tracing into popular frameworks (Spring Boot, Express, Django) with zero code changes. Feels like magic, and mostly works. Manual instrumentation gives you finer control - adding custom spans, attributes, or metrics where the automatic approach falls short.

OpenTelemetry Collector

A vendor-agnostic proxy that receives, processes, and exports telemetry data. Instead of configuring each application to send data directly to your backend, you route everything through the Collector. It decouples your instrumentation from your backend choice and lets you transform, filter, or enrich data in transit. Switching from Jaeger to Tempo? Change the Collector config. Your applications don’t need to know.

Receiver

The Collector component that ingests telemetry data from your applications. Receivers support various protocols and formats: OTLP, Jaeger, Zipkin, Prometheus, and many more. You can run multiple receivers simultaneously, which is invaluable when migrating from one instrumentation library to another. Migration without downtime.

Processor

The middle layer of the Collector pipeline that transforms data between receiving and exporting. Processors can batch data for efficiency, filter out noisy spans, sample only a percentage of traces, or enrich telemetry with additional attributes. Order matters. Processors execute sequentially, and a poorly placed filter can drop data before it gets enriched.

Exporter

The Collector component that sends processed telemetry to your backend of choice. Exporters exist for virtually every observability platform: Jaeger, Prometheus, Datadog, Splunk, you name it. You can even configure multiple exporters simultaneously - send traces to Jaeger for debugging and to long-term storage for compliance. Why choose?

Connector

A relatively new Collector component that acts as both an exporter and a receiver, bridging two pipelines. A connector can analyze incoming spans and generate metrics from them - counting errors or measuring latency without separate instrumentation. Deriving one signal from another.

APM (Application Performance Monitoring)

A broad category of tools that monitor and manage the performance and availability of your applications. APM solutions typically combine traces, metrics, and logs into a unified experience with dashboards, alerting, and root cause analysis. Every vendor defines APM slightly differently, but the goal is always the same: know when your app is slow or broken, and understand why.

Prometheus

The de facto standard for metrics collection in the cloud-native world. Prometheus uses a pull model - it scrapes metrics from your applications at regular intervals, rather than waiting for them to be pushed. Its query language, PromQL, is powerful but has a learning curve that makes regex look approachable.

Grafana

A visualization and dashboarding platform that can display data from virtually any source. Grafana doesn’t store data itself - it connects to Prometheus, Loki, Tempo, and dozens of other backends. If Prometheus is the engine, Grafana is the dashboard on your car. Pretty, informative, and the part everyone actually looks at.

Dashboard based on Node Exporter Full pre-made template

Loki

A log aggregation system, inspired by Prometheus. Unlike Elasticsearch, Loki indexes only metadata (labels) rather than the full text of log lines. Much cheaper to operate. The trade-off? Full-text search is slower. But in practice, filtering by labels and then scanning is fast enough for most use cases.

Tempo

A distributed tracing backend, designed for massive scale at minimal cost. Tempo stores traces in object storage (like S3) without requiring any indexing infrastructure. The catch: you need a trace ID to look up a trace. But integrations with Loki and Grafana make discovery seamless, so it’s less of a problem than it sounds.

Jaeger

An open-source distributed tracing platform, originally built by Uber. Jaeger helps you visualize request flows and latency bottlenecks in complex distributed systems.

Zipkin

One of the earliest open-source distributed tracing systems, inspired by Google’s Dapper paper. Zipkin pioneered many concepts that are now standard, including the B3 propagation format - a set of HTTP headers (X-B3-TraceId, X-B3-SpanId, etc.) that carry trace context between services. Still actively maintained and has a loyal following, though newer tools have captured more mindshare.

Mimir

A highly scalable, long-term storage backend for Prometheus metrics. Prometheus is great for short-term storage, but Mimir can handle months or years of metrics data across multiple tenants.

Thanos

Another approach to scaling Prometheus, focused on long-term storage and global querying across multiple Prometheus instances. Thanos sits alongside your existing Prometheus servers and uploads their data to object storage, providing a unified query layer. If Mimir replaces Prometheus, Thanos extends it.

Alloy

Open-source telemetry collector, previously known as Grafana Agent. Collects metrics, logs, traces, and profiles, then ships them to your backends.

Pyroscope

A continuous profiling platform. While traces tell you where time is spent across services, Pyroscope tells you where time is spent within a single process - down to the function and line of code. Some call profiling the “fourth pillar” of observability. I’d say the jury is still out, but Pyroscope makes it accessible in production without significant overhead.

Source: Grafana Pyroscope documentation

Commercial APM platforms (Datadog, Splunk, New Relic)

Full-stack observability solutions that bundle metrics, traces, logs, dashboards, and alerting into a single managed product. They all solve the same problem: you don’t want to run your own observability infrastructure. The trade-off is cost and vendor lock-in. OpenTelemetry helps with the lock-in part - instrument once, switch backends later.

ELK Stack (Elasticsearch, Logstash, Kibana)

The classic open-source trio for log management. Elasticsearch stores and indexes, Logstash ingests and transforms, Kibana visualizes. For years, ELK was the answer to “where do I search my logs?” Powerful, but running Elasticsearch at scale is practically a full-time job. The Grafana stack (Loki + Grafana) is a lighter alternative - Loki skips full-text indexing, which trades search flexibility for much lower operational cost. ELK predates OpenTelemetry and focuses primarily on logs, while OTel covers all three signals and is backend-agnostic. That said, you can absolutely feed OTel data into Elasticsearch.

Kibana

The visualization layer of the Elastic Stack (formerly ELK: Elasticsearch, Logstash, Kibana). Provides dashboards and exploration tools for data stored in Elasticsearch, particularly logs. While Grafana has become the default for metrics, Kibana remains the go-to for teams heavily invested in Elasticsearch.

Further reading

• Demystifying OpenTelemetry