Crafting Your Own Compiler: From Python Logic to High-Speed WebAssembly

In this tutorial, participants will learn how to build in Python a compiler that translates a scaled-down programming language called chiqui_forth into executable WebAssembly (Wasm) code.

Wasm is a relatively new technology that can be used to create client-side and standalone applications that are efficient, secure, and portable. It’s an intermediate language for a stack-based virtual machine that uses a just-in-time (JIT) compiler to produce native machine code. Wasm provides a portable compilation target for languages such as C/C++, Rust, C#, and many others.

Although writing a compiler for a full blown programming language is a daunting task, it’s possible to write swiftly a Wasm targeting compiler for a fairly small language, chiqui_forth in this case. This tiny language is a subset of the Forth programming language, which first appeared in 1970. It’s a stack-oriented language that is programmed using Reverse Polish Notation (RPN), which happens to be quite easy to convert into Wasm code.

By the end of this session you’ll have a new mental model for software:

No prior knowledge of compiler design, Wasm, or web development is required.

WebAssembly is a binary format designed for a stack machine, serving as a virtual Instruction Set Architecture (ISA) that defines a standardized, abstract machine language for a sandboxed CPU. Engineered from its inception as a compiler target rather than a handwritten language, it allows developers to compile software from various high-level source languages into a highly portable format.

In both the binary and textual formats, the fundamental unit of code in WebAssembly is a module. In the text format, a module is represented as one big S-expression. S-expressions are a long-established, simple textual format for representing trees, allowing us to view a module as a tree of nodes that describe the module’s structure and its code.

Let’s see what an S-expression looks like. Each node in the tree goes inside a pair of parentheses: ( …​ ). The first label inside the parentheses tells you what type of node it is, and after that there is a space-separated list of either attributes or child nodes. For example:

In this example the node types are module, func and export. The root of the s-expression is the module node. This node has func as its only child. The func node has two children: the export node and the return attribute. Finally, the export node has only one child, which is the "myFunction" attribute. Indentation is used here to clarify the tree-structure of the s-expression.

WAT supports two types of comments:

Examples:

Comments are ignored by the Wasm assembler and are used strictly for code documentation.

WebAssembly programs natively support multiple categories of data types. The core numeric types are:

These four core types are used as prefixes in most WebAssembly instructions. They are also used when declaring variables, parameters, and function return types.

A stack is a data structure with two operations: push and pop. Items are pushed (inserted) onto the stack and subsequently popped (removed) from the stack in last in, first out (LIFO) order.

WebAssembly execution is defined in terms of a stack machine where the basic idea is that every instruction pushes and/or pops a certain number to/from a stack. For example, to add the numbers 7 and 5, the following three steps need to be carried out on the stack:

An illustration of the three steps described above can be seen in Figure 2 .

The actual WAT code to do this arithmetic operation would look like this:

In WebAssembly, functions serve as the primary execution blocks. A function executes its logic by pulling arguments from the stack, performing its operations, and leaving the results on the stack for subsequent instructions.

Executing a Wasm module within Python using the wasmtime runtime requires a structured lifecycle: establishing an isolated execution environment, compiling the module, and instantiating it. Serving as a continuation of the previous example, the following code demonstrates this complete workflow by loading the resulting WAT file, instantiating it without external host imports, extracting its exported function, and safely marshaling Python arguments and results across the runtime boundary.

Add to the example.wat file a new function called fah_to_cel that takes as argument a 64-bit floating point value that corresponds to a temperature \(F\) in degrees Fahrenheit and converts it to degrees Celsius.

Add the necessary code to the wat_launcher.py file to check that the computed results are correct using the following values:

Add to the example.wat file a new function called quadratic_root that takes three 64-bit floating point values corresponding to the coefficients \(a\), \(b\), and \(c\) of the quadratic equation \(ax^2+bx+c = 0\), and computes its first root.

Add the necessary code to the wat_launcher.py file to check that the computed results are correct using the following test cases (assume all inputs yield real roots):

Because computers cannot natively understand human-written code, you must use either a compiler or an interpreter to translate and execute any program written in a high-level language.

A compiler is a program that reads the entire source code all at once and converts it into a separate, standalone file — such as machine code — that is ready to run at maximum speed. In contrast, an interpreter works on the fly. It reads, translates, and executes your code one instruction at a time, in real time, right while the program is running.

To understand the difference, imagine you need to read a book written in Polish, but you only speak English:

Building a compiler is incredibly rewarding because it completely strips away the “magic” behind how programming languages work. By writing one yourself, you get to see exactly how your lines of text are broken down, organized, and turned into something a computer can run. This behind-the-scenes perspective provides a clearer understanding of language behavior and deepens your insight into software execution. Ultimately, it shifts your perspective from simply writing software to fundamentally understanding how things work underneath.

Although writing a compiler for a full-blown programming language like C++, Java, or Rust can be a daunting task that could take months or even years, it is entirely feasible to build a WebAssembly-targeting compiler for a very small language in the time allocated for this tutorial. This tiny language is called chiqui_forth, a subset of the Forth programming language invented by Chuck Moore in 1970.

The following blueprint outlines the execution flow of what will become the chiqui_forth compiler’s main() function. Acting as the entry point, this function will manage the lifecycle of the compilation process by coordinating the transition from the analytical front end to the generative back end. By following this sequential framework, it will handle everything from initial environmental checks and token sanitization to memory mapping, code synthesis, and the final generation of executable target artifacts.

The previous tasks are summarized in Figure 3, where you can see the pipeline for the custom chiqui_forth compiler, mapping out how raw source code passes through a Front End validation and analysis stage before a Back End synthesis stage generates the final WebAssembly target outputs.

Now that we have mapped out the compiler’s high-level architecture, it is time to start building the Front End. Your first task is to implement the command-line interface gatekeeper.

This component is responsible for verifying that the user provided the correct number of arguments when invoking the compiler from the terminal, and extracting the target file path.

Create a Python script called chiqui_forth.py and implement a main() function alongside a helper function named get_source_filepath() that complies with the following:

Once you have implemented the functions and wired them into your main script execution block, you can verify your work directly from the terminal. Test your script with the following three scenarios:

With our command-line gatekeeper in place, we can move to the next phase of the Front End: reading the source file and breaking its contents down into a stream of processable tokens. Since Forth is a space-delimited language, our tokenization process is wonderfully straightforward.

In your chiqui_forth.py script, implement a helper function named read_words(input_file_name) that complies with the following specification:

Once you have implemented the function and updated main(), create a sample chiqui_forth file called simple.4th with the following contents:

Now verify your work directly from the terminal:

Now that we have a clean list of words from our source code, we need to address comments. In Forth, comments are enclosed within parentheses and separated by whitespace. For example, ( this is a comment ) tells the compiler to ignore everything inside.

To clean our token stream before synthesis, your next task is to write a parser component that strips out these comments completely while ensuring the delimiters are perfectly balanced.

In your chiqui_forth.py script, implement a helper function named remove_comments(tokens) that complies with the following specification:

Once you have implemented the function and updated main(), create two additional sample files called bad_comment1.4th and bad_comment2.4th alongside your valid simple.4th file to verify all three execution paths from the terminal:

Now verify your work directly from the terminal:

With our token stream completely sanitized of comments, the compiler needs to distinguish between the different types of words (named operations or commands) present in the source code. In chiqui_forth, a word can either be a predefined operation or a user-defined variable name. To lay the groundwork for our symbol table and memory allocation, we must implement a way to validate whether a given word is a valid identifier.

To map these named operations to their actual target code, the compiler utilizes a global lookup dictionary named OPERATION. This dictionary acts as our predefined instruction table: the keys represent the Forth words the compiler looks for, while the values are lists containing the exact WebAssembly instructions that must be emitted to execute that operation.

Define this OPERATION mapping at the global level of your chiqui_forth.py script:

Next, implement a helper function named is_var_name(token) that complies with the following specification:

Once you have implemented the definition and the function, you can verify its behavior by importing your script into an interactive Python REPL session and running a few test cases:

Now that the compiler can identify valid variable identifiers, it needs to handle the other fundamental category of tokens: numeric literals. According to our syntax definition, any token that represents a valid number is automatically pushed onto the evaluation stack. Before we can generate code for these values, we must write a helper function to reliably detect them.

In your chiqui_forth.py script, implement a helper function named is_number(token) that complies with the following specification:

Once you have implemented the function, you can verify its behavior using an interactive Python REPL session with the following test cases:

Now that your compiler can identify both words and numbers, it needs to analyze the clean token stream to discover which variable names are actually used by the programmer. In Forth, variables are modified using the store operator (!), which is attached directly to the end of a variable’s identifier name (for example, x!). To set up our dynamic memory allocator in a later step, we must first scan the source code and compile a unique collection of all declared variable names.

In your chiqui_forth.py script, implement a helper function named find_vars_used(tokens) that complies with the following specification:

Once you have implemented the function, you can verify its behavior by opening an interactive Python REPL session and running a few test cases:

With our compiler now able to extract the complete set of unique variable names from the token stream, we can begin the code-generation phase for memory allocation. In WebAssembly, variables are treated as function-local storage. Before they can be used within our main program logic, they must be explicitly declared at the top of the function structure using the (local $name type) syntax.

In your chiqui_forth.py script, implement a helper function named declare_vars(vars) that complies with the following specification:

Once you have implemented the function, you can verify its behavior by opening an interactive Python REPL session and running a few test cases:

Now that our compiler can declare variables and recognize numbers, predefined operations, and identifiers, we have arrived at the heart of the translation pipeline: the code generator. This component iterates through the clean token stream and synthesizes the corresponding executable WebAssembly Text (WAT) instructions for the stack machine.

In your chiqui_forth.py script, implement a function named code_generation(tokens) that complies with the following specification:

Once you have implemented the function, you can verify its behavior by opening an interactive Python REPL session and running a few test cases:

We have arrived at the final stage of our compiler pipeline: emitting the target files. Up until now, our synthesized WebAssembly Text (WAT) instructions have only existed as strings inside a Python list. To make our code runnable, we must inject these instructions into a complete WebAssembly module template, save it as a .wat text file, and compile it into a raw binary .wasm file using the wat2wasm() function from the wasmtime module.

At the global level of your script, define the following string template that represents our bare-bones WebAssembly module structure:

This template defines a WebAssembly module that hooks into three host functions (emit, input, and print) — which we will define later in our runtime environment — to grant our compiled code native access to basic terminal input and output operations. The empty curly braces {} act as a structural format placeholder where your merged variable declarations and instruction body lines will be dynamically injected.

Next, implement a helper function named create_target_files(source_path, compiled_lines) that complies with the following specification:

Once you have implemented the function, you can verify its behavior by opening an interactive Python REPL session and running a test case:

After executing the function, check your current directory from the terminal. You should see two brand-new files generated: test_program.wat (which you can open in any text editor) and test_program.wasm (the binary module ready for execution).

If you have successfully completed all exercises C through K then you have built every individual component required for our translation pipeline. Congratulations! Now, it is time to assemble these isolated pieces into a cohesive, working compiler.

To achieve this, we will provide a new version of the main() function that will serve as the central orchestrator, defining a clear boundary between the Front End (which analyzes and cleans the source code) and the Back End (which synthesizes and emits the executable WebAssembly code).

In your chiqui_forth.py script, overwrite the main() function with the following code:

Now that our compiler can generate valid binary .wasm files, we need a runtime environment to execute them and provide the implementation for our host I/O functions. The following complete Python script, chiqui_forth_rt.py, leverages the wasmtime library to instantiate our compiled WebAssembly modules, link the required external host functions (emit, input, and print) directly to native Python operations, and trigger the exported _start function to run the program from the terminal.

With both your compiler orchestrator (chiqui_forth.py) and your host runtime environment (chiqui_forth_rt.py) fully completed, you are ready to test the entire compilation and execution loop. Below are two sample program listings designed to test character output, structural comment stripping, dynamic variable allocation, and interactive terminal input.

Save the following code blocks into individual files within your active development directory, then follow the terminal steps to compile and execute them.

Now that your compiler can cleanly translate and execute basic chiqui_forth programs, it is time to expand its computational capabilities. Right now, our code generator only recognizes a tiny handful of operations. To build a truly practical stack machine, we need to support a complete set of arithmetic and relational operations.

Modify the OPERATIONS mapping dictionary in your chiqui_forth.py script so that the compiler supports all the operation words detailed in the following table. Pay close attention to WebAssembly’s sign-extension suffixes (_s) required for signed integer math and comparisons.

Once you have updated your operational dictionary, verify your compiler’s extended vocabulary by compiling and executing the provided validation program:

Compile the source program:

Run the program:

If your dictionary mappings are structurally sound and mapping to the correct target WAT instructions, the evaluation suite will complete successfully and print the following confirmation message to your console:

Our stack machine needs a mechanism to repeat execution paths. To achieve this, we will implement a do/loop repetition construct. Its syntax and behavior mirror Python’s structured while loops, providing a clean way to perform conditional iteration:

When this construct executes, the \(\textit{condition}\) expression is evaluated first. If the resulting value on top of the stack is zero (representing false), execution immediately breaks out of the construct and jumps to the code following the loop keyword. If the condition is non-zero (true), the \(\textit{body}\) code runs, the \(\textit{condition}\) is evaluated once again, and the process repeats.

Note that a question mark (?) word is mandatory in this syntax to explicitly mark the boundary where the \(\textit{condition}\) block ends and the executable \(\textit{body}\) begins.

The following program example demonstrates how to use this construct to count from 1 to 10:

Modify the OPERATIONS mapping dictionary in your chiqui_forth.py script so that the compiler supports the do, ?, and loop words by translating them to their corresponding WebAssembly structured control flow expressions as detailed below:

To gain a deeper understanding of how these target instructions manipulate structural execution frames, consult the official WebAssembly MDN reference documentation for the block, loop, br, and i32.eqz primitives.

Once your dictionary mappings are in place, verify the pipeline by compiling and running the test case:

Expected output:

To thoroughly evaluate your compiler’s brand-new looping capabilities, try compiling and running the following two sophisticated test programs. Both scripts rely on nested loop structures to compute and render patterns dynamically based on interactive terminal input.

Save these listings to your local directory and test them using any input value from 5 to 20.