Dyon 0.4 is now released!

Dyon is a rusty dynamically typed scripting language, using a lifetime checker instead of garbage collection. It has an object model similar to Javascript, but with option and result types instead of null. The language uses dynamic modules for organizing code, which makes it easy to use for interactive coding.

Previous blog posts:

• Dynamo (Dynamo was the old name, the language is now renamed to Dyon)
• Scripting without garbage collector
• Error handling in Dyon

You can now use Dyon for interactive coding!

I am experimenting with a setup where a Rust application starts a loader script, which then loads and refreshes another module while running. It slows down to 1/10 of the speed when the window is unfocused.

You can try it out yourself! I added instructions to the readme.

About Piston in general

The Piston project is a large collaboration between many programmers to build a modular game engine in Rust. Currently, there are 100 repositories and 176 people who have contributed to various projects.

Because Rust is a new programming language with an unusual model for memory safety, we need to find good patterns for designing libraries in the domain of game development and interactive applications. This means a lot of experimentation with design, but also testing and integration.

Piston has a minimalistic modular core which is used with various window and graphics backends. This design let people can use it for smaller project while development continues toward bigger goals. Many libraries can be used on their own, and window and graphics is completely decoupled.

The Piston project collaborates with other open source projects in the Rust community, such as the Gfx, RustAudio and Servo.

In addition to maintainance of important libraries, we do a lot of research and thinking outside the box. This is possible because Rust scales well with a big project, so we have time to do other stuff instead of fixing bugs. Conrod, Meta-Parsing and Dyon are part of the research projects.

If you have a project that you want to work on as a community project, you are welcome to start it under PistonDevelopers. Everybody who contributes gets access to all the repos, so people can help out across projects and choose what they want to work on. We use a tool, Eco to automate the thinking of breaking changes across the ecosystem.

At the moment Piston is going through a complex upgrade for 2D graphics, but got through the hardest parts and is making progress. Because of a breaking change in Rust, we recommend using beta or nightly until Rust 1.8 is stable. After this upgrade, we will start optimizing 2D rendering.

Dyon is an experimental scripting language without garbage collector that I work on. It borrows ideas from Rust, Javascript and Go. Instead of a garbage collector, it uses a lifetime checker.

Example:

fn foo(a, b: 'a) { // 'b' outlives 'a'
    ...
}

The object model is similar to Javascript, but has no null value.

Version 0.3.0 added option and result for error handling.

In this article, I will explain the difference between null and option/result, and how Dyon steals an idea from an accepted RFC. Rust does not support this feature yet, but I think it is brilliant and looking forward to it.

For an introduction to Dyon, see the two previous blog posts:

• Dynamo (this is the old name, now changed to Dyon)
• Scripting without garbage collector

What is wrong with null?

Everything:

• The program suddenly crashes with a mysterious error message
• It can happen everywhere in the code base where an object is referenced
• Other language features can make it unneccessary

The inventor called it his billion-dollar mistake.

Disclaimer: I do not mention the name because it feels wrong to associate an awesome person with a tiny mistake. However, if I am going to mention anyone, for completely unrelated reasons, then Jeff Rulifson is a hero of mine.

What is right about option/result?

For the same reasons:

• When the program crashes, it tells you what happened
• It can only happen where these types are used, no need to suspect the whole code base
• Does exactly what it is supposed to do

Rust and Haskell are languages where you can code without null, which is great.

So, how does this work in Dyon?

Result

Let us look at a simple program:

fn foo(a) -> {
    if a {
        return err("error!")
    } else {
        return ok("success!")
    }
}

fn main() {
    x := unwrap(foo(false))
    println(x)
}

This prints out “success!”.

Change foo(false) to foo(true) and you get an error:

 --- ERROR --- 
error!
10,17:     x := unwrap(foo(true))
10,17:                 ^

What if we want to add a function that changes “success!” into “victory!”?

fn bar(a) -> {
    x := foo(a)?
    return ok(if x == "success!" { "victory!" } else { x })
}

fn main() {
    x := unwrap(bar(true))
    println(x)
}

When foo(a) returns an error, The ? operator propagates the error, returning from the function.

When foo(a) returns ok(_), it unwraps the value.

Today, you do the same in Rust by using the try! macro. One problem is that it leaves no trace, making it hard to figure out where the error comes from.

An idea I got was to push a trace error message when using the ? operator:

 --- ERROR --- 
error!
In function `bar`
10,10:     x := foo(a)?
10,10:          ^

19,17:     x := unwrap(bar(true))
19,17:                 ^

In Dyon all errors are of the same dynamic type, so adding this feature was not difficult. I created a struct Error that wraps the error message, with an extra field for the trace:

pub struct Error {
    message: Variable,
    // Extra information to help debug error.
    // Stores error messages for all `?` operators.
    trace: Vec<String>,
}

The trace is hidden from the user, only visible when using unwrap. When using unwrap_err, you only get the error message without the trace.

Option

An object is HashMap under the hood, so you can remove and add fields by need.

However, there are situations this is bad:

• When adding wrong keys leads to hard-to-find bugs
• When expressing that a field is present, but has no value

Rust uses an Option type, with None and Some(x). Dyon uses none() and some(x).

The ? operator converts option into result. Change the example into the following:

fn foo(a) -> {
    if a {
        return none()
    } else {
        return some("success!")
    }
}

fn bar(a) -> {
    x := foo(a)?
    return ok(if x == "success!" { "victory!" } else { x })
}

fn main() {
    x := unwrap(bar(true))
    println(x)
}

This given an error:

 --- ERROR --- 
Expected `some(_)`, found `none()`
In function `bar`
10,10:     x := foo(a)?
10,10:          ^

19,17:     x := unwrap(bar(true))
19,17:                 ^

Change bar(true) into bar(false) and it prints “victory!”.

External functions

Since Dyon uses dynamic modules, it is not possible to document it statically. The only way to tell which functions are available is by inspecting it from the inside. functions() gives you a sorted list of all functions with their lifetimes.

Example:

fn main() {
    fs := functions()
    println(fs[0])
}

This prints:

{name: "acos", type: "intrinsic", arguments: [{name: "arg0", lifetime: none()}], returns: true

There are 3 categories of functions:

• intrinsic (part of standard Dyon environment)
• external (custom Rust functions operating on the Dyon environment)
• loaded (imported and local functions)

Here is an example for writing a custom Rust function:

extern crate dyon;

use std::sync::Arc;
use dyon::*;

fn main() {
    let mut dyon_runtime = Runtime::new();
    let dyon_module = load_module().unwrap();
    if error(dyon_runtime.run(&dyon_module)) {
        return
    }
}

fn load_module() -> Option<Module> {
    let mut module = Module::new();
    module.add(Arc::new("say_hello".into()), dyon_say_hello, PreludeFunction {
        arg_constraints: vec![],
        returns: false
    });
    if error(load("source/test.rs", &mut module)) {
        None
    } else {
        Some(module)
    }
}

fn dyon_say_hello(_: &mut Runtime) -> Result<(), String> {
    println!("hi!");
    Ok(())
}

test.rs:

fn main() {
    say_hello()
}

Some thoughts so far

I have a lot of fun working on Dyon. It has a very simple syntax, so my brain does not have to process a lot to read the code.

First class functions are problematic because they require some type checking to be safe. One idea is to limit them to arguments, without the ability to move or live inside objects.

Since you can not reference memory outside the stack, it is very limited of how you can structure the code. I wonder what happens when programs get larger…

I hope you enjoyed this article, and perhaps you might even try Dyon out a bit! Do not recommend using it yet, because there will be plenty of breaking changes.

Eco is a tool for reasoning about breaking changes in Rust ecosystems.

In this article I will explain how to use it, and why it saves tons of hours for a big software project like Piston.

For localized analysis, you might consider Cargo-Outdated.

Releasing libraries

The Piston project uses a simple philosophy for releasing versions:

• Do not break existing code
• Follow semver versioning
• Keep the ecosystem integrated as much as possible

Which sounds obvious and easy to do, right?

It is a lot easier than not following these rules, but it is harder than it sounds.

Small problems become large by multiplication

In Piston we have 100 repositories and over 160 people with access to all the libraries. Among the 160+ people there are very few who push code every day, but those who do pushes a lot. Some people push occationally, and others fixes minor things and help people that got problems.

When you work with over 100 people over a large time period, there will be lot of small problems. There is no single person that can follow the development in detail for the whole project. This is itself a problem, but it is one you can not fix because it is just the way it is.

I read somewhere that human brains can not multiply. People systematically underestimate how small problems become big problems when they scale. It seems right to me, because I get surprised every time I upgrade the ecosystem! It usually takes more work than anticipated, and even when no step is particularly hard, it just takes a lot of time. My feelings summarized:

Wow! This is a lot more libraries than it feels like when everything works!

In a small software project, it is easy to update a library when a breaking change happens in a dependency. For a large enough software project, it is humanly impossible!

This happens because dependencies have a lot of subtle issues that gives them a hairy topology on large scale.

We learned from experience that no human could do it right at this scale. It was not before we started to use Eco that we got it right, and it even told us stuff that would have not been discovered otherwise!

Eco does the thinking - you do the upgrades

Here is a JSON document listing projects that are part of the Piston ecosystem.

Some of these libraries are external. They are maintained under other organizations or by single people.

Eco takes a such list and extracts dependency information directly from the web. Then it runs an analysis algorithm and generates recommended update actions from the dependency information.

When I do upgrades, I type:

$ cargo run --example piston > todo.txt

It takes Eco 2 seconds to gather data and do the analysis for the whole Piston ecosystem. An improvement at least 1000x times faster than thinking through this manually.

What I get is a list like this:

     Running `target/debug/examples/piston`
{
  "sdl2_mixer": {
    "order": 4,
    "bump": {
      "old": "0.15.0",
      "new": "0.16.0"
    },
    "dependencies": {
      "sdl2": {
        "bump": {
          "old": "0.15.0",
          "new": "0.16.0"
        }
      }
    },
    "dev-dependencies": {
    }
  },
  "piston3d-gfx_voxel": {
    "order": 5,
    "bump": {
      "old": "0.7.0",
      "new": "0.8.0"
    },
    "dependencies": {
      "piston-gfx_texture": {
        "bump": {
          "old": "0.8.0",
          "new": "0.10.0"
        }
      },
  ...

Then I work my way through the list, deleting the “bump” information as I go. When crates.io is updated for a project, I delete it from the list.

Even when Eco does the thinking, it might take many days to upgrade Piston when there are lots of breaking changes. Currently, we are doing upgrades that have been going on for weeks! The more complex these upgrades are, the longer it takes. However, Eco makes it possible to do such upgrades without loosing track of progress.

The nice thing is that when somebody makes some changes, I can re-run the analysis to get an updated view. People working on the Piston project or the external ones does not need to know Eco at all!

Ignore version

Disclaimer: This example is taken a real situation with all the subtleties that follow.

nwin is chief maintainer of the popular Image library. It is a pure Rust alternative for encoding and decoding various image formats. Some codecs live in other repos, such as Gif.

Currently, Gif has version 0.8.0, while Image uses 0.7.

When Eco performs an analysis, it recommends Image to be updated to 0.8.0 with the new version of Gif. I want to wait until nwin thinks it is time to update Image, or I could do the update myself after a while. However, that might take weeks, depending on how much nwin, I and others have stuff to do. The problem is, that Eco thinks Image should be 0.8.0 everywhere and now, which causes a lot of breaking changes. We would like Eco to take it easy for this particular upgrade for a while.

To fix this, I insert an “ignore-version” field:

"gif": {
    "url": "https://raw.githubusercontent.com/PistonDevelopers/image-gif/master/Cargo.toml",
    "ignore-version": "0.7.0"
},

This makes Eco ignore all projects that uses Gif version 0.7.0.

• The maintainer can focus on the stuff that needs to be done without having to publish too often
• I can control how much workload pressure there is on libraries higher up in the dependency graph
• People using these libraries might appreciate how we shuffle tasks around to avoid even more frequent updates

Override version

Sometimes people publish a new version on crates.io, but forget to push their changes. In Piston we do not publish before the change is merged, but occationally we make mistakes, and this happens in external projects as well.

The “override-version” field can be used to replace the version in Cargo.toml with a manual one, such that upgrades can be made without waiting for the master branch to be updated.

Programming in the large

A fundamental challenge with software engineering is that most problems are easy to fix, but the integration between lots of code gets extremely hard when things scale.

The goals of Piston project are ambitious enough to scale easily up to at least to 200 libraries. Now it is in the beginning phase, but because Rust is such a good language for this type of programming, we expect to be able to move on when things stabilize. Even with just a few people working at it any time, we can maintain it easily, but also do a lot of research.

In the 2 years since Piston was started, we have been working on:

• Game engine core design
• Image formats
• Image processing
• UI
• AI behavior trees with parallel semantics
• Sprite scene animation
• Testing of Gfx and Glium vs raw OpenGL
• Made some smaller games (I made 2, one about surviving sea snakes, another about surviving sea birds!)
• A Minecraft renderer
• A Minecraft server
• Skeleton animation for 3D characters
• Vector math for internal use
• Quaternion and dual quaternion maths
• Testing and fixing bugs in various window backends
• 2D graphics
• Rust plugin for Visual Studio
• Meta parsing (a domain specific language for describing how to parse a text document, including itself)
• 3D formats such as Collada, Wavefront-OBJ and OpenGEX
• Collecting information about physically based rendering
• Audio (now developed outside Piston, under the RustAudio organization and others)
• Analysis of ecosystems
• Path semantics (a new formal way of inference about algorithms, now my personal research project)
• Many bugs reported to the Rust compiler team
• Collaborating with the Servo project
• Rust bindings to FreeType, will soon experiment with alternatives
• Mixed economy algorithm (one I hope to test as a virtual currency concept for MMOs)
• Camera controllers
• Scripting language without garbage collector

The core of Piston is the smallest possible for a game engine and have so far worked very well.

There has been a lot of progress on ironing out the details of 2D graphics, and still a lot work left to do. It is far from an easy task, given the multiple goals of flexibilibility, modularity, performance and consistency across lower level APIs. Upgrades are really tough because it affects so many projects. Sometimes I just have to work on other stuff to get a break from thinking about the same problems all the time!

We do not know how big Piston will become, and try to use external projects when reasonable. If a project grows too large for Piston, we might even move it to an external organization.

Yet, it is easy to forget that some things are only possible if you have the right tools. There are many ways to solve small problems, but there are fewer ways to make it scale.

My hat off to the people who have made Rust a such a great language!