The image library is now pure Rust

Yesterday, oyvindln made a PR to image-png to replace flate2 with deflate, a DEFLATE and zlib encoder written in safe rust. The PR was merged, and currently I am ecoing the new version in the Piston ecosystem.

This means that the image library is now pure Rust! C is gone!

Just think of it: An image library supporting several popular image formats, written from scratch in Rust, over a period of 3 years!

To celebrate this moment, here is an overview of the people who made this possible (listing major components):

Notice! When library was moved, some people were left out, but hopefully they are in the list of authors. There are also lot of people who contributed indirectly by testing and feedback. Thanks to you all!

A special credit goes to these 2 fine people:

  • ccgn, who started the project in 2014
  • nwin, who has been the top maintainer since then

During the early start of the Piston project, in a storm of rustc breaking changes in 2014, these two stood together as pillars, keeping the project afloat on top of a build script hacked together in Make and Bash, before Cargo came and saved us all from drowning.

OK, perhaps not that dramatic, but those breaking changes were intense.

Btw, we welcome new contributors!

Happy New Year from the Piston Project!

Welcome 2017, may the earth make another beautiful ellipse around the sun! I wish you all Happy New Year!

I just published Dyon 0.14.0, now with a new feature “link loop”.

Making code generation and templating easier

You can read more details about the design of the link loop here.

Yesterday, I was trying to figure out how Dan Amelang’s Nile programming language works. It is part of the effort of ViewPoint Research Institute (VPRI) to reinvent and revolutionize computing. VPRI does a lot of exciting stuff, and currently I am looking into Nile to get some ideas about 2D graphics.

The Piston project already uses a lot of meta parsing, inspired by OMeta2 used to create Nile. Piston-Meta makes it easy to build custom text formats, and the meta documents can also be shared and reused between projects.

For example, Eco is a tool for analyzing breaking changes in Rust ecosystems, and uses the same meta data converter code on two different types of formats: Cargo.toml and JSON dependency graph. Meta parsing does not have to be a fancy technique, it can be useful in bread-and-butter programming too!

Dyon also uses the same meta language to describe its language syntax. The benefit of using this technique is that you get closer to the intrinsic complexity of the domain you are working in. Piston-Meta is a very dense language, just useful enough be used for parsing, and nothing else. Nile was designed to describe how 2D graphics works, but it has some very promising ideas: What about 3D, sound, etc.?

Here is an example of a Nile function/stream (source):

CalculateBounds : Bezier >> (Point, Point)
    min =  999999 : Point
    max = -999999 : Point
    ∀ (A, B, C)
        if ¬(A.y = B.y ∧ B.y = C.y)
            min' = min ◁ A ◁ B ◁ C
            max' = max ▷ A ▷ B ▷ C
    >> (min, max)

Functions in Nile are quite different than in a typical programming language. Every function can have input and output streams in addition to arguments. Because these are processed in parallel, they do not deal with state the same way as typical functions do. One Nile function corresponds to 3 C-functions, divided into “prologue”, “body” and “epilogue”.

I am trying to figure out how to translate this technique to Rust, but also want to see how it is like to write a small compiler in Dyon. So far I had to change the syntax a little bit, since Piston-Meta has no support for whitespace sensitive syntax. If somebody wants to port OMeta2 to Rust, please do it!

Dyon uses a link type to generate code, which is a list that can only contain bool, f64 and str. It is used for code generation and template programming.

For example, here is some Dyon code that prints out Rust code:

fn main() {
    println(gen_struct("Foo", [{name: "bar", type: "f64"}]))
}

gen_struct(name, fields: [{}]) = link {
    "pub struct "name" {\n"link i {
    "    pub "fields[i].name": "fields[i].type",\n"
    }"}\n\n"
}

In the example above we have a link { ... } block and a link i { ... } loop (new in 0.14). The block creates a link, and the link loop joins all link items created inside the body.

In case you wonder what link i { ... } means, it is sugar for link i len(fields) { ... }. Dyon looks inside the body of the loop and figures out which list that uses i.

The performance is pretty good. On my laptop this program runs in 25 seconds:

fn main() {
    x := link i 100_000_000 {i", "}
}

It creates 0, 1, 2, 3, ... up to 100 millions.

Other things happening

There is a lot of discussion going on in the Conrod project. You are welcome to join!

People are now starting to work more the image library. If you want to help, see if you can solve some of the issues. oivindln got now a working deflate encoder in Rust, perhaps we can get rid of C entirely?

Dyon 0.13 is released!

Dyon is a rusty dynamically scripting language, borrowing ideas from Javascript, Go, mathematical notation and of course Rust!

In this post I will go in depth about two features of Dyon, and discuss them from the perspective that I use and need them.

Dynamically loaded modules

One of the things that makes Dyon special is the way of organizing code using dynamically loaded modules. When scaling up big projects in Rust, I noticed that each project required a configuration file to tell which dependencies it has. This is useful, but introduces extra work when moving code around. Another problem is when I work on some game, and want to refresh the game logic while running, there are many ways to do this and it depends on the specific project.

Sometimes I want to experiment with different algorithms and compare them side by side. While a version control system is great, it does not allow you the playful coding style I like. The projects I work on tend to focus on exploring some idea, and this leads to a dozen files spread across folders which does not fall into a neat namespace hierarchy. I do not want to think too much about how to name stuff, it gets in the way for productivity.

A Dyon script file does not know which other files it depends on, so you need to tell it for each project.

Setting up dependencies is part of a loader script, that loads modules and connect them together.

For example:

fn main() {
    find := unwrap(load("src/find.dyon"))
    unop_f64 := unwrap(load("../dyon_lib/unop_f64/src/lib.dyon"))
    unop_f64_lib := unwrap(load(
        source: "src/unop_f64_lib.dyon",
        imports: [unop_f64]
    ))
    set_alg := unwrap(load("../dyon_lib/set_alg/src/lib.dyon"))
    main := unwrap(load(
        source: "src/main.dyon",
        imports: [unop_f64_lib, set_alg, find]
    ))
    call(main, "main", [])
}

Horrible, right? Yet, there are some upsides:

  1. Easy to change the loader script to reload code on the fly
  2. Easy to generate code at loading and running time
  3. Easy to control preloading of modules that does not need reloading
  4. Easy to rewrite and refactor code from Dyon to Rust
  5. One configuration file per project

In most languages, you need some complicated system for these things, but in Dyon you learn how do it as a beginner.

The 0.13 version adds some important changes:

  1. Modules are now represented as Arc<Module> behind a Arc<Mutex<Any>> pointer
  2. External functions (Rust code) are resolved directly instead of depending on the module

The reason for 1) is when closures escape module boundary. Variables in Dyon are 24 bytes on 64 bit architectures, which was makes it possible to fit in more stuff like 4D vectors and secrets. I squeezed in a pointer to the closure environment stored beside the closure pointer. This keeps the track of the information required to find the relative location of loaded functions. The Arc<Module> pointer makes it possible to create the closure environment faster, instead of cloning the whole module for each closure.

The reason for 2) is when I use another pattern:

fn main() {
    window := load_window()
    image := load_image()
    video := load_video()
    input := load_input()
    draw := unwrap(load(
        source: "src/draw.dyon",
        imports: [image]
    ))
    sample := unwrap(load(
        source: "src/sample.dyon",
        imports: [draw]
    ))
    animation := unwrap(load(
        source: "src/animation.dyon",
        imports: [sample]
    ))
    main := unwrap(load(
        source: "src/programs/checkerboard.dyon",
        imports: [window, input, animation, video]
    ))
    call(main, "main", [])
}

In the example above load_window is an external function (written in Rust) that returns a module. The returned module containst other external functions, which then get imported into the other modules.

To make this logic work, I decided to use a direct function pointer and wrap it in FnExternalRef. The downside is that you can not track down the location of the external function when something bad happens. I am crossing my fingers and hope this will not be a big problem (famous last words).

Try expressions

A new feature in 0.13 is the ability to use try to turn failures into err and success into ok.

Link to design notes

Some background of why I care about this so much: I am working on path semantics, which is a big dream I have.

A path is a function that relates one piece of a function to another function in a space called “path semantics”, because it is a space of meaning (semantics). Normally there is a bunch of paths for a function, that might differ for arguments and return value, but they all need to work together to get to the other function.

When the same path is used everywhere, it is called a “symmetric path”.

Symmetric paths are extremely mathematically beautiful and rare.

Example:

and[not] <=> or

which is equal to, but not meaning the same as the equation:

not(and(X, Y)) <=> or(not(X), not(Y))

I just occured to me that this is one of De Morgan’s laws.

Here is another example:

concat[len] <=> add

which is equal to, but not meaning the same as the equation:

len(concat(X, Y)) <=> add(len(X), len(Y))

You can use these laws to do transformations like these:

add(X, Y)
concat[len](X, Y) // using a symmetric path to reach the other function
// there exists `Z` and `W` such that `X = len(Z)` and `Y = len(W)`
concat[len](len(Z), len(W))
len(concat(Z, W))

I want to create an algorithm to explore the space of path semantics efficiently. There is no currently proof technique I know to derive them, except to “make a hypothesis and falsify”, kind of like how scientists do. The only way to find which functions that works with others is by trying.

Here is a function that tests for a symmetric path candidate:

/// Infers whether a path can be used for a function
/// in a symmetric way for both arguments and result.
sym(f: \([]) -> any, p: \([]) -> any, args: []) =
    all i {is_ok(try \p([args[i]]))} &&
    is_ok(try \p([\f(args)]))

Let me take this apart an explain the pieces:

  • sym is the function name
  • f: \([]) -> any is an argument f of type closure taking [] (array) and returning any
  • p: \([]) -> any is an argument p of type closure taking [] (array) and returning any
  • args: [] is an argument args of type [] (array)
  • \p([args[i]]) calls the closure p
  • \p([\f(args)]) calls the closure f and then p
  • all i { ... } is a mathematical loop that automatically infers the range [0, len(args)) from the body

The \ notation is used in Dyon because type checking happens in parallel with AST construction, plus some other design choices that closures always return a value. I needed a design that was easy to refactor and different for normal functions and closures.

The problem was that this function gets called many times with different closures, and sometimes it fails and crashes because the types are wrong.

try converts all failures into err, such that the function sym can detect a path candidate.

Now, you might think, why not use try and catch blocks?

Dyon already has a way of dealing with errors: res, like Result in Rust but with a trace and any as error type. There is no need to add a second way of dealing with errors, when the first method works just fine.

For Rust programmers, the try keyword can be a bit confusing, since they might be thinking of the try! macro. In Dyon, you use the ? operator instead of try!, just like in Rust. I decided to use it because it feels closer to what we mean with “trying” in natural language.

For example, when a function foo returns res, you can write:

fn foo() -> res { ... }

fn bar() -> res {
    a := try foo()?
    ...
}

This is executed as try (foo()?), returning err early, so you can handle the case where it failed. Interestingly, it gives the same result as (try foo())?.

Older Newer