gfx_graphics is now ready for real world testing! The repository contains example code.

What is it?

gfx_graphics is a back-end for piston-graphics that uses Gfx for rendering.

This is how it looks like:

let mut g2d = G2D::new(&mut device);
for e in EventIterator::new(&mut window, &event_settings) {
    use piston::event::RenderEvent;
    e.render(|_| {
        g2d.draw(&mut renderer, &frame, |c, g| {
            c.rgb(1.0, 1.0, 1.0).draw(g);
            c.rect(0.0, 0.0, 100.0, 100.0).rgb(1.0, 0.0, 0.0).draw(g);
            c.rect(50.0, 50.0, 100.0, 100.0).rgba(0.0, 1.0, 0.0, 0.3).draw(g);
            c.trans(100.0, 100.0).image(&image).draw(g);
        });

        device.submit(renderer.as_buffer());
        renderer.reset();
   });
}

About Gfx

Gfx is a safe, low level 3D API on top of OpenGL that makes rendering on multi-threads possible by using Renderer objects. It is designed to be fast at modern 3D APIs like Metal, Mantle, DirectX 12, next-gen OpenGL etc.

When you write a shader, you can bind the data with normal Rust structs, and Gfx checks the parameter types for you.

The Piston project is working closely together with Gfx hackers to make 2D and 3D as fast as possible for cross platform development in Rust.

About piston-graphics

piston-graphics is a library for rendering sprites and simple 2D shapes. It uses immutable context types with compiler optimization to remove state overhead. This was the library that started the Piston project.

You don’t have to push and pop matrix transforms, and you can fork a context as you wish. The context is not bound to the graphics back-end, so whenever you call .draw(back_end) it will render. This means you can render to multiple back-ends at the same time, if you need to.

Writing your own back-end is easy, by implementing the BackEnd trait. piston-graphics is designed to work with the simplest shaders possible, and performs triangulation on the CPU. Texture support is optional, so implementers can test it gradually.

piston-graphics is used in Conrod, an immediate mode graphics API.

One goal of the Piston project is using Rust’s type system to write back-end agnostic libraries. Nothing in the Piston core depends on anything besides the Rust standard library. At the moment we lack a pure Rust font library, so if you want to help out with Conrod, please see if you can do something about it!

How fast is gfx_graphics?

Nobody knows exactly how fast it is yet. But it is fast!

Measuring performance in a graphics API is a hard problem, because if you optimize for a single case, then it might run slower in total. One way to find out is writing real games and applications, and swap the back-end while running.

The only benchmark available is a game called Sea Snake Escape. It renders 65536 triangles on the sea snakes each frame, updated through a dynamic buffer.

On my computer, it runs at 60 fps with Gfx and 30 with OpenGL. The reason Gfx is faster is most likely because it uses state caching for alpha blending. For Gfx, this is 3932160 triangles per second. Every single one is triangulated and transformed on the CPU. Which means, for things like a GUI framework or simple 2D games, this is pretty good.

Still, there is room for optimization both in Gfx and piston-graphics.

Recently I added a DeformGrid object to piston-graphics

Here is a demo of it in action:

Example code is in the “deform” app under piston-examples.

I have been using this algorithm for years in a 2D animation software. The algorithm is called “Rigid Moving Least Squares Deformation”.

The algorithm takes 42 lines of code:

for m in range(0, nx) {
    for n in range(0, ny) {
        let ip = m + n * nx;
        let vx = m as f64 * units_h + x;
        let vy = n as f64 * units_v + y;
        let mut sum_wi = 0.0;
        let mut p_star_x = 0.0; let mut p_star_y = 0.0;
        let mut q_star_x = 0.0; let mut q_star_y = 0.0;
        for i in range(0, num) {
            let pix = ps[i][0]; let piy = ps[i][1];
            let qix = qs[i][0]; let qiy = qs[i][1];
            let vl = (pix - vx) * (pix - vx) + (piy - vy) * (piy - vy);
       
            let w = if vl < eps && vl > -eps { 1.0 / eps } else { 1.0 / vl };
            sum_wi += w;
            p_star_x += w * pix; p_star_y += w * piy;
            q_star_x += w * qix; q_star_y += w * qiy;
            wis[i] = w;
        }

        p_star_x /= sum_wi; p_star_y /= sum_wi;
        q_star_x /= sum_wi; q_star_y /= sum_wi;
        let mut fr_x = 0.0; let mut fr_y = 0.0;
        let vpx = -(vy - p_star_y); let vpy = vx - p_star_x;
        for i in range(0, num) {
            let pix = ps[i][0]; let piy = ps[i][1];
            let qix = qs[i][0]; let qiy = qs[i][1];
            let pi_hat_x = pix - p_star_x; let pi_hat_y = piy - p_star_y;
            let qi_hat_x = qix - q_star_x; let qi_hat_y = qiy - q_star_y;
            let ai11 = pix * vpy - piy * vpx;
            let ai21 = pi_hat_y * vpy + pi_hat_x * vpx;
            let ai12 = pix * (-vpx) - piy * vpy;
            let ai22 = pi_hat_y * (-vpx) + pi_hat_x * vpy;
            fr_x += wis[i] * (qi_hat_x * ai11 + qi_hat_y * ai21);
            fr_y += wis[i] * (qi_hat_x * ai12 + qi_hat_y * ai22);
        }

        let vl = vpy * vpy + vpx * vpx;
        let fl = fr_x * fr_x + fr_y * fr_y;
        let vl = if fl == 0.0 { 0.0 } else { (vl / fl).sqrt() };
        fr[ip][0] = fr_x * vl + q_star_x;
        fr[ip][1] = fr_y * vl + q_star_y;
    }
}

The variable names are chosen to reflect the original paper. One difference is that our algorithm uses a fixed “alpha” for performance reasons.

I wish computer scientists provided such example code with research papers!

Usage

Deformed images are common on objects that require organic animation, such as characters or plants. Here is an example where it used to create a game with the look of the old He-Man cartoon movies. It was animated and exported to sprites, so the deforming does not happen in real time.

How does it work?

Perhaps you are familiar with matrix transformations? A matrix transformation is a general linear map, which means it can transform while keeping lines straight. The algebra on matrices is called linear algebra and is one of the central topics in mathematics.

While it seems difficult on the surface, it all boils down to: + - * / .sqrt() and some loops. These are very simple operations, which are put together in patterns, and then comes out as a deformed image!

Instead of transforming every pixel, we transform a grid and render the image on the GPU.

Algorithm walkthrough

Let’s look at the first part:

// Get the coordinate in the grid.
let vx = m as f64 * units_h + x;
let vy = n as f64 * units_v + y;
// Initialize variables to sum something.
let mut sum_wi = 0.0;
let mut p_star_x = 0.0; let mut p_star_y = 0.0;
let mut q_star_x = 0.0; let mut q_star_y = 0.0;
// For each control point.
for i in range(0, num) {
    // Get original control point P.
    let pix = ps[i][0]; let piy = ps[i][1];
    // Get deformed control point Q.
    let qix = qs[i][0]; let qiy = qs[i][1];
    // Get distance^2 from original control point.
    let vl = (pix - vx) * (pix - vx) + (piy - vy) * (piy - vy);

    // Get 1 / distance^2, with some trick to prevent division by zero.
    // This is a "weight" of the current point.
    let w = if vl < eps && vl > -eps { 1.0 / eps } else { 1.0 / vl };
    // Sum up weights.
    sum_wi += w;
    // Multiply weight with the point, will use this value later.
    p_star_x += w * pix; p_star_y += w * piy;
    q_star_x += w * qix; q_star_y += w * qiy;
    // Remember the weight for later.
    wis[i] = w;
}

// Divide by sum of weights to get something called "weighted average".
p_star_x /= sum_wi; p_star_y /= sum_wi;
q_star_x /= sum_wi; q_star_y /= sum_wi;

Notice that the weight is 1 / distance^2. This means that the closer you are to a control point, the more influenced you are by that point.

“Weighted average” tells us how much a point is influenced by all control points. It gives an ideal position where the 1 / distance^2 weights are balanced. We need two weighted averages, one for the original and one for the deformed grid.

Since the second part is most complex, let’s wait with it until the end. Here is the last part:

// Normalization trick to get length of vector.
let vl = vpy * vpy + vpx * vpx;
let fl = fr_x * fr_x + fr_y * fr_y;
let vl = if fl == 0.0 { 0.0 } else { (vl / fl).sqrt() };
// Add vector to weighted average in deformed coordinates.
fr[ip][0] = fr_x * vl + q_star_x;
fr[ip][1] = fr_y * vl + q_star_y;

This gives us the final deformed point.

Notice the second step computes something to add to the weighted average! How can this be?

You can override dependencies with Cargo to experiment by putting some 0.0 in front of fr_x and fr_y. If you run this, the image will become a shrinked shape with spikes attached to the control points.

This tells us that the effect of the second part of the algorithm wears off when we get close to a control point! The second part of the algorithm is just to straighten out rest of the warped image.

Notice that (vl / fl).sqrt() saves us one square root operation. This is because to get a length of a vector, you take the square root of the dot product with itself. Since we are dividing one length of a vector with another, we can move the square root operation to after the division, while getting the same result.

Now to the second part:

// The new deformed position is given by a sum.
let mut fr_x = 0.0; let mut fr_y = 0.0;
// Get normal vector from original weighted average to position.
let vpx = -(vy - p_star_y); let vpy = vx - p_star_x;
for i in range(0, num) {
    // Get original control point P.
    let pix = ps[i][0]; let piy = ps[i][1];
    // Get deformed control point Q.
    let qix = qs[i][0]; let qiy = qs[i][1];
    // Vector from original weighted average to original control point.
    let pi_hat_x = pix - p_star_x; let pi_hat_y = piy - p_star_y;
    // Vector from deformed weighted average to deformed control point.
    let qi_hat_x = qix - q_star_x; let qi_hat_y = qiy - q_star_y;
    // Compute a 2x2 matrix using the normal vector,
    // and the vector from original weighted average to original control point.
    let ai11 = pix * vpy - piy * vpx;
    let ai21 = pi_hat_y * vpy + pi_hat_x * vpx;
    let ai12 = pix * (-vpx) - piy * vpy;
    let ai22 = pi_hat_y * (-vpx) + pi_hat_x * vpy;
    // Sum transformed vectors in deformed coordinates by weights.
    fr_x += wis[i] * (qi_hat_x * ai11 + qi_hat_y * ai21);
    fr_y += wis[i] * (qi_hat_x * ai12 + qi_hat_y * ai22);
}

Notice that we don’t use the deformed control points before the very last step. We could precompute the matrix, but that would require one matrix for every point. Storing that much memory might even be slower! The algorithm is O(N * M * P) where N is grid width, M is grid height and P is number of control points. This remains unchanged even if you precompute the matrices.

Enjoy!

Piston is a Rust game engine developed by the open source organization PistonDevelopers.

http://www.piston.rs/

We are looking for contributors! How to contribute to Piston

At the core of Piston is the event loop, which polls events from the window. You can write your own back-end by implementing the Window trait. There exists back-ends for SDL2 and GLFW already. The library you need to write your own back-end is here.

This week I’ve been working on a new design for events, which uses a trick with Rust’s Any and TypeId.

• Existing code should work as normal, but the new design is preferred because it is more generic
• The new design should be just as fast when compiled with “-O2”

Old style

use piston::Input;
use piston::input::Press;
...
match e {
    Input(Press(button)) => { ... }
    ...
}

New style

use piston::event::PressEvent;
...
e.press(|button| { ... });

How does it work?

Each event is defined as a trait with a blanket impl for all T: GenericEvent. When the method press is called, it creates a unique id associated with the event trait by calling TypeId::of::<Box<PressEvent>>(). This is then passed to with_event on GenericEvent.

impl<T: GenericEvent> PressEvent for T {
    ...
    
    #[inline(always)]
    fn press(&self, f: |Button|) {
        let id = TypeId::of::<Box<PressEvent>>();
        self.with_event(id, |any| {
            match any.downcast_ref::<Button>() {
                Some(&button) => f(button),
                None => fail!("Expected `Button`")
            }
        });
    }
}

If the event supports press events, it will call the closure with an &Any argument. This is then downcasted to the argument type of the event callback. If the type does not match, it will fail with an error message. The methods are inlined because Rust is smart enough to infer whether a TypeId matches.

Definition of GenericEvent:

pub trait GenericEvent {
    fn from_event(event_trait_id: TypeId, args: &Any) -> Option<Self>;
    fn with_event(&self, event_trait_id: TypeId, f: |&Any|);
}

Since the event trait PressEvent and GenericEvent communicates through &Any, we have to run unit tests to make sure they downcast to the right argument type. The assert_event_trait function tests that a given instance of an event is implemented correctly.

#[test]
fn test_input_event() {
    use input;
    use input::Keyboard;

    let ref e = PressEvent::from_button(Keyboard(input::keyboard::A)).unwrap();
    assert_event_trait::<InputEvent, Box<PressEvent>>(e);
    ...
}

Why not just use traits?

When a function calls another function, you need to list all the generic constraints:

fn foo<E: PressEvent + ReleaseEvent + UpdateEvent + ...>(e: &E) { ... }

fn bar<E: PressEvent + ReleaseEvent + UpdateEvent + ...>(e: &E) { foo(e) }

With this approach, if you want to add a HeadTrackerEvent, then you need to update all functions that call functions that call .head_tracker(|head_data| { ... }).

By using GenericEvent, we can do this:

fn foo<E: GenericEvent>(e: &E) { ... }

fn bar<E: GenericEvent>(e: &E) { foo(e) }

The code won’t break just because somebody wants a HeadTrackerEvent, because all events are supported through GenericEvent.

• It solves the problem where native back-ends have custom types of events
• All code written on top of Piston will run on any platform
• New or custom hardware can be supported easily
• Container widgets do not need to be recompiled
• Window back-end can use generic wrappers, for example enum HeadTrack<I: GenericEvent>

Can I impl GenericEvent for native event structures, like SDL2 and GLFW?

Yes.

You probably need to wrap it in a struct tuple, since you need to own either the trait or the type to write an impl. For example pub struct MyEvent(SDL2Event). Then write a window back-end as impl Window<MyEvent> for MyWindow.

Remember to inline the GenericEvent impl and write unit tests!!!

Further work

• Update AI behavior trees to use generic events
• Update cam
• Update Conrod
• Investigate event transformations
• Discuss future directions for input model and integration with glutin
• Network events?