Toward Piston 1.0

Follow @PistonDeveloper at Twitter!

Update: There is a Rust gamedev meetup in Oslo coming up:

https://www.meetup.com/Rust-Oslo/events/261774885/

Sadly, I’m on vacation at that time, but you can meet people from the Amethyst project. Happy coding!

About the Piston core

The Piston core is a set of libraries that defines a core abstraction for user inputs, window and event loop.

If you take a look at the dependency graph in the README, it might look a bit scary:

piston dependencies

In this post I will go through each library, explain what it does and describe the status of stability.

Serde

Serde is a library for serializing and deserializing Rust data structures.

The reason Serde is used by the Piston core, is to support:

  • Sending events over network
  • Store logs of events
  • Record and play back player input for testing and debugging

Today, there is no functionality built into the core for these features. The intention is to handle these cases in separate libraries, if necessary. Piston’s design philosphy is to move complexity away from the core, since it is easier to keep the ecosystem stable when the core is stable.

By using Serde, these libraries gets much easier to write later.

Status: 1.0 (stable)

Piston-Float

This is a set of traits to abstract over f64 and f32 in Rust.

Piston-Float was designed to stabilize Piston’s ecosystem in a time where we needed stability at the bottom of the dependency graph.

Before Piston-Float, we used numeric traits from Rust’s standard library. However, when Rust got close to 1.0, it was decided to move numeric traits into a separate library. At the time, these trait were put together with lots of unstable stuff. They were designed/changing to be more generic than for the primitive float types we needed. The need for stability, plus restricted requirements, made it easier to create a new library.

Status: 1.0 (stable)

Piston-Viewport

A viewport is a rectangle inside the window that sets up the coordinate system for a graphics API.

This library has a simple struct Viewport that stores viewport information. It is used to pass viewport information from render events to 2D graphics backends.

Piston’s 2D graphics library is designed to be decoupled from the core. You can use it without the core, or you can use the core without the 2D graphics libraries. By using a common viewport structure, it is easier to make these libraries work together.

Status: 1.0 (stable)

Bitflags

A Rust macro to generate structures that behave like a set of bitflags.

A set is a mathematical structure that supports operations like intersection, union, etc. Normally, a C-programmer uses some tricks with bit masks to do work with bitflags. However, in idiomatic Rust we like to keep things type safe, which is why this macro exists.

This macro is used by the structure for modifier keys in the core library for user input.

Status: 1.0 (stable)

PistonCore-Input

This library is separated from the others in the core, since a lot of libraries only need to know how to handle user input and do not need to know about the window or event loop abstractions.

For example, when you press a keyboard key or a click a mouse button, this information is stored in a structure that is defined in this library.

The same goes for gamepad or joystick events, touch events, render, update and idle events etc.

While designing the Piston core, we did some research about the problems of modelling user input. What we found was the following:

  • New hardware for user input is not widespread, because software does not support it
  • Software does not support new hardware, because it is not widespread

The way we solve this problem, is to not restrict software to a particular model, but to use traits when handling events. This is why this library contains a lot of event traits, in order to make libraries work with new hardware.

The current design has been kept unchanged for a long time. There are no unstable dependencies. Without further redesign, it is ready to be stabilized.

There might be a few changes in the future, before stabilization.

Status: 0.25 (unstable)

Shader-Version

A helper library for detecting and picking compatible shaders.

Graphics APIs, such as OpenGL, Vulcan, Metal, DirectX etc. comes with shader languages. The shader languages have their own version numbers, which works with various versions of the API.

To pick which shader to use can be tricky, so this library does this for you.

Today, only OpenGL/GLSL is supported by this library. It is because OpenGL was the primary graphics API used at the time of design. Besides, for newer graphics APIs, it is also common to use cross-compiling.

In the future, it is likely that the library becomes replaced by another in the core. New graphics APIs come out all the time, so it will be difficult to stabilize this library. Also, detecting and picking compatible shaders is not needed in the core, only graphics API versions.

Status: 0.3 (unstable)

PistonCore-Window

This library contains the window abstraction.

The most common usage is window backends for GLFW, SDL2, Glutin etc.

However, a window in Piston might be more abstract and higher level than a cross platform window API.

It can be a custom designed backend for server-client logic, or it can be a convenience wrapper, such as Piston-Window.

This library requires some changes before stabilization.

Status: 0.38 (unstable)

PistonCore-Event Loop

This library contains a game event loop.

The event loop supports fixed time step updates, extrapolated time for rendering, benchmark mode etc.

A common problem I see people have, is figuring out the right event loop settings for their game. The defaults are designed for a First-Person-Shooter, which is pretty bad for many other kinds of games and applications. In order to address this issue properly, some changes are needed.

Status 0.43 (unstable)

Piston

This library re-exports the core libraries. That’s all.

Status: 0.43 (unstable)

Notes

The Piston core has been in use for so long time, relatively unchanged, so it has not been much focus around it.

  • Most of the interests experienced people have are in various modular libraries, not in the core
  • Experienced people don’t think too much about the core, which makes it hard to see through the eyes of beginners

For example, I noticed that some projects uses Piston-Window in libraries. It is designed to be a convenience wrapper, not an abstraction. The Piston core is more suitable for generic libraries.

I tried to put information about this in various places, but not sure if it is efficient enough. This is a common problem because the project is so big: Even if something is documented somewhere, it does not necessarily mean people will see it.

Transitioning to Image-rs

Follow @PistonDeveloper at Twitter!

This blog is about the transition of image libraries from The Piston Project to a new organization: Image-rs

Why Image Libraries are Moving

Today, the transitioning of the popular image library to the new Image-rs will be completed.

This is the first of several libraries that be moved, which I and other maintainers feel belong under the new organization. Now that the main issues and concerns have been resolved for the image library, I expect that much less time is required to move other libraries.

There are several reasons these libraries are moving:

  • Better security
  • More focus
  • More activity

In addition to these important reasons, there is an opportunity for new people to lead.

Publishing Permissions

The permissions to publish on crates.io for these crates will move from the Publish team in The Piston Project to the Publish team under Image-rs.

I (bvssvni) plan to participate in the new organization as a contributor.

Today, crates.io requires private owners to add or remove other owners. For security reasons, my own owner permissions will be removed and replaced by the owners of Image-rs.

Owner and Leadership Transition

Previously, the image libraries were part of The Piston Project and therefore under supervision of myself (bvssvni).

Under the new organization, there are 3 active contributors to image libraries that will serve this role under the transition, in case anything unexpected happens (alphabetic order):

  • fintelia
  • HeroicKatora
  • theotherphil

Among those 3, HeroicKatora has been working on inviting policy for the new organization, which is important for setting a future direction of these projects.

Personal Comments on the Transition

Due to two recent occurences of suspicious activity, my focus has been on the security of The Piston Project. This has delayed other work and made me aware of how much time it takes to just run The Piston Project and AdvancedResearch.

  • People working on image libraries deserve the best working environment it is possible to make
  • Now fundamental image libraries are in place, The Piston Project should continue focus on research and experimenting
  • Security is not just important for safety, but also to reduce waste of time

It makes completely sense to me to move these libraries to a new organization.

The way it looks now for Image-rs, I am proud to be part of the new organization.
I hope that other people will feel the same and welcome the change.

What is Happening 7

Follow @PistonDeveloper at Twitter!

This blog post is a brief summary of what happened the past 8 months in the Piston project.

Piston (Core)

List of contributors (72)

Conrod

Conrod is an UI framework that makes it easy to program UIs in Rust.

List of contributors (89)

Image

Image is a very popular image library with pure-Rust encoders and decoders.

The versions 0.20 and 0.21 were released with changes you can read here. (Too long to be listed here)

List of contributors (134)

Imageproc

Imageproc is a library for image processing.

List of contributors (26)

Dyon

Dyon is a scripting language with lifetime checker instead of garbage collection, a similar object model to Javascript and lots of other features useful for gamedev.

  • Added 4D matrices
  • In-loops added for easier communication between threads
  • Improved docs
  • Improved edge cases in syntax, type checker and runtime
  • Faster parsing overall due to Piston-Meta upgrade
  • Internal refactoring of instrinsics

List of contributors (10)

Turbine

Turbine is a long term project to develop a game engine with built-in editor.

Currently some components are developed separately and tested, for later be used in the game engine.

AdvancedResearch

A part of Piston project is research, which moved to its own organization when unrelated to game development.

The reading sequences have been re-organized into two parts:

On the software side, the most exciting news are:

The work on the Control Problem of AI and Zen Rationality resulted in LOST (Local Optimal Decision Theory). The environment which this algorithm performs optimally falls outside classical Instrumental Rationality and might therefore be used to study Zen Rationality (extended reasoning about higher order goals). This decision theory describes an analogue of a “panic” state for formal agents. One mathematical property of this algorithm is that there is no expected utility loss by changing the decision theory, which means that it might serve as a building block in Operator Triggered Intermediate Decision Theories. Such agents might have higher level of mathematical provable safety. It was also proven that AI boxing is only decidable if the world is divided into finite states (paper).

Some other areas of progress:

Example of ongoing topic: Semantics of choices

Due to the many new papers and ideas floating around, I choose here to focus on a particular topic to give you a taste of what kind of research is going on: The semantics of making a choice and how it might be grounded physically.

In one way a choice is very simple, in another way it is very deep and complex.

The basic problem of understanding what choices are, is that they are very easy to understand starting from the axiom of path semantics, but they are very hard to understand as something grounded in physical reality. According to path semantics, this grounding must exist in order for mathematics to make sense, but we don’t know what it is.

A choice might be thought of as an action where one forgets something (or erases a resource), and then “feed in” some new information that is required to obtain a new resource, in order to make new choices. Ideas from a discussion with Adam Nemecek resulted in the paper about Adversarial Paths.

Much about adversarial paths is not easy to understand, but it is very useful as a building block for other mathematical languages. For example, this syntax can be used formalize Adversarial Discrete Topology which might be used to e.g. understand game theory and agents in environments of unknown complexity.

Some important property that is known informally about choices, is the following:

The major difference between Turing machines and informal theorem proving is that Turing machines eliminate choices.

The word “decidable” often appears related to Turing machines, but it might be misguided as it refers to some enabling property. On the other hand, “decidable” might be better understood as a disabling property in terms of eliminating choices. It goes much deeper than just talking about algorithms that are not decidable vs algorithms that are decidable.

Much of the semantics that makes mathematical useful is based on some concept of choice in one way or another, but any realization of an automatic procedure of such semantics requires a reduction of choice into a decidable computable process. Hence, viewing choices as a form of resource, there exists some kind of game (as in game theory) that is intrinsic to the nature of mathematics and computers, which does not arise from the choice of an opponent, but from semantics of expressions in various mathematical languages.

The semantics of choices relative to other parts of mathematics is hard, but physical grounding of choices is even harder.

In trying to understand extreme anthropic observer selection effects, it is speculated that this constraint of eliminating choices is an emergent phenomena. We might be living inside that a universe that has strange underlying physical laws that permits some violation of the principle “making a choice”, while all semantics that we use in practice (including computers and physical machines) is a result of viewing this physical reality through a lense of an extreme observer effect.

While the foundations for choices is a mystery at this point, it serves as a useful concept for arguing for uniqueness of mathematical emergent semantics such as the unit interval on real numbers. Unit intervals on real numbers is fundamental for fields such as geometry, where it appears in the domain of homotopy maps. We want to know whether basic assumptions about the physical universe are unique in some sense, such that there can not exist fundamentally different universes with some kind of “strange semantics of geometry”. Better understanding of this topic could bring us closer to understand how to think about physical laws where choices are not eliminated.

Improved Organizational Security

After detected suspicious behavior on Christimas Eve 2018, I (bvssvni) reviewed security policies and made the following organizational security improvements:

  • Reduced attack surface by separating write access to repositories from crates.io publishing permissions
  • Restricted conditions of getting write access to repositories
  • Restricted conditions of getting crates.io publishing permissions
  • Recommended maintainers to add collaborators to specific repositories when needed

Since everying goes through PRs, most people do not need the write access.

When reviewing logs of all projects, I concluded that most contributions are made by a handful of people with specialist knowledge. It seems that most work in recent years requires specialist knowledge or familiarity with specific codebases, such that the amount of available work for people who want to contribute in general, but is not able to specialize, is shrinking. Most people would be better off just by making PRs to specific projects, and the likelihood that they need repeated access is shrinking over time.

Due to the shifting focus from general contributions toward specialism, there is less need for increasing the amount of members with write access. The majority of projects is now entering a maintenance and stabilization period.

Important Organization Notice: Expecting Decreased Available Work Without Specialist Knowledge During Next 5 Years

The two major activities of the Piston project - maintenance and research - where research always required specialist knowledge, plus the choice of using a modular architecture, means that it is likely that in the next 5 years, more and more low hanging fruit will be picked and the amount of available work without specialist knowledge is expected to decrease.

Most active contributors are heavily invested in specific projects and new contributors are often interested in existing specific projects or creating new ones that require specialist knowledge. This means that we might benefit from making it easier to do specialist work over time, and general organizational development is expected to have less benefit beyond serving this role.

Even version updates have been done in batches without the need for major changes, which is done most efficiently using Eco’s output by one person and by passing on this information to projects that might use it for integrating updates with plans for new releases. This is expected to be done approximately once a month or less in the future.

The need for architect software design is practically zero, as it seems people find it economically efficient to combine their own set of preferred libraries to solve a specific problem, instead of making major design changes to an existing one to solve a more general class of problems. Most changes in design are focused on keeping backward compability, to keep the same set of problems solvable through modularity.

The need for bug fixing is very low despite the size of the project. Excellent language design of Rust and good developer tools are likely to be blamed.

This means that the Piston project must either:

  1. Focus more on special knowledge and projects
  2. Increase ambitions for the project overall

The economically efficient trend seems to be 1) from observing what people chose to focus on in the past. It is therefore unlikely that we will increase the ambition overall and instead focus more on maintenance with specialist knowledge plus research.

Research has most potential of opening up new available work, but this is a slow going process.

The characteristics of the need of specialist knowledge is the following:

  1. Less need for marketing the project or attracting new members
  2. More need for training people interested in becoming specialists

The organization might therefore benefit from work on documenting and representing topics that makes it easier to specialize for new people.

Older Newer