Tag Archives: development

Piqi in Spil Games – unifying our interface definitions

Piqi is a language we’ve been using to define our internal and external interfaces at Spil Games. We’ve benefited from its versatility as a project, and this blog post will try to show how.

Background (how Spil renders portals)

Simplified request flow example

Simplified request flow example

  1. Browser requests a page (http://www.agame.com/).
  2. Request goes though a CDN, load balancers, and ends up in Widget Platform. Widget platform is a proprietary system (written in Erlang, find out more about that in this talk from Erlang User Conference 2013) which renders HTML. It gets all information needed from a RESTful Spil API (HTTP/JSON).
  3. Spil API receives the request, does the authentication/authorization, and passes it further to backend services. Spil API is connected to backend services via native Erlang; spapi-router is used for service discovery and routing. Google protocol buffers is the payload format.

You can recognize that this is a typical Service-Oriented Architecture (more
in a talk in Erlang Factory 2013). We have a few dozen independent services, which expose an API that other services (or even browsers from front-end) can call. The way interfaces are defined influences many things: coding a client, coding a server, even the architectural design in early stages of service planning.

Why is this important?

Having many services poses a few challenges:

  • Every service has its own interface, which, in order to be efficiently used, has to be clearly documented:
    1. Input, output types with examples.
    2. Error conditions.
    3. Human-readable remarks.
  • Since a lot of data is being sent all the time, it’s critical to minimize the data encoding overhead. JSON is good enough for user-facing systems (nicely compatible with web browsers and very easy to use and understand). Protocol buffers provides many advantages for internal traffic: type checking, compatibility, very efficient packing.
  • At service level, input and output should be type-checked according to definition; that avoids lots of silly bugs. It also ensures documentation is kept up to date, because interface definition is documentation.

Technically our requirements for the interface definition are:

  1. Specify functions and input/output types in machine-understandable format.
  2. Input and output verification should be automatic – developer should not care.
  3. Generate documentation from interface definition.
  4. Generated boilerplate, so developer can focus on implementing the business side of the service.
  5. Compatibility is critical. It must be possible to evolve schema and reason about it.

Nice-to-have:

  • Use the same language (and tools with it) for external and internal services. It is easier to maintain one system than two. Keeping this has a lot of advantages:
    1. Once you establish a front-end protocol, you’re able to use this as a base for a back-end protocol. This increases operative efficiency and reduces learning time for developers.
    2. “Internal” and “External” definitions can be shared. For example, both can have common date/time types.
    3. Code generators, documentation generators, etc. can be shared, suppying knowledge for both realms.
  • To minimise demand on CPU and network, use a more efficient protocol (i.e. binary) for internal services. For example, Google protocol buffers, Apache Thrift, Cap’n Proto.

Quick Piqi example

We treat Piqi as Interface Definition Language (IDL), even though it’s much more than that. It helps us define interfaces and work with them. Piqi acts as a language bridge, which can “talk” in different protocols, namely JSON, XML and Google protocol buffers.

Below is the example, echo service definition and implementation:

Implementation:

With just these few lines of code, we automatically gain the ability call this service using HTTP/JSON, HTTP/XML, HTTP/protobuf and Erlang native/protobuf. Example query using HTTP/JSON:

And a corresponding .proto file, which is used to construct payload directly
to the function:

Output site in compact JSON is 109 bytes, whereas size of the equivalent payload in protocol buffers is 23 bytes. Space saving more than a factor of 4.

This function is just as easy to perform using Erlang and protocol buffers directly, without needing to add any extra line of code. This also creates automatic interface documentation:

Generated documentation from piqi rpc file

Automatically generated documentation for Echo Service

Piqi vs Piqi-rpc

Piqi is a language-agnostic tool and can work with any language which supports Google protocol buffers. Language independent.

Piqi-rpc is a library for Erlang applications which enables to create web-services (like in the example above). Language dependent – Erlang.

That being said, Piqi has found its uses mostly in the worlds of Erlang and OCaml, but it is important to know that it can be useful for much wider environments.

Recap

Piqi is the language we use to define interfaces for all our internal and external services. Each service is a piqi-rpc application, exporting its functionality via both JSON/HTTP and Erlang/protobuf. Each service contains its own definition, which can be reused by other services. Generated documentation, which people at all stacks (from front-end javascript to back-end developers) find very convenient.

Piqi and Spil Games

We started adopting Piqi somewhere in mid-2011. Piqi and piqi-rpc are running in production since mid-2012. It is cheerfully generating and parsing millions of queries per day, and it has helped make our work a lot easier and more efficient.

Interested in finding out more? We’re hiring!