MySQL StatsD project on Github

Done is better than perfect.

That is one of Spil Games Engineering’s principles and it is really something I embrace. Almost one and a half year ago, Engineering started to use StatsD and Graphite for collecting and graphing performance metrics and one year ago I got fed up that there was, except for a couple of abandoned projects, no real drop in solution for getting a daemon sending performance metrics via StatsD into Graphite . So I created my own daemon on a Friday afternoon in Python and it was indeed done and not perfect.

Continue reading

OpenStack Swift & many small files

At Spil Games we have been running Swift for more than two years now and are hosting over 400 million files with an average file size of about 50 KB per object. We have a replica count of three so there are 1.2 billion files to be stored on the object servers. Generally speaking, Swift has turned out to be a solid object storage system. We did however run into some performance issues. Below we’ll describe how we analyzed and solved them.

Continue reading

Making RUM actionable

Real User Measurements (or Monitoring) has seen a steep increase in use in the past year. Companies see it as the next Holy Grail in web performance monitoring and are shifting their attention to it. With current RUM techniques, it’s still impossible to pinpoint what the cause of a slowdown is. How do you get insights from the RUM data and how fast can you notice shifts in trends? The second post in our series about RUM will dive into it. Missed the first about the general introduction of RUM? Click here.

Continue reading

Big data processing with Disco

Those who deal with big data probably know about Disco – a distributed computing framework aimed to provide a MapReduce platform for big data processing Python applications. We are proud to say that we are one of the largest users of Disco in the Netherlands.

As an owner of multiple high-traffic portals with lots of content served by CDN providers we want to ensure that the data on our portals loads fast. We have recently rolled out a Disco based solution that helps us to deal with this issue and continuously monitors availability and load performance of our content.

We put the basics of MapReduce paradigm, key points of writing MapReduce jobs with Disco and highlights of our solution together into one workshop. We showed participants how easy it is to develop their own big data applications that process a billion samples per day.

If the topic sounds attractive to you – you’re welcome to have a look at the slides we used during the workshop:

Database TCO presentation

Speaking about MySQL UG NL Meetups – during the Q1 Meetup in February (hosted by Spil Games) one of our DBAs gave a presentation on database TCO (Total Cost of Ownership). We would like to share it here.

Apart from covering the topic of determining database maintenance costs the presentation included some ideas on possible improvements as well as findings of our own small case study. The slides can be found here:

Feel free to ask questions and give comments.

Presenting at the MySQL UG NL Meetup Q2

Next Friday (31st of May) the second MySQL Meetup User Group NL of this year will be hosted by Snow IT in Geldermalsen. It is great to see that various companies are hosting the meetup and that the diversity and number of people attending is increasing. In total three presentations will be given:

  • Choosing the Best Sharding Policy – Doran Levari (ScaleBase, using a video link)
  • Performance Monitoring with Statsd and Graphite – Art van Scheppingen (Spil Games)
  • Basic MySQL performance tuning for sysadmins – Daniël van Eeden (Snow)

Our presentation will be a condensed version of the MySQL Performance Monitoring using Graphite and Statsd presentation that I gave at the Percona Live MySQL Conference in Santa Clara. The main difference is that I’m not getting into the details why we decided to use Graphite but it will only focus on the implementation itself.

Hope to see you next Friday!

OpenStack with Open vSwitch and Quantum (Folsom) on Scientific Linux

We decided to run OpenStack on Scientific Linux 6, a Red Hat derivative. Although this was not a reference platform (this was before Red Hat announced it would support OpenStack), we decided not to go with Ubuntu due to various factors. Knowledge and availability of an in-house infrastructure tailored to Red Hat based machines were the most important deciding factors.

With the availability of OpenStack packages through the EPEL repository it made running OpenStack on Scienfic Linux a manageable endeavour. Installing was basically just following the install guide.

Continue reading

Spil Games’ Engineering blog

Welcome to the new blog of Spil Games’ Engineering department. We noticed we are consuming a lot of information from blogs ourselves and we learn a lot from them. They also give us the inspiration to change the way we are doing things ourselves.

Although our websites are quite well known (,,, for example), Spil Games isn’t. However, this doesn’t mean that we don’t do cool stuff :-). This blog is meant to contribute to the tech community as a whole. We’re going to participate in tech and engineering discussions like RUM performance data versus synthetic performance data and we’re going to show what we’ve done with OpenStack and how we’ve done it. This site will also contain more information on the department itself, how we are organized, what our culture is like and who the people are that are working here.