RUM at Spil Games

The Performance team has a lot of tools to optimize the portals and RUM is one of them. This post is the first of a series in which we’ll take a look at what RUM is, what it can do and what it can’t.

What is RUM?

Real User Measurements (RUM) is a way to get near realtime feedback from your end-users on predefined metrics. The collecting is easy: gather your desired metrics at the end-user and then beacon it back to the server. To minimize the amount of connections you can choose to buffer the metrics first

Why RUM?

RUM shows the actual experience and performance of your sites from your end-user’s point-of-view. That’s all there is to it. No more biased synthetic monitoring with fast connections on powerful PCs, just the real deal. RUM doesn’t have to be “just” about performance and timings. Counts of certain user-initiated actions can also be valuable for business. For example, the amount of resizing of a user’s browser can give an insight if resources should be spend on making the site responsive beyond the first load.

Why not RUM?

RUM does not give detailed information about the components that led to a certain metric, like time-to-onload. With the introduction of the Resource Timing API and the User Timing API this will change, but it’s still too much in the future. RUM also suffers from other programs that run on the end-user’s PC or other external factors. Examples are virus scanners kicking in, someone on the network using up most of the bandwidth, people connecting to your site via a mobile connection on an aircraft and so on. This skews the results and outliers should be expected.

A RUM for every situation

There are multiple kinds of RUM, each having a separate focus area.

  • Simple page load: This often only tracks onload time.
  • Page load with correlations: A combination of onload time with other metrics, like bounce rate.
  • Custom event tracking: Custom events can also be measured. This is useful for tracking Above-the-Fold loading time, which is subjective for most people.

RUM at Spil Games

We use Soasta’s LogNormal and Google Analytics to get a grasp on our standard RUM. These tools handle most common use-cases and they allow to drilldown into the data. Insights into performance of sites in a certain country, with a specific bandwidth or on a specific kind of page are all very easy to gain. This data is a valuable asset.

However, these tools are not meant for custom metrics: how long does the loading of an advertisement take, how fast is the execution of this script or how often is one of the dependencies (like Unity or Flash) not installed?

We’ve created two custom RUM solutions: Spil Event Tracker and Graphite Data Collector. The Spil Event Tracker (SET) is used for business decisions and long term stategic planning. The Graphite Data Collector is a more operational tool that we use to see if something is out of the ordinary.

In the next post on RUM, we’ll discuss the workings of and experiences with LogNormal, SET and Graphite Data Collector and how we make the insights actionable.