## [Web] The Numbers Game

Posted: January 18, 2011 in Web

Yesterday I took part in a bit of masochism.  I argued on Twitter.  Quite frankly: Twitter sucks for a debate.  Organized thoughts are often truncated and cut into multiple snippets, which are often interrupted and interjected.  You have to be insane to try to carry on a meaningful conversation on Twitter, but here we are.

At heart of the lively debate was whether or not X-Fire stats mean anything.  Some say they do, but I disagree.  Sure, they mean something to the users of X-Fire.  More people using X-Fire means just that, and only that:  more people using X-Fire.  Anything other than that is merely inference.

The crux of that opinion comes from a single word: Representative.  In order for you to be able to extrapolate bigger numbers from smaller numbers, the smaller numbers have to be representative.

For example:  If you know that 3,800 people using X-Fire play WAR, it is impossible to infer the number of WAR players unless you know what % of WAR players use X-Fire.  3800 = Y * X (where X is the percent of players and Y is the total players).  The problem with X-Fire is, you don’t know what X is.  X varies from game to game. X varies from month to month.  X can even vary from day to day.

The fact that X can change without adding/losing a subscription makes it hard to rely on those numbers as fact.  The “Number of X-Fire users playing WAR” simply represents…wait for it…. the “Number of X-Fire users playing WAR”.  That’s it.  If that number goes up, it may mean more subs.  It may not.  What if an entire guild is told they should be running X-Fire?  They all install it.  That number goes up. Does it mean more people are using X-Fire in WAR? Yep.  Does that mean more people are playing WAR?  Nope.  See where I’m going?  The same argument can be used for polls conducted on the forums.  Do they represent the players on the forum? Yep.  Do the represent players overall? Newp.  Quite a few developers have outright stated that a majority of players don’t use the forums.

Now let’s get back to the word “representative.” I used the Nielsen ratings yesterday in my argument.  They’re a truly representative system, if not a very small subset.  What they lack in numbers, they make up for in accuracy.  While everyone has different viewing habits, the fact that they proportionately depict the overall makeup of the US and A makes it a meaningful statistic.  Sure, you may not like or agree with the results of the Nielsen ratings, but they’ve proven to work in the past, and are overall reliable.

How X-Fire differs though, is that you don’t have equal representation.  You have a particular sect of gamers. While albeit 17 million is a beefy subscriber base, it doesn’t necessarily reflect the casual gamers out there.  It may, it may not.  It may not reflect the players who have crappy machines and don’t want any overhead running.  It may…it may not.  The only thing X-Fire accurately depicts….is the number of players using X-Fire.

1. An ex-war player says:

I agree you shouldn’t try to extrapolate xfire numbers to actual players with any degree of confidence, but it’s not obvious to me that if you average xfire numbers over some reasonable timeframe (maybe a week or so) and get some sense of real trends e.g. response to the RvR packs in 1.4. Basically, you could average out the number of people that start and stop using xfire (I never used it, don’t know anyone else that did, but then I don’t use facebook or twitter or a lot of other social media things that xfire seems to fit into either). I think the bigger question is whether xfire provides a representative sample of the entire WAR community on the week to month timescale, which I don’t know how you’d answer without access to private mythic data. It seems you can argue that point either way depending on what you want to believe, or just make the basic point that all the xfire numbers describe is trends in WARs popularity for a certain demographic (xfire users).

• Rancid says:

I honestly don’t know if it does represent the WAR community. I can’t imagine it does. The fact that you can’t ever be sure, means it isn’t a good source of data to use as “fact.” And you’re right about trending. You can use it to pinpoint the Xfire demographic, but that’s it.

• An ex-war player says:

Just to follow up on the Elementalisty comment below, can you explain why you don’t think X-Fire is a sufficiently representative sample of WAR, or less so than for other games which apparently do show correlations to official releases? Just like it’s hard to imagine it’s a perfect sampling of the community, I dont see why it should be completely unrepresentative either.

What would really be useful if there were numbers from a few sites, not just X-Fire.

• Rancid says:

Because there’s no way to correlate a relationship between subs and players. There’s not been a confirmation like WoW releasing player counts. If you had a baseline to setup a ratio…maybe. You just don’t. And no, box sales don’t really represent paid subscribers. Not these days, anyway.

2. Wasdstomp says:

Well Census works pretty well if you have enough dedicated players using it, and submitting the data to the site. It still fails in some respects, and doesn’t truly give you real numbers.

You never know if someone just logs on all their alts for a minute or two to trade with their other chars, get something out of the bank or etc.

I think there is another site out there that tracks renown activity. I think that is mostly what players would be interested in seeing, but it still has its faults. How many of those are afkers, renown bonus farmers, and etc.

The only time you know real numbers is when the game closes, and you know subscribers are zero.

• Rancid says:

Yeah, that was Winkl’s population monitor. He was working out some kinks the Herald code. Then Mythic pulled the plug on the Herald. Some say to snuff his stat-gathering. Some say incompetence. I don’t know the real answer, honestly.

3. First, it is X-Fire…not crossfire as you continue to state incorrectly.
As to your example. No one believes for a moment that you can get ACTUAL foolproof data out of the numbers on X-Fire…not even me, who plays a “game” with the numbers and shows how the data could be conclusive…but, has stated over and over, it is only a game.
What X-Fire DOES show time AND again is trends. When someone screws up (Champions Online being a perfect example of a game that continued to fall and fall after the first 3 months of release or the drop off in players on X-fire in WAR after patch 1.4) or someone has a major population and revenue growth (LOTRO)…we can see it on X-Fire.
I expect once Champions Online goes free to play (and the X-Fire client shows that data, as it is broken as of today thanks to an update to CO)…that we WILL see an increase in players and played time on X-Fire.
As to your continued argument of “You can use it to pinpoint the Xfire demographic, but that’s it.”, I kindly disagree as well. Time and again X-Fire has shown correlation with the data released by companies. Turbine being the example here again. When “Triple Revenue” was announced, it was also noted that player numbers tripled on X-Fire gaming time as well.
An X-Fire user is a GAMER, and GAMERS play games. If Nielsen can take a slice of the population to represent millions of viewers, why can’t 17 million registered GAMERS likewise show a trend in a games likability. You are willing to believe that Nielsen can be 100% pinpoint accurate with a 2% sample, but based on game sales of various products, Xfire is close to 10-12% or even MORE of the gaming public..and it shows nothing? I disagree with that.
To outright continue to shut out X-Fire data as 100% useless though is close minded-ness. There is no doubt X-Fire is not perfect…but, time and again, it HAS shown how trends in games work.

• Rancid says:

Fixed the “Crossfire” issue. Thanks for the heads up. 😛

Ok, to the meat:

The biggest issue I have with your game is that you hold the coefficient constant across all games. That flies in the face of logic. Who is to say that the same types of players play each game?

You rely heavily on inductive reasoning which may or may not be accurate. You make the data fit for one model (such as the LOTR Triple Revenue) and then extrapolate it to other games (WAR). How can you view that as even remotely accurate? I can’t, which is why I have a beef.

I also have issue with the fact you lump all GAMERS into one category and assume that 17 million gamers is a proportionally accurate representation of ALL gamers. I have a bigger beef with that. It’s like saying that the movie “Star Wars: Episode 1” sold a million tickets on day 1. It must be a smashing success for the entire population? Well, it was insanely popular with the nerd community, but not any other demographic. You fail to see that just because you have a large sample, you don’t have a representative (read: proportional) sample.

• “The biggest issue I have with your game is that you hold the coefficient constant across all games. That flies in the face of logic. Who is to say that the same types of players play each game?”

First, we WILL note it is a game, and has no foothold in reality, though for some reason you continue to believe this…
But, if we look at what the gamers of X-Fire are doing, it is expressing THEIR opinion of what games THEY wish to play. Based on opinion polling, which states “An opinion poll is a survey of public opinion from a particular sample”, we can then “extrapolate generalities” from said data.
So, we have 17 million gamers that we can look at and get an idea of what games seem to be more popular…thus the “trending”.
What we ARE seeing is a trend among 17 million gamers, which can be played with to get some extraneous data….generalities if you will.
Just most of this discussion is truly moot. You keep speaking like cold, hard facts are being provided, when nothing of the sort is being done.

Which brings us to…

“I also have issue with the fact you lump all GAMERS into one category and assume that 17 million gamers is a proportionally accurate representation of ALL gamers.”

No, what I HAVE done is look at what game is played the most according to the X-Fire network, which happens to be an MMO. I then took the PREPONDERANCE of evidence. which has shown time and again that certain interests breed interest in commonalities, and applied an algorithm based on this commonality.
If a gamer for example likes an FPS, there is a VERY good chance they will like another FPS. Maybe not…but, the chance exists. If you like Pizza from one pizza place, there may be chance you may like pizza from another place. How will you know? You may TRY it.
Thus, based on the percentage of players on X-Fire playing an MMO, there is a good chance that that player may like another MMO on the list.
So, yes, in all actuality, that “percentage” could give me a good representation of what games are being played right now by most gamers.
As I stated earlier (and researching it even more), Nielsen can base all tastes of all TV viewers based on a less than 1% choice of random viewers.
So, even if 2% of the full X-Fire audience plays MMO’s on the X-Fire network, I can extrapolate data from that audience just like Nielsen can gain it’s data.
If Nielsen can decide I like American Idol and it is the #1 show for over 6 years, yet I have NEVER watched it physically on my TV set and wish it would go away…you honestly continue to think Nielsen’s data is better than mine? Especially when the #1 game listed on X-Fire is WoW, which happens to be one of the largest grossing, biggest selling and most played MMO IN HISTORY and can be PROVEN.

I understand this is strictly all about WAR and my continued observance of it’s downfall according to X-Fire. And if it wasn’t for other factors like a second service (Raptr) also showing a DOWN trend and the overall rumble of discussion on forums and blogs as well as various news media sites all discussing the error of WAR’s ways and Mythic’s final nail…then I would see your argument.
But, it is hard to not see the almost two month downing of Realmwar which fed data to Winkls for population stats, specifically after it had been noted Winkls found an issue with “ghosted” players, as suspicious. This could mean stats would show the population on a downward spiral, which would be evidence that could corroborate further my findings on X-Fire.

Just remember….it is a game after all.