The perils of comparison studies

On Friday a number of UK publications posted news articles about a Deutsche Bank report saying that London’s public transport fares are the highest in the world.

The comparison was of a monthly public transport pass.

Most of the news reports listed only the top three most expensive cities; London, Dublin and Auckland, and the cheapest: Mumbai.

I wondered where Melbourne ranked. The BBC report was the only one to list beyond the top three; it showed Melbourne at number 9, with a monthly Pass costing £82 (US $106).

It was also the only one to link to the source Deutsche Bank report. The report is not about public transport specifically, it’s about cost of living in numerous categories, in 47 cities. (Note: 47 cities is not the entire world.)

Looking through the Deutsche Bank report, they’d got figures from various places. The public transport fares had come from a web site called Expatistan, which compares cost of living indicators — indeed, Deutsche Bank credits them for most of their pricing data:

Most of our price data is collected from Expatistan (www.expatistan.com). We would like to give special thanks to the founder Gerardo Robledillo for permitting us to use this data and for assisting us in collating historical data from the website.

Clicking through to have a look at their page about Melbourne, I found they had a monthly public transport pass listed at A$136.

The problem is, this is wrong. A 30-day Pass actually costs $147.60 (excluding the cost of the Myki ticket). It looks like their price is about two years out of date.

Edit: It’s been pointed out that a 28-day Pass is about the right price. But 28 days isn’t a month (usually!)

Expatistan’s information is comprehensive… but it’s also crowd-sourced. This can be problematic — Deutsche Bank notes:

Much of our data is from sources that utilize crowdsourcing techniques to collect and aggregate price data. While this methodology provides them with regular data updates from a large sample, there may be issues with data quality and consistency over time. … Do note thus that prices, changes and ranks should be considered representative with considerable room for measurement and sampling error.

A few clicks and I was able to update the Melbourne price on Expatistan, but it’s not being accepted until I guess others enter the same data.

Trains at Milsons Point station

Sydney was ranked 7th most expensive by Deutsche Bank. Expatistan says it’s A$147, but it’s not clear how this was calculated given there is no monthly pass available. The weekly cap on Opal is A$60, so a monthly cap would be at least $240 for four weeks.

Even the provenance of the London figure quoted in all those articles is uncertain. The Deutsche Bank figure quoted is US$174. The Expatistan figure is UKP £134, which matches the USD figure. But £134 doesn’t match anything on the current official price list – the closest logical match is a zone 1 Monthly ticket, which is £126.80.

Zone 1 only covers central London. The New York City price doesn’t match the official price (USD $121) either, perhaps because it changed in March. But even if it did, that’s not really a valid comparison, as the NYC fare covers the entire subway and local bus network.

So really the fare comparison part of the study is problematic.

Perhaps I’m over-thinking it, but when I did fare comparisons in the past for the PTUA, I went to official sources of fare information, and I was as careful as possible to ensure I was comparing apples with apples.

In this case, the media quotes Deutsche Bank, who used Expatistan, who trust in their users and their algorithms. And data points I’ve checked are wrong, and/or, in my opinion, not equivalent enough to be compared.

The lesson here is: Don’t believe everything you read.

And when a study boldly states a conclusion, do a little digging, to understand how they got there.

If you enjoyed this post, please consider leaving a comment. You can subscribe via feed reader RSS, or subscribe by email. You can also Follow me on Twitter, or Like the blog on Facebook.

5 Replies to “The perils of comparison studies”

  1. And most importantly, don’t trust apparently ironclad figures from big-name sources based purely on the organisational reputation of the source in other fields. This kind of ‘source laundering’ (Big name A cites smaller name B cites Joe Random C) is more common than you might think, and occurs particularly when a ‘gun for hire’ consultant or quango steps outside their primary area of expertise without actually bringing in other experts as associates or subcontractors.

  2. I’m pretty sure Wellington is the most expensive place to use public transport in the world. There, a 2-3km trip costs you $5. The monthly pass for the inner zone (equivalent in area to Zone One here) is $150 in the local currency, steps up to $230 and $300, and then goes all the way up to $400 per month (AU$374).

  3. Re: Sydney / Opal: @ $60pw the monthly cost is ($60*52w)/12m = $260 p.m.
    Disclosure: I rented for many years, many years ago.

Leave a Reply

Your email address will not be published.