That hasn’t stopped “screen-scraping” of data or simply typing in paper timetables (from Urban Mapping to many listed here). Unfortunately, the legal basis for scraping is complex, which creates significant risks for anyone building a business. For example, earlier this year, airline Ryanair requested the removal of all their data from Skyscanner, a flight price comparison site that gathers data by scraping airlines’ websites. How many airlines would need to object to their data being scraped before a “price comparison” service becomes unusable?

User-generated mapping content is evolving, often to circumvent restrictive distribution of national mapping. Services include OpenStreetMap and the recently announced Google Map Maker.

Micro-blogging, primarily through Twitter, has started to show the potential of individual travellers to report information about their journeys: Ron Whitman‘s Commuter Feed is a good example. Tom Morris has also experimented with London Twitter feeds.

This article outlines why the “social web”/tech-entrepreneur sector may wish to stop trying to use official sources of data, and instead apply the technology it understands best: People.

The Big Picture

I will use the example of UK local bus data to summarise the strategic issues for data providers. I can only presume the issues are similar elsewhere (comments welcome).

Explaining exactly who the data providers are is one of the many problems of trying to extract and use the data. I would provide more detail, but the topic is somewhat sensitive. The most critical point in the chain that constructs and distributes the data are local authorities – sub-regional public bodies, typically those responsible for large cities, conurbations or counties. They process the data, but are not under any statutory requirement to do so (no national government legislation requires it).

There are a number of issues for the existing data providers:

1. Mindset of centralised control: Most operators, public authorities, and other agencies, still have a mindset of centralised control of information, delivered to users via the method the agency believes is appropriate. This is heavily driven by the belief that only the agency can be accountable or impartial, and that incorrect information supplied by an uncontrolled third party is likely to damage the image of local transport service and generally reflect badly on the agency.
2. Mindset of local: Most agencies are locally focused, locally orientated. It seems logical for them to commission a fully-functioning website or piece of information delivery software that is specific to their city, because their target market is local. There’s a lack of global perspective: An agency will typically commission a system that is specific to their city, even when 95% of the features would work for any city, and 90% are already in existing global products.
3. Not appreciating trends in delivery channels: There is still an attitude of “we’ll provide a website”, without a comprehension that the number of channels for delivery of information is exploding far faster than any one agency can hope to construct bespoke user interfaces to cater for. Mobile devices, integration into social software. There would probably a market for a “WiFi-enabled” alarm clock that would ring later if your morning train had been delayed: We simply can’t define the limits for how this information might be used.
4. Not appreciating trends in cost: Even large, well-funded agencies are starting to fall behind the technology. The cost of systems (many millions of dollars invested year on year in some cases) is starting to hurt. Logically the global system should win out, because one city is very much like another: There is considerable scope for sharing systems costs.

What It Means

Long term we are heading for global providers of information, that pool data from local sources. That will be forced by the cost of technology. This can be seen in technology costs driving things like agglomeration in the groceries sector (such as Walmart) over the last 30 years. Also in the move from customised mainframe computing, to shared operating systems and platforms (such as Windows). This will be worse, because the number of systems will be simultaneously exploding alongside the complexity of those systems.

As these issues become progressively better understood, data will become more centralised. Even in agencies where (in my opinion) uniqueness and absolute control are culturally in-breed, such as London Transport/TfL, cost will eventually win the argument.

However, centralised data handling does not automatically make the data open. Quite the opposite.

Contracted Provision

Currently, effective control of data is with local government. Many individuals within local government will naturally attempt to block any change that might leverage power away from them and their organisation. “Job protection” is an over-simplification, but helps explain the underlying position. But by contracting data handling and presentation to a third-party contractor, local government would gain the technological “economies of scale” (assuming the contractor won many contracts from different authorities) and notionally maintain control.

Use of third-party contractors is already common within the local government sector, particularly for Information Technology.

An example can be seen in Edinburgh City Council’s Traffic Map. In spite of how it appears, the information isn’t powered directly by Edinburgh City Council or Google. Instead it is part of Mott MacDonald‘s Common Data Management Facility, providing services under contract to many different local authorities.

In the UK public transport arena, Trapeze is a good example of the gradual agglomeration of data handling within a few large businesses, where historically many small software providers could be found.

The example above provides key driver information, and is somewhat useful, but is it the best outcome? I suspect not. Contracts tend to be priced highly, because local government clients are high risk: Their political control means that they can change their strategic direction and requirements unexpectedly. At best, customer feedback loops through local authorities are slow and politicised. At worst the design of the system will reflect the arbitrary views of a self-proclaimed expert (such as myself). Even if you think it is perfect, there is no scope for choice or creativity. Choice is good and need not be expensive.

Social Provision

Instead of using official data, why not let users reconstruct it? User-generated content is cheaper to create than information from professionally staffed sources: Since very many contributors do so little work, no individual expects payment. User-generated content can be just as accurate too, although this is not automatic: For example, a strong community will subject everything to peer review, weeding out poor information and contributors.

This is not an entirely theoretical position. There is a largely untapped human resource, just waiting to help.

The transport enthusiasts (transit fans, “spotters”) already collate and produce some extremely high quality information about certain technical aspects of operations and services. For example, sites such as LondonBusRoutes.net contain detail on the bus route timing and vehicle allocation (type and number of buses), which transpires to be difficult to extract from official sources. While it may be argued that these sites simply repackage official information, their very existence is a testament to the strength of underlying community.

Casual observation of people delayed on trains or in traffic suggests they derive some comfort from picking up their mobile (cell) phone and telling someone about it. Something they can do, in a scenario they otherwise have no control over. Their desire to communicate the same information to drivers or users 10 miles behind them (who might be able to re-plan their route, should they know) is untested. But the potential is intriguing.

Nobody has entirely worked out how to use these people; yet.

Battle Lines

If the social web/tech-entrepreneur sector chooses to fight the “status quo” head on, it does so against large multi-national IT providers who support clients with historically entrenched positions. Not a contest that favours the underdog.

If the tech’ “upstarts” can find a way to use this human resource effectively, they will ultimately provide a more cost-effective solution than the traditional “government IT” sector can offer. Integrate that user-generated information into the wider consumer internet, and the machinery of government simply won’t be able to justify its historic position of pouring millions into systems it controls. The “social web”/tech-entrepreneur sector wins.

The upstarts do not need perfect source data, if the implementation of results is considered to be better by users. The early Xephos vs TransportDirect comparisons provide some evidence. The success or failure of the social web/tech-entrepreneur sector is ultimately dependant on whether they can provide better information than official sources, using the resources and skills they have available to them.

Disclaimer: The contents of this article reflect my own personal analysis of the situation. This does not directly reflect advice to, or views of, government or anyone else involved in the handling and provision of public transportation data.

This article describes how the UK’s public transport data is being integrated into Google. It questions why data is being made available based solely on the business model adopted. It explores the real value of this information, and presents a case for the liberalisation of data.

Readers unfamiliar with the topic area should read my earlier Introduction to UK Local Public Transport Data, which contains non-technical background information, and defines many of the terms used (such as “local”). The original research for this was done in June/July 2007, so may now be slightly out of date.

The illustration on the right is the Google part of a visual representation of web trends, based on the Tokyo metro map, by Information Architects Japan.

On this page:

• Information contained within Google
• Who pays?
• There’s no such thing as a free lunch
• Free data = free promotion
• Impartiality and accuracy
• Choice
• Thinking outside the box
• Google Transit?
• Further Reading
• Notes
• Acknowledgements

Also in this series of articles:

• Introduction to UK Local Public Transport Data

Information contained within Google

There is currently stop-based scheduled departure information in Google Maps for bus services in Buckinghamshire. Zoom right in on Aylesbury town centre, and you will see bus stop icons. Click to see the next departure times. A version of the full Google Transit for most of the South East region (21/22 authorities) is on trial, but not yet public. (The regional definition of “South East”, excludes London.) This will allow point-to-point journeys to be planned and visualised on the map. Both of those are run by Google themselves, who are determining the pace of implementation.

Traveline South East is the only regional consortium currently involved. However, there is growing interest amongst the wider professional community – for example Peter Stoner’s (Traveline Coordinator) What impact will Google Transit have for UK journey planners? (PDF) in ATCO News (the magazine of association representing local authority public transport coordination officers).

Data converters exist from ATCO-CIF and TransXChange (see the Introduction to UK Local Public Transport Data) into a Google format. The format is essentially sets of CSV files – making it terribly hard to extend. Inevitably the format will keep on changing, as new networks are added, all with their odd little quirks.

There is no railway information, since there is no rail industry agreement. Trains are a significant local public transport mode within South East England, so excluding them greatly weakens the usefulness of the information.

There are other uses of Google Maps Application Programmer’s Interface (API) – for example, Kizoom has used it in Leicestershire. I’ll discuss that “mashup” approach later. I mention it here because it is easy to confuse a “system run by Google” with a “system that uses Google technology”.

Who pays?

The current approach should be regarded as an experiment. But it may accidentally set a dangerous precedent.

There is no exchange of money: Google get to use the data for free, but don’t charge for running the service. It is regarded as a complementary distribution channel [1] – an extension of a search engine, which directs people on to services like TransportDirect for more detailed information. There is a contract, but it is not terribly binding on either party. Essentially, Traveline could pull the data, Google could stop offering the service.

This isn’t a free-for-all: Anyone else that wants to follow has to negotiate with each regional Traveline consortium separately. The terms provisionally adopted in the South East are that the service must be free to the end-user, and the data represented faithfully.

Since Summer 2007 it has been possible to licence Traveline data for commercial use, for a price. Availability of Traveline data hasn’t yet been widely publicised.

Spot the problem? Two different commercial licensing models are emerging for the same data: One for commercial applications that are free to the end-user, and one for applications that adopt a different business model.

There’s no such thing as a free lunch

While Google’s service may be free to the end-user, that doesn’t make Google a non-profit or charity. It shouldn’t be given preferential treatment, simply because of its choice of business model.

Companies like Google and Yahoo are trying to find out as much about their users as possible [2]. Queries about public transport journeys are like “manna from heaven”. Time and place to ridiculous levels of detail. Fares will start to indicate who. Why you can take a good guess at if you know enough about the time and place.

What’s the commercial value to this? Right now, advertising. Target people very precisely. Someone just asked how to get to Leicester Square (London) in the evening? Start selling them last minute theatre tickets, or suggestions for restaurants or nightclubs. It’s like having someone hand out your fliers at the station entrance, but a lot cheaper. It’s a logical part of a wider trend towards behavioural advertising.

But that’s just the start. Some other uses for this data are discussed below.

So there is a solid commercial basis to Google’s “free” service, for those far-sighted enough to look.

But Google is far from the only commercial organisation here:

• Search competitors Yahoo and Microsoft all have mapping projects.
• The UK has plenty of businesses with experience of building/operating journey planners: Xephos and Journeyplan already offer the public such services themselves. But there are several others who might evolve from the current business model based on bidding for contracts.
• The US has independents like HopStop, who provide journey planning information in many US cities, and might be expected to expand.

The market is theoretically highly contestable, particularly once it is viewed as part of a wider market for information, beyond simple “bus times”. Unfortunately, giving data freely to one business, while charging another, is not a good way to foster competition.

Free data = free promotion

So information about public transport services might have commercial value. Does that mean it should be sold?

It seems we have forgotten that this information is gathered to inform intending travellers. The easier it is for people to use this information – in whatever form – the more likely it is that they will travel using public transport. That contributes to policy objectives, profits, long-term sustainability. Why do I feel like I need to explain this?

Airlines pay to have their data used by others. This extract is from CheapFlights.co.uk, the UK’s leading air travel website:

“Do the agents and airlines on your site pay to keep their prices there? Yes. Apart from the joy of receiving their money, we do find it works much better. This is not as hypocritical as it sounds: in our first year (1996) we had a mixture of paying and non-paying contributors, and we invariably found that the paying ones took far more trouble to give us a fuller range of prices, and to keep them more up-to-date. With so many advertisers, we are able to carry a fully comprehensive range of cheap prices and so are able to fulfill our mission to be the No.1 guide to travel deals.”

Why such a difference in approaches? Local public transport has little motivation to sell itself. And when it does, the decision making process is cluttered by many organisations with slightly different aims. In contrast aviation needs to sell itself to survive, reaps benefits from doing so, and has a more streamlined decision making process.

Aviation does use a commission model to sell travel, which local public transport does not. But that is not important to CheapFlights.co.uk: They aren’t actually selling air travel themselves.

Impartiality and accuracy

The belief that only the public sector can be impartial in the provision of information is common within government.

For local bus services, it seems (there are no doubt other interpretations) to stem from a misguided expectation that “head-to-head” competition would become common following deregulation (in 1986). Head-to-head means two or more operators serving the same route, at about the same time, offering customers a genuine choice for their journey. In reality, very few routes sustain sufficient volumes of passengers to financially support more than one operator in the long run. Short run, such competition does break out. It is rarely profitable, so does not last. Across the whole of the UK, there are only a handful of routes where aggressive head-to-head competition has persisted, south Manchester being the best example. In most cases where there is a choice of operators, competition has matured: As far as competition legislation allows, operators co-operate with one another by providing information about one-another’s services.

So while it is possible for a commercial business to seek an advantage by misrepresenting (or arguably, merely ignoring) another commercial business, the scope for such tactics is small.

Accuracy is more of a problem. If one were to adopt an open, free-for-all approach, inevitably some of the services offered would be biased or wrong. Inaccurate information here isn’t just annoying. If you miss the last bus home because the time you were given was 5 minutes too late, that’s a serious problem. Mostly for you; but also for the commercial operator, for your local politician, for wider transport policy. It is not just a problem for the organisation that gave the bad information.

Internet history is fascinating in this area. 7 years ago most major search engines delivered paid listings. The results of a search were biased based on how much money businesses (mostly) had paid the search engine. Not as adverts, as results. Yet over time people opted instead to use search engines that biased results by the quality of the information: People are, incredibly, not stupid, and will eventually tend to seek out the best source of information. Services that transpire to be hopelessly wrong or biased fall out of use by the majority, so long as there are adequate competitors to turn to.

If there is choice in the market for information, the risk of bad information being used should be reduced: Uncertainty and distrust will naturally lead users to check several sources until they become confident in their preferred source.

Choice

But that isn’t the only reason choice is important.

Chris Anderson popularised the “long tail“. A tiny proportion of internet services will attract the most of traffic, yet the aggregate of all the “unpopular” services may still be very important. The implications are that the more choice you offer, the greater total usage you will have: Aiming one product to be “the best” will always miss opportunities, even if it is successful. And sometimes approaches/products that you didn’t expect to be popular will become so. Particularly if you guide users between the different options. The internet changes the cost model – additional choices that would be prohibitively expensive as physical products, can be delivered at very marginal cost electronically.

So choice is cheap; the more choice, the more use; the more use, the more paying punters.

All this is counter-intuitive for those (I include myself here) who would rather specify perfection, and then spend all our resources trying to attain it.

Without wishing to re-open old wounds, can monolithic projects like TransportDirect really be judged as a success? After 7 years the quality of the information is good, but not without “teething problems“. The official Transport Direct Evaluation shows that a plethora of other travel information services attract more users than TransportDirect (even Transport for London, which only serves about 15% of the UK population). Since that evaluation makes scant mention of the cost, I’ve been forced to summarise it in the box below:

Superficial TransportDirect cost analysis
Every user session (basically, an enquiry) costs £0.20 (40 cents) to answer [3]. Crudely discounting the £40+ million ($80 million) development costs over an operational life of ten years (this is the internet, right?), raises the total cost per enquiry to £0.60 ($1.20). To put those figures in perspective, the total revenue from each of those enquires is likely to be of the order of £2 ($4) [4]. The vast majority of that revenue is paying for core operating costs – like drivers, vehicles, fuel and terminals. A commercial operator spending 30% of their revenue communicating basic information like when their service departs, will be out of business rather quickly, unless they are attracting very high proportions of genuinely new customers [5].

The case for government commissioned super-portals does not sound terribly convincing.

Thinking outside the box

A key benefit of choice and a more open approach, is that it fosters creativity and innovation in how the information is presented.

Let’s look at a few examples.

Mashups

I mentioned the “mashup” (colloquial term) earlier. These combine data from multiple sources and in turn create a unique service or visual output. TubeJP is a simple example (extract shown right). It was built as a non-commercial project by Dean Larman-Moore. It uses elements from Google Maps, some live data from Transport for London (TfL), journey and location point data from timetables and maps, and bespoke journey planner code. Nothing especially revolutionary, but one can see how different data sources can be wrapped together. Critically, this isn’t something TfL are offering from their popular journey planner. TubeJP is different. For better or worse, its existence offers users a choice – in this case, an independent service that maps their route, or an official one that does not.

Choice is something TfL is trying to offer: Its journey planner offers almost 30 different user options. For example, the user can opt to select certain modes, or indicate their willingness to walk. TransportDirect’s “door-to-door” journey planner provides ample evidence of the futility of trying to cover all the possibilities – almost 60 choices are presented, yet one can easily conceive many more rational user preferences. There are strong parallels to the development of search engines, shown in the box below. In both cases, it would be easy to operate a series of websites, each of which presents the same information differently. For example, the difference between a kid’s “Buz4u” service and a “Granny Planner” (the names need work, forgive me) is technically a domain name, an algorithm tweak, and a style sheet. But each would pick up users alienated by the “ultimate” (that is, compromise-based design) travel planner. As TubeJP demonstrates, the motivation doesn’t have to come from government or operators – even without access to data sources, fresh ideas will emerge.

Online journey planner and search engine design parallels
The design of online journey planners strongly parallels early internet search engines. These typically presented the user with options or complex methods of writing queries – which most users completely ignored. However, different search engines appeal to different individuals, in part because of the differences in results they deliver. For example, Google now dominates searches by affluent working-age adults, but holds much lower market share among those aged over 70. Ultimately the solution is a better search. But to quote Larry Page, co-founder of Google, “The ultimate search engine would basically understand everything in the world, and it would always give you the right thing. And we’re a long, long ways from that.” In the meantime, a choice of search engines serves a useful purpose.

Facebook applications

The proliferation of social networking sites is hard to ignore: MySpace, Facebook, Bebo, and others. Around 20% of the UK population visit a social networking site each month [6]. Likely heavily skewed towards the young, who are among the greatest users of public transport. These are developing such that third-party services can be delivered to people within these networks. The best example is Facebook’s application interface. An example in theory: a group could decide visit a location, and a journey planning application automatically calculate personalised journeys for everyone in the group, such that they arrive at the same time from wherever they are now. The delivery of information becomes a natural part of the group’s interactions.

Already Facebook applications exist for organising activities like car-sharing (ride sharing). For some large organisations, these social networking sites could become a replacement for the corporate intranet. That trend would create huge potential for services like automated company travel plans (advising employees on the best option for travel to work).

Data integration

There is much debate as to whether Facebook-like services are likely to develop as platforms in their own right (much like DOS in the 1970s), or whether “walled gardens” (one provider distributing content) will always fail because the wider internet will always offer the most choice (as has been the case thus far in internet history).

I think they will raise the bar of expectations, such that people will be less likely to expect to go to specific websites each time they want certain information.

Perhaps Tim Berners-Lee’s Semantic Web will finally emerge? Each time you combine information sources, you have the potential to create some new online service – a new way of communicating information. So perhaps the internet (and related services) will blossom like a fractal. Some branches will be attractive, others less so. Some won’t get explored at all. But if we acknowledge the long tail, the fractal is genius: Somewhere in there will be the best solution for any one individual, in any one set of circumstances.

As internet-type information increasingly flows through mainstream personal mobile communication devices (it has been about to happen for the last decade, so it might start soon), the uses of this data will become progressively harder to predict. If the augmented reality concepts of the foresight projects like the Metaverse transpire to be correct, intending travellers will learn about local public transport networks by simply glancing at the street. Concepts like visiting a website or “texting” for information (using SMS to query times at a location), will become “so 2000…”

Google Transit?

This isn’t about Google at all. The issues Google Transit raises merely go to the core of what we do with local public transport data. This is all part of a much wider data liberalisation/control agenda. In the case of public transport information, there are real benefits to data liberalisation, even if that means there is no way to recover the costs of gathering the data:

• The more choice of information, the more usage, the more patronage. The cost of providing that choice may be very marginal.
• If data integration is the future, then potential travellers will expect it. Ultimately, if you want to keep and build patronage, you don’t have a choice.

In addition, freely available information is likely to get built in while the technology is developing and maturing, rather than wait for the public mainstream to adopt a new technology and then build information into it.

There’s a risk that:

• The process stalls because it unravels too many “difficult areas”,
• The public sector becomes implicit in the creation of a private monopoly, because the approach is inequitable, or
• The whole exercise might be viewed “a bit like commissioning a travel planner”, and all the other potential benefits lost.

The local public transport sector has a really great opportunity to be right at the centre of an information-driven location-orientated hyper-communicating society. Let’s hope it can see it as an opportunity, and not a threat.

There are many aspects of the topic that this article does not address, such as alternative approaches to gathering and distributing data, and the role of specific organisations within the process: Why should local public transport information be a public sector function, when most operations are commercial? Rather than provide “the solution”, the article aims to feed debate on the reasons why we do what we do.

Further Reading

• Introduction to UK Local Public Transport Data – a non-technical introduction to the United Kingdom’s electronic local public transport data.
• Google Transit Feed Specification – information for developers at Google Code.
• Google Transit Trip Planner Group – discussion, and some documentation.
• JourneyPlan: Delivering the data – Mac Logan (Journeyplan) demonstrates a role for private developers.
• Transport Direct Evaluation – official evaluation project, including background information about competitors to TransportDirect.

Notes

• [1] Based on the comments of Roger Slevin (Traveline South East). This may not necessarily reflect the views of all those involved.
• [2] Bradley Horowitz’s The Tech Lab essay explores some of Yahoo’s approaches to this topic.
• [3] “Non-capital funding” – i.e. revenue cost – has a budget of £2 million in 2005/06. While the service may be growing in popularity, so far during 2007 the number of user sessions per week has been hovering just below 200,000. That’s about 10 million user sessions per year. Hence a revenue cost of £0.20 per session. Assuming 100 million enquiries over 10 years (which assumes no growth, so is likely a slight underestimate, but this calculation is crude anyway), and a development cost of $40 million (likely an underestimate of the final cost, since TransportDirect is still being enhanced), that’s an additional £0.40 per enquiry, or £0.60 total.
• [4] The £2 figure is very, very, approximate. I don’t know the balance of modes (bus/rail/car), the extent to which one-way, multiple-journey or long-distance journeys are being enquired about, the proportion of single travellers vs groups, or the extent to which TransportDirect is disproportionately used by those paying reduced fares.
• [5] The Transport Direct Evaluation cites a net mode shift due to using TransportDirect of 5% of surveyed users (that’s individual people, not enquiries). The net total clouds the patterns within: For every three users attracted to public transport, one is actually lost. Unfortunately, assessing the motivations behind travel behaviour change in one simple survey question is rather simplistic. For example, someone might seek travel information because they have already subconsciously decided to investigate alternatives. The sample is likely to be heavily biased.
• [6] Comscore reports 10 million visitors to each of the two most popular sites (Bebo and MySpace) during July 2007. That’s a third of the online population, a sixth of the total UK resident population. A proportion of users will visit more than one site, so the overall proportion of people using social networking sites in general, is higher. The proportions making significant use of these services will be lower. Nielsen//NetRatings (PDF) notes that in May 2007, over half (53%) of Facebook’s UK visitors also visited MySpace, as did 43% of Bebo’s visitors. 26% of UK MySpace visitors also visited Bebo, 26% also visited Facebook.

Acknowledgements

The author is grateful to Roger Slevin (as project manager for Traveline South East) for some of the factual information about Google Transit in the UK. The opinions expressed here are entirely those of the author, Tim Howgego.

(Further comments are now locked due to heavy spamming.)