That hasn’t stopped “screen-scraping” of data or simply typing in paper timetables (from Urban Mapping to many listed here). Unfortunately, the legal basis for scraping is complex, which creates significant risks for anyone building a business. For example, earlier this year, airline Ryanair requested the removal of all their data from Skyscanner, a flight price comparison site that gathers data by scraping airlines’ websites. How many airlines would need to object to their data being scraped before a “price comparison” service becomes unusable?

User-generated mapping content is evolving, often to circumvent restrictive distribution of national mapping. Services include OpenStreetMap and the recently announced Google Map Maker.

Micro-blogging, primarily through Twitter, has started to show the potential of individual travellers to report information about their journeys: Ron Whitman’s Commuter Feed is a good example. Tom Morris has also experimented with London Twitter feeds.

This article outlines why the “social web”/tech-entrepreneur sector may wish to stop trying to use official sources of data, and instead apply the technology it understands best: People.

The Big Picture

I will use the example of UK local bus data to summarise the strategic issues for data providers. I can only presume the issues are similar elsewhere (comments welcome).

Explaining exactly who the data providers are is one of the many problems of trying to extract and use the data. I would provide more detail, but the topic is somewhat sensitive. The most critical point in the chain that constructs and distributes the data are local authorities – sub-regional public bodies, typically those responsible for large cities, conurbations or counties. They process the data, but are not under any statutory requirement to do so (no national government legislation requires it).

There are a number of issues for the existing data providers:

1. Mindset of centralised control: Most operators, public authorities, and other agencies, still have a mindset of centralised control of information, delivered to users via the method the agency believes is appropriate. This is heavily driven by the belief that only the agency can be accountable or impartial, and that incorrect information supplied by an uncontrolled third party is likely to damage the image of local transport service and generally reflect badly on the agency.
2. Mindset of local: Most agencies are locally focused, locally orientated. It seems logical for them to commission a fully-functioning website or piece of information delivery software that is specific to their city, because their target market is local. There’s a lack of global perspective: An agency will typically commission a system that is specific to their city, even when 95% of the features would work for any city, and 90% are already in existing global products.
3. Not appreciating trends in delivery channels: There is still an attitude of “we’ll provide a website”, without a comprehension that the number of channels for delivery of information is exploding far faster than any one agency can hope to construct bespoke user interfaces to cater for. Mobile devices, integration into social software. There would probably a market for a “WiFi-enabled” alarm clock that would ring later if your morning train had been delayed: We simply can’t define the limits for how this information might be used.
4. Not appreciating trends in cost: Even large, well-funded agencies are starting to fall behind the technology. The cost of systems (many millions of dollars invested year on year in some cases) is starting to hurt. Logically the global system should win out, because one city is very much like another: There is considerable scope for sharing systems costs.

What It Means

Long term we are heading for global providers of information, that pool data from local sources. That will be forced by the cost of technology. This can be seen in technology costs driving things like agglomeration in the groceries sector (such as Walmart) over the last 30 years. Also in the move from customised mainframe computing, to shared operating systems and platforms (such as Windows). This will be worse, because the number of systems will be simultaneously exploding alongside the complexity of those systems.

As these issues become progressively better understood, data will become more centralised. Even in agencies where (in my opinion) uniqueness and absolute control are culturally in-breed, such as London Transport/TfL, cost will eventually win the argument.

However, centralised data handling does not automatically make the data open. Quite the opposite.

Contracted Provision

Currently, effective control of data is with local government. Many individuals within local government will naturally attempt to block any change that might leverage power away from them and their organisation. “Job protection” is an over-simplification, but helps explain the underlying position. But by contracting data handling and presentation to a third-party contractor, local government would gain the technological “economies of scale” (assuming the contractor won many contracts from different authorities) and notionally maintain control.

Use of third-party contractors is already common within the local government sector, particularly for Information Technology.

An example can be seen in Edinburgh City Council’s Traffic Map. In spite of how it appears, the information isn’t powered directly by Edinburgh City Council or Google. Instead it is part of Mott MacDonald’s Common Data Management Facility, providing services under contract to many different local authorities.

In the UK public transport arena, Trapeze is a good example of the gradual agglomeration of data handling within a few large businesses, where historically many small software providers could be found.

The example above provides key driver information, and is somewhat useful, but is it the best outcome? I suspect not. Contracts tend to be priced highly, because local government clients are high risk: Their political control means that they can change their strategic direction and requirements unexpectedly. At best, customer feedback loops through local authorities are slow and politicised. At worst the design of the system will reflect the arbitrary views of a self-proclaimed expert (such as myself). Even if you think it is perfect, there is no scope for choice or creativity. Choice is good and need not be expensive.

Social Provision

Instead of using official data, why not let users reconstruct it? User-generated content is cheaper to create than information from professionally staffed sources: Since very many contributors do so little work, no individual expects payment. User-generated content can be just as accurate too, although this is not automatic: For example, a strong community will subject everything to peer review, weeding out poor information and contributors.

This is not an entirely theoretical position. There is a largely untapped human resource, just waiting to help.

The transport enthusiasts (transit fans, “spotters”) already collate and produce some extremely high quality information about certain technical aspects of operations and services. For example, sites such as LondonBusRoutes.net contain detail on the bus route timing and vehicle allocation (type and number of buses), which transpires to be difficult to extract from official sources. While it may be argued that these sites simply repackage official information, their very existence is a testament to the strength of underlying community.

Casual observation of people delayed on trains or in traffic suggests they derive some comfort from picking up their mobile (cell) phone and telling someone about it. Something they can do, in a scenario they otherwise have no control over. Their desire to communicate the same information to drivers or users 10 miles behind them (who might be able to re-plan their route, should they know) is untested. But the potential is intriguing.

Nobody has entirely worked out how to use these people; yet.

Battle Lines

If the social web/tech-entrepreneur sector chooses to fight the “status quo” head on, it does so against large multi-national IT providers who support clients with historically entrenched positions. Not a contest that favours the underdog.

If the tech’ “upstarts” can find a way to use this human resource effectively, they will ultimately provide a more cost-effective solution than the traditional “government IT” sector can offer. Integrate that user-generated information into the wider consumer internet, and the machinery of government simply won’t be able to justify its historic position of pouring millions into systems it controls. The “social web”/tech-entrepreneur sector wins.

The upstarts do not need perfect source data, if the implementation of results is considered to be better by users. The early Xephos vs TransportDirect comparisons provide some evidence. The success or failure of the social web/tech-entrepreneur sector is ultimately dependant on whether they can provide better information than official sources, using the resources and skills they have available to them.

Disclaimer: The contents of this article reflect my own personal analysis of the situation. This does not directly reflect advice to, or views of, government or anyone else involved in the handling and provision of public transportation data.

Advantages of Web 2.0

Web 2.0 is characterised by the:

• low cost of acquiring large numbers of users,
• ability to generate revenue through advertising/e-commerce,
• use of online metrics as feedback loops in product development,
• sustainable long term profitability (at least for some).

Dave McClure did not actually try and define the term, which was probably wise. Generally the term is applied to websites and services where users collaborate or share content.

Web 2.0 has a number of advantages (although it could be argued that some of these apply to earlier iterations of the internet too):

• APIs – the ability to act as a web-based service, rather than just a “website”.
• PC-like interface, albeit still 5 years behind contemporary PC interfaces.
• RSS feeds (for data sharing) and widgets (user interfaces embedded elsewhere).
• Use of email mailing lists for retaining traffic. While email certainly isn’t a “web 2.0″ technology, his argument is that email is increasingly overlooked as a means of retaining website visitors.
• Groups of people acting as a trusted filter for information over the internet.
• Tags (to give information structure) and ratings (to make better content stand out).
• Real-time measurement systems rapidly giving feedback. Key is the immediacy of the information, and the ability to evolve the web service to reflect that.
• Ability to make money from advertising, leads and e-commerce. While true since about 1995, the web user-base is now far larger, so the potential to leverage revenue also greater.

Metrics for Startups

I believe the ability to very accurately analyse website usage, implement changes, and then analyse the results, is a key advantage of web-based services. It is an advantage often overlooked by information technology professionals and programmers. I’m not sure why – possibly because web service developers:

• don’t appreciate how hard/expensive gathering equivalent information is in other sectors of the economy, or
• are scared to make changes in case they loose business, and/or believe their initial perception of what “works” to be optimum, or
• just lack the pre-requite analytical curiosity to investigate?

Or perhaps Web 2.0 just isn’t mature enough yet for developers to have to worry too much about optimisation: A new concept for a site will probably either fail horribly or generate super-normal profits. The sector isn’t yet competing on very tight margins, where subtle optimisation can make or break profitability. Of course, optimisation of websites can deliver substantial changes in user behaviour. For example, I have found that a relatively subtle change to the position of an advert can alter the revenue generated by over 20%.

Dave McClure developed the AARRR model. AARRR segments the five stages of building a profitable user-base for a website:

1. Acquisition – gaining new users from channels such as search or advertising.
2. Activation – users’ first experience of the site: do they progress beyond the “landing page” they first see?
3. Retention – do users come back?
4. Referral – do users invite their friends to visit?
5. Revenue – do all those users create a revenue stream?

For each stage, the site operator should analyse at least one metric. The table below gives some possible metrics for each stage, with a sample target conversion ratio (the proportion that reach that stage).

These metrics become critical to the design of the product. Poor activation conversion ratio? Work on the landing page(s): Guess at an improvement, test it out on the site, analyse the feedback, and iterate improvements. Gradually you’ll optimise performance of the site.

I find this attempt to structure analysis and relate it back to core business performance, very interesting. However, the sample metrics can be improved on a lot, depending on the nature of the site. For example, to track virality (referral), I might watch the monthly number of del.icio.us adds, or monitor the number of new links posted on forums (Google’s Webmaster tools allow that). Tracking users all the way through the tree from arrival to revenue generation needs to done pragmatically where revenue is generated from very infrequent “big-ticket” sales: With minimal day-to-day data, it can take a long time to determine whether a change genuinely has improved long-term revenue, or whether natural fluctuations in day-to-day earnings just contrived to make it a “good day/week/month”.

Now I know this approach works, but why it works is less clear. We might like to think that we are genuinely improving the user experience, and maybe we are. However, it could be argued that merely the act of change is perceived by users as an improvement – a variation of the Hawthorne effect. The counter argument to the Hawthorne effect can be seen on sites with low proportions of repeat visitors: The majority of those experiencing the improvement will not know what was implemented before.

History of Social Networking

Dave McClure’s interpretation of the timeline of the development of social networking sites is as interesting for what it includes, as for what it omits: No Geocities; no usenet; no forums; no MUDs… The following timeline shows key services in chronological order, except without dates – all the services shown were created within the last ten years:

• Email lists (Yahoo Groups)
• 1.0 Social Networks (Friendster) – these early network established the importance of up-time (service reliability) and the ability of users to manipulate pages.
• Blogs – links between weblogs acting as networks.
• Photos and video (Flickr, YouTube) – created a sense of community, and allowed tagging/grouping of content.
• 2.0 Social Networks (LinkedIn)
• Feeds and shared social information (Upcoming.com event planner)
• Applications and widgets – the ability to embed data about a user’s friends in applications is probably “the most powerful change on the internet in the last ten years”.
• Hosted platforms (OpenSocial, Facebook) – most services are likely to allow 3rd-party developers to provide applications on their platforms.
• Vertical communities (Ning) – ultimately this may develop such that a service like Facebook acts as a repository for a user’s online identity, while specific groups of people gather on other networks.
• Availability of information – a single sign-on, with automatic data transfer between services.

The future may be “Social Prediction Networks”. This is a variation on the theme of using trusted networks to filter content: Instead of Blogging meets Search, I characterise Social Prediction Networks as Digg meets Facebook. Shrewd observers will note Facebook has already implemented Digg-like features, while simultaneously topic-specific, community-orientated Digg-clones are being launched. People gather into interest groups around a topic, and then through use of tagging and rating, the community filters content. The system effectively predicts what other people in the group will find useful. This may be an optimum approach for groups above the Dunbar number (or an equivalent number representing the maximum number of people a person can form stable relationships with).

Interesting Aspects of Facebook

Three were discussed:

1. Social graph (friend list) – email and SMS (mobile phone) service providers have rich data on the frequency of communication between people, yet aren’t using this information to form social networks. Dave noted that two major email service providers, Yahoo and AOL, are currently struggling to thrive – this could be an avenue for their future development.
2. Shared social activity streams – knowledge of what your friends think is important. Friends are more likely to influence you than people you do not know.
3. API/Platform – dynamic behaviour and links across your social network.

Further Observations

Will growth in social networks continue? Yes – the friend list adds value to the content.

Will others compete? Probably, as a “long-tail” of networks, likely topic-specific.

Can social networks be monetarized better? Currently social networking services generate far less revenue than search services. The challenge for social networking sites is to move towards the wealthy territory of search services. At the same time, search services are moving towards becoming more like social networking sites.

How can traditional companies engage with social networking sites? Social networking sites work best for sales where a product has a strong aspect of peer pressure in the decision to buy. The most important advice is not to create a copy of a website: Instead provide less complex content that uses social networks to draw users to a website.

Applications for social networks tend to be over-complicated, normally because programmers attempt to implement functions found in software they have previously written for other platforms or websites. Generally the successful applications are very simple. Some developers have opted to break complex applications into a series of smaller applications, and use the virality of social networking sites to build traffic for one application from another.

Social network applications are exceptionally viral. They can gain users very rapidly, yet also loose users just as fast. Much of this virality comes from feeds, which typically alert friends when a user installs an application. Within a few years the feed is likely to be based on actual usage of an application.

Facebook now allows applications to be added to “fan pages” (or product pages) – so individual users need not now be forced to install an application to use it.

Those using email lists for retention are best to focus on the title of the email, and not the content. Merely make it easy to find a URL in the content. The key decision for the reader is whether to open the email. What the email says is almost irrelevant – they’ve already decided to visit the site based on the title.