Social, Mobile and Global – the new human race

This is my talk from the MENA – ICT event in Jordan. The event was great. Almost 1000 attendees from all over the Middle East and North Africa (MENA) talking about Information and Communications Technology (ICT).

The Valley of Death

Charge+of+the+Light+Brigade+Cavalry+ChargeThis poem seem particularly pertinent to the current state of Silicon Valley. Incubators on ones side of the valley. Nothing in the Valley. Growth Capital on the other side.

Most startups are dead as soon as they exit their 3 month stint. The exceptions are rare yet notable.

WHAT HAPPENED TO VENTURE CAPITAL?

The Charge of the Light Brigade

by Alfred Tennyson.


Half a league, half a league,
Half a league onward,
All in the valley of Death
Rode the six hundred.
"Forward the Light Brigade!
Charge for the guns!" he said.
Into the valley of Death
Rode the six hundred.

Forward, the Light Brigade!"
Was there a man dismay'd?
Not tho' the soldier knew
Some one had blunder'd.
Theirs not to make reply,
Theirs not to reason why,
Theirs but to do and die.
Into the valley of Death
Rode the six hundred.

Cannon to right of them,
Cannon to left of them,
Cannon in front of them
Volley'd and thunder'd;
Storm'd at with shot and shell,
Boldly they rode and well,
Into the jaws of Death,
Into the mouth of hell
Rode the six hundred.

Flash'd all their sabres bare,
Flash'd as they turn'd in air
Sabring the gunners there,
Charging an army, while
All the world wonder'd.
Plunged in the battery-smoke
Right thro' the line they broke;
Cossack and Russian
Reel'd from the sabre-stroke
Shatter'd and sunder'd.
Then they rode back, but not,
Not the six hundred.

Cannon to right of them,
Cannon to left of them,
Cannon behind them
Volley'd and thunder'd;
Storm'd at with shot and shell,
While horse and hero fell,
They that had fought so well
Came thro' the jaws of Death,
Back from the mouth of hell,
All that was left of them,
Left of six hundred.

When can their glory fade?
O the wild charge they made!
All the world wonder'd.
Honor the charge they made!
Honor the Light Brigade,
Noble six hundred!

 

The impact of mobile growth on web advertising. Is Android bad for Google?

alternate textGoogle and Facebook can’t turn off the mobile deluge

I just posted on TechCrunch. The article focuses on the Facebook S1 filing and in particular on the risks section that covers the growth of the mobile internet and its potential impact on the web business model.

Facebook itself has been very clear that its advertising revenues are exclusively derived from its web site, and also that an increasing amount of its usage comes from mobile in general and smart phones in particular.

Buried in the article is this point:

Google’s present – and Facebook’s future – involves the painful fact that the very success of mobile platforms in helping human beings be productive, on the go, has a negative impact on the desktop-based advertising programs of the past 10 years. Mobile growth impacts web advertising revenues, except of course for Apple who make money from hardware and software and so benefits from these trends. The reason is simple. We do less ad-centric activities on mobile than we did on the web. And we are less likely to click away on an ad when we are focused on a specific goal on a largely single window device.

The implication of this point is that, absent an advertising solution for mobile, Google’s success in distributing Android, as well as the rise of the iPhone, are directly damaging to Google’s legacy business model. Now, it isn’t as if anybody can turn the mobile internet off or slow its growth. So Google has no option but to be a significant player in mobile, and has no option but to try and drive the revenues it derives from mobile harder than the pace of slow down in web based revenues that result from the trend. But to accomplish that Google, and Facebook will need to innovate in advertising. Web based display ads, text ads and others are really not able to translate effectively to mobile without seriusly undermining the user experience.

This is one of the areas we are focused on at just.me.

Cloud based services – the future of the Internet

Fred Wilson has a response today to Eric Schmidt’s declaration in Edinburgh that Google+ is an “identity service”. He asks and answers his own question.

“whom Google built this service for? You or them. And the answer to why you need to use your real name in the service is because they need you to.”

Of course Facebook is also an identity service. Facebook Connect is the means of distributing it. And of course Facebook too is built using real names because “they need you to”.

At this level FaceBook and Google have much in common, and both are vying for us to use them for online authentication. Facebook is far ahead of course.

Late yesterday I posted an opinion piece as a guest author on TechCrunch. It is about the uncertain future of web services as mobile devices proliferate globally. We will soon all have awesome identity machines in our pocket. They will be capable of being used to authenticate us (even using 2 step authentication). Any cloud-based 3rd party identity system will be unnecessary.

The future of identity is distributed, under user control, and owned and managed by the user from their device. It will be capable of supporting anonymity and real names and will be able to be trusted by sites requiring you to authenticate. The idea of any 3rd party dictating how you can present yourself online will no longer be applicable. Of course, it still has to be built…..

Having said that, there is absolutely nothing wrong with Google and/or Facebook building an identity system that dictates how we present ourselves. Our choice is to use it or not…..We don’t have to.

De-portalization and Internet Revenues I

I am re-posting this from the edgeio blog. Mainly because I think it has current relevance and will in future have historical value.

I’m not certain the edgeio blog will continue to exist, so this is the new home for the post. Since it was originally written we have seen the rise of Adbrite, Glam, Sugar Publishing, Digg and other businesses based on understanding the proliferation of publishing, reading habits, and advertising away from the big portals. I also note that Chris Anderson of Long Tail fame commented on the post, something I failed to notice originally. So, here is the original post

This post is a little more philosophical than most that you will see here. It provides a little bit of background as to why edgeio is in the business of bringing together, organizing and distributing listings to the edge of the network. In short it is because we believe that the Internet is moving away from big centralized portals, which have gathered the lions share of Internet traffic, towards a pattern where traffic is generally much flatter. The mountains, if you will, continue to exist. But the foothills advance and take up more of the overall pie. Fred Wilson had a post earlier this week about the de-portalization of the Internet which is essentially making the same point when seen from the point of view of Yahoo.


Update: 11am Pacific, Sunday 10 December

Several commentators are seeing the word “de-portalization” (first coined by Fred Wilson) and reading “end of portals”. To be clear, and apologies if I wasn’t already, de-portalization represents a change in the relative weight of portals in a traffic sense, and the emergence of what I call the “foothills” as a major source of traffic. This will affect money flows. Portals will remain both large and will continue to grow. But relatively less than the traffic in the foothills. The foothills will monetize under greater control of its publishers and the dollar value of its traffic is already large and will get much larger.


The following 3 graphics illustrate what we believe has happened already and is likely to continue.

The first picture is a rough depiction of Internet traffic before the flattening

2004 and all that

The second picture is a rough depiction of today – with the mountains still evident, but much less so

The rise of the foothills

The third picture is where these trends are leading. To a flatter world of more evenly disributed traffic.

The future pattern of web traffic

Some of the consequences of this trend are profound. Here are our top 10 things to watch as de-portalization continues..

1. The revenue growth that has characterized the Internet since 1994 will continue. But more and more of the revenue will be made in the foothills, not the mountains.
2. If the major destination sites want to participate in it they will need to find a way to be involved in the traffic that inhabits the foothills.
3. Widgets are a symptom of this need to embed yourself in the distributed traffic of the foothills.
4. Portals that try to widgetize the foothills will do less well than those who truly embrace distributed content, but better than those who ignore the trends.
5. Every pair of eyeballs in the foothills will have many competing advertisers looking to connect with them. Publishers will benefit from this.
6. Because of this competition the dollar value of the traffic that is in the foothills will be (already is) vastly more than a generic ad platform like Google Adsense or Yahoo’s Panama can realize. Techcrunch ($180,000 last month according to the SF Chronicle) is an example of how much more money a publisher who sells advertising and listings to target advertisers can make than when in the hands of an advertiser focused middleman like Google.
7. Publisher driven revenue models will increasingly replace middlemen. There will be no successful advertiser driven models in the foothills, only publisher centric models. Successful platform vendors will put the publisher at the center of the world in a sellers market for eyeballs. There will be more publishers able to make $180,000 a month.
8. Portals will need to evolve into platform companies in order to participate in a huge growth of Internet revenues. Service to publishers will be a huge part of this. Otherwise they will end up like Infospace, or maybe Infoseek. Relics of the past.
9. Search however will become more important as content becomes more distributed. Yet it will command less and less a proportion of the growing Internet traffic.
10. Smart companies will (a) help content find traffic by enabling its distribution. (b) help users find content that is widely dispersed by providing great search. (c) help the publishers in the rising foothills maximize the value of their publications.

Discussion

Kevin Burton
Techmeme
Mike Arrington
Syntagma
Keith Teare’s Weblog
Dan Farber at ZDNet
Mark Evans
Fred Wilson
Ivan Pope at Snipperoo
Tech Tailrank
Collaborative Thinking
David Black
Surfing the Chaos
Ben Griffiths
Dave Winer (great pics)
Kosso’s Braingarden
Dizzy Thinks
Mark Evans

Dare Obasanjo

OpenID and Data Portability

Nicolas Popp – a leading advocate of Open Identity and data solutions – posted on his VeriSign blog today following the rather heated discussions that have ensued since Google announced its Friend Connect product recently.

Nico’s employer – VeriSign – along with Microsoft, Yahoo, Google, AOL and others, is a member of the board of the OpenID foundation.Nico’s primary argument (emphasis mine) is that:

Undoubtedly, data portability is the natural child of federated identity (more on that in a future post). Personal and social data are an important part of any consumer identity’. Like identifiers, credentials and profile attributes, social graphs, activity streams belong to the end user who created them in the first place. In the long run, consumers will require full control, privacy, security and portability over such personal information. Therefore, the identity technical community must engineer a new and comprehensive identity portability layer. The new layer needs to broaden the tradition notion of identity federation beyond names, passwords and profile to encompass the full gamet of personal and social data. Furthermore, this new layer must support a plurality of identity service providers who can compete and distinguish themselves by the quality of their service and the user experience that they provide. Freeing our data off Web portals and social networks by creating a new service layer dominated by one single service provider is hardly trading one master for another.

I am in full agreement with this approach. And .. as coincidence would have it, last week I registered the domain name – itsmygraph.com – with a view to beginning to participate in this discussion. I have an early draft of my thoughts. They are at sites.itsmygraph.com. But as a teaser – here is my high level view of the evolution of Internet Users:

I would love to get feedback on your thoughts about the future of data portability and its relationship to OpenID and OAuth.

My personal view is that Michael Arrington had it right when he said recently:

I’ll say what the OpenID Foundation cannot, for political reasons – It’s time for these companies to do what’s right for the users and fully adopt OpenID as relying parties. That doesn’t fit in with their strategy of owning the identity of as many Internet users as possible, but it certainly fits in with the Internet’s very serious need for an open, distributed and secure single log in system (OpenID is all three).

If and when the Big Four become relying parties, the floodgates will truly open and there will be no looking back. And until they do that, I’m not buying that they really support what OpenID is trying to accomplish.

The Conversation:

Techmeme
TechCrunch
ReadWriteWeb

Google and the newspapers

t_google_logo_hires-1.jpg

Over the long labor day weekend Google announced a serious change in the way Google News will relate to the various wire services and the newspaper industry. The change could have a dramatic impact on the traffic Google News sends to newspaper web sites. There have been several commentaries on the developments and Techmeme has been a great source tracking them.


The New York Times, ironically carrying a Reuters syndicated article, said


                Google is playing host to articles from four news agencies, including The Associated Press, the company said Friday, setting the stage for it to generate advertising revenue from Google News.

            The news agencies — the Press Association of Britain, Canadian Press, Agence France-Presse and The A.P. — now have their articles featured with the organizations’ own brands on Google News. The companies have agreed to license news feeds to Google.
            The five-year-old Google News service previously searched the Web to uncover links to news articles from thousands of sources, and clustered links on similar subjects together.”

The impact of the decision was also well understood by the Reuters writer:

            “Because of Google’s campaign to simultaneously reduce duplicate articles, the original wire service article is likely to be featured in Google News instead of versions of the same article from newspaper customers, sapping ad revenue to those newspapers.”

It is worth viewing this incredible serious news in context. Newspapers are reeling from a very recent and sudden drop in classified advertising revenues. Many of the top publications saw falls of more than 20% in Q2 alone. Last week the Washington Post’s Sam Diaz published a story covering the most recent trends. Techmeme’s gathering together of the conversation is here. The Washington Post piece is here. The conclusion, which Don Dodge articulates well is that:

    “The Next Big Thing – Online classified ads, local search, and mobile search are huge markets, with no dominant leader, and lots of opportunity for innovation. New business models will emerge and a new set of leaders will reap billions in profits…pennies at a time.”

Or, to put the same thing another way, newspaper classified revenues are likely to continue to decline as online classifieds revenue grows. If true this will represent a blow to the newspapers to compound the impact of the Google News decision.

 

My own research – and as ceo of edgeio I have a strong stake in the game so I have done a lot – suggests that these trends are both strong and relatively irreversible. But it also suggests that online classified revenues are not yet growing sufficiently to stem the decline in print classified revenues.

Here is the entire US advertising spend since 2001. It has grown every year.

The newspapers share of this spend has declined.

More graphically:

Now, within that decline there has been a growth of online revenues for the newspapers as the following chart shows:

But, this growth has been too small to make a big difference. If the newspapers are to survive they need to attract more traffic and more advertisers than they have so far been capable of. Dodge suggests that this will be achieved by becoming more “local”, even “hyper-local”. I have to say that I believe he may prove to be right, but not for many years. There is no current evidence that individuals are adopting a strong local mindset when using the Internet. Rather the contrary. The growth of blogs like Techcrunch (disclosure: i am a shareholder) GigaOm, ReadWriteWeb, PaidContent.org and many others suggest that online behavior is shaped more by ones work/interests/hobbies/passions than where one lives. Vertical editorial content is growing in traffic. Local editorial is not growing so fast.

Google has benefitted most from the growth of vertically focused traffic, as the following chart shows. Search advertising – including that embedded in hundreds of thousands of publishers sites, has grown whilst classified advertising has stayed flat as a share of Internet Advertising over the 2001-2006 period.

 

Classifieds are of course not only local. Magazines have perfected the ability to target ads at vertical interests for many years. Boat ads in boating magazine, job ads in almost every vertically focused publication, car ads in classic car magazines, furniture ads in House and Home publications and so on.

 

Beyond all of this there is still a big opportunity for newspapers to grow revenue.

As the chart alongside shows, the newspapers still command a giant number of absolute dollars. There is time to figure out how to do classifieds online. The stand-alone classified sites (eBay, Craigslist, Monster, Careerbuilder) have built awesome businesses that mirror offline publications like Autotrader in the US and Exchange and Mart in the UK. They are stand-alone, classified-only environments.  Together they accounted for only $3.1 billion in revenues in 2006.

There is a much larger pie to be figured in discovering how to embed classified advertising in editorialized environments in a manner that is targeted at the readers of those publications.  In other words, to do online what the magazine industry has done so well offline. This may well be an opportunity greater than $15 billion a year in new revenues.

So… it is still all to be played for, and it will be fun being involved. The online classified players, the newspapers and the magazines are all likely to be involved as this plays out. And all stand to grow revenues.

The Pareto Principle is nonsense.

In response to the current discussion on Techmeme and TailRank hipmojo writes that the Pareto principle is in play on the internet and that no matter how much we want it to be otherwise 80% of online advertising will go to 20% of the web sites.

When the dust settles, the top 20% of websites will get 80% of ad revenues. It’s that simple. Portals might change in shape, form or nature, but whatever they represent loosely will still get the bulk of revenues and traffic.

With respect, that is nonsense. Since the advent of Google Adsense the shape of internet advertising spend has mirrored the flattening of traffic I speak of on the edgeio blog. Almost half of Google’s revenue comes from Adsense. And about 75% of the dollars earned through Adsense stay with the publishers whose sites the ads run on. Clearly the lions share of the money spent through Google is shared about 50-50 with the publishers in the “foothills”.

It may be worth listening to the Google Earnings calls on Earningscast to validate this.

That is why Google talks so much about “inventory”. That is, traffic from outside google.com. The size and cost of this inventory is a major variable and the need to grow it helps us to understand deals like the one with YouTube.

If you roll the clock back to the pre-Adsense days when DoubleClick ruled, and online advertising was only going to large sites, it is a huge change in monetization and traffic flows. Give Google credit for this.

One of the things my piece argues is that there is a new trend on top of this established one – publisher monetization of their own content through direct relationships to advertisers (job boards, sponsorships and Techmeme like ad units being examples).

Sure the portals are still big but the collective foothills are as big now, and will be a lot bigger in the future.

De-portalization and Internet revenues

Last week Fred Wilson did a post on a phenomena he called de-portalization. I think he is right on the money.

I just posted a piece on the edgeio blog that picks up on that theme and discusses the consequences of the trend.

The top 10 consequences are:

1. The revenue growth that has characterized the Internet since 1994 will continue. But more and more of the revenue will be made in the foothills, not the mountains.
2. If the major destination sites want to participate in it they will need to find a way to be involved in the traffic that inhabits the foothills.
3. Widgets are a symptom of this need to embed yourself in the distributed traffic of the foothills.
4. Portals that try to widgetize the foothills will do less well than those who truly embrace distributed content, but better than those who ignore the trends.
5. Every pair of eyeballs in the foothills will have many competing advertisers looking to connect with them. Publishers will benefit from this.
6. Because of this competition the dollar value of the traffic that is in the foothills will be (already is) vastly more than a generic ad platform like Google Adsense or Yahoo’s Panama can realize. Techcrunch ($180,000 last month according to the SF Chronicle) is an example of how much more money a publisher who sells advertising and listings to target advertisers can make than when in the hands of an advertiser focused middleman like Google.
7. Publisher driven revenue models will increasingly replace middlemen. There will be no successful advertiser driven models in the foothills, only publisher centric models. Successful platform vendors will put the publisher at the center of the world in a sellers market for eyeballs. There will be more publishers able to make $180,000 a month.
8. Portals will need to evolve into platform companies in order to participate in a huge growth of Internet revenues. Service to publishers will be a huge part of this. Otherwise they will end up like Infospace, or maybe Infoseek. Relics of the past.
9. Search however will become more important as content becomes more distributed. Yet it will command less and less a proportion of the growing Internet traffic.
10. Smart companies will (a) help content find traffic by enabling its distribution. (b) help users find content that is widely dispersed by providing great search. (c) help the publishers in the rising foothills maximize the value of their publications.

Discussion

Kevin Burton
Techmeme
Mike Arrington
Syntagma
Dan Farber at ZDNet
Mark Evans
Fred Wilson
Ivan Pope at Snipperoo
Tech Tailrank
Collaborative Thinking
David Black
Surfing the Chaos
Ben Griffiths
Dave Winer (great pics)
Kosso’s Braingarden
Dizzy Thinks
Mark Evans

Is scraping and crawling stealing?

A spat has blown up over the weekend regarding Oodle and Vast.com “scraping” content from 3rd party sites and re-purposing it inside their environments. This essay is my reaction to the spat. As a founder of edgeio I clearly have an interest in the answer to the question. edgeio does not scrape or crawl. All of its content is permission based (published using the “listing” tag; uploaded directly into edgeio OR published on edgeio directly to a personal listings blog that we host).

However, there is more at stake here than competitive issues between edgeio on the one hand and Vast/Oodle on the other. The wider issue is whether or not scraping (which is very like crawling and indexing except it reads displayed content not files) constitutes stealing of data.

The following is taken from an article on ClickZ:

“This is called stealing content…there’s no advantage to me to have them steal,” commented Laurel Touby, founder and CEO of media industry site mediabistro.com, upon learning that Vast.com had linked from its search results to full mediabistro.com job listings pages, even though those pages require registration when accessed on the mediabistro.com site.

Vast.com CEO Naval Ravikant said Vast.com’s crawlers do not automatically register or login to sites, so they must have found passage through the mediabistro.com system via a legitimate entryway.

So let’s try and address this broader issue. Firstly this is a new discussion. Nobody accuses Google of stealing the data that is in it’s index (except book publishers of course). Why not? Well, because Google primarily indexes the “visible” web. That is to say, sites that are linked to from other sites and are not behind a password protection system of any kind; and even then it respects directives in a file called robots.txt where a publisher can ask not to be indexed. And secondly, Google does not display entire documents (although its cache is getting very close to doing so and may give rise to similar discussions in future). Rather it points to the original source for reading/viewing the content. Thus the business model of the original publisher is left in tact.

With the emergence of vertical search aggregators, especially in the commerce space, the issue of ownership and permission become far more pronounced. Why? Because the data represents an inventory, and often an “invisible” web inventory – that is to say, behind a password protected site. The effort to aggregate that inventory into a central marketplace is done without permission of the owner of the inventory. Whether password protected or not this is going to give rise to disputes like the one between Craigslist and Oodle a little while ago.

There is no need to invent new means of dealing with this. But there is a need for good behavior. Crawlers always should respect robots.txt. Scrapers are different. The spiders can read displayed contentl directly and do not crawl the file system. As such they can bypass robots.txt. If scrapers respected robots.txt then a publisher could effectively put its content out of the reach of the crawlers. It isn’t clear at this point whether the scrapers do respect robots.txt files. A better solution is to use RSS for syndication rather than crawling and scraping. More on this below.

The second issue is whether the item level link from a result set points to the original source or to a hosted copy of the original. Oodle and Googlebase had a difference of opinion about this issue. Content publishers will care what the answer is.

The third issue with scraping is a quality issue. On its home page Vast.com states:

All results are automatically extracted by crawling the web
Vast.com cannot guarantee the accuracy or availability of the results

And the oodle blog notes that its index:

…only includes listings that are fresh and relevant: we keep track of all the listings we’ve seen and auto-expire old ones that are still online and exclude things that look like listings but aren’t (reviews, spam, etc.).

The issue here is twofold. To stay current with a live inventory of listings is hard. To even attempt to do so creates a need to crawl and index very aggressiveley, and the results are often not good. Craigslist’s gripe with Oodle was at least in part driven by its experience with Oodle’s crawlers. They were apparently polling and sucking content very aggressiveley and needed to in order to stay current. If you do not poll aggressiveley your index gets even more out of sync with the original source than it already is.

It seems to me that RSS is a custom made solution to these problems. Scraping and Crawling are the wrong tools.

If publishers who wish their content to be syndicated to a third party publish an RSS feed and the third party consumes the feed we have a) a permission based syndication system; b) a real time ability to update inventory. edgeio made the decision to follow the Craigslist model whereby a listing is explicitly requested by a publisher. Publishers of listings, from your Mom to a large site, are a community, made up of many smaller communities. A central listings service (CLS) should be a service to that community. Permission based, real-time publishing, via RSS, is the right tool for the job. Over time this is a highly scalable solution. Publishers can opt in and out at will.

I predict many more accusations of stealing insofar as the industry continues to mine the “invisible” web, and the specialist web, via scraping and crawling.

And finally, edgeio publishes RSS feeds of every item (either individual items, or our entire inventory). Oodle and Vast are not competitors, but distribution partners. Our data is more valuable insofar as more people see it. That will happen if the data is placed in more environments. So, take it, for free. But please, do not scrape it or crawl it. Just read the RSS feeds. That is why we have them.

Last word goes to And Beal’s The Marketing Pilgrim blog. After reading the ClickZ piece he says:

Ouch!

Certainly Vast is not alone in convincing classified sites that they’re helping them bring new visitors, but if the classified search engines are to see a bright future, they’ll need to secure strong partnerships with their partner sites.

My emphasis!

Update:

Tech.memeorandum link for this subject.