Connected Knowledge

Yesterday, I encountered an interesting, chance juxtaposition of ideas from two people I’ve long admired, University of Michigan economist Scott Page and virtual reality pioneer Jaron Lanier. Since their comments seemed at such odds (at first blush anyway), I felt compelled to attempt a reconciliation.  Here’s the quandary.  I’ve been reading Jeff Howe’s book Crowdsourcing, a recurring theme of which is Page’s  "Diversity Trumps Ability" theorem.  In a nutshell, the theorem asserts that under certain, fairly common circumstances groups of experts are often outperformed by larger, more diverse groups of individuals with lesser abilities.  As Page puts it “a randomly selected collection of problem solvers outperforms a collection of the best individual problem solvers.”

While mulling over the various examples of diversity trumping ability that pepper Howe’s book, a friend retweeted a headline from the New York Times entitled The Madness of Crowds and an Internet Delusion.  This was too serendipitous to pass up so I clicked through to the article which turned out to be a discussion of Lanier’s new book You Are Not a Gadget. The Times describes the book as “a manifesto against … the glorification of open-source software, free information and collective work at the expense of individual creativity.” This was at odds with my impressions of the dreadlocked, multi-instrumentalist, technology impresario. So why is an elder statesman of Silicon Valley and one of the earliest and most prominent citizens of the quintessential online community The Well hating on Web 2.0?

His biggest complaint is against “hive thinking” and the “digital Maoism” he sees emerging in the participatory web. In an interview on Amazon.com coinciding with the release of his book, Lanier rightfully laments the dynamic of people becoming “artificially caustic, flattering, or otherwise manipulative” in order to stand out from the crowd.  It is hard to argue with this point. Online communities tend to coalesce around affinity rather than diversity, but Lanier also takes issue with the whole notion of “collective intelligence” arguing that the heart of the Diversity Trumps Ability theorem is fundamentally flawed.

I am amazed by the power of the collective to enthrall people to the point of blindness. Collectivists adore a computer operating system called LINUX, for instance, but it is really only one example of a descendant of a 1970s technology called UNIX. If it weren’t produced by a collective, there would be nothing remarkable about it at all.

Meanwhile, the truly remarkable designs that couldn’t have existed 30 years ago, like the iPhone, all come out of "closed" shops where individuals create something and polish it before it is released to the public. Collectivists confuse ideology with achievement.

Lanier does allow that there are some (rather mundane) cases where the The Wisdom of Crowds can be useful, such as product pricing and movie recommendations, but in any situation involving “creativity and imagination,” he says, “a crowd process generally fails”.

I have not yet read You Are Not a Gadget (it was only released on Tuesday) but by all accounts it sounds like a less vitriolic and better reasoned version of Andrew Keen’s argument in The Cult of the Amateur, in essence that the Internet is killing culture, undermining morality and slowly but surely bringing about the general collapse of civilization .  So how does one reconcile Page’s portrayal of the mass of amateurs, what Howe calls “the Billion”, contributing to the collective intelligence of the web with Lanier’s depiction of Web 2.0 as a partisan mob and a new robber baron class (Google, Amazon, etc) whom he calls “Lords of the Cloud”?  In my view, they are both correct from their respective perspectives.

The diversity Page is talking about is not the ethnicity-based diversity HR departments seem to be obsessed with, at least not exclusively or even primarily.  Rather it is a diversity of cognitive strengths.  i.e. talents, problem solving approaches, core knowledge and simple perspective.  Say you form two teams to play “Who wants to be a Millionaire”, one composed exclusively of Nobel Laureate chemists and one of randomly selected college graduates.  The raw IQ scores of the Nobel Laureate team may be higher than that of the random alumni team, but their knowledge will be so specialized, their approach to problem solving so homogeneous, they will be unlikely to outperform the less expert, more generalized team in areas outside of their particular expertise.  Princeton’s Russell K. Nieli summarizes this phenomenon by saying “divergent yet complimentary talents often produce better and more creative outcomes than convergent and more overlapping ones, regardless of how superlative the latter might be.”

This actually supports Lanier’s condemnation of web-enabled herd mentality.  When he says “Only the quirkiness of considered individual expression can cut through the nonsense of the mob,” Lanier is stating a fundamental condition of successful collaboration.  Howe sums this up nicely in his discussion of the investment service Marketocracy.

Deliberation is the enemy of Collective Intelligence because it reduces diversity.  As individuals confer, they also reach consensus.  One of the chief conditions that allow the crowd to make smart predictions or come up with novel approaches to a problem is autonomy: each makes his or her choice independently.

“Diversity Trumps Ability” does not equate to “Design by Committee”. It is, in fact, its antithesis. By opening problem solving tactics to a broader, more diverse talent pool, whether through crowdcasting, prediction markets or simple social media, you actually reduce the risk of group think by engaging talent outside of the committee (or lab team, or working group or advisory council) bringing to bear a new and likely novel perspective and approach. The vast majority of successful Innocentive challenge solvers are working outside of their own field.  For example, most problems classified as “chemical” are actually solved by trained electrical engineers.  The trick is that if a chemistry-oriented solution would work, it would have been found by the in-house chemists originally working the problem. This approach can often find a viable and even optimal solution faster than in-house, brute force brain power alone.  Several of the Innocentive participants Howe interviewed say that if they don’t have a solution in mind by the time they finish reading the problem, they move on to something else. The secret to collective intelligence and the innovation it can foster is to tap into what the crowd already knows. Bill Joy’s famous aphorism “No matter who you are, most of the smartest people work for someone else” is more applicable than ever.

So what about the "user-generated nonsense" of Keen’s Cult of the Amateur and the mediocrity it tends to engender?  It can be challenging to cut through the clutter of the collective, but fortunately bounded collective intelligence, as I’ve been describing here, tends to be self-correcting.  If a sub-optimal consensus forms due to negotiation, deliberation or inertia, another approach will likely be voiced and eventually take its place. Most consumer markets, the software industry in particular, have demonstrated that if a product isn’t satisfying the needs of its users, a better, faster, cheaper rival will appear to fill the gap and can ultimately take over that market space relatively quickly.  Adam Smith’s Invisible Hand  is just as influential on the Web as it is on Wall Street.  There will always be forays down blind alleys and flocks of red herring, but ultimately natural selection wins out.  Page captures this dynamic succinctly. “The reason diversity trumps ability in a problem solving scenario is because you can always throw the idiots off the bus.”