New Open Data in Science Task Group Proposed.

February 1st, 2010 No comments

I received an email over the weekend from Jonathan Gray, Community Coordinator for The Open Knowledge Foundation, announcing the formation of a new task group on Open Data in Science.

The Task Group will:

  • Act as a central point of reference and support for individual scientists and research institutions interested in open data in science.
  • Identify practices of early adopters and document existing open scientific datasets.
  • Develop principles for open data in science, and legal and technical guidance for opening up scientific data.
  • Conduct and disseminate research into incentives and obstacles to opening up scientific data.

The group will strive to strengthen the international network of individuals and organizations who support open data in science – bringing together leading figures in this field as well as young researchers from a diverse range of scientific disciplines.

The final proposal is still forthcoming, but this could prove to be an extremely valuable resource to Open Science.






Bringing the Goundswell Behind the Firewall

January 27th, 2010 No comments

Enterprise 2.0

Book Review:
Enterprise 2.0: New Collaborative Tools for Your Organization’s Toughest Challenges

Andrew McAfee
Harvard Business School Press, 2009
ISBN-10: 1422125874, ISBN-13: 978-1422125878


In 2003 publisher Tim O’Reilly began using the term “Web 2.0” to describe the emergence of the “participatory web.”  Blogs, Wikis, reviews and recommendations were turning passive consumers into active contributors.  O’Reilly sponsored a conference around the concept and a buzzword was born.  For the past couple of years, MIT’s Andrew McAfee has been championing the use of these tools behind the firewall as a means to foster innovation and efficiency within organizations.  McAfee has summarized his research into the participatory enterprise in his new book Enterprise 2.0: New Collaborative Tools for Your Organization’s Toughest Challenges.

Enterprise 2.0 isn’t yet a full-fledge buzzword, but it’s getting there.  Enterprise social computing platforms are all the rage (there’s even an Enterprise 2.0 Conference) and vendors like SocialText, Jive and even Microsoft have rushed new platforms to market that promise to usher in a new age of openness and collaboration for even the stodgiest of corporations. A lot of organizations are drinking the Kool-Aid and rolling out these tools without much thought as to what they hope to accomplish, much less how Web 2.0 will help. This was the fatal flaw of the knowledge management frenzy of the 90s.  Too much emphasis was placed on tools and technology and too little on the people they were supposed to benefit.  This point is not lost on McAfee.  It is in fact the central premise of his outstanding book.

 

Enterprise 2.0 is not primarily a technological phenomenon.  … The appearance of these novel tools is a necessary but not sufficient condition for allowing new modes of interaction, collaboration, and innovation. … the mechanisms of emergence are organizational and managerial, rather than purely technical.  In other words, leaders can’t simply assume that healthy communities will self-organize and act in a coherent and productive manner after Web 2.0 tools are deployed.

 

Throughout the book, McAfee uses case studies of organizations such as  Google, Serena Software, VistaPrint and the US Intelligence Community, to demonstrate the central concepts of Enterprise 2.0, their benefits and pitfalls and how best to approach their adoption. He defines Enterprise 2.0 as “the use of emergent social software platforms by organizations in pursuit of their goals.” These Emergent Social Software Platforms (ESSP) encompass all of the tools making up Web 2.0, but the concept goes beyond a roster of applications and services.

 

Social software enables people to rendezvous, connect or collaborate through computer mediated communication and to form online communities.  Platforms…are digital environments in which contributions and interactions are globally visible and persistent over time.   Emergent means that the software is freeform and contains mechanisms like links and tags to let the patterns and structure inherent in people’s interactions become visible over time.

 

Since any new paradigm is only as good as the number of catchy acronyms it fosters, McAfee organizes the six most common technical features of these tools under the sobriquet SLATES (Search, Links, Authoring, Tagging, Extensions, Signals). The first four of these features are by now familiar to most people on the web, but the terms Extensions and Signals may not resonate immediately, even though we likely use them daily.  Extensions are the semi-automated classifications and recommendations that drive the “you may also like” features of  Amazon.com, Netflix and StumbleUpon.com.  (ie.  “If you liked this, then by extension you would like this.”)  Signals are simply notifications of new content such as RSS feeds and SharePoint alerts.  While this list of characteristics provides a useful framework for the discussion of Enterprise 2.0  the real value of McAfee’s work is his analysis of how they can be used to achieve an organization’s goals.

Central to McAfee’s conception of ESSPs and the connected enterprise is the concept of tie strength as first articulated in Mark Granovetter’s 1973 study “The Strength of Weak Ties”  (SWT).  I discussed SWT briefly in an earlier post, but in a nutshell tie strength refers to the Bullseye different levels of relationships we maintain in both our personal and professional lives. These range from close friends (strong ties) to casual acquaintances (weak ties).  McAfee also accounts for those people who would be valuable associates if only we knew about them.  Taken together these various types of interpersonal relationships form the “Enterprise 2.0 Bull’s-Eye”.

Most enterprises attempting to promote collaboration focus on teams, usually through providing new tools or sponsoring rope-climbing retreats.  The idea has always been to build tight-knit groups; to reinforce strong ties.  McAfee argues that while these relationships are critical and should be supported, say through the adoption of collaborative authoring environments, weak ties should not be neglected.  The problem is that everyone in a tight-knit group already knows everyone else.  It is far less likely that all my close friends or teammates (strong ties) have relationships with my casual acquaintances (weak ties) the people on the outer-edge of my social network.  Even though my association with these folk of the fringe is tenuous, those relationships provides a link between my core group and others that might not otherwise be available. “Strong ties are unlikely to  be bridges between networks,” McAffee says. “Weak ties are good bridges”  He goes on to cite Granovetter.

 

The weak tie between [a person] and his acquaintance, therefore, becomes not merely a trivial acquaintance tie but rather a crucial bridge between the two densely knit clumps of close friends…these clumps would not, in fact be connected to one another at all were it not for the existence of weak ties. …  social systems lacking in weak ties will be fragmented and incoherent.  New ideas will spread slowly, scientific endeavors will be handicapped, and subgroups separated by … geography or other characteristics will have difficulty reaching a modus vivendi.

 

McAfee also draws on the work of anthropologist Robin Dunbar to bolster his claims about the corporate value of social computing.  According to Dunbar the theoretical maximum social group size for humans is between 100 and 230 people, probably settling at around 150.

 

The figure of 150 seems to represent the maximum number of individuals with whom we can have a genuinely social relationship, the kind of relationship that goes with knowing who they are and how they relate to us.  Putting it another way, it’s the number of people you would not feel embarrassed about joining uninvited for a drink if you happened to bump into them in a bar.

 

By reducing the time and effort it takes to keep track of other people’s activities, both personal and professional, social computing platforms have the potential to dramatically increase the number of people with whom you can have a social relationship, in essence increasing your memory capacity for potentially valuable contacts. Organizations which are too focused on strong ties fail to see the value of these “low-density” networks.  As a result, the platforms that nurture and sustain them, such as Facebook, LinkedIn and Twitter, are viewed with suspicion and are generally considered an unprofitable time-suck for their staff.  McAfee’s case studies give some excellent counter examples, such as Serena Software’s decision to move their entire corporate Intranet to Facebook and bring in the staff’s teenage children to train employees on “Facebook Fridays.”

Organizations often attempt to tap into the third ring of McAfee’s bull’s-eye by creating elaborate, centralized expertise directories.  My experience with such directories has always been that they are extremely difficult to initially populate and even harder to maintain.  As a result, their utility drops off quickly after launch, unless people can be induced to take ownership of their own profiles and keep them up to date on their own initiative.  That is not to say they are not a worthwhile endeavor, but McAfee highlights some interesting alternatives (or at least supplements) that may already be in place, such as document repositories, but that are underutilized.  McAfee demonstrates the successful conversion of potential ties into productive collaborations in both the Google and US Intelligent Community case studies and elaborates on those efforts throughout the book.

The introduction of many of these tools may seem to be old-hat to many readers, but it is a necessary level-setting for the broad readership to which the book seems targeted.  McAfee’s tool overviews are effective and engaging.  They provide all the orientation necessary for newcomers while still treating veterans to fresh insights.  This is important as the real value of the book is found in Part II: Succeeding With Enterprise 2.0. In these final three chapters, McAfee addresses the standard objections to adopting Web 2.0 tools (Won’t staff just use it to gossip? What if somebody says something bad about our company?  Doesn’t this expose us legally?) A long list of objections is presented and while the risks can’t necessarily eliminated, they can generally be adequately mitigated.  On balance, the benefits more than counterbalance the cost. McAffee sums up his advice by saying, “For most organizations, in fact, I believe that these benefits outweigh all the risks.”

Chapter seven lays out a roadmap for succeeding with Enterprise 2.0.  McAfee is quick to point out that successfully adopting these technologies and techniques “can’t be reduced to a single step-by-step recipe.”  Rather he offers recommended steps and a few warnings.  First and foremost is managing your own expectations of how quickly or fully people will embrace a new way of working even if it does make their lives easier.  A new tool or process needs to provide at least a ten-fold improvement over the status-quo in order to be accepted. McAfee cites the research of Harvard Marketing professor John Gourville to support this assertion.

 

Many products fail because of a universal, but largely ignored, psychological bias: People irrationally overvalue benefits they currently possess relative to those they don’t. … [This] leads to a clash in perspectives: Executives, who irrationally overvalue their innovations, must predict the buying behavior of consumers, who irrationally overvalue existing alternatives.  The results are often disastrous: Consumers reject new products that would make them better off, while executives are at a loss to anticipate failure.  This double-edged sword is the curse of innovation.

 

McAfee advises a go-slow approach with a “long haul view.” Determining what you are trying to accomplish in advance of deploying tools is critical, as is reconciling yourself to the fact that the benefits will likely take quite some time to fully realize.  Even when those benefits do materialize they may difficult to quantify.  McAfee advises against attempting to calculate a Return On Investment for Enterprise 2.0 initiatives.  It is much more important to measure progress.  As he says “these [initiatives] require sustained management attention, not just a periodic contest among business cases in which the highest ROI figure wins.”  As I prepare my budget and capital funding requests for the coming year, I will be be passing this book along to both my superiors and subordinates with this passage underlined.


Purchase Enterprise 2.0: New Collaborative Tools for Your Organization’s Toughest Challenges at Amazon.






The Social Technographics Ladder Gets a Bit Longer

January 21st, 2010 No comments

In my last post, I talked a bit about the need for a “benevolent dictator” to shepherd the implementation, adoption and perpetuation of social computing initiatives.  Often the person or persons in that oversight role will have a vision or a set of goals, but rarely do they have a clear plan of how to achieve those goals.  Determining what success means and how to measure it has proven even more elusive.  Web analytics has blossomed from a cottage industry into a full fledged discipline as a way to inform strategies on the web, but the field has yet to get its arms around social media in any systematic way.  Back in April of 2007 Forrester Research analysts Josh Bernoff and Charlene Li took a first stab at it by introducing the notion of “Social Technographics.”

Li and Bernoff discuss Social Technographics and its application at length in their book  Groundswell: Winning in a World Transformed by Social Technologies but the idea itself is pretty simple. As Li writes in her original blog post:

 

Many companies approach social computing as a list of technologies to be deployed as needed – a blog here, a podcast there – to achieve a marketing goal.  But a more coherent approach is to start with your target audience and determine what kind of relationship you want to build with them, based on what they are ready for.

 

Forrester approached this by defining six levels of participation on the participatory web. These range from “Spectators” who consume social media but do not contribute, up to “Creators,” those who publish a web page, maintain a blog or upload media to site like YouTube.  These participation levels are more roles or profiles rather than more customary segmentations. Each level may or may not overlap.  As Bernoff notes, “people participate in multiple behaviors, and not everyone at a higher level on the ladder actually does everything in the lower rungs.”

SocTechGraph

Over the past couple of years, this ladder of participation has proven to be a useful way to analyze the target audiences for social media efforts and inform strategies accordingly. On Tuesday, Forrester expanded its ladder to include a new level of participation, the “Conversationalist.”  This is an individual who participates at least weekly in microblogging, such as Twitter, LinkedIn status changes or Facebook “what’s on your mind?” updates.

What makes the ladder compelling is the consumer data upon which it is based.  The rule of thumb for Web 2.0 users has always been “1×9x90”.  That is, 1% of the user population create and contribute content, 9% comment and critique and the remaining 90% passively consume.  Forrester’s Social Technographics show this to be too simplistic.  Consumers tend to move back and forth between rungs on the ladder depending on the context.  By writing this blog, I am firmly on the Creator rung but only in the context of my professional life.  I’m an avid music lover and follow many artist blogs and websites but don’t generally contribute beyond rating mp3s, so in that context, I’m a Collector.  I’m a bit of a news junkie so I often visit The Economist, CNN, and the New York Times, but I don’t think I’ve ever posted a comment, so in the news world I’m purely a Spectator.

Li and Bernoff advocate using the ladder to develop your social media roadmap and overall strategy.

 

I’ve used the "participation ladder"to help figure out which social strategies to deploy first – and also how to encourage users to "climb up", so to speak, from being Spectators to becoming more engaged. It’s my belief that not everyone is cut out from the start to be a Creator; nor is everyone inclined to jump with both feet into social networking. Companies seeking to engage customers with these new tools need to understand where their audiences are with this categorization.

 

I believe this approach may be even more appropriate for collaborative efforts within the enterprise.  Any sizable enterprise is basically the Internet in microcosm.  Most Enterprise 2.0 effort fail when they do not take this into account.  Too often everyone on staff is expected to assume the duties of a Creator when many simply aren’t cut out for it, though I believe it is possible to bring them along over time, to help the “climb up” as Li puts it.  For example, someone may not be ready to write a lengthy knowledgebase article, but might be perfectly comfortable microblogging their status and questions on Yammer or instant messaging with GoogleTalk.  The point is that as these tools are rolled out across an enterprise the different types of participation need to be taken into account, planned for and accommodated.  The new and improved Social Technographics framework is a good place to start.






The Benevolent Dictator Principle

January 17th, 2010 No comments

A couple of years ago, media incubator NewAssignment.net partnered with Wired magazine in a decidedly meta project called Assignment Zero.  The idea was to use crowdsourcing to produce an in depth study of crowdsourcing.  NewAssignment would “have a crowd of volunteers write the definitive report on how crowds of volunteers are upending established businesses, from software to encyclopedias and beyond.”  Like many enterprises venturing into Web 2.0 territory, the sponsors did not put much thought into how this would happen.  They would simply provide the tools and topics and the magic of the crowd would take care of the rest. “It’s like throwing a party,” Assignment Zero editor Lauren Sandler told the New York Times in 2007. “You program the iPod, mix the punch and dim the lights and then at 8 o’clock people show up. And then who knows what is going to happen?”

The goal was to produce at least 80 high-quality, feature length articles. At the completion of its 12 week run the project had produced only seven.  The problem was not due to a lack of interest or enthusiasm.  According to a  post-mortem published in Wired magazine the project attracted over 500 potential contributors in its first week of operation. Rather, Assignment Zero’s difficulties arose from the assumption that the flood of volunteers would self-organize and produce the desired content.  This is the problem with most ‘let’s just throw it out there’ implementations of collaborative technologies.  If you build it they will come, but they are more likely to stand around looking at each other than to start a Field of Dreams-esque pickup game.  Eventually, most will loose interest and wander off.  This is precisely what happened to Assignment Zero.

Halfway through the project, the sponsors realized they were in trouble, but they were learning.  The first wave of volunteers had largely disappeared and most of the topic pages had been abandoned.  The problem was that assignment zero put the organizational burden on the volunteers themselves.  “What we learned,” said Jay Rosen one of the project sponsors, “is that you have to be waaaay clearer in what you ask contributors to do. Just because they show up once doesn’t mean they’ll show up over and over. You have to engage them right away.”  The most critical form of engagement is to provide guidance, direction and structure.

In my last post, I sited Jaron Lanier’s concerns with the rush to Web 2.0.  In an essay for the Edge entitled Digital Maoism, he captures the lesson Assignment Zero eventually learned.

 

Every authentic example of collective intelligence that I am aware of also shows how that collective was guided or inspired by well-meaning individuals. These people focused the collective and in some cases also corrected for some of the common hive mind failure modes.

 

In open-source software development, this is role is known as the benevolent dictator. In Assignment Zero’s case this took the form of 30 volunteer professional editors who were trained on the tools, educated on the goals of the project and assigned to manage various topics. In essence, each research area to be explored became a fiefdom (but a pleasant fiefdom) with its own benevolent dictator working under the direction of a board of governors.

This may sound like it flies in the face of the open enterprise and web 2.0 in general, but what I’m describing is not a traditional command and control structure, but rather a guide and nurture ethic more on the order of Robert Greenleaf’s servant leadership model.  Successful crowdsourcing efforts depend on the passion of participants or they will fail.  After all, you can make an assignment, but if the contributor’s heart isn’t in it you are likely to get less than stellar results, if you get any at all.  And while we may be creating fiefdoms, the borders are always open.  People can come and go as they please.

This is the second lesson Assignment Zero learned.  Once you have set the direction and boundaries, the community controls the scope and direction of the project. In the Wired post-mortem, David Cohn, another Assignment Zero editor, recalled “we had to jettison most of the topics we’d started off with. Instead, we concentrated on the topics that people were most clearly interested in.”  Instead of the 80 feature articles Assignment Zero was hoping for, they ended up with only seven, but this may not be the failure it seems.  In addition to the formal articles, the project conducted and posted 80 in depth interviews, at least 60 of which were of a professional caliber.

While Assignment Zero may not have produced the definitive study of crowdsourcing it intended, at least not in the form originally conceived, it did demonstrate and document some important lessons about collaborative technologies.  First, you must provide guidance and direction.  Second, that guidance and direction cannot be rigid or overbearing.  Give the crowd, be it a marketing team, a group of developers or an online community, an overall vision, set some boundaries, educate them in the tools and resources and then only intervene when necessary to keep things on track.

Some companies are starting to understand this as evidenced by an interview with SunGard President and C.E.O. Cristóbal Conde in today’s New York Times.  Conde has embraced Enterprise 2.0 in a big way for SunGard.  His personal tool of choice is the Twitter-like microblogging tool Yammer.  In his view, these technologies flatten the organization in critical ways.  The boss becomes more of a first-among-equals shaping and facilitating the conversation rather than being a top down commander.

 

By having technologies that allow people to see what others are doing, share information, collaborate, brag about their successes — that is what flattens the organization. I think the role of the boss is to then work on those collaboration platforms, as opposed to being the one making the decisions. It’s more like the producer of the show, rather than being the lead.

I think too many bosses think that their job is to be the lead, and I don’t. By creating an atmosphere of collaboration, the people who are consistently right get a huge following, and their work product is talked about by people they’ve never met. It’s fascinating.

 

Out of this dynamic, a true meritocracy can emerge within the organization.  Too often, people with good ideas go unheard and people with bad ideas are heeded all because of where they each sit in the org chart.  Conde views Web 2.0 as an opportunity to apply both of the lessons of Assignment Zero, proving himself to be a benevolant dictator in the finest sense.

 

You have to work on the structure of collaboration. How do people get recognized? How do you establish a meritocracy in a highly dispersed environment?

The answer is to allow employees to develop a name for themselves that is irrespective of their organizational ranking or where they sit in the org chart. And it actually is not a question about monetary incentives. They do it because recognition from their peers is, I think, an extremely strong motivating factor, and something that is broadly unused in modern management.

 

So is this a contradiction?  Yet another paradox of the Web 2.0 world? I’m interested in hearing your thoughts on this balance between governance and openness.  What experiences have you had with engaging the crowd both inside the firewall an out.  Have you participated in a community under a benevolant (or perhaps a despotic) dictator?  Let me know.

The Madness of Crowds or the Brilliance of the Billion?

January 13th, 2010 1 comment

Yesterday, I encountered an interesting, chance juxtaposition of ideas from two people I’ve long admired, University of Michigan economist Scott Page and virtual reality pioneer Jaron Lanier. Since their comments seemed at such odds (at first blush anyway), I felt compelled to attempt a reconciliation.  Here’s the quandary.  I’ve been reading Jeff Howe’s book Crowdsourcing, a recurring theme of which is Page’s  "Diversity Trumps Ability" theorem.  In a nutshell, the theorem asserts that under certain, fairly common circumstances groups of experts are often outperformed by larger, more diverse groups of individuals with lesser abilities.  As Page puts it “a randomly selected collection of problem solvers outperforms a collection of the best individual problem solvers.”

While mulling over the various examples of diversity trumping ability that pepper Howe’s book, a friend retweeted a headline from the New York Times entitled The Madness of Crowds and an Internet Delusion.  This was too serendipitous to pass up so I clicked through to the article which turned out to be a discussion of Lanier’s new book You Are Not a Gadget. The Times describes the book as “a manifesto against … the glorification of open-source software, free information and collective work at the expense of individual creativity.” This was at odds with my impressions of the dreadlocked, multi-instrumentalist, technology impresario. So why is an elder statesman of Silicon Valley and one of the earliest and most prominent citizens of the quintessential online community The Well hating on Web 2.0?

His biggest complaint is against “hive thinking” and the “digital Maoism” he sees emerging in the participatory web. In an interview on Amazon.com coinciding with the release of his book, Lanier rightfully laments the dynamic of people becoming “artificially caustic, flattering, or otherwise manipulative” in order to stand out from the crowd.  It is hard to argue with this point. Online communities tend to coalesce around affinity rather than diversity, but Lanier also takes issue with the whole notion of “collective intelligence” arguing that the heart of the Diversity Trumps Ability theorem is fundamentally flawed.

Jaron Lanier

I am amazed by the power of the collective to enthrall people to the point of blindness. Collectivists adore a computer operating system called LINUX, for instance, but it is really only one example of a descendant of a 1970s technology called UNIX. If it weren’t produced by a collective, there would be nothing remarkable about it at all.

Meanwhile, the truly remarkable designs that couldn’t have existed 30 years ago, like the iPhone, all come out of "closed" shops where individuals create something and polish it before it is released to the public. Collectivists confuse ideology with achievement.

Lanier does allow that there are some (rather mundane) cases where the The Wisdom of Crowds can be useful, such as product pricing and movie recommendations, but in any situation involving “creativity and imagination,” he says, “a crowd process generally fails”.

I have not yet read You Are Not a Gadget (it was only released on Tuesday) but by all accounts it sounds like a less vitriolic and better reasoned version of Andrew Keen’s argument in The Cult of the Amateur, in essence that the Internet is killing culture, undermining morality and slowly but surely bringing about the general collapse of civilization .  So how does one reconcile Page’s portrayal of the mass of amateurs, what Howe calls “the Billion”, contributing to the collective intelligence of the web with Lanier’s depiction of Web 2.0 as a partisan mob and a new robber baron class (Google, Amazon, etc) whom he calls “Lords of the Cloud”?  In my view, they are both correct from their respective perspectives.

The diversity Page is talking about is not the ethnicity-based diversity HR departments seem to be obsessed with, at least not exclusively or even primarily.  Rather it is a diversity of Scott E. Pagecognitive strengths.  i.e. talents, problem solving approaches, core knowledge and simple perspective.  Say you form two teams to play “Who wants to be a Millionaire”, one composed exclusively of Nobel Laureate chemists and one of randomly selected college graduates.  The raw IQ scores of the Nobel Laureate team may be higher than that of the random alumni team, but their knowledge will be so specialized, their approach to problem solving so homogeneous, they will be unlikely to outperform the less expert, more generalized team in areas outside of their particular expertise.  Princeton’s Russell K. Nieli summarizes this phenomenon by saying “divergent yet complimentary talents often produce better and more creative outcomes than convergent and more overlapping ones, regardless of how superlative the latter might be.”

This actually supports Lanier’s condemnation of web-enabled herd mentality.  When he says “Only the quirkiness of considered individual expression can cut through the nonsense of the mob,” Lanier is stating a fundamental condition of successful collaboration.  Howe sums this up nicely in his discussion of the investment service Marketocracy.

Deliberation is the enemy of Collective Intelligence because it reduces diversity.  As individuals confer, they also reach consensus.  One of the chief conditions that allow the crowd to make smart predictions or come up with novel approaches to a problem is autonomy: each makes his or her choice independently.

“Diversity Trumps Ability” does not equate to “Design by Committee”. It is, in fact, its antithesis. By opening problem solving tactics to a broader, more diverse talent pool, whether through crowdcasting, prediction markets or simple social media, you actually reduce the risk of group think by engaging talent outside of the committee (or lab team, or working group or advisory council) bringing to bear a new and likely novel perspective and approach. The vast majority of successful Innocentive challenge solvers are working outside of their own field.  For example, most problems classified as “chemical” are actually solved by trained electrical engineers.  The trick is that if a chemistry-oriented solution would work, it would have been found by the in-house chemists originally working the problem. This approach can often find a viable and even optimal solution faster than in-house, brute force brain power alone.  Several of the Innocentive participants Howe interviewed say that if they don’t have a solution in mind by the time they finish reading the problem, they move on to something else. The secret to collective intelligence and the innovation it can foster is to tap into what the crowd already knows. Bill Joy’s famous aphorism “No matter who you are, most of the smartest people work for someone else” is more applicable than ever.

So what about the "user-generated nonsense" of Keen’s Cult of the Amateur and the mediocrity it tends to engender?  It can be challenging to cut through the clutter of the collective, but fortunately bounded collective intelligence, as I’ve been describing here, tends to be self-correcting.  If a sub-optimal consensus forms due to negotiation, deliberation or inertia, another approach will likely be voiced and eventually take its place. Most consumer markets, the software industry in particular, have demonstrated that if a product isn’t satisfying the needs of its users, a better, faster, cheaper rival will appear to fill the gap and can ultimately take over that market space relatively quickly.  Adam Smith’s Invisible Hand  is just as influential on the Web as it is on Wall Street.  There will always be forays down blind alleys and flocks of red herring, but ultimately natural selection wins out.  Page captures this dynamic succinctly. “The reason diversity trumps ability in a problem solving scenario is because you can always throw the idiots off the bus.”