A physics of ideas

"One of the main obstacles to productivity today is the growing problem of information overload. Information overloads results because we lack effective tools for automatically organizing information collections into meaningful and relevant chunks. "

Introduction: This paper provides an overview of a new approach to measuring the physical properties of ideas as they move in real-time through information spaces and populations such as the Internet. It has applications to search, information filtering, personalization, ad targeting, knowledge discovery and text-mining, market research, trend analysis, intelligence gathering, organizational behavior and social/cultural studies. One of the main obstacles to productivity today is the growing problem of information overload. Information overload results because we lack effective tools for automatically organizing information collections into meaningful and relevant chunks. For many years I have been thinking about a new way to approach this problem that is based on some ideas in classical physics. For quite some time I didn't say anything about it because it seemed like good material for a patent, but now I've decided it would be better to just put this in the public domain since I would rather have it be prior-art than patented by anyone. This is fundamental and useful and everyone should benefit from it.

In this article I propose the beginning of what might be called "a physics of ideas." My approach is based on mapping classical physics to memes that move through information spaces over time. The key to this is to measure the momentum of ideas as they move through space and time, and thus the momentums of documents that contain them. This provides a means to quantify the strength and trajectory of ideas as they move through a given corpus (and by inference, through the populations that create and consume the documents in that corpus) — and this enables us to start applying classical physics to empirically measure and understand the dynamics of ideas that are shaping our world. In other words, we can start to objectively analyze interactions between ideas as well as the impact that various ideas have on people, organizations and events in the "real world" and in turn the impact that those things have back on ideas.

Ideas are perhaps the single most powerful force shaping our world today, other than the climate. Humanity's behaviors are nothing but the results of various ideas — the phenotype of the ideas that are actually at work in the population at a given time. This is true for any organization, and even for individuals — ultimately much if not all of their behavior is conditioned by their ideas. So if we can form a science of ideas that enables us to begin to quantify and analyze their dynamics, we can start to gain new insights into human behavior and the "hidden" forces shaping our world today. My approach is to find a way to map what is going on in the realm of ideas to existing methods in classical physics — I want to make it possible to treat ideas as ideal particles in a Newtonian universe. It may then be possible to use the wealth of techniques that physicists have developed for analyzing the dynamics of particle systems to understand the dynamics of ideas within and between individuals and groups.

BACKGROUND

But first some background about how I came up with this idea…In 1993 I worked as an analyst at Individual, Inc., back in the pre-Web days. In that job I was part of a sophisticated information filter. Individual published filtered personalized newsfeeds. They aggregated content from thousands of sources and then filtered it into strategic newsfeeds tailored to the interests of their customers. You may have used Newspage or Heads Up, in the past. Chances are, if you did, I was your analyst. The way that the Individual system worked was that first a set of AI agents did a first pass on the incoming content to sort it into buckets. These buckets were routed to a team of human analysts with expertise in the relevant fields. The analysts would then go through the articles in the buckets to prioritize them, remove duplicates or items that had come through in previous articles as well as items that did not belong, and add in any items that should be included.

I want to make it possible to treat ideas as ideal particles in a Newtonian universe.

What this meant in practical terms for me as an analyst was that every night from about 8 PM until 1 AM I had to personally read through around 1600 news articles. My beat was emerging technology, software, broadband, online-services, multimedia and satellite applications, so I enjoyed it (yes, I am a serious news hound!). But still it was a challenge to keep on top of such a fire hose. Not only did I have to figure out what was important and how to prioritize it, but I also had to remember if I had ever seen and published anything about a given subject before in the previous year. By trial and error I evolved a solution to this problem. In summary, what I realized was that whether or not something is relevant is much more subtle than merely keyword matching! A good example can be found in nature — specifically frogs. Frogs have interesting visual systems. They are tuned to focus on things that move. They are most sensitive to size and velocity, but they also notice changes in velocity. Things that are small and that…

By Nova Spivack