Artificial Intelligence is still (somewhat) of a buzzword, though in practice much of what is likely to be implemented in this current business cycle under that all-encompassing umbrella has been. Visual recognition systems, check. Speech recognition, check. Expert systems, check. Self-driving vehicles, check. Recommendation engines, yup. Business Intelligence tools with a healthy smattering of data science, check, check, check. None of them are quite perfect yet (and arguably some are a long way towards achieving even practical applicability) but the shape of the immediate future, say over the next five years or so, is pretty clear.
Understanding Survivor Bias
So far, most software is just beginning to enter the specialized artificial intelligence (SAI) era. A good working definition for SAI is that it is software where the logic is determined not by a programmer, but by the analysis of data by a particular process and the deployment of a model that's derived from that analysis. Put another way, let's say that you have an application that takes a photograph of a person against a known scale and uses certain pre-specified rules to determine whether that person is male or female, such as height, weight distribution, length of hair, the presence or absence of certain clothing or articles and so forth.
An algorithmic approach to this problem is to create buckets that identify specific dimensions or qualifying traits, then determine whether a given person falls into a particular configuration of buckets. This is not, by itself, artificial intelligence, though it is the basis for a great number of expert systems.
However, let's say that a human being creates a dataset that contains these same buckets (or facets), then uses a machine learning algorithm to compare other individuals to this base set, producing statistical measures of fitness. The model that was created is now no longer dependent upon a programmer's direct input (though indirect input is a different question), but rather has become an algorithm determined by statistical fitness based upon the originating data. Confirmation of that fit can then go into refining the model, creating an adaptive framework that better converges on the "actual" distribution of traits.
This statistical (also known as stochastic) approach is a hallmark of AI. A model is created based upon data, the predictive use of that model on new data helps to refine the model, rinse and repeat. At least that's the theory. In practice, most SAI systems area highly dependent upon having good quality, unbiased data, something that is, in fact, in remarkably short supply. Professional pollsters, who arguably have been doing AI work for a while now, will tell you just how hard it is to keep bias from keeping into their work.
A simple example of bias can be seen in political polling. The vast majority of polling until comparatively recently used telephone polling to determine where a particular candidate stood with the electorate. Prior to 2008, this wasn't that much of a problem - most people had landlines, and mobile phones were found only in the hands of a very small and generally elite minority.
That situation changed dramatically over the next decade, to the extent that landlines are well on their way to becoming extinct today. Yet it took a while for pollsters to realize this, with the upshot being that, as measured by their source data, the electorate was becoming significantly older and more conservative, simply because those were the people least likely to make the transition to cell phones. Add into this the increasingly precise AI abilities of cell phones themselves to determine whether a given call was a spam call (and many people consider such polling as spam), and this skewed the bias even more.
Shooting Down Aircraft
This is an example of survivor's bias and shows how, even when attempts are made to seek solid data, it is possible for such stochastic methods to create bad models - and from there, bad policy when those models are implemented as decision makers. The name actually stems from an analysis conducted by statistician Abraham Wald during World War II. Wald reviewed a report made by the Navy that illustrated the bullet holes in aircraft that came back from missions.
The initial report made the recommendation to put armor (expensive to both install and to fly with) on those areas, but Wald successfully made the counter-argument: those aircraft had made it home despite taking damage in those places, which suggested strongly that aircraft that had taken damage elsewhere were unable to get back at all. His recommendation was to armor those parts of bombers that seemed relatively pristine, and the suggestion, once implemented, saw a significant improvement in sortie survival rates.
This survivor bias, unfortunately, occurs everywhere and can be especially pernicious when dealing with models that have a number of different variables. For instance, the AI software that makes approval recommendations for house loans is predicated upon historical credit records. However, good credit is often determined as a function of having a solid, regular stream of money. If loans historically were not made to minorities (they were figuratively shot down) due to racial bias or similar factors, then the software that uses that historical data will not have as many instances of minority house owners. As such, the software will be biased pre-emptively to deny credit, even if a given buyer otherwise has excellent credit.
It is, in fact, this particular conundrum that faces recommendation engines, which are generally machine-learning AIs. When you select a video on Netflix or an eBook on Amazon, this identifies you as being interested in a specific set of buckets, and the recommendation engine then finds the closest content to that configuration that it can. If your preferred medium is paranormal steampunk detective novels, then after a while, everything that you see is a variant of a paranormal steampunk detective novel. They are the survivors. However, after a while, you may get tired of this particular genre and try something in the thriller genre, yet your impression of the store is that there are no good thriller novels simply because they are not survivors.
This is one reason that such recommendations are increasingly adaptive, often incorporating outliers in a search to liven up the mix a bit. Put another way, a good machine learning system needs a systematic way to forget information that is stale.
Beyond that, such machine learning systems are also increasingly able to detect emergent patterns early. This is especially critical for caching. Machine learning, building a model on the fly with incoming data, is a comparatively expensive operation. Rather than running such a comparison every time someone makes a recommendation (which would bring either Netflix or Amazon to its knees) most recommendation engines analyze data, build the model, then cache that model in some kind of an indexed system such as a relational database, triple store or NoSQL store. Caches usually expire over a certain period of time, but certain types of queries can trip a hard wire that indicates to the system that the pre-cached content may be stale and needs to be recomputed.
The Dog That Did Not Bark In the Night
This ability to detect and property interpret anti-patterns is now becoming one of the hot areas of AI research. Sherlock Holmes fans are likely aware of the "dog that did not bark in the night" when a prized racehorse went missing. The pattern was that the dog barked when strangers approached, the anti-pattern was that the dog did not bark during the theft, which in turn led Holmes to the realization that the dog in question knew the thief well.
Again, this is a restatement of survivor's bias, and it's remarkably difficult to incorporate precisely because the antipatterns are emergent, unexpected and often signaled only by an absence of information.
This does not mean that machine learning is worthless, far from it, in fact. Machine learning is a very powerful tool for doing initial categorization. Moreover, it can also "kick out" those particular cases that don't readily fit within existing models, which means that they can be treated as outliers and analyzed on that basis. Outliers have always been problematic - they could represent statistical noise, but they also could be indicative of emerging trends.
However, before specialized AI gets too deeply embedded in various decision loops, it is well worth the (human) analysis to determine whether the training data was suitably representative and complete, and given some indication of this, how likely that bias is present. Without that critical piece of human governance, depending upon machine language AI is foolhardy at best and dangerous at worst.
Specialized Artificial Intelligence, especially as manifest in both Machine Learning and Deep Learning systems, represents a qualitative shift in the way that we use software systems. We are moving away from the idea that computer logic is created directly by human programmers, moving instead toward the notion that computer systems are increasingly deriving their own knowledge through the continual process of systemic modeling and categorization based upon streams of data.
However, as this form of programming becomes more pervasive, we should be diligent in recognizing that bias, especially survivor bias, can skew our expectations and give us false models that in turn may exclude or injure people who aren't even aware of their use. The danger that this introduces is in many respects far more profound and insidious than the threat of automation on employment, largely because it is so subtle and invisible.
Kurt Cagle is Managing Editor for Cognitive World, and is a contributing writer for Forbes, focusing on future technologies, science, enterprise data management, and technology ethics. He also runs his own consulting company, Semantical LLC, specializing on Smart Data, and is the author off more than twenty books on web technologies, search and data. He lives in Issaquah, WA with his wife, Cognitive World Editor Anne Cagle, daughters and cat (Bright Eyes).