Yes, Virginia, There is New Mathematics...
"Gargantuan data sets" are creating a problem for scientists, especially when new technologies help capture large data sets involving human genome sequencing, monitoring of etransactions, or radio signals from outer space. But what's the problem...more data is supposed to be good?
The problem is that the standard use of statistical predictions using maximum likelihood estimation (MLE) no longer seem to work with these "HighD" data sets involving hundreds of billions of information bits. In fact, the MLE is producing unreliable predictions, or estimates. As one author put it, the paradox occurs "because any one data point swims in an increasingly immense sea, it's not likely to be representative."
So what to do? Charles Lawrence, a mathematician at Brown University, and his graduate student Luis Carvalho have created a new class of "centroid" estimators that actually produce data points as reliable estimates. And, new mathematics has been created!
Source: ScienceDaily, March 2, 2008
