Luc Devroye

This text presents a wide-ranging and rigorous overview of nearest neighbor methods, one of the most important paradigms in machine learning. Now in one self-contained volume, this book systematically covers key statistical, probabilistic, combinatorial and geometric ideas for understanding, analyzing and developing nearest neighbor methods. Gérard Biau is a professor at Université Pierre et Marie Curie (Paris). Luc Devroye is a professor at the School of Computer Science at McGill University (Montreal).

This text presents a wide-ranging and rigorous overview of nearest neighbor methods, one of the most important paradigms in machine learning. Now in one self-contained volume, this book systematically covers key statistical, probabilistic, combinatorial and geometric ideas for understanding, analyzing and developing nearest neighbor methods. Gérard Biau is a professor at Université Pierre et Marie Curie (Paris). Luc Devroye is a professor at the School of Computer Science at McGill University (Montreal).

Pattern recognition presents one of the most significant challenges for scientists and engineers, and many different approaches have been proposed. The aim of this book is to provide a self-contained account of probabilistic analysis of these approaches. The book includes a discussion of distance measures, nonparametric methods based on kernels or nearest neighbors, Vapnik-Chervonenkis theory, epsilon entropy, parametric classification, error estimation, free classifiers, and neural networks. Wherever possible, distribution-free properties and inequalities are derived. A substantial portion of the results or the analysis is new. Over 430 problems and exercises complement the material.

Thls text ls about one small fteld on the crossroads of statlstlcs, operatlons research and computer sclence. Statistleians need random number generators to test and compare estlmators before uslng them ln real l!fe. In operatlons research, random numbers are a key component ln !arge scale slmulatlons. Computer sclen- tlsts need randomness ln program testlng, game playlng and comparlsons of algo- rlthms. The appl!catlons are wlde and varled. Yet all depend upon the same com- puter generated random numbers. Usually, the randomness demanded by an appl!catlon has some bullt-ln structure: typlcally, one needs more than just a sequence of Independent random blts or Independent uniform [0,1] random vari- ables. Some users need random variables wlth unusual densltles, or random com- blnatorlal objects wlth speclftc propertles, or random geometrlc objects, or ran- dom processes wlth weil deftned dependence structures. Thls ls preclsely the sub- ject area of the book, the study of non-uniform random varlates. The plot evolves around the expected complexlty of random varlate genera- tlon algorlthms. We set up an ldeal!zed computatlonal model (wlthout overdolng lt), we lntroduce the notlon of unlformly bounded expected complexlty, and we study upper and lower bounds for computatlonal complexlty. In short, a touch of computer sclence ls added to the fteld. To keep everythlng abstract, no tlmlngs or computer programs are lncluded. Thls was a Iabor of Iove. George Marsagl!a created CS690, a course on ran- dom number generat!on at the School of Computer Sclence of McG!ll Unlverslty.

Density estimation has evolved enormously since the days of bar plots and histograms, but researchers and users are still struggling with the problem of the selection of the bin widths. This text explores a new paradigm for the data-based or automatic selection of the free parameters of density estimates in general so that the expected error is within a given constant multiple of the best possible error. The paradigm can be used in nearly all density estimates and for most model selection problems, both parametric and nonparametric. It is the first book on this topic. The text is intended for first-year graduate students in statistics and learning theory, and offers a host of opportunities for further research and thesis topics. Each chapter corresponds roughly to one lecture, and is supplemented with many classroom exercises. A one year course in probability theory at the level of Feller's Volume 1 should be more than adequate preparation. Gabor Lugosi is Professor at Universitat Pompeu Fabra in Barcelona, and Luc Debroye is Professor at McGill University in Montreal. In 1996, the authors, together with Lászlo Györfi, published the successful text, A Probabilistic Theory of Pattern Recognition with Springer-Verlag. Both authors have made many contributions in the area of nonparametric estimation.

Users of density estimation methods still struggle with selection of bin widths. This text explores a paradigm for data-based or automatic selection of free parameters of density estimates in general so that expected error is within a given constant multiple of best possible error.

Pattern recognition presents one of the most significant challenges for scientists and engineers, and many different approaches have been proposed. The aim of this book is to provide a self-contained account of probabilistic analysis of these approaches. The book includes a discussion of distance measures, nonparametric methods based on kernels or nearest neighbors, Vapnik-Chervonenkis theory, epsilon entropy, parametric classification, error estimation, free classifiers, and neural networks. Wherever possible, distribution-free properties and inequalities are derived. A substantial portion of the results or the analysis is new. Over 430 problems and exercises complement the material.