Table Of ContentTest-Driven Machine Learning
Control your machine learning algorithms using
test-driven development to achieve quantifiable
milestones
Justin Bozonier
BIRMINGHAM - MUMBAI
Test-Driven Machine Learning
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: November 2015
Production reference: 1231115
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78439-908-5
www.packtpub.com
Credits
Author Project Coordinator
Justin Bozonier Paushali Desai
Reviewers Proofreader
Lars Marius Garshol Safis Editing
Alexey Grigorev
Indexer
Tejal Daruwale Soni
Commissioning Editor
Dipika Gaonkar
Graphics
Jason Monteiro
Acquisition Editors
Divya Poojari
Llewellyn Rozario Production Coordinator
Melwyn Dsa
Content Development Editor
Nikhil Potdukhe Cover Work
Melwyn Dsa
Technical Editors
Rupali R. Shrawane
Copy Editor
Yesha Gangani
About the Author
Justin Bozonier is a data scientist living in Chicago. He is currently a Senior Data
Scientist at GrubHub. He has led the development of their custom analytics platform
and also led the development of their first real time split test analysis platform
which utilized Bayesian Statistics. In addition he has developed machine learning
models for data mining as well as for prototyping product enhancements. Justin's
software development expertise has earned him acknowledgements in the books
Parallel Programming with Microsoft® .NET as well as Flow-Based Programming, Second
Edition. He has also taught a workshop at PyData titled Simplified Statistics through
Simulation.
His previous work experience includes being an Actuarial Systems Developer at
Milliman, Inc., contracting as a Software Development Engineer II at Microsoft,
and working as a Sr. Data Analyst and Lead Developer at Cheezburger Network
amongst other experience.
Savannah Bozonier—the best partner I've ever had in life. Time and
again she has made room in her life so I can push myself to do things
that take an immense amount of time. Things like writing this book.
My friends and colleagues for their support and help which
culminated in this book: Tom Hayden, Drew Fustin, and Andrew
Slotnick.
My mentors across the years—Chad Boyer, Kelly Leahy,
Robert Ream, James Thigpen, and Loren Bast.
My parents—I don't know what it's like to be told I can't do
something. My life reflects that in every way.
About the Reviewers
Lars Marius Garshol has worked as a consultant, product developer, and open
source developer for two decades. He added Unicode support to the Opera web
browser, edited a number of ISO standards, and developed the query language
tolog. Later, he worked as an enterprise architect and an R&D developer. He is the
developer of Duke, an open source tool for identifying near-duplicate database
records. He wrote Definitive XML Application Development, published in 2002.
Currently he is a software engineer at Schibsted Products & Technology in Oslo,
Norway. He's working on a book on Norwegian farmhouse ale.
Alexey Grigorev is an experienced software developer and data scientist with five
years of professional experience. In his day-to-day job, he actively uses R and Python
for data cleaning, data analysis, and modeling. He believes that testing is not only an
integral part of software development, but it is also very useful for building machine
learning models.
www.PacktPub.com
Support files, eBooks, discount offers, and more
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.
com and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on
Packt books and eBooks.
TM
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital
book library. Here, you can search, access, and read Packt's entire library of books.
Why subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print, and bookmark content
• On demand and accessible via a web browser
Free access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view 9 entirely free books. Simply use your login credentials for
immediate access.
Table of Contents
Preface v
Chapter 1: Introducing Test-Driven Machine Learning 1
Test-driven development 2
The TDD cycle 3
Red 4
Green 4
Refactor 4
Behavior-driven development 5
Our first test 6
The anatomy of a test 8
Given 9
When 9
Then 9
TDD applied to machine learning 10
Dealing with randomness 11
Different approaches to validating the improved models 16
Classification overview 16
Regression 16
Clustering 17
Quantifying the classification models 17
Summary 19
Chapter 2: Perceptively Testing a Perceptron 21
Getting started 21
Summary 33
[ i ]
Table of Contents
Chapter 3: Exploring the Unknown with Multi-armed Bandits 35
Understanding a bandit 35
Testing with simulation 36
Starting from scratch 37
Simulating real world situations 40
A randomized probability matching algorithm 43
A bootstrapping bandit 45
The problem with straight bootstrapping 48
Multi-armed armed bandit throw down 50
Summary 52
Chapter 4: Predicting Values with Regression 53
Refresher on advanced regression 53
Regression assumptions 54
Quantifying model quality 54
Generating our own data 55
Building the foundations of our model 56
Cross-validating our model 64
Generating data 65
Summary 66
Chapter 5: Making Decisions Black and White with Logistic
Regression 67
Generating logistic data 68
Measuring model accuracy 70
Generating a more complex example 71
Test driving our model 72
Summary 77
Chapter 6: You're So Naïve, Bayes 79
Gaussian classification by hand 79
Beginning the development 81
Summary 99
Chapter 7: Optimizing by Choosing a New Algorithm 101
Upgrading the classifier 101
Applying our classifier 106
Upgrading to Random Forest 111
Summary 115
[ ii ]
Table of Contents
Chapter 8: Exploring scikit-learn Test First 117
Test-driven design 118
Planning our journey 118
Creating a classifier chooser (it needs to run tests
to evaluate classifier performance) 119
Getting choosey 131
Developing testable documentation 134
Decision trees 134
Summary 137
Chapter 9: Bringing It All Together 139
Starting at the highest level 141
The real world 158
What we've accomplished 165
Summary 165
Index 167
[ iii ]
