Python for data science for dummies

Introduction

About This Book

Foolish Assumptions

Icons Used in This Book

Beyond the Book

Where to Go from Here



Part 1: Getting Started With Data Science and Python

Chapter 1: Discovering the Match between Data Science and Python

Defining the Sexiest Job of the 21st Century
Considering the emergence of data science
Outlining the core competencies of a data scientist
Linking data science, big data, and AI
Understanding the role of programming
Creating the Data Science Pipeline
Preparing the data
Performing exploratory data analysis
Learning from data
Visualizing
Obtaining insights and data products
Understanding Python's Role in Data Science
Considering the shifting profile of data scientists
Working with a multipurpose, simple, and efficient language
Learning to Use Python Fast
Loading data
Training a model
Viewing a result


Chapter 2: Introducing Python's Capabilities and Wonders

Why Python?
Grasping Python's Core Philosophy
Contributing to data science
Discovering present and future development goals
Working with Python
Getting a taste of the language
Understanding the need for indentation
Working at the command line or in the IDE
Performing Rapid Prototyping and Experimentation
Considering Speed of Execution
Visualizing Power
Using the Python Ecosystem for Data Science
Accessing scientific tools using SciPy
Performing fundamental scientific computing using NumPy
Performing data analysis using pandas
Implementing machine learning using Scikit-learn
Going for deep learning with Keras and TensorFlow
Plotting the data using matplotlib
Creating graphs with NetworkX
Parsing HTML documents using Beautiful Soup


Chapter 3: Setting Up Python for Data Science

Considering the Off-the-Shelf Cross-Platform Scientific Distributions
Getting Continuum Analytics Anaconda
Getting Enthought Canopy Express
Getting WinPython
Installing Anaconda on Windows
Installing Anaconda on Linux
Installing Anaconda on Mac OS X
Downloading the Datasets and Example Code
Using Jupyter Notebook
Defining the code repository
Understanding the datasets used in this book


Chapter 4: Working with Google Colab

Defining Google Colab
Understanding what Google Colab does
Considering the online coding difference
Using local runtime support
Getting a Google Account
Creating the account
Signing in
Working with Notebooks
Creating a new notebook
Opening existing notebooks
Saving notebooks
Downloading notebooks
Performing Common Tasks
Creating code cells
Creating text cells
Creating special cells
Editing cells
Moving cells
Using Hardware Acceleration
Executing the Code
Viewing Your Notebook
Displaying the table of contents
Getting notebook information
Checking code execution
Sharing Your Notebook
Getting Help


Part 2: Getting Your Hands Dirty With Data

Chapter 5: Understanding the Tools

Using the Jupyter Console
Interacting with screen text
Changing the window appearance
Getting Python help
Getting IPython help
Using magic functions
Discovering objects
Using Jupyter Notebook
Working with styles
Restarting the kernel
Restoring a checkpoint
Performing Multimedia and Graphic Integration
Embedding plots and other images
Loading examples from online sites
Obtaining online graphics and multimedia


Chapter 6: Working with Real Data

Uploading, Streaming, and Sampling Data
Uploading small amounts of data into memory
Streaming large amounts of data into memory
Generating variations on image data
Sampling data in different ways
Accessing Data in Structured Flat-File Form
Reading from a text file
Reading CSV delimited format
Reading Excel and other Microsoft Office files
Sending Data in Unstructured File Form
Managing Data from Relational Databases
Interacting with Data from NoSQL Databases
Accessing Data from the Web


Chapter 7: Conditioning Your Data

Juggling between NumPy and pandas
Knowing when to use NumPy
Knowing when to use pandas
Validating Your Data
Figuring out what's in your data
Removing duplicates
Creating a data map and data plan
Manipulating Categorical Variables
Creating categorical variables
Renaming levels
Combining levels
Dealing with Dates in Your Data
Formatting date and time values
Using the right time transformation
Dealing with Missing Data
Finding the missing data
Encoding missingness
Imputing missing data
Slicing and Dicing: Filtering and Selecting Data
Slicing rows
Slicing columns
Dicing
Concatenating and Transforming
Adding new cases and variables
Removing data
Sorting and shuffling
Aggregating Data at Any Level


Chapter 8: Shaping Data

Working with HTML Pages
Parsing XML and HTML
Using XPath for data extraction
Working with Raw Text
Dealing with Unicode
Stemming and removing stop words
Introducing regular expressions
Using the Bag of Words Model and Beyond
Understanding the bag of words model
Working with n-grams
Implementing TF-IDF transformations
Working with Graph Data
Understanding the adjacency matrix
Using NetworkX basics


Chapter 9: Putting What You Know in Action

Contextualizing Problems and Data
Evaluating a data science problem
Researching solutions
Formulating a hypothesis
Preparing your data
Considering the Art of Feature Creation
Defining feature creation
Combining variables
Understanding binning and discretization
Using indicator variables
Transforming distributions
Performing Operations on Arrays
Using vectorization
Performing simple arithmetic on vectors and matrices
Performing matrix vector multiplication
Performing matrix multiplication


Part 3: Visualizing Information

Chapter 10: Getting a Crash Course in MatPlotLib

Starting with a Graph
Defining the plot
Drawing multiple lines and plots
Saving your work to disk
Setting the Axis, Ticks, Grids
Getting the axes
Formatting the axes
Adding grids
Defining the Line Appearance
Working with line styles
Using colors
Adding markers
Using Labels, Annotations, and Legends
Adding labels
Annotating the chart
Creating a legend


Chapter 11: Visualizing the Data

Choosing the Right Graph
Showing parts of a whole with pie charts
Creating comparisons with bar charts
Showing distributions using histograms
Depicting groups using boxplots
Seeing data patterns using scatterplots
Creating Advanced Scatterplots
Depicting groups
Showing correlations
Plotting Time Series
Representing time on axes
Plotting trends over time
Plotting Geographical Data
Using an environment in Notebook
Getting the Basemap toolkit
Dealing with deprecated library issues
Using Basemap to plot geographic data
Visualizing Graphs
Developing undirected graphs
Developing directed graphs


Part 4: Wrangling Data

Chapter 12: Stretching Python's Capabilities

Playing with Scikit-learn
Understanding classes in Scikit-learn
Defining applications for data science
Performing the Hashing Trick
Using hash functions
Demonstrating the hashing trick
Working with deterministic selection
Considering Timing and Performance
Benchmarking with timeit
Working with the memory profiler
Running in Parallel on Multiple Cores
Performing multicore parallelism
Demonstrating multiprocessing


Chapter 13: Exploring Data Analysis

The EDA Approach
Defining Descriptive Statistics for Numeric Data
Measuring central tendency
Measuring variance and range
Working with percentiles
Defining measures of normality
Counting for Categorical Data
Understanding frequencies
Creating contingency tables
Creating Applied Visualization for EDA
Inspecting boxplots
Performing t-tests after boxplots
Observing parallel coordinates
Graphing distributions
Plotting scatterplots
Understanding Correlation
Using covariance and correlation
Using nonparametric correlation
Considering the chi-square test for tables
Modifying Data Distributions
Using different statistical distributions
Creating a Z-score standardization
Transforming other notable distributions


Chapter 14: Reducing Dimensionality

Understanding SVD
Looking for dimensionality reduction
Using SVD to measure the invisible
Performing Factor Analysis and PCA
Considering the psychometric model
Looking for hidden factors
Using components, not factors
Achieving dimensionality reduction
Squeezing information with t-SNE
Understanding Some Applications
Recognizing faces with PCA
Extracting topics with NMF
Recommending movies


Chapter 15: Clustering

Clustering with K-means
Understanding centroid-based algorithms
Creating an example with image data
Looking for optimal solutions
Clustering big data
Performing Hierarchical Clustering
Using a hierarchical cluster solution
Using a two-phase clustering solution
Discovering New Groups with DBScan


Chapter 16: Detecting Outliers in Data

Considering Outlier Detection
Finding more things that can go wrong
Understanding anomalies and novel data
Examining a Simple Univariate Method
Leveraging on the Gaussian distribution
Making assumptions and checking out
Developing a Multivariate Approach
Using principal component analysis
Using cluster analysis for spotting outliers
Automating detection with Isolation Forests


Part 5: Learning From Data

Chapter 17: Exploring Four Simple and Effective Algorithms

Guessing the Number: Linear Regression
Defining the family of linear models
Using more variables
Understanding limitations and problems
Moving to Logistic Regression
Applying logistic regression
Considering when classes are more
Making Things as Simple as Naïve Bayes
Finding out that Naïve Bayes isn't so naïve
Predicting text classifications
Learning Lazily with Nearest Neighbors
Predicting after observing neighbors
Choosing your k parameter wisely


Chapter 18: Performing Cross-Validation, Selection, and Optimization

Pondering the Problem of Fitting a Model
Understanding bias and variance
Defining a strategy for picking models
Dividing between training and test sets
Cross-Validating
Using cross-validation on k folds
Sampling stratifications for complex data
Selecting Variables Like a Pro
Selecting by univariate measures
Using a greedy search
Pumping Up Your Hyperparameters
Implementing a grid search
Trying a randomized search


Chapter 19: Increasing Complexity with Linear and Nonlinear Tricks

Using Nonlinear Transformations
Doing variable transformations
Creating interactions between variables
Regularizing Linear Models
Relying on Ridge regression (L2)
Using the Lasso (L1)
Leveraging regularization
Combining L1 & L2: Elasticnet
Fighting with Big Data Chunk by Chunk
Determining when there is too much data
Implementing Stochastic Gradient Descent
Understanding Support Vector Machines
Relying on a computational method
Fixing many new parameters
Classifying with SVC
Going nonlinear is easy
Performing regression with SVR
Creating a stochastic solution with SVM
Playing with Neural Networks
Understanding neural networks
Classifying and regressing with neurons


Chapter 20: Understanding the Power of the Many

Starting with a Plain Decision Tree
Understanding a decision tree
Creating trees for different purposes
Making Machine Learning Accessible
Working with a Random Forest classifier
Working with a Random Forest regressor
Optimizing a Random Forest
Boosting Predictions
Knowing that many weak predictors win
Setting a gradient boosting classifier
Running a gradient boosting regressor
Using GBM hyperparameters


Part 6: The Part of Tens

Chapter 21: Ten Essential Data Resources

Discovering the News with Subreddit
Getting a Good Start with KDnuggets
Locating Free Learning Resources with Quora
Gaining Insights with Oracle's Data Science Blog
Accessing the Huge List of Resources on Data Science Central
Learning New Tricks from the Aspirational Data Scientist
Obtaining the Most Authoritative Sources at Udacity
Receiving Help with Advanced Topics at Conductrics
Obtaining the Facts of Open Source Data Science from Masters
Zeroing In on Developer Resources with Jonathan Bower


Chapter 22: Ten Data Challenges You Should Take

Meeting the Data Science London + Scikit-learn Challenge
Predicting Survival on the Titanic
Finding a Kaggle Competition that Suits Your Needs
Honing Your Overfit Strategies
Trudging Through the MovieLens Dataset
Getting Rid of Spam E-mails
Working with Handwritten Information
Working with Pictures
Analyzing Amazon.com Reviews
Interacting with a Huge Graph


Index

About the Author
John Mueller is a freelance author and technical editor. He has writing in his blood, having produced 99 books and more than 600 articles to date. The topics range from networking to home security and from database management to heads-down programming. During his time at Cubic Corporation, John was exposed to reliability engineering and has had a continued interest in probability ever since.



Luca Massaron is a data scientist specialized in organizing and interpreting big data and transforming it into smart data by means of the simplest and most effective data mining and machine learning techniques. Because of his job as a quantitative marketing consultant and marketing researcher, he has been involved in quantitative data since 2000 with different clients and in various industries. Luca was able to quickly rank among the top 10 Kaggle data scientists.