Working with Pandas:
Python data analysis library
Matt Shirley
May 1 2014
Overview
- What is (are?) Pandas
- Pandas Series & Dataframe
- Reading data from CSV
- (RE)indexing your data
- Using Pandas to plot your data
What is Pandas?
- Pandas: Panel Data System
- Pandas provides tools for:
- Cleaning data
- Analyzing data
- Simple regression of data
- Ploting data
- Data in Pandas are organized in table-like structures
- Pandas is built on Numpy as a datastructure backend
What is Pandas?
- Pandas allows creation of indexed two-dimensional arrays
- Similar to
dataframeobject in R language - Basic building block is a
Series- dictionary-like object with an index
Seriesobjects are combined as columns to produce aDataFrameDataFrameis an indexed, named two-dimensional array- think of it like an Excel workbook
Installation
pip install pandas or conda install pandas
Pandas
In [120]:
from pandas import Series, DataFrame
Recipe for a Series:s = Series(data, index=index)
- data can be a
dictor a numpyndarray
In [121]:
fruits = {'red': 'apple',
'yellow': 'banana',
'green': 'kiwi',
'purple': 'grape'}
series_fruits = Series(fruits, index=['red', 'yellow', 'green', 'purple'])
series_fruits
Out[121]:
Series are like list and dict:
In [122]:
series_fruits[:2]
Out[122]:
In [123]:
series_fruits['red']
Out[123]:
In [124]:
try:
series_fruits['orange']
except KeyError:
print('Key does not exist!')
Series with numeric content can be treated like a Numpy ndarray:
In [125]:
from string import ascii_letters
from numpy.random import randn
numbers = Series(randn(10), index=tuple(ascii_letters[:10]))
In [126]:
numbers
Out[126]:
In [127]:
numbers + numbers
Out[127]:
In [128]:
numbers*4
Out[128]:
In [129]:
numbers**2
Out[129]:
In [130]:
numbers.sum()
Out[130]:
In [131]:
numbers.describe()
Out[131]:
In [132]:
veg = {'red': 'tomato',
'yellow': 'squash',
'purple': 'eggplant'}
series_veg = Series(veg, index=['red', 'yellow', 'purple'])
series_veg
Out[132]:
Combine Series instances to make a DataFrame:
In [133]:
produce = DataFrame({'fruits': series_fruits, 'vegetables': series_veg})
produce
Out[133]:
- You'll notice that Pandas
DataFramehandles the missing 'green' vegetable by substitutingNaN. NaN, "not a number" is a stand-in for missingnessNaNis excluded from operations performed on a dataframe- for example, missing numbers would be excluded from a column sum or mean
In [134]:
produce.dropna()
Out[134]:
In [135]:
produce.fillna('brocolli')
Out[135]:
In [136]:
produce.fillna(method='ffill')
Out[136]:
In [198]:
produce.unstack()
Out[198]:
In [199]:
produce.reindex(['yellow', 'red', 'purple', 'green'])
Out[199]:
In [137]:
from pandas import read_csv
movies = read_csv('movies.csv')
movies.head()
Out[137]:
In [138]:
movies['rating'].mean()
Out[138]:
In [139]:
movies['length'].mean()
Out[139]:
In [189]:
list(movies.columns.values)
Out[189]:
In [140]:
movies.describe()
Out[140]:
In [158]:
by_year = movies.set_index('year')
by_year.sort_index(inplace=True)
by_year.loc[1990:1991].head()
Out[158]:
In [142]:
movies[['rating', 'year']].boxplot(by='year')
Out[142]:
In [143]:
movies[['budget', 'year']].boxplot(by='year')
Out[143]:
In [156]:
by_year.loc[1990:1999][['budget']].hist(bins=20)
Out[156]:
In [157]:
by_year.loc[1960:1969][['budget']].hist(bins=20)
Out[157]:
In [196]:
movies[['rating', 'mpaa']].boxplot(by='mpaa')
Out[196]:
In [192]:
movies[(movies['Drama'] == 1) & (movies['Romance'] == 1)][['rating', 'mpaa']].boxplot(by='mpaa')
Out[192]:
In [194]:
movies[(movies['Drama'] == 1) & (movies['Comedy'] == 1)][['budget', 'mpaa']].boxplot(by='mpaa')
Out[194]:
