import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import numpy as np
from scipy import stats
np.random.seed(10)
Let's simulate some housing discrimination tester data. Suppose a testing organization hired 20 white people and 20 black people, trained them to behave identically and give identical answers to landlord questions, and sent them into a large housing complex to inquire about one-bedroom apartments. The testers were instructed to ask how much rent is on one-bedrooms in the complex, but not to ask for an application---and then to record how much they were quoted, and whether they were offered an application anyway.
We'll assume that whether they're offered an application is a bernouli variable, i.e., a coin flip, and that the rent they're quoted is normally distributed. We'll simulate some racist data, i.e., where the probability that black people get offered an application is lower, and the mean rent is higher.
def generate_test(race, mean_rent, prob_application):
rent_charged = np.trunc(stats.norm.rvs(loc=mean_rent, scale=20, size=1))
application_offered = stats.bernoulli.rvs(p=prob_application, size=1)
return {"race": race, "rent": rent_charged[0], "application": application_offered[0]}
testers = []
for white_tester in range(20):
testers.append(generate_test("white", 500, 0.7))
for black_tester in range(20):
testers.append(generate_test("black", 550, 0.4))
testers
df = pd.DataFrame(testers)
df.head()
df.describe()
The first thing we'll do is a simple t-test for difference of means here. The data isn't paired (i.e., in our hypothetical research design, the housing organization didn't send two people who were matched on other characteristics, one white and one black, in to test.) so we can use the independent t-test.
t, p = stats.ttest_ind(df[df.race == "white"].rent, df[df.race == "black"].rent)
print(p)
print(t)
It looks like we can reject the null hypothesis that white and black rents are the same. How about offering applications? There, we have a binary categorical variable, so we should go for something like a chi-squared test.
I actually brushed over some complexities in the chi-squared test in the catalogue of hypothesis tests lesson. There are three different flavors of the chi-squared test: a test for independence, for homogeneity, and for goodness of fit. The goodness-of-fit one is for testing whether a set of categorial data came from a hypothesis discrete distribution. We don't need that here.
The difference between the test for independence and the test for homogeneity is a bit subtle. The most common explanation of the difference is that the test for independence is for research designs where you collect two observations for each member of the sample (i.e., their race and whether they were offered an application); whereas the test for homogeneity is for research designs where you sample the subpopulations separately, i.e., sample a bunch of white people and a bunch of black people and then observe who got an application. Here's a good explanation of the flavors of chi-squared tests.
The good thing is that, mathematically, the tests of independence and homogeneity are the same, the only difference is in interpretation. So we can use what we've already learned.
crosstab = pd.crosstab(df["race"], df["application"])
crosstab
chi2, p, dof, expected = stats.chi2_contingency(crosstab)
print(p)
It looks like we can't quite reject (at the standard 0.05 level) the null hypothesis that white and black folks are given applications similarly---which should suggest something to our testing organization about maybe having picked a research design with a bigger sample size, since the differences look pretty stark in the crosstab.
But, of course, Fisher's exact test is also available to us, so let's do one more test just to see what we get. (In scientific context, you should really precommit to the tests you'll carry out before doing so, since, let's recall, that if you look at the data a bunch of times, you might get significant results just by chance. So what I'm about to do is technically a bit dirty and I really should have just done this first.)
oddsratio, p = stats.fisher_exact(crosstab)
print(p)
Oh look, I managed to torture a significant result out of the data. See above as to why this is cheating. But also, probably reach for Fisher's exact test first. Also, we could have gone for a z test of independence of proportions---google it if you're curious.
Paired data¶
The hypothetical test I described doesn't quite match the examples given in many of our housing discrimination cases. Often, these organizations use paired/matched testers instead. That is, they send one black person and one white person close together in time, and where they match on everything else that might be relevant to the landlord (income, number of pets, general personality and good looks, etc.)
This violates one of the assumptions of both the standard t-test and the chi-squared/fisher tests, namely that the samples are independent. But we have tests for this purpose too, including the paired t-test and McNemar's test. (I didn't cover McNemar's test in the catalogue of hypothesis tests, but it's the equivalent of the chi-squared test for paired data.) I'll also break out the Wilcoxon signed-rank test, discussed at the end of the catalogue of hypothesis tests, just because I can.
Rather than simulate new data, let's just change the assumption of our implicit data. Let's assume they were paired, and that the pairings correspond to equivalent positions in the subsets of the DataFrame we already have (i.e., that the first white person was paired with the first black person, and so forth).
t, p = stats.ttest_rel(df[df.race == "white"].rent, df[df.race == "black"].rent)
print(p)
Mcnemar's test is buried in statsmodels, and it has a really obnoxious API that doesn't take Pandas dataframes, and returns some idiotic object called a "bunch." It's very annoying. But we can work with it.
(Note: if you actually find yourself doing this kind of research on a regular basis, I'd probably recommend switching from Python to R; as a language it's better suited to this kind of bread-and-butter statistics, although less well-suited to lots of the other stuff we do in this class. Once you understand Python, it's pretty easy to learn R.)
from statsmodels.stats.contingency_tables import mcnemar
print(mcnemar(np.array(crosstab)))
statistic, p = stats.wilcoxon(df[df.race == "white"].rent, df[df.race == "black"].rent)
print(p)
Again we see that we have more convincing evidence for a difference in rents than for a difference in application-offering rates. Looks good enough to file suit to me...
Now I'm going to save this dataset, so I can make you figure out how to work with it in class!
df.to_csv("simulated_housing_test.csv", index=False)