In data science, p-value is used to determine statistical significance of the result. It gives the probability of a statistical model that, when the null hypothesis is true, the statistical summary would be the same or greater than the observed result.
In Simple linear regression, we predict a quantitative response Y on the basis of a singular predictor variable X. It assumes that there is a linear relationship b/w X and Y. Mathematically, it can be written as (we are regressing Y on X):
Here, the model coefficients and represents the slope and intercept in the linear model and is the error term.
We perform Hypothesis tests on the coefficients to check the validity of a claim made about the population. The most common hypothesis tests involves testing the null hypothesis of
: There is no relationship between X and Y
versus the alternate hypothesis
: There is some relationship between X and Y
To test the null hypothesis, we need to determine whether ,our estimate for , is sufficiently far from zero that we can be confident that is non-zero. We compute t-statistic which measures the number of standard deviations that is away from .
Consequently, it’s simple to compute the probability of observing any value equal to
|t| or larger, assuming (null hypothesis is true). This probability is called p-value. The p-value is defined as the probability, under the null hypothesis , of obtaining a result equal to or more extreme than what was actually observed.
A small p-value indicates that there is an association b/w the predictor and the response. We reject the null hypothesis – i.e. a relationship exists b/w X and Y. Typically, p-value cutoffs for rejecting the null hypothesis is 5% or 1%. Most authors refer to statistically significant as p-value < 0.05 and statistically highly significant as p-value < 0.001 (less than one in a thousand chance of being wrong). The cutoff, denotes the significance level of the test.
No hypothesis test is 100% certain. Because the test is based on probabilities, there is always a chance of drawing an incorrect conclusion. Two types of errors are possible.
The statistical hypothesis testing process can be summarized in the following steps:
Let’s understand it through an example. A recent study estimated that 20% of all college students in US smoke. The Head of Health Services at Goodheart University suspects that the proportion of smokers may be lower there. In hopes of confirming the claim her claim, she chooses a random sample of 400 Goodheart students, and finds that 70 of them are smokers.
The p-value is often misunderstood as being the probability that the null hypothesis is true.
The second conclusion does not imply that I accept , but just that I don’t have enough evidence to reject it.