The p-value can be used as an alternative to the rejection region, providing the minimum level of significance at which the null hypothesis can be rejected. The smaller the p-value, the stronger the alternative hypothesis is supported.

Government agencies often use p-values ​​to increase the credibility of a study or report. For example, the U.S. Census Bureau requires that any analysis with a p-value greater than 0.10 must be accompanied by a statement that the difference is not statistically different from zero. The Census Bureau also sets standards for which p-values ​​are acceptable for various publications. [1]

Key Points

• The P-value is a statistical measurement used to test a hypothesis based on observed data.
• The P value measures the probability of obtaining an observed result if the null hypothesis is true.
• The lower the P value, the greater the statistical significance of the observed difference.
• A p-value of 0.05 or less is generally considered statistically significant.
• A P value can be used as an alternative to or in addition to a preselected confidence level for a hypothesis test.

Understanding P-values

P-values are typically found using p-value tables or spreadsheet/statistical software. These calculations are based on assumed or known probability distributions for the specific statistic being measured. Sample size determines the reliability of the observed data and directly affects the accuracy of the p-value calculation. Hypothesis testing using the p-value method Hypothesis testing using the calculated p-value method The calculated p-value is calculated from the deviation between the observed value and the selected reference value and takes into account the probability distribution of the statistic. The greater the difference between the two values, the smaller the corresponding p-value.

Mathematically, the p-value is calculated using integral calculus, which calculates the area under the probability distribution curve for all values ​​of the statistic that are at least as far away from the reference value as the observed values ​​are from the reference value, relative to the total area under the probability distribution curve. Standard deviation quantifies the spread of data points from the mean and plays an important role in calculations.

The calculation of the p-value varies depending on the type of test performed. Three test types describe the position on a probability distribution curve: left-tail, right-tail, or two-tail tests. In each case, the degrees of freedom play a crucial role in determining the shape of the distribution and in the calculation of the p-value.

Simply put, the greater the difference between two observations, the less likely it is that the difference is due to simple random chance, which is reflected in a lower p-value.

P value method for hypothesis testing

The P-value method for hypothesis testing uses a calculated probability to determine whether there is evidence to reject the null hypothesis. This determination relies heavily on the test statistic, which summarizes information from the sample relevant to the hypothesis being tested. The null hypothesis, also called a conjecture, is an initial statement about the population (or the data-generating process). The alternative hypothesis states whether the population parameter is different from the value of the population parameter stated in the conjecture.

In practice, the significance level is stated in advance to determine how small the p-value must be to reject the null hypothesis. Because different researchers use different significance levels when examining a question, it can sometimes be difficult for readers to compare results from two different tests. P-values ​​provide a solution to this problem.

IMPORTANT: Even a low p-value is not necessarily proof of statistical significance, as there is still the possibility that the observed data are the result of chance. Only repeated experiments or studies can confirm whether a relationship is statistically significant.

For example, suppose a study comparing the returns on two specific assets is conducted by different researchers who use the same data but use different significance levels. Researchers might reach opposite conclusions about whether assets differ.

If one researcher uses a 90% confidence level and another researcher requires a 95% confidence level to reject the null hypothesis, and if the p-value for the observed difference between the two returns is 0.08 (corresponding to the 92% confidence level), then the first researcher will find a statistically significant difference between the two assets, while the second researcher will find that there is no statistically significant difference between the returns.

To avoid this problem, researchers can report p-values ​​for hypothesis tests and allow readers to interpret statistical significance on their own. This is called the p-value method of hypothesis testing. An independent observer can note the p-value and decide for themselves whether this represents a statistically significant difference.

Example of P value

One investor claims their portfolio's performance is comparable to the Standard & Poor's (S&P) 500 Index. To determine this, investors conducted a two-tailed test.

The null hypothesis states that the portfolio's returns are comparable to the S&P 500's returns over the specified period, while the alternative hypothesis states that the portfolio's returns are not comparable to the S&P 500's returns - If the investor performs a one-tailed test, the alternative hypothesis would state that the portfolio's returns are less or greater than the S&P 500's returns.

p-value hypothesis testing does not necessarily use a pre-selected confidence level at which investors should reset the null hypothesis that returns are equivalent. Instead, it provides a measure of how much evidence there is to reject the null hypothesis. The smaller the p-value, the more evidence there is against the null hypothesis.

Therefore, if an investor finds a p-value of 0.001, there is strong evidence against the null hypothesis, and the investor can confidently conclude that the portfolio's returns are not comparable to the S&P 500's returns.

While this doesn't provide an exact threshold as to when investors should accept or reject the null hypothesis, it does have another very practical advantage. P-value hypothesis testing provides a straightforward way to compare the relative confidence an investor can have in selecting several different types of investments or portfolios relative to a benchmark such as the S&P 500 Index.

For example, given two portfolios, A and B, that perform differently than the S&P 500 with p-values ​​of 0.10 and 0.01, respectively, investors can be more confident that portfolio B, with a lower p-value, will actually show consistently different results.

Is a P value of 0.05 significant?

A p-value less than 0.05 is generally considered statistically significant, in which case the null hypothesis should be rejected. A p-value greater than 0.05 means that the deviation from the null hypothesis is not statistically significant and the null hypothesis is not rejected.

What does a P-value of 0.001 mean?

A p-value of 0.001 indicates that, if the null hypothesis being tested is indeed true, there is a one in a thousand chance of observing an outcome that is at least as extreme as the actual outcome. This leads the observer to reject the null hypothesis because either a highly rare data outcome is observed or the null hypothesis is incorrect.

How to use P-value to compare 2 different results of a hypothesis test?

If you have two different results, one with a p-value of 0.04 and the other with a p-value of 0.06, then the result with a p-value of 0.04 will be considered statistically more significant than the result with a p-value of 0.06. In addition to this simplified example, you can compare a p-value of 0.04 to a p-value of 0.001. Both are statistically significant, but the 0.001 example provides stronger evidence against the null hypothesis than the 0.04 example.

Bottom line

The p-value is a measure of the significance of the observed data. When researchers find a clear relationship between two variables, there's always the possibility that the correlation could be a coincidence. p-value calculations help determine whether an observed relationship is likely due to chance.

References

[1] U.S. Census Bureau. “Statistical Quality Standard E1: Analyzing Data.”

About TheQuantHackers

TheQuantHackers is a cutting-edge community composed of a group of people from the world's top universities and quantitative finance practitioners, dedicated to exploring the infinite possibilities in the fields of artificial intelligence (AI) and quantification (Quant). Our team members come from world-renowned universities such as the University of Cambridge, Oxford University, Harvard University, ETH Zurich, Peking University, and the University of Science and Technology of China. Our external consultants come from first-class companies such as Microsoft, HSBC, Citadel, Man Group, Citi, Jump Trading, and top domestic private equity companies.