Statistics is the study of uncertainity. One way to deal with uncertainity is by probabilities.
Read the first post The Bayesian Thinking - I.
P[rolling a four] = 1/6.
P[rolling a four] = 1/6.
P[die is fair]will be different than that person’s. It interprets probability as a measure of belief that an individual may possess about the occurrence of an event i.e. you may have prior belief about an event, but those beliefs are likely to change in case of new evidence. The Bayesian probablity provide tools to update subjective beliefs in light of new evidence (data).
Let’s take an example of finding probability of coin being fair. Let’s say an individual has a prior belief that the coin is fair. If after a number of tosses, say 500, the coin comes up head 400 times then the individual’s belief will be updated to coin is unfair.
Thus in Bayesian interpretation, the probability is a summary of an individual’s opinion. The different individuals may have different opinion (prior belief), as they might have different data and ways of interpreting it. However, as both of theses individuals get new information (data), their potentially differing beliefs leads to the same posterior belief.
According to the Frequentist theory, only repeatable events have probabilities. In the Bayesian framework, probability simply describes uncertainty.
Frequentist notion is objective while the Bayesian one is subjective.
In frequentist statistics, parameters () are fixed as they are specific to the problem, and are not subject to random variablility so probability statements about them are not meaningful while data () is random. On the other hand, in Bayesian statistics, parameters () are random (unknown) thus direct probability statements about parameters are made and data (X) is fixed.
e.g. Consider the following statements:
Statement 1 is a frequentist one. It means that if we were to repeat the analysis many times, using new data each time, then in 5% of samples where the null hypothesis is true it will be (falsely) rejected (but nothing is stated about this sample). Statement 2 is Bayesian. It applies on the basis of this sample (as a degree of belief).
The Bayesian methods work for any number of data points (i.e. for any ), while frequentists method work only when number of data points is greater than number of parameters (i.e. ).
The frequentist train their model using maximum liklihood principle ( i.e. they try to find the parameter that maximizes the liklihood, the probability of data given parameters. However, the Bayesian will try to compute the posterior probability, the probability of parameters given data, using Bayes theorem.