# 2.2_Basic_Statistics_for_Data_Mining

원본: file:///Users/taemin/Desktop/lecture/DM/%E1%84%8C%E1%85%A1%E1%84%85%E1%85%AD/2.2_Basic_Statistics_for_Data_Mining.pptx

PPTX 텍스트와 삽입 이미지 OCR. 수식 및 그림 전체 레이아웃은 원본과 영상을 대조해야 합니다. WDP 원본은 보존하지만 OCR하지 않았습니다.

## 슬라이드 1

데이터마이닝을
 위한 기본 통계 개념

### 삽입 이미지 OCR (미검수)



![[media/image3.png]]

## 슬라이드 2

Agenda
https://towardsdatascience.com/8-fundamental-statistical-concepts-for-data-science-9b4e8a0c6f1c

### 삽입 이미지 OCR (미검수)

3147
¾¾
V6L2 +
x26a-d)
1Y5-a)/
CaAH49)=Y
4
〈y

![[media/image4.png]]

![[media/image5.png]]

## 슬라이드 3

Statistical sampling

### 삽입 이미지 OCR (미검수)

1. Statistical sampling
In statistics, the entire set of raw data that you may have available for a test
or experiment is known as the population. For a number of reasons, you
cannot necessarily measure the patterns and trends across the entire
population. For that reason statistics allows us to take a sample, perform
some computations on that set of data, and using probability and some
assumptions we can with a certain degree of certainty understand trends
for the entire population or predict future events.
Statistics allows us to take a sample, or portion of the
population, pertorm some computations on that set
of data and using probability and some assumptions
we can with a certain degree of certainty understand
trends for the entire population
Let's say, for example, that we want to understand the prevalence of a
disease such as breast cancer in the entire population of the United
Kingdom. For practical reasons, it is not possible to screen the entire둡
population. Instead, we may take a randoun sample and measure the
prevalence among them. Assuming our sample is sufficiently randomised
and representative of the entire population we can achieve a measure of
prevalence.

![[media/image6.png]]

## 슬라이드 4

Statistical Features

### 삽입 이미지 OCR (미검수)

A box plot perfectly illustrates what we can do with basic statistical
features:
•When the box plot is short it implies that much of your data points are
similat, since there are many values in a small range
• When the box plot is tall it implies that much of your data points are
quite difterent, since the values are spread over a wide range
• If the median value is closer to the bottom then we know that most of
the data has lower values. It the median value is closer to the top then
we know that most of the data has higher values. Basically, if the median
line is not in the middle of the box then it is an indication of skewed
data.
• Are the whiskers very long? That means your data has a high standard
deviation and variance i.e the values are spread out and highly
varying. If you have long whiskers on one side of the box but not the
other, then your data may be highly varying only in one direction.
All of that information from a few simple statistical features that are easy to
calculate! Try these out whenever you need a quick yet informative view of
your data.
Statistical features is probably the most used statistics concept in data
science. It's often the first stats technique you would apply when exploring
a dataset and includes things like bias, variance, mean, median, percentiles,
and many others. It's all fairly easy to understand and implement in code!
Check out the graphic below for an illustration.
- maximum
2.0
1.5
1.0
median -
5
third quartile
IOR
first quartile
minimum
0
A basic box plot
The line in the middle is the median value of the data. Median is used over
the mean since it is more robust to outlier values. The furst quartie is
essentially the 25th percentile; i.e 25% of the points in the data fall below
that value. The third quartile is the 7Sth percentile; i.e 75% of the points in
the data tall below that value. The min and max values represent the upper
and lower ends of our data range.

![[media/image8.png]]

![[media/image7.png]]

## 슬라이드 5

Descriptive statistics

### 삽입 이미지 OCR (미검수)

Descriptive statistics, as the name suggests, helps us to describe the data. In
other words, it enables us to understand the underlying characteristics. It
doesn't predict anything, make assumptions or infer anything it simply
provides a description of what the data sample we have looks like.
Descriptive statistics are derived from calculations, often called patameters.
These include things like the:
• Mean - the central value, commonly called the average.
• Median - the middle value if we ordered the data from low to high and
divide It exactly m halt.
• Mode- the value which occurs most often.

![[media/image9.png]]

## 슬라이드 6

Distributions

### 삽입 이미지 OCR (미검수)

0.10
0.08
0.06-
0.04-
0.02-
0.00
0
20
40
Recency (months)
8
Distribution of the frequency of blood donations. Image by author.
80
One of the most important distributions is the normal distribution,
commonly referred to as the bell curve due to its shape. It is symmetrical in
shape with most of the values custering around the central peak and the
further away values distributed equally on each side of the curve. Many
variables in nature will form a normal distribution such as peoples heights
and IO scores.
Descriptive statistics are useful but they can often hide important
information about the data set. For example, if a data set contains several
numbers that are much larger than the others then the mean may be
skewed and will not give us a true representation of the data.
A distribution is a chart, often a histogram, that displays the frequency with
which each value appears in a data set. This type of chart gives us
information about the spread and skewness of the data.
A distribution will usually form a curve-like graph. This may be skewed
more to the lett or right.
0.0012 -
0.0010ㅓ
0.0008 ㅓ
0.0006 -
0.0004 1
0.0002 -
0.0000
0
2000
4000
6000
8000 10000 12000
Monetary (C.C. blood)
Distribution of blood volume from the “transfusion" data set. Image by author.
In some cases, the Curve may not be as smooth.
4
3
2
1
0
-0.4
-02
A normal distribution. Image by author.
0.2
0.4

![[media/image11.png]]

![[media/image10.png]]

![[media/image12.png]]

## 슬라이드 7

Probability

### 삽입 이미지 OCR (미검수)

4. Probability
Probability, in simnple terms, is the likelihood of an event occurring. In
statistics, an event is the outcome of an experiment which could be
something like the rolling of a dice or the results of an AB test.
Probability fot a single event is calculated by dividing the number of events
by the number ot total possible outcomes. In the case ot, say, rolling a six on
a dice there are 6 possible outcomes. So the chance of rolling a six is 1/6 =
0.167, sometimes this is also expressed as a percentage so 16.7%.
Events can be either independent or dependent. With dependent events, a
prior event influences the subsequent event. Let's say we have a bag of
M&M's and we wanted to determine the probability of randomly picking a
red M&M. If every time we removed the selected M&M from the bag the
probability of picking red would change due to the effect of prior events.
Independent events ate not affected by prior events. In the case of the bag
of M&M's if each time we selected one we put it back in the bag. The떄
probability of selecting ted would remain the same each time.
Whether an event is independent ot not is important, as the way in which
we calculate the probability of multiple events changes depending on the
type.
The probability of multiple independent events is calculated by simply
multiplying the probability of each event. In the example of the dice toll,
say we wanted to calculate the chance of rolling a 6 three times. This would
look like the following:
1/6= 0.167 1/6 = 0.167 1/6 = 0.167
0.167 * 0.167 * 0.167 = 0.005
The calculation is different for dependent events, also known as
conditional probability. If we take the M&M's example, imagine we have a
bag with only two colours red and yellow, and we know that the bag
contains 3 red and 2 yellow and we want to calculate the probability of
picking two reds in a row. on the first pick, the probability of picking a red
1s 3/5 = 0.6. on the second pick we have removed one M&M, whichs
happened to be red, so our second probability calculation is 2/4 = 0.5. The
probability of picking two reds in a row is therefore 0.6 * 0.5 = 0.3.

![[media/image13.png]]

![[media/image14.png]]

## 슬라이드 8

Probability Distributions

### 삽입 이미지 OCR (미검수)

• A Uniform Distribution is the most basic of the 3 we show here. It has a
single value which only occurs in a certain range while anything outside듏
that range is just O. It's very much an "on or off" distribution. We can
also think of it as an indication of a categorical variable with 2
categories: 0 or the valle. Your categorical vartable might have multiple
values other than 0 but we can still visualize it in the same was as a
piecewise function of multiple uniform distributions.
• A Normal Distribution, commonly referred to as a Gaussian
Distribution, is specifically defined by its mean and standard deviation.
The mean value shifts the distribution spatially and the standard
deviation controls the spread. The import distinction from other
distributions (e.g poisson) is that the standard deviation is the same in
all directions. Thus with a Gaussian distribution we know the average
value of our dataset as well as the spread of the data i.e is it spread over
a wide range ot is it highly concentrated around a few values.
• A Poisson Distribution is similar to the Normal but with an added
factor of skewness. With a low value for the skewness a poisson
distribution will have relatively uniform spread in all directions just like
the Notmal. But when the skewness value is high in magnitude then the
spread of our data will be different in different directions; in one
direction it will be very spread and in the other it will be highly
concentrated.
We can define probability as the percent chance that some event will occur.
In data science this is commonly quantified in the range of O to 1 where 0
means we are certain this will not occur and 1 means we are certain it will
occur. A probability distribution is then a function which represents the
probabilities of all possible values in the experiment. Check out the graphic
below for an illustration.
Poisson Distribution
0.40
0.35 -
0.30 1
0.25 -
0.20 -
0.15 -
0.10 -
0.05 -
0.00어
10
15
Intensity = 1
Intensity = 4
Intensity = 10
Intensity = 15
25
0.25 7
-Uniform(1, 6)
-Uniform(4, 12)
0.2-
0.15-
0.1 -
0.05
0+
0
2
6
8
12

![[media/image16.png]]

![[media/image15.png]]

![[media/image19.png]]

![[media/image18.gif]]

![[media/image17.png]]

## 슬라이드 9

Probability and Likelihood
Probability shows 
how much 
likely something is to happen
measures 
the goodness of fit
 of a statistical model to a sample of data 
Probability
Likelihood
Probability of crayfish life between 30 and 40 years?
Likelihood of crayfish life of 20 with 
𝜇
: 25 
𝜎
: 5 ?
 
30 ~ 40 years
→
Probability density function
𝜇
: 25 
𝜎
: 5
 

### 삽입 이미지 OCR (미검수)

OIER

![[media/image1280.png]]

![[media/image20.png]]

![[media/image1300.png]]

![[media/image21.jpeg]]

## 슬라이드 10

Probability and Likelihood
Probability shows 
how much 
likely something is to happen
measures 
the goodness of fit
 of a statistical model to a sample of data 
Probability
Likelihood
𝑃𝑟
(
y
|
Θ
)
 
𝐿
(
Θ
|
y
)
 
density
density
Θ
 
→
 
𝜇
: 25 
𝜎
: 5
 

### 삽입 이미지 OCR (미검수)



![[media/image1320.png]]

![[media/image20.png]]

![[media/image1340.png]]

![[media/image1330.png]]

## 슬라이드 11

Bias

### 삽입 이미지 OCR (미검수)

As we have previously discussed in statistics we frequently use samples of
data to make estimates about the whole data set. Similarly, for predictive
modelling, we will use some training data and attempt to build a model that
can make predictions about new data.
Bias is the tendency of a statistical or predictive model to over or
underestimates a parameter. This is often due to the method used to obtain
a sample or the way that errors are measured. There are several types of
bias commonly found in statistics. Here is a brief description of two of them.
1. Selection bias - this occurs when the sample is selected in a non-
random way. In data science, an example may be stopping an AB test
early when the test is running or selecting data for training a machine
learning model fromn one time period which could mask seasonal effects.
2. Confirmation bias - this occurs when the person performing some
analysis has a predetermined assumption about the data. In this
situation, there can be a tendency to spend more time examining
vatiables that are likely to support this assumption.

![[media/image22.png]]

## 슬라이드 12

Variance

### 삽입 이미지 OCR (미검수)

As we discussed earlier in this article the mean in a sample of data is the
central value. Variance measures how far each value in the data set is from
the mean. Essentially it is a measurement of the spread of numbers in a data
Set.
Standard deviation is a common measure of variation for data that has a
normal distribution. It is a calculation that gives a value to represent how
widely distributed the values are. A low standard deviation indicates that
the values tend to he quite close to the mean, Whilst a high standard
deviation indicates that the values are more spread out.
If the data does not follow a normal distribution then other measures of
variance are used. Commonly the interquartile range is used. This
measurement is derived by first ordering the values by rank and then
dividing the data points into four equal parts, called quartiles. Each
quartile describes where 25% of the data points Iie according to the
median. The interquartile range is calculated by subtracting the median for
the two central quarters, also known as @l and 03.
Monetary(c.c blood)
......
•
10
Noreary(Ce bcod)
A boxplot provides a useful visualisation of the interquartile range. Image by author.
12k

![[media/image23.png]]

## 슬라이드 13

Bias/Variance tradeoff

### 삽입 이미지 OCR (미검수)

The concepts of bias and variance are very important for machine learning.
When we build a machine learning model we use a sample of data known as
the training data set. The model learns patterns in this data and generates a
mathematical function that is able to map the correct target label or value
(y) to a set of inputs (X).
When generating this mapping function the model will use a set of
assumptions to better approximate the target. For example, the linear
regression algorithm assumes a linear (straight line) relationship between
the input and the target. These assumptions generate bias in the model.
As a computation, bias ts the difference between the mean prediction generated
by the model and the true value.
If we were to train a model using different samples of training data we
would get a variation in the predictions that are returned. Variance in
machine learning is a measure of how large this difference is.
In machine learning bias and variance make up the overall expected error
for our predictions. In an ideal world, we would have both low bias and low
variance. However, in practice minimizing bias will usually result in an
increase in variance and vice versa. The bias/variance trade-off describes
the process of balancing these two errors to minimise the overall error for a
model.

![[media/image24.png]]

## 슬라이드 14

Correlation

### 삽입 이미지 OCR (미검수)

Correlation is a statistical technique that measures relationships between
two varlables. Correlation is assumed to be linear (forming a line when
displayed on a graph) and is expressed as a number between +1 and -1,
this is known as the correlation coefficient.
A correlation coefficient of +1 denotes a perfectly positive correlation
(when the value for one variable increases the value of the second variable
also increases), a coefficient of O denotes no correlation and a coefficient of
-1 denotes a perfect negative correlation.
...
Statistics is a broad and comnplex field. This article is meant as a brief
introduction to some of the most commonly used statistical techniques in
data science. Often data science courses assume prior knowledge of these
basic concepts or start with descriptions that are overly complex and
difficult to grasp. Ihope this article will act as a refresher fot a selection of
basic statistical techniques used in data science Prior to moving onto more
advanced topics.
Thanks for reading!

![[media/image25.png]]

## 슬라이드 15

Dimensionality Reduction

### 삽입 이미지 OCR (미검수)

Another way we can do dimensionality reduction is through feature
pruning. With feature pruning we basically want to remove any features we
see will be unimportant to our analysis. For example, after exploring a
dataset we may find that out of the 10 features, 7 of them have a high
correlation with the output but the other 3 have very low correlation. Then
those 3 low correlation features probably aren't worth the compute and we
mnight just be able to remove them fromn our analysis without hurting the
OLLPut.
The most common stats technique used for dimensionality reduction is PCA
which essentially creates vector representations of features showing how
important they are to the output i.e their correlation. PCA can be used to do
both of the dimensionality reduction styles discussed above. Read more
about it in this tutorial.
The term Dimenstonality Reduction is quite intuitive to understand. We have
a dataset and we would like to reduce the number of dimensions it has. In
data science this is the number of feature variables. Check out the graphic
below for an illustration.
dimension:
TU POSITIONS
2 dimensions)
100 postton
•.
3 dimensions:
1000 positionst
Dimensionality Reduction
The cube represents our dataset and it has 3 dimensions with a total of
1000 points. Now with today's computing 1000 points is easy to process,
but at a larger scale we would run into problems. Howevet, just by looking
at our data from a 2-Dimensional point of view, such as from one side of the
cube, we can see that it's quite easy to divide all of the colours from that
angle. With dimensionality reduction we would then project the 3D data
onto a 2D plane. This effectively reduces the number of points we need to
compute on to 100, a big computational saving!

![[media/image27.png]]

![[media/image26.png]]

## 슬라이드 16

Over and Under Sampling

### 삽입 이미지 OCR (미검수)

Over and Under Sampling are techniques used for classification problems.
Sometimes, our classification dataset might be too heavily tipped to one
side. For example, we have 2000 examples for class 1, but only 200 for class
2. That'lI throw off a lot of the Machine Learning techniques we try and use
to model the data and make predictions! Our Over and Under Sampling Can
combat that. Check out the graphic below for an illustration.
Undersampling
Oversampling
Copies of the
minority class
Samples of
majonity dlass
Original dataset
Original dataset
Under and and Over Sampling
In both the left and right side of the image above, our blue class has far
more samples than the orange class. In this case, we have 2 pre-processing
options which can help in the training of our Machine Learning models.
Undersampling means we will select only some of the data from the
mnajority class, only using as many examples as the minority class has. This
selection should be done to maintain the probability distribution of the
class. That was easy! We just evened out our dataset by just taking less
samples!
Oversampling means that we will create copies of out minority class in order
to have the same number of examples as the majority class has. The copies
will be made such that the distribution of the minority class is maintained.
We just evened out our dataset without getting any more data!

![[media/image28.png]]

## 슬라이드 17

Bayesian Statistics

### 삽입 이미지 OCR (미검수)

But what if someone were to tell you that the specific die that was given to
you was loaded to always land on 6? Since frequency analysis only takes into
account prior data, that evidence that was given to you about the die being
loaded is not being taken into account.
Bayesian Statistics does take into account this evidence. We can illustrate
this by taking a look at Baye's theorem:
Prior Probability
Likelihood of the evidence 'E'
if the Hypothesis 'H' is true
P(H) *P(EIH)
P(HB) =-
P(E)
Posterior Probability of ‘H'
Priori probability that the evidence
given the evidence
itself is true
Baye's Theoram
The probability P(H) in our equation is basically our frequency analysis;
given our prtor data what is the probability of our event occutring. The
P(E H) in our equation is called the ttkelthood and is essentially the
probability that our evidence is correct, given the information from out
frequency analysis. For example, lf you wanted to roll the die 10,000 times,
and the first 1000 rolls you got all 6 you'd start to get pretty confident that
that die is loaded! The P(E) Is the probability that the actual evidence is
true. It I told you the die is loaded, can you trust me and say it's actually
loaded or do you think it's a trick?!
Fully understanding why we use Bayesian Statistics requires us to first
understand where Frequency Statistics fails. Frequency Statistics is the type
of stats that most people think about when they hear the word
"probability"”. It involves applying math to analyze the probability of some
event occurring, where specifically the only data we compute on is prior
data.
Let's look at an example. Suppose I gave you a die and asked you what were
the chances of you rolling a 6. Well most people would just say that it's 1 in
6. Indeed if we were to do a frequency analysis we would look at some data
Where someone rolled a die 10,000 times and compute the frequency of
each number rolled; it would roughly come out to 1 in 6!

![[media/image30.png]]

![[media/image29.png]]

## 슬라이드 18

How to obtain Generalization?
 
How to avoid an overfitting problem with small data set?
 Collecting enough data
   
→ 
too expensive
 Data augmentation
 Bayesian approach
 Regularization

### 삽입 이미지 OCR (미검수)

i) MSE
ii) penalty term

![[media/image32.png]]

![[media/image31.jpeg]]

![[media/image34.png]]

![[media/image33.png]]