Skip to content
Sem 5 Prep
← BI & Data ScienceR: cleaning, aggregating and plotting

Formulas for this chapter

Missing values

mean(x, na.rm = TRUE) cor(x, y, use = "complete.obs")

Any calculation over a column that may have gaps. Never substitute zero for a missing value.

NA
Value exists in the world but is absent from the data
na.rm = TRUE
Drop NAs from both the sum and the count

Group summaries

aggregate(y ~ g, data = df, FUN) df[which.max(df$y), ]

Totals, averages or counts within each level of a category, and pulling the whole record of the largest value.

y ~ g
The number to summarise, broken down by the grouping column
FUN
sum, mean, length, max: the function applied within each group
which.max
Position of the maximum, so it belongs in the row slot

The four base charts

hist(x, main =, xlab =, col =) one variable's spread barplot(h, names.arg =, col =) categories compared plot(x, y, col =, pch = 19) two variables together boxplot(y ~ group, data =, col =) spread within groups

Choose from the question being asked. Aggregate before barplot, and scatter before correlating.

main / xlab / ylab
Title and axis labels
pch
Point shape; 19 is a solid dot
names.arg
Bar labels, taken from the aggregated group names

Correlation and its heatmap

r = sum((x - xbar)(y - ybar)) / sqrt( sum(x - xbar)^2 x sum(y - ybar)^2 ) cor(x, y, method = "pearson") cor(num_data) # the matrix corrplot(cor(num_data), method = "color") # the heatmap distinct pairs = n(n - 1) / 2

After the scatter plot, and before any regression. A block of strong correlation among predictors is a multicollinearity warning.

r
-1 to +1, no units, measures the straight-line relationship only
diagonal
Always 1: a variable against itself
n
Number of numeric columns in the matrix
Step 1 of 20
The ideaTheory

One missing number poisons the average

Four delivery times: 5, 7, unknown, 10 days. Ask R for the mean and it answers NA.

That is not a bug. R is refusing to guess. The true mean depends on the number you do not have, so the only honest answer is "unknown" until you tell R what to do about it.

days <- c(5, 7, NA, 10) mean(days) # NA mean(days, na.rm = TRUE) # 7.333

Adding na.rm = TRUE is you taking responsibility for dropping it.

← BackContinue →