---
title: "Introduction to DataExplorer"
author: "Boxuan Cui"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to DataExplorer}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
library(rmarkdown)
library(DataExplorer)
library(data.table)
library(nycflights13)
library(networkD3)

knitr::opts_chunk$set(fig.width = 6, fig.height = 6, fig.align = "center", warning = FALSE)
```

<script src="d3.min.js"></script>

This document introduces the package *DataExplorer*, and shows how it can help you with different tasks throughout your data exploration process.

There are 3 main goals for *DataExplorer*:

1. [Exploratory Data Analysis (EDA)](https://en.wikipedia.org/wiki/Exploratory_data_analysis)
1. [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)
1. Data Reporting

The remaining of this guide will be organized in accordance with the goals. As the package evolves, more content will be added.

## Data
We will be using the [nycflights13](https://cran.r-project.org/package=nycflights13) datasets for this document. If you have not installed the package, please do the following:

    install.packages("nycflights13")
    library(nycflights13)

There are 5 datasets in this package:

* airlines
* airports
* flights
* planes
* weather

If you want to quickly visualize the structure of all, you may do the following:

```{r plot-str-template, eval=FALSE}
library(DataExplorer)
data_list <- list(airlines, airports, flights, planes, weather)
PlotStr(data_list)
```

```{r plot-str-run, echo=FALSE}
data_list <- list(airlines, airports, flights, planes, weather)
diagonalNetwork(
  PlotStr(data_list, print_network = FALSE),
  width = 700,
  height = 550,
  fontSize = 12,
  margin = list(
    "left" = 80,
    "right" = 150
  )
)
```

You may also try `PlotStr(data_list, type = "r")` for a radial network.

---

Now let's merge all tables together for a more robust dataset for later sections.

```{r merge-data}
merge_airlines <- merge(flights, airlines, by = "carrier", all.x = TRUE)
merge_planes <- merge(merge_airlines, planes, by = "tailnum", all.x = TRUE, suffixes = c("_flights", "_planes"))
merge_airports_origin <- merge(merge_planes, airports, by.x = "origin", by.y = "faa", all.x = TRUE, suffixes = c("_carrier", "_origin"))
final_data <- merge(merge_airports_origin, airports, by.x = "dest", by.y = "faa", all.x = TRUE, suffixes = c("_origin", "_dest"))
```

## Exploratory Data Analysis
Exploratory data analysis is the process to get to know your data, so that you can generate and test your hypothesis. Visualization techniques are usually applied.

You can easily check the basic statistics with base R, e.g.,

```{r eda-base-r, eval=FALSE}
dim(final_data)
summary(final_data)
object.size(final_data)
```

### Missing values
Real-world data is messy. After running the basic descriptive statistics, you might be interested in the missing data profile. You can simple use `PlotMissing` function for this.

```{r eda-plot-missing}
PlotMissing(final_data)
```

You may also store the missing data profile with `missing_data <- PlotMissing(final_data)` for additional analysis.

### Distributions
To visualize distributions for all discrete features:

```{r eda-bar-discrete, fig.width=10, fig.height=6}
BarDiscrete(final_data)
```

To visualize distributions for all continuous features:

```{r eda-histogram-continuous, fig.width=10, fig.height=6}
HistogramContinuous(final_data)
```

You may also visualize just one feature using this function:

```{r eda-bar-discrete-2}
BarDiscrete(final_data$manufacturer)
```

```{r eda-histogram-continuous-2}
HistogramContinuous(final_data$seats)
```

### Correlation
To visualize correlation heatmap for discrete and continuous features:

```{r eda-correlation, fig.width=8, fig.height=8}
CorrelationDiscrete(final_data)
CorrelationContinuous(final_data, use = "na.or.complete")
```

## Feature Engineering
Feature engineering is the process of creating new features from existing ones. Newly engineered features often generate valuable insights.

For functions in this section, most data objects need to be set to  [data.table](https://cran.r-project.org/package=data.table). However, you don't need to know data.table package, simply set the object back to the original class after feature engineering.

### Replace missing values
Missing values may have meanings for a feature. Other than imputation methods, we may also set them to some logical values. For example, for discrete features, we may want to group missing values to a new category. For continuous features, we may want to set missing values to a known number based on existing knowledge.

In *DataExplorer*, this can be done by `SetNaTo()`. The function automatically matches the argument for either discrete or continuous features, i.e., if you specify a number, all missing continuous values will be set to that number. If you specify a string, all missing discrete values will be set to that string. If you supply both, both types will be set.

```{r fe-set-na-to, collapse=TRUE}
library(data.table)
final_dt <- data.table(final_data)
SetNaTo(final_dt, list(0L, "unknown"))

```

### Group sparse categories
From the bar charts above, we observed a number of discrete features with sparse categorical distributions. Sometimes, we want to group the low-frequency categories to a new bucket, or reduce the number of categories to a reasonable range. `CollapseCategory()` will do the work.

Take *manufacturer* feature for example, suppose we want to group the long tail to another category. We could try with bottom 20% (by count) first:

```{r fe-collapse-category-count-trial}
CollapseCategory(data = final_dt, feature = "manufacturer", threshold = 0.2)
```

As we can see, manufacturer will be shrinked down to 5 categories, i.e., BOEING, EMBRAER, unknown, AIRBUS and OTHER. If you like this threshold, you may specify `update = TRUE` to update the original dataset:

```{r fe-collapse-category-count-update, results='hide'}
CollapseCategory(data = final_dt, feature = "manufacturer", threshold = 0.2, update = TRUE)
BarDiscrete(final_dt$manufacturer)
```

Instead of shrinking categories by frequency, you may also group the categories by another continuous metric. For example, if you want to bucket the carrier with bottom 20% distance travelled, you may do the following:

```{r fe-collapse-category-metric-trial}
CollapseCategory(data = final_dt, feature = "name_carrier", threshold = 0.2, measure = "distance")
```

Similarly, if you like it, you may add `update = TRUE` to update the original dataset.

```{r fe-collapse-category-metric-update, results='hide'}
CollapseCategory(data = final_dt, feature = "name_carrier", threshold = 0.2, measure = "distance", update = TRUE)
BarDiscrete(final_dt$name_carrier)
```

### Drop features
After viewing the feature distribution, you often want to drop features that are insignificant. For example, features like *dst_origin* has only one value, and it doesn't provide any valuable information. You can use `DropVar()` to quickly drop features. The function takes either names or column indices.

```{r fe-drop-var, eval=FALSE, collapse=TRUE}
DropVar(final_dt, c("dst_origin", "dst_dest", "tzone_dest"))
DropVar(final_dt, c(34, 41, 42))
```

## Data Reporting
To organize all the data profiling statistics into a report, you may use the `GenerateReport()` function. It will run most of the EDA functions and output a html file.

```{r dr-generate-report, eval=FALSE}
GenerateReport(final_data)
```
