---
title: "dataPreparation"
header-includes: \usepackage{booktabs}
date: '`r Sys.Date()`'
output:
  rmarkdown::html_vignette:
    number_sections: yes
  
vignette: >
  %\VignetteIndexEntry{Tutorial} 
  %\VignetteEngine{knitr::rmarkdown} 
  %\VignetteEncoding{UTF-8}
---

```{r results='hide', message=FALSE, warning=FALSE, echo=FALSE}
library(dataPreparation)
library(data.table)
library(knitr)
library(kableExtra)
library(pander)
options(knitr.table.format = "html") 
Sys.setlocale("LC_TIME", "C")

# A Prefix nulling hook.
# source: https://stackoverflow.com/questions/22524822/how-can-i-remove-the-prefix-index-indicator-1-in-knitr-output
# Make sure to keep the default for normal processing.
default_output_hook <- knitr::knit_hooks$get("output")

# Output hooks handle normal R console output.
knitr::knit_hooks$set( output = function(x, options) {

  comment <- knitr::opts_current$get("comment")
  if ( is.na(comment) ) comment <- ""
  can_null <- grepl( paste0( comment, "\\s*\\[\\d?\\]" ),
                     x, perl = TRUE)
  do_null <- isTRUE( knitr::opts_current$get("null_prefix") )
  if ( can_null && do_null ) {
    # By default R print output aligns at the right brace.
    align_index <- regexpr( "\\]", x )[1] - 1
    # Two cases: start or newline
    re <- paste0( "^.{", align_index, "}\\]")
    rep <- comment
    x <- gsub( re, rep,  x )
    re <- paste0( "\\\n.{", align_index, "}\\]")
    rep <- paste0( "\n", comment )
    x <- gsub( re, rep,  x )
  }

  default_output_hook( x, options )

})

knitr::opts_template$set("kill_prefix"=list(comment=NA, null_prefix=TRUE))
```

This vignette introduces *dataPreparation* package (v0.2), what it offers, how simple it is to use it.



# Introduction
## Package presentation
Based on  [data.table](http://r-datatable.com) package, **dataPreparation** will allow you to do most of the painful data preparation for a data science project with a minimum amount of code.


This package is

  * fast (use `data.table` and exponential search)
  * RAM efficient (perform operations by reference and column-wise to avoid copying data)
  * stable (most exceptions are handled)
  * verbose (log a lot)

`data.table` and other dependencies are handled at installation.

## Main preparation steps

Before using any machine learning (ML) algorithm, one needs to prepare its data. Preparing a data set for a data science project can be long and tricky. The main steps are the followings:

  * **Read**: load the data set (this package don't treat this point: for csv we recommend `data.table::fread`)
  * **Correct**: most of the times, there are some mistake after reading, wrong format... one have to correct them
  * **Transform**: creating new features from date, categorical, character... in order to have information usable for a ML algorithm (aka: numeric or categorical)
  * **Filter**: get rid of useless information in order to speed up computation
  * **Handle NA**: replace missing values
  * **Pre model transformation**: Specific manipulation for the chosen model (handling NA, discretization, one hot encoding, scaling...)
  * **Shape**: put your data set in a nice shape usable by a ML algorithm
  
Here are the functions available in this package to tackle those issues:

Correct                     | Transform                | Filter              | Pre model manipulation| Shape              
---------                   |-----------               |--------             |---------------------- | ------------------------
unFactor                    | generateDateDiffs        | fastFilterVariables | fastHandleNa          | shapeSet           
findAndTransformDates       | generateFactorFromDate   | whichAreConstant    | fastDiscretization    | sameShape          
findAndTransformNumerics    | aggregateByKey           | whichAreInDouble    | fastScale             | setAsNumericMatrix 
setColAsCharacter           | generateFromFactor       | whichAreBijection   |                       | one_hot_encoder
setColAsNumeric             | generateFromCharacter    |                     |                       |
setColAsDate                | fastRound                |                     |                       |
setColAsFactor              |                          |                     |                       |
							
All of those functions are integrated in the **full pipeline** function `prepareSet`.

In this tutorial we will detail all those steps and how to treat them with this package using an example data set.

## Tutorial data
For this tutorial, we are going to use a *messy* version of [adult](https://archive.ics.uci.edu/ml/datasets/adult) data base. 

```{r comment="#",  null_prefix=TRUE}
data(messy_adult)
print(head(messy_adult, n = 4))
```

We added 9 really ugly columns to the data set:

  * 4 dates with various formats, or time stamps, and NAs
  * 1 constant column
  * 3 numeric with different decimal separator
  * 1 email address

The same info can be contained in two different columns.

# *Correct* functions

## Identifying factor that shouldn't be
It often happens when reading a data set that R put string into a factor even if it shouldn't be. In this tutorial data set, `mail` is a factor but shouldn't be. It will automatically be detected using `unFactor` function:

```{r comment="#",  null_prefix=TRUE}
print(class(messy_adult$mail))
messy_adult <- unFactor(messy_adult)
print(class(messy_adult$mail))
```


## Identifing and transforming date columns
The next thing to do is to **identify** columns that are dates (the first 4 ones) and **transform** them.

```{r echo = FALSE, results='hide', comment="#",  null_prefix=TRUE}
setDT(messy_adult)
store <- copy(messy_adult[,.(date1, date2, date3, date4)])
```
```{r, comment="#",  null_prefix=TRUE}
messy_adult <- findAndTransformDates(messy_adult)
```
Let's have a look to the transformation performed on those 4 columns:
```{r echo = FALSE, results='asis', comment="#",  null_prefix=TRUE}
setnames(store, paste0(names(store), "_prev"))
toPlot <- cbind(head(store, n=6), data.frame("transfo" = rep("  =>", 6)), head(messy_adult[,.(date1, date2, date3, date4)], n = 6))

kable(toPlot) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)

```

As one can see, even if formats were different and somehow ugly, they were all handled.

## Identifying and transforming numeric columns
And now the same thing with numeric
```{r echo = FALSE, results='hide', comment="#",  null_prefix=TRUE}
store <- copy(messy_adult[,.(num1, num2, num3)])
```
```{r, comment="#",  null_prefix=TRUE}
messy_adult <- findAndTransformNumerics(messy_adult)
```
```{r echo = FALSE, results='asis', comment="#",  null_prefix=TRUE}
setnames(store, paste0(names(store), "_prev"))
toPlot <- cbind(head(store, n=6), data.frame("transfo" = rep("  =>", 6)), head(messy_adult[,.(num1, num2, num3)], n = 6))

kable(toPlot) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)

```

So now our data set is a bit less ugly.

# *Filter* functions
The idea now is to identify useless columns:

  * *constant* columns: they take the same value for every line,
  * *double* columns: they have an exact copy in the data set,
  * *bijection* columns: there is another column containing the exact same information (but maybe coded differently) for example col1: Men/Women, col2 M/W.

## Look for constant variables
  
```{r, results = 'hold', comment="#",  null_prefix=TRUE}
constant_cols <- whichAreConstant(messy_adult)
```
## Look for columns in double
```{r, results = 'hold', comment="#",  null_prefix=TRUE}
double_cols <- whichAreInDouble(messy_adult)
```

## Look for columns that are bijections of one another
```{r, results = 'hold', comment="#",  null_prefix=TRUE}
bijections_cols <- whichAreBijection(messy_adult)
```

To control this, let's have a look to the concerned columns:
```{r comment="#",  null_prefix=TRUE, echo = FALSE}
kable(head(messy_adult[, .(constant, date3, date4, num1, num3, education, education_num)])) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```
Indeed:



  * *constant* was build constant, it contains only 1,
  * *num1* and *num3* are equal,
  * *date3* and *date4* are separated by 2 days: date4 doesn't contain any new information for a ML algorithm,
  * *education* and *education_num* contains the same information one with a key index, the other one with the character corresponding. `whichAreBijection` keeps the character column.

## Filter them all
To directly filter all of them:
```{r results = "hold", comment="#",  null_prefix=TRUE}
ncols <- ncol(messy_adult)
messy_adult <- fastFilterVariables(messy_adult)
print(paste0("messy_adult now have ", ncol(messy_adult), " columns; so ", ncols - ncol(messy_adult), " less than before."))
```

4 useless rows have been deleted. Without those useless columns, your machine learning algorithm will at least be faster and maybe give better results.



# *Transform* functions
Before sending this to a machine learning algorithm, a few transformations should be performed. 

The idea with the functions presented here is to perform those transformations in a RAM efficient way.

## Dates differences
Since no machine learning algorithm handle Dates, one needs to transform them or drop them. A way to transform dates is to perform differences between every date. 

We can also add an analysis date to compare dates with the date your data is from. For example, if you have a birth-date you may want to compute age by performing today - birth-date. 

```{r comment="#",  null_prefix=TRUE}
messy_adult <- generateDateDiffs(messy_adult, cols = "auto", analysisDate = as.Date("2018-01-01"), units = "days")
```
```{r echo=FALSE, comment="#",  null_prefix=TRUE}
kable(cbind(data.frame("..." = rep("  ...", 6)), head(messy_adult[, (ncol(messy_adult) - 4):ncol(messy_adult), with = FALSE], n = 6))) %>%
  kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

## Transforming dates into aggregates
Another way to work around dates would be to aggregate them at some level. This time `drop` is set to `TRUE` in order to drop date columns. 
```{r comment="#",  null_prefix=TRUE}
messy_adult <- generateFactorFromDate(messy_adult, cols = "auto", type = "quarter", drop = TRUE)
```
```{r echo=FALSE, comment="#",  null_prefix=TRUE}
kable(cbind(data.frame("..." = rep("  ...", 6)), head(messy_adult[, (ncol(messy_adult) - 2):ncol(messy_adult), with = FALSE], n = 6))) %>%
  kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

## Generate features from character columns

Character columns are not handled by any machine learning algorithm, one should transform them. Function `generateFromCharacter` build some new feature from them, and then drop them.

```{r comment="#",  null_prefix=TRUE}
messy_adult <- generateFromCharacter(messy_adult, cols = "auto", drop = TRUE)
```
```{r echo=FALSE, comment="#",  null_prefix=TRUE}
kable(head(messy_adult[, .(mail.notnull, mail.num, mail.order)], n = 6)) %>%
  kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```


## Aggregate according to a key
To model something by country; one would want to to compute an aggregation of this table in order to have one line per country.

```{r comment="#",  null_prefix=TRUE}
agg_adult <- aggregateByKey(messy_adult, key = "country")
```
```{r echo=FALSE, comment="#",  null_prefix=TRUE}
kable(cbind(head(agg_adult[,c(1,13,23,35,45)]), data.frame("..." = rep("  ...", 6)))) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

Every time you have more than one line per individual this function would be pretty cool.

## Rounding
One might want to round numeric variables in order to save some RAM, or for algorithmic reasons:
```{r comment="#",  null_prefix=TRUE}
messy_adult <- fastRound(messy_adult, digits = 2)
```

```{r echo=FALSE, comment="#",  null_prefix=TRUE}
kable(cbind(head(messy_adult[, 1:6, with = FALSE], n = 6), data.frame("..." = rep("  ...", 6)))) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

# Handling NAs values
Then, let's handle NAs
```{r comment="#",  null_prefix=TRUE}
messy_adult <- fastHandleNa(messy_adult)
```
```{r echo=FALSE, comment="#",  null_prefix=TRUE}
print(cbind(head(messy_adult[,1:4, with = FALSE], n = 4), data.frame("..." = rep("  ...", 4)), head(messy_adult[,15:ncol(messy_adult), with = FALSE], n = 4)))
```


It set default values in place of NA. If you want to put some specific values (constants, or even a function for example mean of values) you should go check `fastHandleNa` documentation.


# Shape functions
There are two types of machine learning algorithm in R: those which accept data.table and factor, those which only accept numeric matrix. 

Transforming a data set into something acceptable for a machine learning algorithm could be tricky. 

The `shapeSet` function do it for you, you just have to choose if you want a *data.table* or a *numerical_matrix*.


First with *data.table*:

```{r comment="#",  null_prefix=TRUE}
clean_adult = shapeSet(copy(messy_adult), finalForm = "data.table", verbose = FALSE)
print(table(sapply(clean_adult, class)))
```

As one can see, there only are, numeric and factors.

Now with *numerical_matrix*:

```{r comment="#",  null_prefix=TRUE}
clean_adult <- shapeSet(copy(messy_adult), finalForm = "numerical_matrix", verbose = FALSE)
```

```{r echo=FALSE}
kable(cbind(head(clean_adult[,1:6]), data.frame("..." = rep("  ...", 6)))) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

As one can see, with `finalForm = "numerical_matrix"` every character and factor have been binarized.



# Full pipeline
Doing it all with one function is possible: 

To do that we will reload the ugly data set and perform aggregation.

```{r warning = FALSE, comment="#",  null_prefix=TRUE}
data("messy_adult")
agg_adult <- prepareSet(messy_adult, finalForm = "data.table", key = "country", analysisDate = Sys.Date(), digits = 2)
```

As one can see, every previously steps have been done.

Let's have a look to the result

```{r echo=FALSE, comment="#",  null_prefix=TRUE}
print(paste0(ncol(agg_adult), " columns have been built; for ", nrow(agg_adult), " countries."))
kable(cbind(head(agg_adult[,1:7]), data.frame("..." = rep("  ...", 6)))) %>%
   kable_styling(bootstrap_options = c("striped", "hover"), full_width = FALSE, font_size = 12)
```

# Description

Finally, to generate a description file from this data set, function `description` is available.

It will describe, the set and its variables. Here we put `level=0` to have some global descriptions:
```{r, tidy=TRUE, tidy.opts=list(length.cutoff=10), comment="#",  null_prefix=TRUE}
description(agg_adult, level = 0)
```



# Conclusion
We presented some of the functions of *dataPreparation* package. There are a few more available, plus they have some parameters to make their use easier. So if you liked it, please go check the package documentation (by installing it or on [CRAN](https://CRAN.R-project.org/package=dataPreparation/dataPreparation.pdf))


We hope that this package is helpful, that it helped you prepare your data in a faster way.


If you would like to give us some feedback, report some issues, add some features to this package, please tell us on [GitHub](https://github.com/ELToulemonde/dataPreparation/issues). Also if you want to contribute, please don't hesitate to contact us.



