
\documentclass{article}

\usepackage{url}
\usepackage{natbib}
\usepackage{graphics}
\usepackage{amsmath}
\usepackage{indentfirst}
\usepackage[utf8]{inputenc}
\usepackage{fullpage}

% \VignetteIndexEntry{Transform-dataset}

\begin{document}

\SweaveOpts{eps = FALSE, pdf = TRUE}
\SweaveOpts{prefix.string=fig/transform}
<<include=FALSE,echo=FALSE>>=
  if(!file.exists("fig")) dir.create("fig")  
@
\setkeys{Gin}{width=1\textwidth}

<<foo,include=FALSE,echo=FALSE>>=
options(keep.source = TRUE, width = 100)
foo <- packageDescription("Rcell")
@

\title{Data manipulation with Rcell (Version \Sexpr{foo$Version})}
\author{Alan Bush}
\maketitle

\section{Introduction}

Once you have your data loaded into \textbf{R}, you can filter it and plot it as shown in ``Getting Started with Rcell''. To read that document type in the console

<<eval=FALSE,echo=TRUE>>=
vignette('Rcell')
@

But many times we want to do some manipulation or transformations on the data before plotting it. In this document you'll see how this can be done using \textbf{Rcell}.

\section{Transforming variables}

If you haven't done so, load the \textbf{Rcell} package and the example dataset with 

<<echo=TRUE,eval=FALSE>>=
library(Rcell) 
data(ACL394filtered)
@

<<include=FALSE,echo=FALSE,eval=TRUE>>=
library(Rcell)
data(ACL394data)
X$images$path<-factor(system.file('img', package='Rcell'))
@

The easiest way to modify your dataset is to create new variables from existing ones. For example, its desirable to correct the fluorescence measure of a cell by the background fluorescence. To do this for the YFP channel we can use the \verb@f.bg.y@ variable, that contains the most common value (mode) for pixels not associated with any cell. If a cell has no fluorophores, we expect it to have a total fluorescence equivalent to \verb@f.bg.y@ times the number of pixels of the cell, \verb@a.tot@. So the background corrected fluorescence can be calculated as \verb@f.tot.y - f.bg.y*a.tot@. To creare a new variable called \verb@f.total.y@ with the corrected value for fluorescence we can use the \verb@transform@ funcion. As all other \textbf{Rcell} functions, the first argument is the \verb@cell.data@ object to tranform.

<<include=TRUE>>=
X<-transform(X,f.total.y=f.tot.y - f.bg.y*a.tot)
@

Once created, you can use the new variable as any other variable of the dataset. You can create several variables in a single call to \verb@transform@, as shown next for the fluorescence density variables.

<<include=TRUE>>=
X<-transform(X,f.density.y=f.tot.y/a.tot,f.density.c=f.tot.c/a.tot)
@

You can keep track of the variables you've created with the \verb@summary@ function, that will display among other things the ``transformed'' variables with their definition.

<<eval=FALSE,include=TRUE>>=
summary(X)
@

\section{Merging variables}

Sometimes there is no formula to specify the new variable you want to create. For example, you might want to create a variable that describes the treatment each position received. In the example dataset (\verb@help(ACL394)@) each position received a different dose of alpha-factor pheromone, according to the Table~\ref{tab:t01}. 

<<echo=FALSE,results=tex>>=
library(xtable)
mytable<-data.frame(pos=with(X,unique(pos)),alpha.factor=rep(c(1.25,2.5,5,10,20),each=3))
print(xtable(mytable, table.placement = "tbp",caption="example data.frame to merge",label="tab:t01"), include.rownames=FALSE)
@

You can create this table in Excel\footnote{or from R: \textbf{\texttt{mytable<-data.frame(pos=with(X,unique(pos)),alpha.factor=rep(c(1.25,2.5,5,10,20),each=3))}}} and save it as a tab delimited text file. If you name it ``mytable.txt'', then you can loaded into \textbf{R} with \verb@read.table@. The best option is to save the file in your working directory, or to change your working directory to where you saved the file (see \verb@?setwd@).   

<<eval=FALSE,include=TRUE>>=
mytable<-read.table("mytable.txt",head=TRUE)
@  

If the first row of your text file contains the column names (recommended), you have to set \emph{head} to \verb@TRUE@ in \verb@read.table@. Once loaded you can add the new data to your dataset using the \verb@merge@ function. This function looks for common variables between \verb@X@ and \verb@mytable@ and, if it finds them it merges the dataset according to those common variables. Be aware that the names of the columns of \verb@mytable@ have to match EXACTLY\footnote{\textbf{R} is case-sensitive so \textbf{\texttt{"pos"}} is different to \textbf{\texttt{"Pos"}}} to the variables of \verb@X@\footnote{You can see these variables with \textbf{\texttt{summary(X)}}}. In this case it will merge by \emph{pos}. You can also specify the variable to merge by with the \emph{by} argument.

<<include=TRUE>>=
X<-merge(X,mytable)
@

\section{Transform By}

A common transformation is normalization, i.e. dividing the value of a variable by the ``basal'' level. For example, we might be interested in the fold icrease of YFP fluorescence through time. So we need to divide the measured value at each time by the value at time cero, and we need to this for every cell. How can we do this? The steps we should follow are the following:
\begin{enumerate}
\item Divide the dataset by cell, creating a table for each cell. 
\item Indentify the value of fluorescence for time cero.
\item Create a new variable by dividing the fluorescence at each time by the value at time cero.
\item Join the cells datasets back together to retrieve the original dataset with the new variable.  
\end{enumerate}

All these steps are done by the function \verb@transform.by@, but it requires information on how each step should be done. For the first step, it needs to know how to partition the dataset. This is specified by passing a quoted list of variable, whos combination of levels specify a group. For example, if you want to divide the datset by position, the second argument of \verb@transform.by@ should be \verb@.(pos)@. If you want to divide your dataset by cell use \verb@.(pos,cellID)@. Note that cells in different position can have the same cellID, so the combination of \verb@pos@ and \verb@cellID@ uniquely identifies a cell. The variable \verb@ucid@ (for Unique Cell ID) is another way to uniquely identify a cell. 
Next we need to specify the name of the new variable to be created (\verb@norm.f.tot.y@ for example), and the definition for this variable, \verb@f.tot.y/f.tot.y[t.frame==0]@. With the square brackets we are selecting the value of \verb@f.tot.y@ when \verb@t.frame@ is cero. Remember to use the logical operator \verb@==@ and not the assignation operator \verb@=@ within the brackets.

<<include=TRUE>>=
X<-transform.by(X,.(pos,cellID),norm.f.tot.y=f.tot.y/f.tot.y[t.frame==0])
@

You can see the raw and normalized data in Figure~\ref{fig:f01}. Another way to normalize the data, is dividing by the mean of the first three values.

<<include=TRUE>>=
X<-transform.by(X,.(pos,cellID),norm2.f.tot.y=f.tot.y/mean(f.tot.y[t.frame<=2]))
@

\begin{figure}
\begin{center}
<<fig=TRUE,echo=FALSE,width=9,height=4.5>>=
grid.newpage()
pushViewport(viewport(layout = grid.layout(1, 20)))
print(
cplot(X,f.tot.y~t.frame,group=ucid,geom="line",subset=pos==29)
, vp = viewport(layout.pos.row = 1, layout.pos.col = 1:10))
print(
cplot(X,norm.f.tot.y~t.frame,group=ucid,geom="line",subset=pos==29)
, vp = viewport(layout.pos.row = 1, layout.pos.col = 11:20))
@
\caption{Left: raw single cell time course for YFP fluorescence. Right: Same data normalized to each cells value at time cero.}
\label{fig:f01}
\end{center}
\end{figure}

\section{Aggregating your data}     

To calculate summary statistics you can use the \verb@aggregate@ function, that returns an aggregated table. That means that the value of each cell of this aggregated table is caculated from more than one cell of the original table. For example you might be intereset in getting the mean YFP fluroescence for each pheromone dose. \verb@aggregate@ accepts two notations that give equivalent results.

<<eval=FALSE,echo=TRUE>>=
aggregate(X,.(alpha.factor),select="f.total.y")
aggregate(X,f.total.y~alpha.factor) #formula notation
@
<<include=TRUE,echo=FALSE>>=
aggregate(X,f.total.y~AF.nM)
@

You can calculate other statistics using the \emph{FUN} argument, and you can include more than one variable. Here we calculate the median for \verb@f.tot.y@, \verb@f.tot.c@ and \verb@a.tot@. Note the use of the wildcard in the \emph{select} argument.  

<<include=TRUE>>=
aggregate(X,.(alpha.factor),select=c("f.tot.*","a.tot"),FUN=median)
@

The partition of the dataset can be done by more than one variable, for example by dose and time. Using the function \verb@funstofun@ from the \textbf{reshape} package, you can calculate more than one statistic at once.

<<include=TRUE>>=
aggregate(X,f.density.y~t.frame+alpha.factor,FUN=funstofun(median,sd),subset=t.frame%%3==0)
@

     
\section{Evaluating expressions in your dataset}     

Using the \verb@with@ function, you can evaluate a expression in a enviroment created from your dataset. That means that you can use the names of your variables directly, without any prefix. 
For example to calculate the mean of \verb@f.tot.y@ from position 1

<<include=TRUE>>=
with(X,mean(f.tot.y[pos==1]))
@

If you don't use \verb@with@ you have to write the full identifier of the variable, and the code becomes longer and harder to understand. For exaple, the same result can be obtained with

<<include=TRUE,eval=FALSE>>=
mean(X$data$f.tot.y[X$data$pos==1])
@

\section{Exporting your data}

Although you can do much of your analysis using Rcell functions, you might need to export the data to some other application or use another package within \textbf{R}. To retrieve the entire dataset in a data.frame, use the double square brackets notation. This returns the registers that pass the QC.filter.

<<eval=FALSE,echo=TRUE>>=
df<-X[[]]
@
    
This dataset is usually big, and has many varaibles or registers you are not interested in. You can subset the datset as you would a data.frame (but using double brackets)   
    
<<eval=FALSE,echo=TRUE>>=
df<-X[[pos==1,c("cellID","f.tot.y","a.tot")]]
@

You can then save the data.frame to a file with \verb@write.table@, or use it in another \textbf{R} package.  

For some kinds of data analysis you need your data in a different form than the one \textbf{Rcell} uses. You can use the \verb@reshape@ function to reshape your data. For instance, a common restructuring is to display time as different columns, and individual cells as different rows. You can obtain this sort of data.frames with the following command.

<<include=TRUE>>=
reshape(X,pos+cellID~variable+t.frame,select="f.tot.y",subset=pos<=2&cellID<=10&t.frame%%2==0)
@

see \verb@help(reshape.cell.data)@ for more details. 


 
 







\begin{thebibliography}{}

\bibitem[Pau et al. (2010)]{pau2010}
Pau, Fuchs et al. (2010).
\newblock EBImage: an R package for image processing with applications to cellular phenotypes.
\newblock \emph{Bioinformatics}, 26(7):979-981.

\bibitem[Colman-Lerner et al. (2005)]{acl2005}
Colman-Lerner, Gordon et al. (2005).
\newblock Regulated cell-to-cell variation in a cell-fate decision system.
\newblock \emph{Nature}, 437(7059):699-706.

\bibitem[Chernomoretz et al. (2008)]{cpmcb2008}
Chernomoretz, Bush et al. (2008).
\newblock Using Cell-ID 1.4 with R for Microscope-Based Cytometry.
\newblock \emph{Curr Protoc Mol Biol.}, Chapter 14:Unit 14.18.

\end{thebibliography}

\end{document}
