pandoc, markdown and pander

Pandoc + markdown seem to be a great way of documenting my work.

Markdown syntax is very simple and allows to add basic formatting and figures to an otherwise simple text document, without obfuscating the actual text.

Then I simply compile the document using the pandoc command:

pandoc -o document.docx document.md
pandoc -o document.pdf  document.md

There are some more tricks, of course, and plenty of output formats are possible. One thing I was struggling with was that images in docx files were much too large. It turns out that the PNG graphics I generate from PDFs (which, in turn, come from R) lacked the information about density units. I was using the convert program from ImageMagick, and it turns out it is necessary to add the option -units PixelsPerInch:

convert -density 300 -units PixelsPerInch image.pdf image.png

Another thing that I found useful was the pander package. Of course, there is this whole science of generating dynamic documents and reports from R using Sweave or knittr, but at the moment I rather produce two files: a commented R pipeline and, separately, a report in markdown format.

(The reason for not using knittr is that I given that I work with some very large data sets that sometimes take ages to compute, I would have to work out the details of cacheing and handling code that takes a while to execute. Also, I want to have a document with all commands, for me, and report without any R code for everyone else).

Pander allows to create nice tables in R that can be directly copied and pasted to a markdown document (of course, pander is so much more, but this is my main use at the moment):

pander(foo, 
       emphasize.strong.cols=1, justify="left", 
       style="simple", digits=2, split.tables=Inf)

I was astonished how nice the resulting word file is. The PDFs, which are produced by TeX/LaTeX, I think, are actually more trouble, for example because LaTeX disregards my order of figures and tables, they are all floating objects and there is no easy way to change this from within the document.

Creating a graph with variable edge width in Rgraphviz

This was waaaaay more complicated than necessary. Figuring it out took me almost a whole day.

In essence, there is the graph package in R, which provides graph objects and methods, and there is the Rgraphviz package, which allows you to plot the graphs on the screen. They work well.

library(graph)
library(Rgraphviz)
nodes <- c( LETTERS[1:3] )
edgesL <- list( A=c("B", "C"), B=c("A", "C"), C=c("B", "A" ) )
graph <- new( "graphNEL", nodes= nodes, edgemode="undirected", edgeL=edgesL )
rag <- agopen( graph, "" )
plot(rag)

Here the output:

example0

So far, so good. If I wanted to color the edge from A to C red, here is what I could do:

eAttrs <- list( color=c( "A~C"="red" ) )
rag <- agopen( graph, "", edgeAttrs=eAttrs )
plot(rag)

example2

The attribute “color” works well. The man page for AgEdge gives us other attributes, specifically, “lwd” which specifies the width of the edge. However, it is not possible to set the edge widths using the above method. I found that the following code works for me:

setEdgeAttr <- function( graph, attribute, value, ID1, ID2 ) {

  idfunc <- function(x) paste0( sort(x), collapse="~" )
  all.ids <- sapply( AgEdge(graph), 
    function(e) idfunc( c( attr(e, "head"), attr( e, "tail" ))))

  sel.ids <- apply(cbind( ID1, ID2 ), 1, idfunc )

  if(!all(sel.ids %in% all.ids)) stop( "only existing edges, please" )
  sel <- match( sel.ids, all.ids )

  for(i in 1:length(sel)) {
    attr( attr( graph, "AgEdge" )[[ sel[i] ]], attribute ) <- value[i]
  }

  return(graph)
}

Example:

rag <- agopen( graph, "" )
rag <- setEdgeAttr( rag, "lwd", c(5, 20), c("B", "B"), c( "A", "C" ) )
plot(rag)

example4

What a colossal waste of my time. However, I need a visualization with graphs; and it needs to take a custom node drawing function as an argument, so there.

Adding authentication to a shiny server

Umph, that was a tough one. I spent ages figuring out how to do it correctly. I have a server running apache (on port 80) and shiny on port (say) 11111. Shiny has its own document root, and within this root, we have a shiny app, say, “example”. So to view this app you need to type http://server:1111/example/. So far, so good. What I wanted, though, was (i) some kind of password protection for the app, and (ii) calling the app from the URL http://server/example/. Turns out it can be done, but it was not trvial.

First, I modified configuration of the shiny server to listen only to the specified port and only to the localhost; this prevents anyone from any other machine to connect to shiny:

server {
  listen 11111 127.0.0.1;
  location / {
    site_dir /srv/shiny-server;
    log_dir /var/log/shiny-server;
    directory_index off;
  }
}

Now to apache. In the httpd.conf file, I have added The following:

         <VirtualHost *:80>

            Redirect /example /example/
            ProxyPass /example/ http://127.0.0.1:11111/example/
            ProxyPassReverse /example/ http://127.0.0.1:11111/example/

            <Location /example>
                AuthType Basic
                AuthName "Enter your login name and password"
                AuthUserFile /etc/httpd/htpasswd.users
                Require valid-user
            </Location>

         </VirtualHost>

This makes apache work as a proxy to the shiny server; however, with the added benefit of a simple authentication for the shiny contents.

It took me quite some time to figure out that without the Redirect directive above, http://server/example/ works, but http://server/example (without the slash) doesn’t.

Finally, I created new users with htpasswd.

tagcloud: creating tag / word clouds

May I present my new package: tagcloud. Tag clouds is for creating, um, tag clouds (aka word clouds). It is based on the code from the wordcloud package, but (i) has no tools for analysing word frequencies, instead (ii) focuses on doing better tag clouds. As to (ii), it adapts to the geometry of the window better and can produce different layouts. Also, with the extrafont package, it can use just any fonts you can think of:

sample8

The general syntax is simple. The function tagcloud takes one mandatory argument, the tags to display — a character vector. In the example below, I use the font names available through the extrafont packageยน, but it can be anything.

library( extrafont )
tags <- sample( fonts(), 50 )
tagcloud( tags )
dev.copy2pdf( file= "sample1.pdf", out.type= "cairo" )

Note the use of cairo in the above, otherwise the PDF does not include the fonts and the result is not impressive. Here is the result:

sample1

OK, let’s add some weights and colors. Also, wouldn’t it be cool if each font name was displayed in the actual font?

weights <- rgamma( 50, 1 )
colors <- colorRampPalette( brewer.pal( 12, "Paired" ) )( 50 )
tagcloud( tags, weights= weights, col= colors, family= tags )

sample2

How about mixed vertical and horizontal tags?

tagcloud( tags, weights= weights, col= colors, family= tags, 
fvert= 0.5 )

sample3

The fvert parameter specifies the proportion of tags that should be displayed vertically rather than horizontally.

Or a different layout?

tagcloud( tags, weights= weights, col= colors, 
family= tags, algorithm= "fill" )

sample4

Tagclouds comes with additional tools. Firstly, you have editor.tagcloud — a very minimalistic interactive editor. You need to store the object invisibly returned from tagcloud:

tc <- tagcloud( tags, weights= weights, col= colors, family= tags )
tc2 <- editor.tagcloud( tc )
plot( tc2 )

With strmultline you can break up long, multi-word tags into multi-line tags:

tagcloud( strmultline( tags ), weights= weights, col= colors, family= tags )

The result is as follows (notice “Andale Mono” or “DejaVu Sans Light”):

sample5

Finally, smoothPalette makes it easy to generate a gradient from numbers. Imagine that we want to code some other numeric information (this could be a p-value, for example) with a smooth gradient from light grey (low value) to black (high value):

newvar <- runif( 50 )
colors2 <- smoothPalette( newvar )
tagcloud( tags, weights=weights, col=colors2, family= tags )

By default, smoothPalette uses a grey-white gradient, but it can actually use any kind of color palette.

sample6


1: In order to use the fonts installed on your system, you need to import them — preferably as root — using the extrafont package. At least in my Ubuntu installation you should provide the paths to where your TTF fonts are installed, for example:

library( extrafont )
font_import( paths= "/usr/share/fonts/truetype/" )

Troubles with sapply

Say your function in sapply returns a matrix with a variable number of rows. For example

ff <- function( i ) matrix( i, ncol= 3, nrow= i )
pipa <- sapply( 1:3, , simplify= "array" )

sapply is too stupid to see the pattern here (or maybe I don’t know how to cast the return value into an appropriate shape…). The result of the above is a list:

[[1]]
     [,1] [,2] [,3]
[1,]    1    1    1

[[2]]
     [,1] [,2] [,3]
[1,]    2    2    2
[2,]    2    2    2

[[3]]
     [,1] [,2] [,3]
[1,]    3    3    3
[2,]    3    3    3
[3,]    3    3    3

However, we can turn this list of matrices into a simple matrix using Reduce:

pipa <- Reduce( function( ... ) rbind( ... ), pipa )

biomaRt

The biomaRt package allows to query the gigantic ensembl data base directly from R. Since it happens only once in a while, I find myself reading the biomaRt vignette every time I’m using it. Here are the crucial points for the most common application.

ensembl <- useMart( "ensembl", dataset="hsapiens_gene_ensembl" )
f <- listFilters( ensembl )
a <- listAttributes( ensembl )

The data frames f and a hold the available filters and attributes, respectively. To retrieve information, we need to specify by what key we are searching (filters), what keys do we want to retrieve from the database (attributes), what are the values of our search keys (values) and from which mart we want to retrieve the information.

g <- c( "CCL2", "DCN", "LIF", "PLAU", "IL6" )
d <- getBM( 
        attributes= c( "entrezgene", "description", "uniprot_genename" ), 
        filters= "hgnc_symbol", 
        values= g, 
        mart= ensembl )

This returns the following:

  entrezgene                                                      description uniprot_genename
1       6347    chemokine (C-C motif) ligand 2 [Source:HGNC Symbol;Acc:10618]             CCL2
2       1634                            decorin [Source:HGNC Symbol;Acc:2705]              DCN
3       3569 interleukin 6 (interferon, beta 2) [Source:HGNC Symbol;Acc:6018]              IL6
4       3976         leukemia inhibitory factor [Source:HGNC Symbol;Acc:6596]              LIF
5       5328   plasminogen activator, urokinase [Source:HGNC Symbol;Acc:9052]             PLAU

OK, one problem that presents itself is that if there are multiple matching values for a given key, you are getting multiple lines for that key. It may be useful to aggregate this information.

g <- c( "BTC" )
d <- getBM( attributes= c( "uniprot_genename", "description", "ensembl_gene_id" ), filters= "hgnc_symbol", values= g, mart= ensembl )

will return two lines:

  uniprot_genename                                description ensembl_gene_id
1              BTC betacellulin [Source:HGNC Symbol;Acc:1121] ENSG00000261530
2              BTC betacellulin [Source:HGNC Symbol;Acc:1121] ENSG00000174808

We can use aggregate to collapse the ensembl IDs:

ff <- function( x ) { x <- x[ !is.na( x ) & x != "" ] ; paste( unique( x ), collapse= ";" ) }
d <- aggregate( d, by= list( d$uniprot_genename ), FUN=ff )[,-1]

The ff function ignores empty strings and NAs. Also, we remove the first column as it is added by the aggregate function. The result:

  Group.1 uniprot_genename                                description                 ensembl_gene_id
1     BTC              BTC betacellulin [Source:HGNC Symbol;Acc:1121] ENSG00000261530;ENSG00000174808

Pathway analysis in R and BioConductor.

There are many options to do pathway analysis with R and BioConductor.

First, it is useful to get the KEGG pathways:

library( gage )
kg.hsa <- kegg.gsets( "hsa" )
kegg.gs2 <- kg.hsa$kg.sets[ kg.hsa$sigmet.idx ]

Of course, “hsa” stands for Homo sapiens, “mmu” would stand for Mus musuculus etc.

Incidentally, we can immediately make an analysis using gage. However, gage is tricky; note that by default, it makes a pairwise comparison between samples in the reference and treatment group. Also, you just have the two groups — no complex contrasts like in limma.

res <- gage( E, gsets= kegg.gs2, 
       ref= which( group == "Control" ), 
       samp= which( group == "Treatment" ),
       compare= "unpaired", same.dir= FALSE )

Now, some filthy details about the parameters for gage.

  • E is the matrix with expression data: columns are arrays and rows are genes. If you use a limma EList object to store your data, this is just the E member of the object (rg$E for example). However, and this is important, gage (and KEGG and others) are driven by the Entrez gene identifiers, and this is not what you usually have when you start the analysis. To get the correct array, you need to

    • select only the genes with ENTREZ IDs,
    • make sure that there are no duplicates
    • change the row names of E to ENTREZ IDs
  • gsets is just a list of character vectors; the list names are the pathways / gene sets, and the character vectors must correspond to the column names of E.
  • ref and samp are the indices for the “reference” and “sample” (treatment) groups. This cannot be logical vectors. Only two groups can be compared at the same time (so for example, you cannot test for interaction).
  • compare — by default, gage makes a paired comparison between the “reference” and “sample” sets, which requires of course to have exactly the same number of samples in both sets. Use “unpaired” for most of your needs.
  • same.dir — if FALSE, then absolute fold changes are considered; if TRUE, then up- and down-regulated genes are considered separately

To visualise the changes on the pathway diagram from KEGG, one can use the package pathview. However, there are a few quirks when working with this package. First, the package requires a vector or a matrix with, respectively, names or rownames that are ENTREZ IDs. By the way, if I want to visualise say the logFC from topTable, I can create a named numeric vector in one go:

setNames( tt$logFC, tt$EID )

Another useful package is SPIA; SPIA only uses fold changes and predefined sets of differentially expressed genes, but it also takes the pathway topology into account.