Dimont: Difference between revisions

From Jstacs
Jump to navigationJump to search
No edit summary
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
by Jan Grau, Stefan Posch, Ivo Grosse, and Jens Keilwagen
by Jan Grau, Stefan Posch, Ivo Grosse, and Jens Keilwagen


ChIPper is a universal tool for de-novo motif discovery. ChIPper has successfully been applied to ChIP-seq, ChIP-exo and protein-binding microarray (PBM) data.
Dimont is a universal tool for de-novo motif discovery. Dimont has successfully been applied to ChIP-seq, ChIP-exo and protein-binding microarray (PBM) data.


We provide ChIPper as a public web-server, a web-application that can be installed in a local Galaxy server, and as a command line program.
We provide Dimont as a public web-server, a web-application that can be installed in a local Galaxy server, and as a command line program.


== ChIPper web-server ==
== Paper ==
ChIPper is available as a public web-server at [http://galaxy.informatik.uni-halle.de galaxy.informatik.uni-halle.de].
If you use Dimont, please cite
 
J. Grau, S. Posch, I. Grosse, and J. Keilwagen. [http://nar.oxfordjournals.org/content/early/2013/09/19/nar.gkt831 A general approach for discriminative de-novo motif discovery from high-throughput data]. ''Nucleic Acids Research'', 41(21):e197, 2013. doi:
10.1093/nar/gkt831
 
== Dimont web-server ==
Dimont is available as a public web-server at [http://galaxy.informatik.uni-halle.de galaxy.informatik.uni-halle.de].
 
== Dimont @ Galaxy toolshed ==
 
Dimont, [[Dimont#DimontPredictor|DimontPredictor]], [[Dimont#DimontGenomeScan|DimontGenomeScan]], and a [[Dimont#Data preparation|data extraction]] tool are also available from the [http://toolshed.g2.bx.psu.edu Galaxy tool shed] in repository [http://toolshed.g2.bx.psu.edu/view/grau/dimont_motif_discovery dimont_motif_discovery]. From the toolshed, all tools can be installed with minimal effort from the Galaxy admin interface.
 
== Dimont @ Chipster ==
 
 
 
Dimont is also provided with [http://chipster.csc.fi Chipster] starting from Chipster release 2.11.


== Download ==
== Download ==


ChIPper is implemented in Java using Jstacs. You can download the command line application as a [http://www.jstacs.de/download.php?which=ChIPper Jar].
Dimont is implemented in Java using Jstacs. You can download the command line application as a [http://www.jstacs.de/download.php?which=Dimont Jar].
In addition, we provide the [http://www.jstacs.de/download.php?which=ChIPperWeb Jar of the Galaxy web-application] for installing it in your local [http://getgalaxy.org Galaxy] server.
In addition, we provide the [http://www.jstacs.de/download.php?which=DimontWeb Jar of the Galaxy web-application] for installing it in your local [http://getgalaxy.org Galaxy] server.


ChIPper will be part of the next public release of the Jstacs library.
Dimont will be part of the next public release of the Jstacs library.


== Running the command line application ==
== Running the command line application ==
Line 51: Line 67:
<td>Value tag (The tag for the value information in the FastA-annotation of the input file)</td>
<td>Value tag (The tag for the value information in the FastA-annotation of the input file)</td>
<td>String</td>
<td>String</td>
</tr>
<tr>
<td><font color="green">sd</font></td>
<td>Standard deviation (The standard deviation of the position distribution centered at the position specified by the position tag, valid range = [1.0, 10000.0], default = 75.0)</td>
<td>Double</td>
</tr>
</tr>
<tr>
<tr>
Line 102: Line 123:
  ...
  ...


where the anchor point is given as 50 for the first two sequences, and the confidence amounts to 515 and 199, respectively. The FastA comment may contain additional annotations of the format <code>key1 : value1; key2: value2;....</code>. We also provide an [http://www.jstacs.de/downloads/chipper-example.fa example] input file.
where the anchor point is given as 50 for the first two sequences, and the confidence amounts to 515 and 199, respectively. The FastA comment may contain additional annotations of the format <code>key1 : value1; key2: value2;....</code>. We also provide an [http://www.jstacs.de/downloads/dimont-example.fa example] input file.


Accordingly, you would need to set the parameter "position" to <code>peak</code> and the parameter "value" to <code>signal</code> for the input file.
Accordingly, you would need to set the parameter "position" to <code>peak</code> and the parameter "value" to <code>signal</code> for the input file.
Line 118: Line 139:
The parameter "delete" defines if BSs of already discovered motifs should be deleted, i.e., "blanked out", from the sequence before searching for futher motifs.
The parameter "delete" defines if BSs of already discovered motifs should be deleted, i.e., "blanked out", from the sequence before searching for futher motifs.


Using the parameter "threads" you can set the number of threads that are used by ChIPper. If you do not provide a value for this parameter, all available threads are used.
Using the parameter "threads" you can set the number of threads that are used by Dimont. If you do not provide a value for this parameter, all available threads are used.


For instance, you can start ChIPper on the [http://www.jstacs.de/downloads/chipper-example.fa example] input file by calling
For instance, you can start Dimont on the [http://www.jstacs.de/downloads/dimont-example.fa example] input file by calling


  java -jar ChIPper.jar home=./ data=chipper-example.fa position=peak value=signal threads=2
  java -jar Dimont.jar home=./ data=dimont-example.fa infix=example position=peak value=signal threads=2


using a PWM as motif model, a uniform distribution as background model and two threads for optimization.
using a PWM as motif model, a uniform distribution as background model and two threads for optimization.


For larger data sets, the standard memory allocation of Java may not be sufficient. You can increase the amount of RAM allocated by the virtual machine using the VM parameters <code>-Xms</code> and <code>-Xmx</code>. To start ChIPper using initially 512MB and a maximum of 2GB of RAM, you call
For larger data sets, the standard memory allocation of Java may not be sufficient. You can increase the amount of RAM allocated by the virtual machine using the VM parameters <code>-Xms</code> and <code>-Xmx</code>. To start Dimont using initially 512MB and a maximum of 2GB of RAM, you call
 
java -Xms512M -Xmx2G -jar Dimont.jar home=./ data=dimont-example.fa infix=example position=peak value=signal threads=2
 
== Data preparation ==
We provide a [http://www.jstacs.de/downloads/extract_data.pl Perl script] for generating an annotated FastA file in the format required by Dimont from chromosome sequences and a BED file.
The chromosome sequences should be provided in a directory containing one FastA file for each chromosome, where the filename should match the chromosome identifier of the BED file with extension <code>.fa</code>. For instance, if chromosome 1 is identified as <code>chr1</code> in the BED file, the corresponding sequence should be available in a file <code>chr1.fa</code>.
The complete list of parameters of this Perl script is
perl extract_data.pl <chromDir> <bedfile> <chromcol> <poscol> <widthcol> <statcol> <outfile>
   
    <chromDir>: the directory containing the chromosome FastAs (as
                typically downloaded from UCSC as "chromFa"): one FastA per
                chromosome, one sequence per file
    <bedfile>:  the file containing the peaks in tabular format,
                e.g., bed, gff, narrowPeak
    <chromcol>: the column of <bedfile> containing the chromosome
    <poscol>:  the column of <bedfile> containing the position relative to
                the chromosome start
    <widthcol>: either i) <int> the column of <bedfile> containing the width of
                          the peak or any region to be extracted
                          symmetrically around <poscol>
                or    ii) f<int> a fixed width of all regions, centered at the
                          position given in <poscol>, e.g., f100 for a fixed
                          width of 100 bp
    <statcol>:  the column of <bedfile> containing the peak statistic
                or a similar measure of confidence
    <outfile>:  the path to the output file, written as FastA
 
== ChIP-seq input data ==
 
We thank Xiaotu Ma, Ashwinikumar Kulkarni, and Michael Q. Zhang for kindly providing the ChIP-seq binding regions compiled for their publication
 
[http://nar.oxfordjournals.org/content/40/7/e50.abstract X. Ma, A. Kulkarni, Z. Zhang, Z. Xuan, R. Serfling, and M. Q. Zhang. A highly efficient and effective motif discovery method for chip-seq/chip-chip data using positional information. Nucleic Acids Research, 40(7):e50, 2012.]
 
Based on these data, we provide [http://www.jstacs.de/downloads/chipseq-data.zip Dimont input data] in the annotated FastA format required by Dimont.
 
If you use these data, please cite above publication and all appropriate original publications related to the data. In the archive, you find a ReadMe file containing the references to the original publications for each of the data sets.
 
== DimontPredictor ==
In addition to the Dimont application, we also provide a command line program and a web-application of a prediction program that can be used to predict binding sites for motifs discovered by Dimont in small (training or test) data sets.
This tool loads a Dimont model from the XML output of a previous Dimont run. In addition to the input parameters explained above, you can specify a threshold on the p-value of predicted binding sites.
DimontPredictor may be useful if you, for instance, want to predict binding sites of a previously discovered motifs in other data sets, or if you want to try different p-values for filtering predictions.
However, you should not use this tool for genomewide scans (cf. below).
 
You can download Jars of the [http://www.jstacs.de/download.php?which=DimontPredictor command line program] and [http://www.jstacs.de/download.php?which=DimontPredictorWeb web-application] of DimontPredictor.
 
The web-application of DimontPredictor is also installed at [http://galaxy.informatik.uni-halle.de galaxy.informatik.uni-halle.de].


java -Xms512M -Xmx2G -jar ChIPper.jar home=./ data=chipper-example.fa position=peak value=signal threads=2
If you like to use the DimontPredictor with [http://www.jstacs.de/index.php/Slim Slim models], you need to download the [http://www.jstacs.de/downloads/SlimDimontPredictor.jar SlimDimontPredictor], which is currently only available as command line application.


== ChIPper predictor ==
== DimontGenomeScan ==
In addition to the ChIPper application, we also provide a command line program and a web-application of a prediction program that can be used to predict binding sites for motifs discovered by ChIPper.
This tool loads a ChIPper model from the XML output of a previous ChIPper run. In addition to the input parameters explained above, you can specify a threshold on the p-value of predicted binding sites.
ChIPperPredictor may be useful if you, for instance, want to predict binding sites of a previously discovered motifs in other data sets, or if you want to try different p-values for filtering predictions.


You can download Jars of the [http://www.jstacs.de/download.php?which=ChIPperPredictor command line program] and [http://www.jstacs.de/download.php?which=ChIPperPredictorWeb web-application] of ChIPperPredictor.  
In addition to the Dimont and DimontPredictor, we also provide a command line program for genomewide scans. This tool loads a Dimont model from the XML output of a previous Dimont run and a genome to be scanned in fastA format.


The web-application of ChIPperPredictor is also installed at [http://galaxy.informatik.uni-halle.de galaxy.informatik.uni-halle.de].
You can download the DimontGenomeScan as a [http://www.jstacs.de/downloads/DimontGenomeScan.jar Jar].


== Installing the web-application ==
== Installing the web-application ==
Line 145: Line 209:


The command line application writes its Galaxy tool definition file itself. If you are in the directory containing the command-line program for Galaxy, you can create the tool definition file by calling
The command line application writes its Galaxy tool definition file itself. If you are in the directory containing the command-line program for Galaxy, you can create the tool definition file by calling
  <code>java -jar ChIPperWeb.jar --create ChIPperWeb.xml</code>
  <code>java -jar DimontWeb.jar --create DimontWeb.xml</code>


Afterwards, this directory contains the tool definition file <code>ChIPperWeb.xml</code>. Now you can register ChIPper in the Galaxy <code>tool_conf.xml</code> file. For details, see the [http://wiki.g2.bx.psu.edu/Admin/Tools/Add%20Tool%20Tutorial Galaxy tutorial for adding new tools].
Afterwards, this directory contains the tool definition file <code>DimontWeb.xml</code>. Now you can register Dimont in the Galaxy <code>tool_conf.xml</code> file. For details, see the [http://wiki.g2.bx.psu.edu/Admin/Tools/Add%20Tool%20Tutorial Galaxy tutorial for adding new tools].

Latest revision as of 09:49, 17 January 2017

by Jan Grau, Stefan Posch, Ivo Grosse, and Jens Keilwagen

Dimont is a universal tool for de-novo motif discovery. Dimont has successfully been applied to ChIP-seq, ChIP-exo and protein-binding microarray (PBM) data.

We provide Dimont as a public web-server, a web-application that can be installed in a local Galaxy server, and as a command line program.

Paper

If you use Dimont, please cite

J. Grau, S. Posch, I. Grosse, and J. Keilwagen. A general approach for discriminative de-novo motif discovery from high-throughput data. Nucleic Acids Research, 41(21):e197, 2013. doi: 10.1093/nar/gkt831

Dimont web-server

Dimont is available as a public web-server at galaxy.informatik.uni-halle.de.

Dimont @ Galaxy toolshed

Dimont, DimontPredictor, DimontGenomeScan, and a data extraction tool are also available from the Galaxy tool shed in repository dimont_motif_discovery. From the toolshed, all tools can be installed with minimal effort from the Galaxy admin interface.

Dimont @ Chipster

Dimont is also provided with Chipster starting from Chipster release 2.11.

Download

Dimont is implemented in Java using Jstacs. You can download the command line application as a Jar. In addition, we provide the Jar of the Galaxy web-application for installing it in your local Galaxy server.

Dimont will be part of the next public release of the Jstacs library.

Running the command line application

For running the command line application, Java v1.6 or later is required.

The arguments of the command line application have the following meaning:

name comment type

home Home directory (The path to the directory containing the input file. Output files are written to this directory as well., default = ./) String
data Input file (The file name of the file containing the input sequences in annotated FastA format (see below)) String
infix Infix (a infix to be used for all output files (model, sequence logos, predicted binding sites)) String
position Position tag (The tag for the position information in the FastA-annotation of the input file) String
value Value tag (The tag for the value information in the FastA-annotation of the input file) String
sd Standard deviation (The standard deviation of the position distribution centered at the position specified by the position tag, valid range = [1.0, 10000.0], default = 75.0) Double
weightingFactor Weighting factor (The value for weighting the data; either a value between 0 and 1, or a description relative to the standard deviation (e.g. +4sd), default = 0.2) Double
starts Starts (The number of pre-optimization runs., valid range = [1, 100], default = 20) Integer
motifWidth Initial motif width (The motif width that is used initially, may be adjusted during optimization., valid range = [1, 50], default = 15) Integer
motifOrder Markov order of motif model (The Markov order of the model for the motif., valid range = [0, 3], default = 0) Integer
bgOrder Markov order of background model (The Markov order of the model for the background sequence and the background sequence, -1 defines uniform distribution., valid range = [-1, 5], default = -1) Integer
ess Equivalent sample size (Reflects the strength of the prior on the model parameters., valid range = [0.0, Infinity], default = 4.0) Double
delete Delete BSs from profile (A switch for deleting binding site positions of discovered motifs from the profile before searching for futher motifs., default = true) Boolean
threads Compute threads (The number of threads that are use to evaluate the objective function and its gradient., valid range = [1, 128], OPTIONAL) Integer

Input sequences must be supplied in an annotated FastA format. In the annotation of each sequence, you need to provide a value that reflects the confidence that this sequence is bound by the factor of interest. Such confidences may be peak statistics (e.g., number of fragments under a peak) for ChIP data or signal intensities for PBM data. In addition, you need to provide an anchor position within the sequence. In case of ChIP data, this anchor position could for instance be the peak summit. For instance, an annotated FastA file for ChIP-exo data comprising sequences of length 100 centered around the peak summit could look like:

> peak: 50; signal: 515
ggccatgtgtatttttttaaatttccac...
> peak: 50; signal: 199
GGTCCCCTGGGAGGATGGGGACGTGCTG...
...

where the anchor point is given as 50 for the first two sequences, and the confidence amounts to 515 and 199, respectively. The FastA comment may contain additional annotations of the format key1 : value1; key2: value2;..... We also provide an example input file.

Accordingly, you would need to set the parameter "position" to peak and the parameter "value" to signal for the input file.

For the initial motif width and the number of pre-optimization runs, we provide default values that worked well in our studies on ChIP and PBM data. However, you may want adjust these parameters to meet your prior information.

The parameter "motifOrder" sets the order of the inhomogeneous Markov model used for modeling the motif. If this parameter is set to 0, you obtain a position weight matrix (PWM) model. If it is set to 1, you obtain a weight array matrix (WAM) model. You can set the order of the motif model to at most 3.

The parameter "bgOrder" sets the order of the homogeneous Markov model used for modeling positions not covered by a motif. If this parameter is set to -1, you obtain a uniform distribution, which worked well for ChIP data. For PBM data, orders of up to 4 resulted in an increased prediction performance in our case studies. The maximum allowed value is 5.

The parameter "weightingFactor" defines the proportion of sequences that you expect to be bound by the targeted factor with high confidence. For ChIP data, the default value of 0.2 typically works well. For PBM data, containing a large number of unspecific probes, this parameter should be set to a lower value, e.g. 0.01.

The "ess" reflects the strength of the influence of the prior on the model parameters, where higher values smooth out the parameters to a greater extent.

The parameter "delete" defines if BSs of already discovered motifs should be deleted, i.e., "blanked out", from the sequence before searching for futher motifs.

Using the parameter "threads" you can set the number of threads that are used by Dimont. If you do not provide a value for this parameter, all available threads are used.

For instance, you can start Dimont on the example input file by calling

java -jar Dimont.jar home=./ data=dimont-example.fa infix=example position=peak value=signal threads=2

using a PWM as motif model, a uniform distribution as background model and two threads for optimization.

For larger data sets, the standard memory allocation of Java may not be sufficient. You can increase the amount of RAM allocated by the virtual machine using the VM parameters -Xms and -Xmx. To start Dimont using initially 512MB and a maximum of 2GB of RAM, you call

java -Xms512M -Xmx2G -jar Dimont.jar home=./ data=dimont-example.fa infix=example position=peak value=signal threads=2

Data preparation

We provide a Perl script for generating an annotated FastA file in the format required by Dimont from chromosome sequences and a BED file. The chromosome sequences should be provided in a directory containing one FastA file for each chromosome, where the filename should match the chromosome identifier of the BED file with extension .fa. For instance, if chromosome 1 is identified as chr1 in the BED file, the corresponding sequence should be available in a file chr1.fa. The complete list of parameters of this Perl script is

perl extract_data.pl <chromDir> <bedfile> <chromcol> <poscol> <widthcol> <statcol> <outfile>
   
   <chromDir>: the directory containing the chromosome FastAs (as 
               typically downloaded from UCSC as "chromFa"): one FastA per
               chromosome, one sequence per file
   <bedfile>:  the file containing the peaks in tabular format, 
               e.g., bed, gff, narrowPeak
   <chromcol>: the column of <bedfile> containing the chromosome
   <poscol>:   the column of <bedfile> containing the position relative to
               the chromosome start
   <widthcol>: either i) <int> the column of <bedfile> containing the width of 
                         the peak or any region to be extracted 
                         symmetrically around <poscol>
               or    ii) f<int> a fixed width of all regions, centered at the
                         position given in <poscol>, e.g., f100 for a fixed
                         width of 100 bp
   <statcol>:  the column of <bedfile> containing the peak statistic
               or a similar measure of confidence
   <outfile>:  the path to the output file, written as FastA

ChIP-seq input data

We thank Xiaotu Ma, Ashwinikumar Kulkarni, and Michael Q. Zhang for kindly providing the ChIP-seq binding regions compiled for their publication

X. Ma, A. Kulkarni, Z. Zhang, Z. Xuan, R. Serfling, and M. Q. Zhang. A highly efficient and effective motif discovery method for chip-seq/chip-chip data using positional information. Nucleic Acids Research, 40(7):e50, 2012.

Based on these data, we provide Dimont input data in the annotated FastA format required by Dimont.

If you use these data, please cite above publication and all appropriate original publications related to the data. In the archive, you find a ReadMe file containing the references to the original publications for each of the data sets.

DimontPredictor

In addition to the Dimont application, we also provide a command line program and a web-application of a prediction program that can be used to predict binding sites for motifs discovered by Dimont in small (training or test) data sets. This tool loads a Dimont model from the XML output of a previous Dimont run. In addition to the input parameters explained above, you can specify a threshold on the p-value of predicted binding sites. DimontPredictor may be useful if you, for instance, want to predict binding sites of a previously discovered motifs in other data sets, or if you want to try different p-values for filtering predictions. However, you should not use this tool for genomewide scans (cf. below).

You can download Jars of the command line program and web-application of DimontPredictor.

The web-application of DimontPredictor is also installed at galaxy.informatik.uni-halle.de.

If you like to use the DimontPredictor with Slim models, you need to download the SlimDimontPredictor, which is currently only available as command line application.

DimontGenomeScan

In addition to the Dimont and DimontPredictor, we also provide a command line program for genomewide scans. This tool loads a Dimont model from the XML output of a previous Dimont run and a genome to be scanned in fastA format.

You can download the DimontGenomeScan as a Jar.

Installing the web-application

The command-line program behind the web-application is a Jar as well, so Java is required on the server running Galaxy. To install this command line program in Galaxy, copy it to the desired destination in the Galaxy tools directory.

The command line application writes its Galaxy tool definition file itself. If you are in the directory containing the command-line program for Galaxy, you can create the tool definition file by calling

java -jar DimontWeb.jar --create DimontWeb.xml

Afterwards, this directory contains the tool definition file DimontWeb.xml. Now you can register Dimont in the Galaxy tool_conf.xml file. For details, see the Galaxy tutorial for adding new tools.