TPP AMZTPP

From SPCTools

(Difference between revisions)

Current revision

AMZTPP is a program developed by the trans-proteomic pipeline team for utilizing cloud computing (specifically Amazon Web Services) to execute multiple different peptide identification programs in parallel on MS/MS data. It can be used to cost effectively analyze large data sets in a short period of time.

 = Installing amztpp =
-The latest version of the TPP cloud tool amztpp can be downloaded from the TPP sourceforge site at http://sourceforge.net/projects/sashimi/files/amztpp.  It is packaged as a gzipped "tar" file and can be unpacked using the standard unix tool tar or on Windows using programs such as 7-zip, WinZip, Stuffit, WinRAR, etc..
+The latest version of the TPP cloud tool amztpp can be downloaded from the TPP sourceforge site at http://sourceforge.net/projects/sashimi/files/amztpp.  It is packaged as a gzipped "tar" file and can be unpacked using the standard Unix tool tar or on Windows using programs such as 7-zip, WinZip, Stuffit, or WinRAR.
 == Prerequisites ==
+Since the amztpp tool is written in Perl you therefore need to install Perl on you system before proceeding.  If you are on Linux it is very likely that you already have Perl installed.  If you are Windows you can download and install the free community edition provided by ActiveState at http://www.activestate.com/activeperl/downloads. ''(ActivePerl 5.14.2 or greater is required.)''
 Ironically TPP is not required in order to use the amztpp tool nor do you need to have any of the open source MS/MS identification programs that it uses installed.  Though you will find having TPP installed will make it much easier to view and interact with the results produced by amztpp.  So its our recommendation that you first install TPP before proceeding following the instructions at [[Software:TPP]].
-Since the amztpp tool is written in Perl you therefore need to install Perl on you system before proceeding.  If you are on Linux it is very likely that you already have Perl installed on your system.  If you are Windows you can download and install the free community edition provided by ActiveState at http://www.activestate.com/activeperl/downloads. ''(ActivePerl 5.14.2 or greater is required)''.
+== Installing amztpp on Windows ==
+The Windows installation process follows the standard procedure for almost all Perl based software.  After ensuring you have perl installed download the amztpp archive from [[https://sourceforge.net/projects/sashimi/files/amztpp/ SourceForge]] and unpack it using a Windows archive tool such as WinZip, 7-zip, or WinRAR. Next open up a Windows "command" window by clicking on the start icon and typing "cmd" in the "Search for programs and files" field.  When the command window opens up change to the folder ("cd") that you unpacked the archive in and make sure the build tool called dmake is installed:
+ C:\AMZTPP-2.4.3> ppm install dmake
+ C:\AMZTPP-2.4.3> dmake
+Now setup for the build of amztpp using the command:
+ C:\AMZTPP-2.4.3> perl Makefile.PL
+ Set up gcc environment - 4.5.2
+ Writing Makefile for AMZTPP
+ Writing MYMETA.yml
+ C:\AMZTPP-2.4.3>
+You'll likely see some messages warning about missing perl modules that are required.  While there are many ways to install these dependencies the easiest to use use ActiveState's ppm installer to install the dependencies via a dmake target as so:
+ C:\AMZTPP-2.4.3> dmake installdeps
+The next step is to test the amztpp build:
+ C:\AMZTPP-2.4.3> dmake test
+A number of tests are skipped by default as they require you to provide your Amazon Credentials.  If you want to run the full test set see the section on how to provide your AWS credentials and set these before running the tests.  Be aware that running tests with your credentials will incur some minor AWS charges.
+Lastly install the program with:
+ C:\AMZTPP-2.4.3> dmake install
 == Installing amztpp on Linux ==
-First download the latest version from sourceforge if you haven't already and unpack the archive:
+First download the latest version from [[https://sourceforge.net/projects/sashimi/files/amztpp/ SourceForge]] if you haven't already and unpack the archive:
  % wget <nowiki>sashimi/files/amztpp/amztpp-latest.tgz</nowiki>
  % tar xvzf amztpp-latest.tgz
- % cd amztpp-2.0
+ % cd amztpp-2.4.3
-There are several ways to proceed from this point. The simplest is to follow the "standard" way of installing perl modules and programs as follows:
+There are several ways to proceed from this point. The simplest is to follow the "standard" way of installing Perl modules and programs as follows:
  % perl Makefile.PL
  Writing MYMETA.yml and MYMETA.json
-If you get one or messages like ''Warning: prerequisite VM::EC2 1.09 not found.'' this indicates that you'll have to install additional modules for amztpp to work.  As before there is a couple of different ways to install them.  The easiest is to use [http://www.cpan.org/ CPAN].  You can run the CPAN commands manually or there's a convenient rule written in the Makefile called installdeps that will do this for you (you'll likely need root privileges to execute this):
+If you get one or messages like ''Warning: prerequisite VM::EC2 1.09 not found.'' this indicates that you'll have to install additional modules for amztpp to work.  There is a couple of different ways to install them.  The easiest is to use [http://www.cpan.org/ CPAN].  You can run the CPAN commands manually but there's a convenient rule written in the Makefile called installdeps that will do this for you (you'll likely need root privileges to execute this):
  % make installdeps
  % perl Makefile.PL PREFIX=/opt/tpp
 |}
-<br>
-Lastly, for some more references on how to install perl modules on linux please see:
-* http://cpan.perl.org/modules/INSTALL.html
-* http://www.perlmonks.org/index.pl?node_id=128077
-* http://perl.jonallen.info/writing/articles/install-perl-modules-without-root
-== Installing amztpp on Windows ==
-The process for installing on Windows is similar to the process on Linux.  After downloading and installing ActiveState's perl download the amztpp archive and unpack it.  Next open up a Windows "command" window by clicking on the start icon and typing "cmd" in the "Search for programs and files" field.  When the command window opens up change to the directory ("cd") that you unpacked the archive to and make sure dmake is installed:
+== Troubleshooting ==
- C:\AMZTPP-2.0>ppm install dmake
+If you run into problems with the installation here are some good references on how to install perl modules:
- C:\AMZTPP-2.0>dmake
-Now setup for the build using the same command as with Linux:
+* http://cpan.perl.org/modules/INSTALL.html
+* http://www.perlmonks.org/index.pl?node_id=128077
+* http://perl.jonallen.info/writing/articles/install-perl-modules-without-root
+* http://www.activestate.com/blog/2010/10/how-install-cpan-modules-activeperl
- C:\AMZTPP-2.0>perl Makefile.PL
+<br>
- Set up gcc environment - 4.5.2
- Writing Makefile for AMZTPP
- Writing MYMETA.yml
- C:\AMZTPP-2.0>
-You'll likely also see some messages warning about missing perl modules that are required.  You can run the following command to the build to use ActiveState's ppm installer to install the dependencies:
- C:\AMZTPP-2.0>dmake installdeps
-The next step is to test the program:
- C:\AMZTPP-2.0>dmake test
-A number of tests are skipped by default as they require you to provide your Amazon Credentials and the tests themselves will incur some minor AWS charges.  If you want to run the full test set see the section on how to provide your AWS credentials and set these before running the tests.
-Lastly install the program with:
- C:\AMZTPP-2.0\dmake install
 = Setting up your Amazon Credentials =
-Almost all of AMZTPP's operations require you to provide credentials to Amazon Web Services to confirm that you are who you say you are and that you do have permissions to do what you are trying to do.  These credentials are know as Amazon's access and secret key.  These keys are used to make secure REST or Query protocol requests to any AWS service API and are created automatically for you when your account is created.
+In order for the TPP to have access to Amazon Web Services you must provide your AWS credentials confirm that you are who you say you are and that you do have permissions to do what you are trying to do.  These credentials are known as Amazon's access and secret key.  These keys are used to make secure REST or Query protocol requests to any AWS service API.
-You can look up what these values are on your [https://aws-portal.amazon.com/gp/aws/securityCredentials Amazon Security Credentials] page.  Your access key and secret key are displayed under the Access Keys tab in the Credentials Section of the page.
+You can create new keys using the [https://console.aws.amazon.com/iam/home?#security_credential Amazon Security Credentials] page.  Your access keys are displayed under the Access Keys section in the Credentials Section of the page.  Secret keys are now no longer displayed.  If you've previously created a access/secret key pair and have forgotten what the secret key is you will need to generate a new key pair.  For more information about setting up your AWS credentials please see [http://blogs.aws.amazon.com/security/post/Tx1R9KDN9ISZ0HF/Where-s-my-secret-access-key Where's my secret access key?]
 There are a couple of different ways to provide these keys when running the amztpp command outlined in the next three sections.
-==== Command line options ====
+==== Using command line options ====
 You can use the command line options --access-key and --secret-key options each time you invoke a command.  This can quickly become onerous. It also carries a degree of risk as your command line history will contain your access and secret keys and other users may be able to see the command you are running.
-==== Environment Variables ====
+==== Using environment variables ====
 By setting environment variables EC2_ACCESS_KEY and EC2_SECRET_KEY (and optionally EC2_REGION).
 You'll have to set these each time you log in or open a command shell unless you setup your login to run these commands.  If so please take the appropriate precautions to protect your keys.
-==== AWS SECRET File ====
+==== Using a secret file ====
 The last and most convenient (but also comes with some risk) is by putting your keys in a .awssecret file.  The amztpp program contains two commands "save" and "delete" to make it easy to do this.  To save your keys you can run the command (on linux or windows):
  amztpp --access-key=<i>your_access_key</i> --secret-key=<i>your_secret_key</i> save
-You can also do this manually on Linux with the commands:
+You can also do this manually on Windows with the commands:
  C:\>echo ''your_access_key''> %LocalAppData%\.awssecret
  % echo chmod 600 ~/.awssecret
-If you use this mechanism always make sure you keep the permissions of this file strict and '''never share it with anyone'''.
+If you use this mechanism always make sure you keep the permissions of this file strict so other users can't read it and '''never share it with anyone'''.
 = Checking your Amazon Credentials =
 the background process is stopped and no instances, messages or files in AWS.
-= AMZTPP Windows Tutorial =
+= Windows Tutorial for amztpp =
+This tutorial will walk you through the steps of executing a MS/MS searches on the cloud using amztpp.
 ==== Step 1: Download and install the test data and database ====
 For this tutorial we'll be using the same dataset used by the TPP tutorial. This is a SILAC-labeled Yeast dataset comprised of 2 runs on a high mass-accuracy Orbitrap instrument, along with a Yeast database appended with decoys.
-We also include a search parameters file. If you haven't already downloaded it you install it by:
+We also include a parameters files for inspect, tandem, myrimatch and omssa for MS/MS identification. You can install it by:
+* Downloading the mzML files, parameter files and database from [http://sourceforge.net/projects/sashimi/files/amztpp/demoAMZTPP.tgz/download Sourceforge] (92.2Mb).
+* Unpack the demo archive using 7zip, Stuffit, unzip or a similar program and set the destination directory to be ''C:\Inetpub\wwwroot\ISB\data''
+If you've successfully installed the demo set you should have a new folder at ''C:\Inetpub\wwwroot\ISB\data\demoAMZTPP''
+==== Step 2: Submit a X!Tandem search of the data to the AWS cloud ====
+To submit searches, you need to use the Window's command line to invoke the program (often called the command shell).
+* First open a new Windows command shell. How to do this varies by the version of your operating system (Windows XP, Windows Vista, Windows 7) but generally you want to go to the start menu and type "cmd" as the program to execute.
+* Change to the demo directory from the first step using the cd command in the command window by typing:
+ C:\>cd \Inetpub\wwwroot\ISB\data\demoAMZTPP
+* Now submit tandem searches of the two MS/MS runs to the cloud and direct that the output will be written to the folder 'tandem':
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP>amztpp tandem OR20080317_S_SILAC-LH_1-1_01.mzML OR20080320_S_SILAC-LH_1-1_11.mzML -o tandem
+This should schedule two searches to be run on the Amazon Cloud.  In the next step you'll learn the command used to start the amztpp background process that will manage all of the tasks required to run your searches on the cloud.
+==== Step 3: Start the amztpp background process ====
+* Run the command to start the background process.  In addition specify that you want a maximum of two EC2 instances (of the default type):
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP>amztpp --ec2-max 2 start
+Assuming no errors the AMZTPP service should have been started up and will now be uploading files (in parallel) to the Amazon Cloud.  Once the files are uploading the service will then launch one or more EC2 nodes to run X!Tandem on the files.  When the results are ready the service will download them to the folder named "tandem".  Please note that the time to upload both mzML files can vary significantly due to your network connections, the internet itself, your proximity to the Amazon region you are using and the load on the Amazon Cloud.
+==== Step 4: Check on the status of your search ====
+[[Image:AMZTPP_Status.jpg|thumb|amztpp status]]
+The amztpp command line program has a "status" command that will output details such as whether the background process is running, the EC2 instances launched and their state, the approximate number of messages in the upload, download and services queues in SQS and a list of some of the files currently stored in S3.  In the command window invoke run:
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP> amztpp status
+If you don't see any EC2 instances running, it may be because the demo's mzML files haven't been completely uploaded yet.  To check to see if the upload is occurring look at the SQS service queues.  If the files are still uploading you'll see that there are pending/active messages in this queue.
+==== Step 5: Launch more searches ====
+Now launch more searches using the OMSSA and Myrimatch search engines with the following commands.  Only the new parameter files will be uploaded to the cloud, as the two mzML files and database files already having been uploaded with the X!Tandem search.
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP>amztpp omssa OR20080317_S_SILAC-LH_1-1_01.mzML OR20080320_S_SILAC-LH_1-1_11.mzML -o omssa
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP>amztpp myrimatch OR20080317_S_SILAC-LH_1-1_01.mzML OR20080320_S_SILAC-LH_1-1_11.mzML -o myrimatch
+When finished the search results will be available in the corresponding omssa and inspect folders.  You can always check on the status of your searches using the amztpp status command.
+==== Step 6: Use the AWS Management Console ====
+The Amazon Web Services management console is a simple and intuitive web-based user interface to all of Amazon's cloud resources.  It can be used to also check on the status of your searches or when things go wrong it can be used to stop services or delete stored data to avoid running up large AWS charges.
+[[Image:AMZTPP EC2 Console.jpg|thumb|EC2 AWS Management Console]]
+Start first by checking on the running nodes using the AWS Management Console.  Using your web browser go to [https://console.aws.amazon.com/ec2/home?region=us-west-2#s=Instances AWS Management Console] and enter your Amazon username and password.  You should then be presented with a screen showing details of your EC2 instances (assuming your searches are still going).  For each instance details such as the state, type of instance, public DNS are shown.  From this page you can also control your instances by selecting one or more instances and then choosing an operation (such as terminate).
+[[Image:AMZTPP_SQS_Console.jpg|thumb|SQS AWS Management Console]]
+Next check on the status of the message queues by clicking on the SQS tab in the console.  The Simple Queue Services (SQS) is a basic messaging system with different queues for sending and receiving messages. The amztpp program uses three such queues (upload, service, and download) to manage the communication between your client computer and the cloud services.  When you submit a search an upload message is queued with the details of the files and parameters to use.  The background service retrieves this message and uploads the files to S3 and then queues a message in the services queue.  EC2 instances retrieve these messages and execute the search program on the uploaded files.  The results are then submitted in another message in the download queue so that the background service on your computer and be notified to retrieve them from S3.
+[[Image:AMZTPP_S3_Console.jpg|thumb|S3 AWS Management Console]]
+Lastly to see what files are being stored in S3 click on the S3 tab in the console. Files in S3 are stored in "buckets" which have to be unique across all AWS accounts.  The amztpp program creates a bucket called TPP-<Amazon user id> for storing files.  Click on the bucket to see its contents in the management console.  Using this interface you can navigate the files stored in S3 and perform many of the common file operations such as adding, deleting, or changing permissions.  You can also view the search logs through this interface as when a EC2 instance shuts down the log files are stored under in the folder /var/log.
+==== Step 7: Cleaning up ====
+When all of your searches are complete (or at any time prior) you can use the amztpp's ''realclean'' command to stop all services and delete all data being stored in Amazon Web Services.  Forgetting to run this command when you are done will result in you being billed for the extra storage.  This command will not delete any data or results stored locally.
+ C:\Inetpub\wwwroot\ISB\data\demoAMZTPP>amztpp realclean
-* [ftp://ftp:a@ftp.peptideatlas.org/pub/PeptideAtlas/Repository/TPP_Demo2009/TPP_Demo2009_mzML_data.zip Download the mzML files] (768Mb) and unzip.
+= Where to Get Help =
-* [ftp://ftp:a@ftp.peptideatlas.org/pub/PeptideAtlas/Repository/TPP_Demo2009/TPP_Demo2009_db_and_tandemParams.zip Download the database and parameters files] (2.1Mb) and unzip.
+==== AMZTPP Manual ====
-* Copy or move the ''yeast_orfs_all_REV.20060126.short.fasta'' file into the folder ''C:\Inetpub\wwwroot\ISB\data\dbase\speclibs'' creating the directories if necessary
+The amztpp program can display both a summary help page and a longer manual page covering all of its many options.  To display the short summary simply run the program without any options or commands.  To get the full manual run the command with the --man option as so:
-* Copy or move the two data files (''OR20080317_S_SILAC-LH_1-1_01.mzML'' and ''OR20080317_S_SILAC-LH_1-1_11.mzML'') as well as the tandem parameters file ''tandem.xml'' into the folder ''C:\Inetpub\wwwroot\ISB\data\demo2009\tandem''. Create this last folder if necessary.
+ C:\>amztpp --man
-==== Step 2: Submit a search of the data to the AWS cloud ====
+==== AMZTPP Frequently Asked Questions (FAQ) ====
-Change to the tandem directory created in the previous step in a command shell and run amztpp to upload the data.
+In addition to this wiki page there is a wiki [[AMZTPP:FAQ]] covering many of the questions that come up when using amztpp.
-For Windows:
+==== SPC Tools Mailing List ====
- C:\>cd \Inetpub\wwwroot\ISB\data\demo2009\tandem
+The SPC Tools Discussion Group [http://groups.google.com/group/spctools-discuss spctools-discuss.googlegroups.com] is a active resource for submitting questions about TPP and tools such as amztpp.
- C:\Inetpub\wwwroot\ISB\data\demo2009\tandem>amztpp tandem OR20080317_S_SILAC-LH_1-1_01.mzML OR20080320_S_SILAC-LH_1-1_11.mzML OR20080317_S_SILAC-LH_1-1_01.mzML OR20080320_S_SILAC-LH_1-1_11.mzML

TPP AMZTPP

From SPCTools

Current revision

Contents

Installing amztpp

Prerequisites

Installing amztpp on Windows

Installing amztpp on Linux

Troubleshooting

Setting up your Amazon Credentials

Using command line options

Using environment variables

Using a secret file

Checking your Amazon Credentials

Windows Tutorial for amztpp

Step 1: Download and install the test data and database

Step 2: Submit a X!Tandem search of the data to the AWS cloud

Step 3: Start the amztpp background process

Step 4: Check on the status of your search

Step 5: Launch more searches

Step 6: Use the AWS Management Console

Step 7: Cleaning up

Where to Get Help

AMZTPP Manual

AMZTPP Frequently Asked Questions (FAQ)

SPC Tools Mailing List

Views

Personal tools

Navigation

support newsgroups

Search

Toolbox