TPP:Cloud

From SPCTools

(Difference between revisions)
Jump to: navigation, search
Revision as of 23:10, 13 July 2011
JoeS (Talk | contribs)
(→TPP High Performance Computing Tools)
← Previous diff		 Current revision
JoeS (Talk | contribs)
(→Getting Started)
Line 3: Line 3:
Amazon Web Services (AWS) is one of the leading providers of cloud computing. It offers a host of different services which include: Amazon Web Services (AWS) is one of the leading providers of cloud computing. It offers a host of different services which include:
-* Elastic Compute Cloud (EC2) - +* Elastic Compute Cloud (EC2) - is a web service that provides resizable compute capacity in the cloud.
* Simple Storage Service (S3) - provides fully redundant data storage infrastructure that can be used to store instrument files, databases and analysis results * Simple Storage Service (S3) - provides fully redundant data storage infrastructure that can be used to store instrument files, databases and analysis results
* Simple Queue Services (SQS) - message queuing system that can be used to schedule and execute jobs * Simple Queue Services (SQS) - message queuing system that can be used to schedule and execute jobs
Line 12: Line 12:
== TPP Cloud Components == == TPP Cloud Components ==
-===== TPP Web Application ===== 
-The TPP Web Application (TWA) is a web application hosted at the [tools.proteomecenter.org] that makes it easy to start and use a virtual server running TPP on the Amazon Web Services. Its an ideal solution for people wanting to try out TPP but don't want to (or can't) install TPP on a local computer. Users can also use TWA to cost effectively access powerful compute resources when their local computer isn't sufficient enough to run TPP. All you need to use this tool is to sign up for an Amazon account, register to use the EC2 product, and provide your Amazon security credentials in the fields provided. More information is available at [[TPP_TWA]]. 
===== TPP Amazon Machine Images ===== ===== TPP Amazon Machine Images =====
Line 20: Line 18:
Amazon machine images are pre-built appliances by the TPP developer team that already have TPP installed on them. These can be used to quickly and easily create virtual servers within the Amazon Elastic Cloud (EC2). Developers are also encouraged to use these images as a basis for their own cloud computing infrastructures. More information is available at [[TPP_Amazon_Machine_Images]]. Amazon machine images are pre-built appliances by the TPP developer team that already have TPP installed on them. These can be used to quickly and easily create virtual servers within the Amazon Elastic Cloud (EC2). Developers are also encouraged to use these images as a basis for their own cloud computing infrastructures. More information is available at [[TPP_Amazon_Machine_Images]].
-===== TPP High Performance Computing Tools =====+===== TPP Web Launcher for Amazon =====
-The hich performance computing (hpc) tool is available to process large datasets using TPP on the cloud. This is an additional command line tool called amztpp that can be downloaded and installed on the same computer you have TPP installed. This tool manages all the details of running MS/MS searches of your local mzML files on one or more Amazon EC2 instance, in parallel. Details on how to install and run this program are available [[here]]+The TPP Web Launcher for Amazon (TWA) is a simple web application hosted at http://tools.proteomecenter.org/twa that makes it easy to start and use a virtual computer running TPP on the Amazon Web Services. Its an ideal solution for individuals wanting to try out TPP but don't want to (or can't) install TPP on a local computer. Users can also use TWA to cost effectively access significant compute resources when there projects exceed their local resources. All you need to use this tool is to sign up for an Amazon account, register to use the EC2 product, and provide your Amazon security credentials in the fields provided. More information is available at [[TPP_TWA]].
 + 
 +===== TPP Amazon Command Line Tool =====
 + 
 +If you have large datasets that you want to search with multiple MS/MS identification algorithms and process these results with TPP then the '''amztpp''' command line tool could be of use to you. This application is a set of command line tools that can be used to execute multiple search engines in parallel on cloud based servers. The complex details of uploading input files and databases, managing paths to files and parameters, starting and stopping instances and scheduling jobs are all managed by the tools in this package. Details on how to install and run this program are available [[TPP_AMZTPP]]
 + 
 +===== TPP Amazon Simulator =====
 + 
 +The TPP Amazon Simulator (amzsim) is a web application hosted at http://tools.proteomecenter.org/aws/amzsim that was developed to simulate the execution of amztpp for the purpose of estimating potential costs and run time for an analysis. Using the simulator users can experiment with different parameters such as the numbers of input files, file sizes, transfer speeds, and EC2 instances.<br>
 +{| style="border-collapse: separate; border-spacing: 2; border-width: 1px; border-style: solid; border-color: #000; padding: 2px; width: 90%; margin-left: auto; margin-right: auto;"
 +|[[Image:Alert.jpg|middle|left]]
 +| All costs calculated in the simulator are purely estimations based on the published Amazon Web Services charges and simple assumptions assumed about the execution of amztpp. Costs may vary significantly from the estimates.
 +|}
 +<br>
 + 
 +== Getting Started ==
 + 
 +In order to use Amazon Web Services you'll need to have an account. If you already have an account with Amazon for shopping for books, electronics, etc. you can use this same account for AWS. All you have to do is sign up for the specific Amazon Web Services that you want to use. Otherwise you can always create an account specifically for AWS. The process for creating a new account is very straightforward and there are many good guides on the web or at Amazon for how to do this. The basic steps are:
 + 
 +# Open the following link http://aws.amazon.com
 +# Sign up for an AWS account using the yellow button in the upper right labeled "Sign Up" or "Sign Up Now".
 +# Fill out the account details and submit the form.
 + 
 +Once your account has been created you'll then need to choose which products you want to use from each of the product's pages. The following are recommended and/or are required by the TPP Cloud components:
 +::* Amazon Elastic Compute Cloud (EC2) - for launching cloud instances of virtual computers
 +::* Amazon Simple Storage Service (S3) - cloud storage and retrieval of any amount of data
 +::* Amazon Simple Queue Services (SQS) - messaging infrastructure for storing messages in queues on the cloud
 + 
 + 
 +{| style="border-collapse: separate; border-spacing: 2; border-width: 1px; border-style: solid; border-color: #000; padding: 4px; width: 90%; margin-left: auto; margin-right: auto;"
 +|[[Image:tip.gif|middle|left]]
 +|Amazon does require that a credit card be associated with your account for billing purposes. But Amazon also provides an alternative mechanism for billing and payments known as [http://docs.amazonwebservices.com/AWSConsolidatedBilling/1.0/AWSConsolidatedBillingGuide.html AWS Consolidated Billing]. This service consolidates payments for multiple accounts within your organization to a single paying account (so your credit card won't get charged). Check with your organization to see if it already has setup consolidated billing or if there is enough interest to warrant setting it up.
 +|}
 +<br>
 + 
 +== Warning about resource usage ==
 + 
 +Use of the TPP Amazon Web Services components will incur charges that the user will be responsible for paying. These charges include S3 storage, SQS messaging, and above all EC2 resource usage. While every effort has been made to avoid any unexpected charges (especially with EC2 by using automatic shutdown scripts) there is always the chance that resources may not be removed and EC2 instances may not be terminated. It is '''YOUR RESPONSIBILITY''' to confirm that all resources uses by these components are removed or terminated. If you do not '''YOU WILL BE CHARGED BY AMAZON FOR ANY RESOURCES UNTIL THEY ARE TERMINATED'''.
 + 
 +It is highly recommended to learn how to check on and delete resources using the the AWS Management Console before using any of the TPP Cloud tools at https://console.aws.amazon.com.
 + 
 +'''[[Neither this program's authors or the Institute for Systems Biology will be liable for any charges incurred through the use of this program.]]'''
 + 
 +== Learn More ==
 + 
 +General Amazon Web Services (AWS) information:
 +* [http://aws.amazon.com Amazon AWS Website]
 +* [http://aws.amazon.com/documentation/ Amazon AWS Documentation]
 +* [http://calculator.s3.amazonaws.com/calc5.html Amazon AWS Pricing Calculator]
 + 
 +Amazon Elastic Compute Cloud (EC2) information:
 +* [http://aws.amazon.com/ec2/ Amazon EC2 Website]
 +* [http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/ Amazon EC2 Getting Started Guide]
 +* [http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/ Amazon EC2 User Guide]
 + 
 +Amazon Simple Storage Service (S3) information:
 +* [http://aws.amazon.com/s3 Amazon S3 Website]
 +* [http://docs.amazonwebservices.com/AmazonS3/latest/gsg/ Amazon S3 Getting Started Guide]

Current revision

Cloud computing is a revolutionary new paradigm shift whereby traditional IT resources such as storage, computing, content delivery, and software are provided through the Internet using simple web services. It brings an elegant solution to the problem of dynamically providing resources and services to perform functions with changing needs.

Amazon Web Services (AWS) is one of the leading providers of cloud computing. It offers a host of different services which include:

  • Elastic Compute Cloud (EC2) - is a web service that provides resizable compute capacity in the cloud.
  • Simple Storage Service (S3) - provides fully redundant data storage infrastructure that can be used to store instrument files, databases and analysis results
  • Simple Queue Services (SQS) - message queuing system that can be used to schedule and execute jobs

TPP now provides a number of ways to utilize cloud computing using Amazon Web Services to cost-effectively analyze large data sets, build your own cloud infrastructures based on TPP, or just run the TPP software without requiring any installation.

Contents

[edit]

TPP Cloud Components

[edit]
TPP Amazon Machine Images

Amazon machine images are pre-built appliances by the TPP developer team that already have TPP installed on them. These can be used to quickly and easily create virtual servers within the Amazon Elastic Cloud (EC2). Developers are also encouraged to use these images as a basis for their own cloud computing infrastructures. More information is available at TPP_Amazon_Machine_Images.

[edit]
TPP Web Launcher for Amazon

The TPP Web Launcher for Amazon (TWA) is a simple web application hosted at http://tools.proteomecenter.org/twa that makes it easy to start and use a virtual computer running TPP on the Amazon Web Services. Its an ideal solution for individuals wanting to try out TPP but don't want to (or can't) install TPP on a local computer. Users can also use TWA to cost effectively access significant compute resources when there projects exceed their local resources. All you need to use this tool is to sign up for an Amazon account, register to use the EC2 product, and provide your Amazon security credentials in the fields provided. More information is available at TPP_TWA.

[edit]
TPP Amazon Command Line Tool

If you have large datasets that you want to search with multiple MS/MS identification algorithms and process these results with TPP then the amztpp command line tool could be of use to you. This application is a set of command line tools that can be used to execute multiple search engines in parallel on cloud based servers. The complex details of uploading input files and databases, managing paths to files and parameters, starting and stopping instances and scheduling jobs are all managed by the tools in this package. Details on how to install and run this program are available TPP_AMZTPP

[edit]
TPP Amazon Simulator

The TPP Amazon Simulator (amzsim) is a web application hosted at http://tools.proteomecenter.org/aws/amzsim that was developed to simulate the execution of amztpp for the purpose of estimating potential costs and run time for an analysis. Using the simulator users can experiment with different parameters such as the numbers of input files, file sizes, transfer speeds, and EC2 instances.

middle
 All costs calculated in the simulator are purely estimations based on the published Amazon Web Services charges and simple assumptions assumed about the execution of amztpp. Costs may vary significantly from the estimates.


[edit]

Getting Started

In order to use Amazon Web Services you'll need to have an account. If you already have an account with Amazon for shopping for books, electronics, etc. you can use this same account for AWS. All you have to do is sign up for the specific Amazon Web Services that you want to use. Otherwise you can always create an account specifically for AWS. The process for creating a new account is very straightforward and there are many good guides on the web or at Amazon for how to do this. The basic steps are:

  1. Open the following link http://aws.amazon.com
  2. Sign up for an AWS account using the yellow button in the upper right labeled "Sign Up" or "Sign Up Now".
  3. Fill out the account details and submit the form.

Once your account has been created you'll then need to choose which products you want to use from each of the product's pages. The following are recommended and/or are required by the TPP Cloud components:

  • Amazon Elastic Compute Cloud (EC2) - for launching cloud instances of virtual computers
  • Amazon Simple Storage Service (S3) - cloud storage and retrieval of any amount of data
  • Amazon Simple Queue Services (SQS) - messaging infrastructure for storing messages in queues on the cloud


middle
Amazon does require that a credit card be associated with your account for billing purposes. But Amazon also provides an alternative mechanism for billing and payments known as AWS Consolidated Billing. This service consolidates payments for multiple accounts within your organization to a single paying account (so your credit card won't get charged). Check with your organization to see if it already has setup consolidated billing or if there is enough interest to warrant setting it up.


[edit]

Warning about resource usage

Use of the TPP Amazon Web Services components will incur charges that the user will be responsible for paying. These charges include S3 storage, SQS messaging, and above all EC2 resource usage. While every effort has been made to avoid any unexpected charges (especially with EC2 by using automatic shutdown scripts) there is always the chance that resources may not be removed and EC2 instances may not be terminated. It is YOUR RESPONSIBILITY to confirm that all resources uses by these components are removed or terminated. If you do not YOU WILL BE CHARGED BY AMAZON FOR ANY RESOURCES UNTIL THEY ARE TERMINATED.

It is highly recommended to learn how to check on and delete resources using the the AWS Management Console before using any of the TPP Cloud tools at https://console.aws.amazon.com.

Neither this program's authors or the Institute for Systems Biology will be liable for any charges incurred through the use of this program.

[edit]

Learn More

General Amazon Web Services (AWS) information:

Amazon Elastic Compute Cloud (EC2) information:

Amazon Simple Storage Service (S3) information:

Retrieved from "http://tools.proteomecenter.org/wiki/index.php?title=TPP:Cloud"
Personal tools