AWS Setup

Obviously, you need an AWS EC2 account in order to get this working. Once you’ve got your account set up, install the AWS command line tools from here (Ignore the negative reviews, they work fine on Linux.). There are a few environment variables to set too. EC2_HOME is where the tools are, a location you’ll need to add to your path. You’ll also need to tell them where your AWS private key and x509 certificate files are (files that should have been generated during AWS registration but, if not, see the AWS x509 docs). Something like the following in your ~/.profile should do the trick:

export EC2_HOME=/home/cassj/.ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=/home/cassj/.ec2/amazon-pk.pem
export EC2_CERT=/home/cassj/.ec2/amazon-x509.pem

You’ll also need to create and register a key-pair to access your EC2 instances. The easiest way to do this is via the AWS management console – it’s fairly self-explanatory. Save the .pem file somewhere on your local machine (I’m using cassj.pem ).

Start Instance

Create a security group for the Galaxy server for which we can open appropriate ports. Then run an instance. I’m using the official Ubuntu Intrepid x86 server AMI as a base: ami-5059be39. I’m also using us-east-1b cos that’s where the EBS volume with all my ChIPseq data lives.

ec2-add-group galaxy -d 'Group for Galaxy Server'
ec2-run-instances   ami-5059be39   --region us-east-1 --availability-zone us-east-1b --key cassj --group galaxy --instance-type m1.small --instance-count 1

Connect to instance

Open up the ssh port (Am just opening it to everyone. Alternatively, you can restrict the IP addresses using CIDR format).

ec2-authorize galaxy -Ptcp -p22 -s 0.0.0.0/0

Run ec2din to check your instance is running and get its address, then ssh in using your keypair, something like:

ssh -i cassj.pem ubuntu@ec2-174-129-174-125.compute-1.amazonaws.com

Install Galaxy

Install Galaxy on your running instance. The following will grab the latest version from the repository and stick it in /galaxy.

sudo apt-get install mercurial
cd /
sudo hg clone http://www.bx.psu.edu/hg/galaxy galaxy
sudo chown -R ubuntu:ubuntu galaxy
cd galaxy
sudo sh setup.sh
sh run.sh

And modify the file universe_wsgi.ini so that the host is set to the appropriate place, eg

host = ec2-174-129-166-230.compute-1.amazonaws.com

Well, that was easy. You seem to need to run run.sh as root initially, but after that it seems to be ok if you run as user ubuntu.

Install Apache for static files

By default Galaxy runs on port 8080. We’ll set up apache running on port 80, tell it to handle any of the requests for static files, to take the load off the Galaxy process and ask it to hand over anything else to Galaxy to deal with. So, install Apache2 and enable mod_rewrite, mod_proxy and mod_proxy_http

sudo apt-get install apache2
sudo a2enmod rewrite
sudo a2enmod proxy
sudo a2enmod proxy_http

In /etc/apache2/sites-available/default this will redirect the stuff handed to Apache to Galaxy:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteRule ^/(.*) http://ec2-174-129-166-230.compute-1.amazonaws.com:8080/$1 [P]

And this will handle the limited number of static files that we want Apache to deal with:

  RewriteRule ^/static/style/(.*) /galaxy/test/static/june_2007_style/blue/$1 [L]
  RewriteRule ^/static/(.*) /galaxy/test/static/$1 [L]
  RewriteRule ^/images/(.*) /galaxy/test/static/images/$1 [L]
  RewriteRule ^/favicon.ico /galaxy/test/static/favicon.ico [L]
  RewriteRule ^/robots.txt /galaxy/test/static/robots.txt [L]
</IfModule>

More info on installing Galaxy can be found on the wiki

Restart apache with sudo /etc/init.d/apache2 restart.

Authorize Apache and Galaxy Ports

ec2-authorize galaxy -Ptcp -p8080 -s 0.0.0.0/0
ec2-authorize galaxy -Ptcp -p80 -s 0.0.0.0/0

Now if you go to http://<Your AWS URL> you should see your Galaxy installation.
It’s not going to be totally functional because we haven’t installed all of the underlying bioinformatics binaries but my plan is to have separate instances doing the actual analysis anyway. That’s tomorrow’s problem though…

background about TFs. ChIPchip, ChIPseq can define binding (with noise), but knowing when binding is functional is less well characterized.

TF binding should be inferred from diverse and complimentary data.

Binding doesn’t imply functionality. Binding and gene regulation are context dependant.

Functional vs Non-functional discrimination? Which factors determine functionality of binding?

Yeast data.
Define binding::
ChIPchip (pvals corres to strength of binding),
PSSM (200bp down and 800 up of the start codon) assoc with a significance score,
Nucleosome Occupancy – significantly lower around binding site.

Bayesian prob of binding based on the 3 sets

ROC curve shows integrated model performs best (known interactions in normal growth conditions with 5-fold cross validation)

Assuming binding is functional if it changes gene expression – as measured with arrays.

integration with gene expression data – find instances where differential binding correlates with differential expression (between normal growth conditions and stress condition).

Looked at functional binding rates in different stress conditions. Functional binding rate is context dependent – differs between stress conditions. Can rank the impact of particular transcription factors in different stress conditions.

Factors determining functionality (from FB and NFB enriched sets).
Distance from the start codon, orientation wrt to direction of transcription. Presence of absence of co-factors on the same promoter.

Feature selection to determine important factors.: Multi-variate random forest classification algorithm. Feature importance score calc based on change in estimation error before and after permuting values of a vector. Identify significant factors for each (condition, TF) pair, calc p value.

Discriminatory factors are different in different conditions, but in most cases, co-factors were most important. Can use this data to determine significant TF-TF co-factor interactions.

Now have 2 sets, 1 enriched in functional binding sites and the other enriched in non-functional.

Only considering metadata – actual data is just in native format and linked to metadata.

Consistent reporting of metadata:
Define scope and minimal info (MIBBI http://mibbi.org), syntax (ISATAB) and semantics (OBO)

http://www.mibbi.org/index.php/Main_Page
http://isatab.sourceforge.net/
http://www.obofoundry.org/

isacreator – standalone java app to annotate and link data on the basis of community-defined metadata ontologies.

poster c19

advanced results analysis and rendering.

data model shared by all analysis tools within a category

standard web interface www.ebi.ac.uk/Tools/category/tool
similarly SOAP API: Tools/services/soap/tool?wsdl
REST API: Tools/services/rest/tool

Soap WS GetParameters(), GetParameterDetails()
REST WS GET /parameters
GET /parameterdetails/param

Running:
SOAP: Run(input) returns job id
GetStatus(jobid) returns RUNNING, FINISHED…
REST:
POST /run
GET /status/jobid

Retrieving results:
SOAP:
GetResultTypes(jobid)
GetResult(jobid, type)
view of result is base64 encoded
REST
GET /resulttypes/jobid
GET /result/jobid/type
get result back with appropriate MIME type

Tools are categorised and subcategorised. Also have context, eg proteins, nucleotide and even more specific.

demo…
shiny
still in beta.

Can get the results as XML (with common data model across tool category so eg. all sequence search XML can be parsed the same)

ELIXIR is a European effort to co-ordinate the infrastructure. Preparatory project – 10-20yr roadmap for infrastructure devel to support research.

EU 32 partners, 13 member states. 4.5mill E funding to define scope, cost of infrastructure.

Goals:
Co-ordinated data resources, integration & interoperability of data, links to data in other domains, open access to data, enhance euro competitiveness in bioscience, address need for increased funding adn its co-ord.

v young science. Funding streams for infrasxr not in place.

stakeholders: users, experimentalists (data provision), resource providers (core & specialist), tool providers (bioinformaticians), funders – govt bodies, EMBL, EU charities, Industry.

Challenges prompting ELIXIR: data growth, global context, large and distributed userbase, preservation & accessibility of data, impact on biosciences, growth of funding

Cost of maintaining data is insignificant compared to cost of data generation. Makes sense to fund.
Integration increasingly important as academic, molecular type data is increasingly needed by medicine, agriculture etc

ESFRI: Biology research infrastructure proposals. ELIXIR will support these.

Reports from initial committee meetings (userbase consultation) due now – will define the scope & remit of ELIXIR. Then work on international agreement for goals, costs then look at how to fund.

Can’t keep everything centralized – need more distrubution. Hub at EBI and nodes in diff member states

Will provide: core and specialist data resources, compute centres, infrastructure for tools and services integration, support for Bio ESFRI projects, community support and training.

DB survey. 170 DBs across EU. Many of the core DBs are at the EBI, but are distributed in the sense that data providers are across Europe. Also many specialist resources across EU. All of these use the core resources as reference data. DB sizes follow power law – most <10GB but a few are huge. All have web browser queries. Some still have email query. about 70% have data downloads and about 30% have programmatic access. 39/170 have some restrictions on data access (legal or practical). A fairly high proportion have no funding. Most of them cost < mill euros. About 40 mill euros a year being spent at the moment on these DBs. Total invest to date is 308 mill euros. 90% have less than 3 year funding security. Most have less than 50K unique users /month, but a few have many more. Most have <5 staff, a few have many. Many don’t have any members of staff. See Poster E41. for details.

So – ELIXIR needs to co-ordinate, prioritise and stabilise funding for these resources.

Databases relatively under control compared to other aspects: Standards and ontologies, Literature, Other domains (medical data, biodiversity data etc), Integration

Don’t need to centralise standards devel, is fine for them to come out of communities, but do need to encourage and publicise standards. OBO.

Lit: integrated, open access text-based lit resource would be nice

compute resources…? Other domains deal with much bigger scale data (CERN), but they have fewer users and bioinfo data is growing at an exponential rate. Can’t chuck NGS data around the web. So – what do we need to keep? Should it be centralised? Probably need biodata grid like CERN (only more complex).
Modularise organisation of dataresources. Build network of biocomp resources. Catalyst devel of web services and cloud computing. More program to data rather than other way round. Work with EU supercomputing centres.

User priorities: integration, format compatibiltiy, website usability.
Short term: acecss – programmatic, web-site, web service, downloads. Develop well-maintained catalogue.
Long term – integration of data and tools. Encourage commercial tools to adopt open standards

Co-ord training.

Comments:

DB developers should have to abide by standards in order to publish / be funded by ELIXIR
Global context is important and ELIXIR will take into account international collab models in funding approaches. Data sharing will be required.
ELIXIR not about providing national infrastructure – this should come from per-country funding. Only interested in pan-EU infrastructure. Prob national nodes would be well set up to also provide pan-EU function though (shared compute etc).

May call for proposals for nodes from EU countries, although no actual funding and no mechanism for deciding which would be accepted yet so proposals a bit hypothetical.

http://www.sanger.ac.uk/software/nmica

scalable. 1000s sequences. 100s motifs highly parallel. extendable

Actually, looking at metamotifs and motif classification in this talk.
Different TF families have different modes of binding.

Can build a metamotif that encodes the distribution across a number of motifs in a family. Like a PWM, but with confidence intervals. Developed a nested sampler to learn these from a set of motifs (eg for bHLH TFs) describes repeating patterns in a set of motifs. Can use these as a prior for motif finding which improves detection of real motif in a test set. (I think…)

recent needs placed on distros: high performance computing, cloud/grid – bootable images with customisations. exact spec of software and env.
increased focus on: libraries – distributing software, data management – using software

debian-med.alioth.debian.org
medical and bioinfo packages. svn package maintenance. Open community
Extension of Deb, not a split.
All packages autobuild from source for > 10 arch
Packages auto-forwarded to Ubuntu.
> 160 packages

current work: more communication of functionality.
GSoC projects – improved cloud prep. Data processing script (?)

Have got an EC2 AMI, but not public yet.

In <language of choice> bind <generic C implementation> using SWIG with Biolib

Limited pool of bioinf programmers and being spread too thin across diff language specific projects. Shared stuff good cos less duplication, more people testing code etc. Inexplicable picture of young girl. Man with beard.

Bioperl on Github. Uses Cmake (as opposed to autoconf? – modular, resolves complex dependencies) and SWIG (rules for generating code, DRY, pattern matching, multi-language support (>20))

Year One:
AffyIO
Staden sequencer trace files
GSL (GNU Science Library)
Rlib R stuff
R/qtl quant genetics
Libsequence sequence analysis
Bio++ sequence analysis

Future:
Automated API doc
more libs (Emboss, NCBI)
more languages
Bio* integration (CPAN, Ruby gems etc)
Distribute as packages

NB: BoF session at 16.50

Note to self: SAMtools?