1 Getting Started with InterMine

Following the steps on this page you will set up an example InterMine. You will:

• Load some real data sets for Malaria (P. falciparum)
• Learn about post-processing after data is loaded
• Deploy a webapp to query the data

2 Before you start

This guide is designed for beginners who have never installed InterMine before. However, you need to have the following prerequisites installed:

• Subversion
• Java SDK
• Ant
• Tomcat
• PostgreSQL

More information on installation and versions can be found here: Installation intructions.

3 What is InterMine?

InterMine is a powerful open source data warehouse system. Using InterMine, you can create databases of biological data accessed by sophisticated web query tools. InterMine can be used to create databases from a single data set or can integrate multiple sources of data. Support is provided for several common biological formats and there is a framework for adding your own data. InterMine includes an attractive, user-friendly web interface that works 'out of the box' and can be easily customised for your specific needs.

Find out more - What is InterMine?

InterMine was developed to enable FlyMine and is now used in other projects. See  FlyMine for a working example, click 'Take a tour' to explore features.

4 Downloading InterMine

We use  subversion to manage and distribute source code. Usually you can check out the latest InterMine branch, for the workshop we will download a zip file of the latest code.

Download the InterMine code into your home directory:

> wget http://www.flymine.org/download/intermine-workshop-sep.tar.gz
> tar zxvf intermine-workshop-sep.tar.gz

You can check out the software from the trunk, but this isn't recommended as the trunk code is under constant development and may not be stable or thoroughly tested.

All InterMine code is freely available under the LGPL license.

5 Getting data

The InterMine checkout includes a tar file with data to load into MalariaMine. These are real, complete data sets for P. falciparum. We will load genome annotation from PlasmoDB, protein data from UniProt and GO annotation also from PlasmoDB.

Copy this to some local directory (your home directory is fine for this workshop) and extract the archive:

> cd
> cp dev/bio/tutorial/malariamine/malaria-data.tar.gz .
> tar -zxvf malaria-data.tar.gz

6 InterMine Installer

We are going to use the InterMine installer to create and build a small example Mine. The Installer hasn't been released to the public and is still in beta. We plan on adding more features to the Installer soon.

6.1 InterMine software

Change into the directory you checked out the InterMine source code to and look at the sub-directories:

> cd dev
> ls

The directories include:

• intermine - the core InterMine code, this is generic code to work with any data model.
• bio - code to deal specifically with biological data, it includes sources to load data from standard biological formats.
• flymine - the configuration used to create FlyMine, a useful reference when building your own Mine.
• testmodel - a non-biological test data model used for testing the core InterMine system.
• imbuild - InterMine's ant-based build system, you shouldn't need to edit anyting here.

All configuration to create a new Mine is held in a directory in dev, your Mine will depend on code in intermine, bio and imbuild. Any Mine needs to be a top level directory in your InterMine checkout.

6.2 Start the InterMine Installer

To start the InterMine Installer, run:

# in dev/intermine/MineManager
> ./run

Using the installer program, you can:

• create new Mines (projects)
• add/remove data sources for your Mine
• add/remove post-processing steps
• explore the current data model

6.3 Installer Menus

The installer has four menus on the top menu bar

1. Mine - manage your Mine
2. Sources - manage data sources for your Mine
3. Model - explore data model
4. Tools - update look of installer program

6.3.1 Project Menu

• New Mine - creates a new Mine (We'll see what this actually does in the next section.)
• Load Mine - edit a previously created Mine
• Save Mine - save any changes you have made to your Mine
• Build Mine - loads all of the selected data sources into the Mine's production database

The project menu lets you create, edit, save and build Mines. Note that these actions are also available via the buttons conveniently located in the lower right panel.

6.3.2 Sources Menu

• Add source - add a data source to your Mine
• Delete source - remove a data source from your Mine
• Post-process - add a post-processing step to your Mine

InterMine comes with several ready-made data sources available to load into your Mine. Please see BioSources for the complete list of available data sources.

The sources menu allows you to add these or your own custom data to your Mine. Note that this step only adds the configuration for each data source, the "build project" step loads the data into the database.

These are steps that run after the data loading is completed. They are used to set calculate/set fields that are difficult to do when data loading or that require multiple sources to be loaded. Please see PostProcessing for a list and detailed description of each post-process.

6.3.3 Model Menu

• View model - view the data model for your Mine

When you first create your Mine, only the core data model (bio/core/core.xml) is loaded. The core model includes elements that are common across all Mines, eg. publication, ontology, and sequence.

Each data source has an "additions" file. An "additions" file is an XML file that extends the model to include any new data types introduced by the data source. We are going to cover this in detail later.

6.3.4 Tools Menu

• Preferences - select the colour scheme for the installer

7 Create MalariaMine

Create a new Mine by selecting the "New Mine" menu item on the "Mine" menu. Fill out the form that appears with the following values:

Enter your database information in the next screen.

Update this form with the information for your mine. The tomcat username and password is "manager".

Click on the "Create Mine" button to create your mine.

You now have a Mine! Let's look at what that action did. Open a new terminal window and run this command:

> cd dev/malariamine
> ls

We will look at each of the sub-directories in much more detail later, they are:

• dbmodel - contains information about the data model to be used and ant targets relating to the data model and database creation.
• integrate - provides ant targets for loading data into the Mine.
• postprocess - ant targets to run post-processing operations on the data in the Mine once it is integrated.
• webapp - basic configuration and commands for building and deploying the web application

In addition there are two default.intermine.xxx.properties files which we won't need to edit and a project.xml file.

Note: most generated directories have a project.properties file and a short build.xml file, these are used by the InterMine build system.

8 Data Model

We're now ready to add data sources to our Mine. See ModelViewer for a detailed explanation of the InterMine data model.

9 Loading Data

9.1 Adding a source

Add the "uniprot" source to your Mine by following these steps:

1. Select the "Add source" option on the "Sources" menu.
2. Fill out the form as below.
3. Click on the "Save Mine" button at the bottom of the screen to save your changes.

• Save your changes using the "Save Mine" button at the bottom of the screen.
• The project.xml file is the configuration file that determines which data sources are loaded when you build your database.
• This file has now been updated for your mine:

> less ~/dev/malariamine/project.xml 

9.2 Running a build

Now that we have configured a data source for our Mine, we can build a database. Run a build by clicking on the "Build Mine" button at the bottom of the page. When you click on that button, a dialog box will appear. Leave the default values and click on "Build Mine" to start the build.

When the build is running you will see the log messages in a new dialog box:

Once the build is complete, you will see a listing of all sources ran, the amount of time it took, and the "OK" button will be enabled.

After the data has loaded, log into the database and view the contents of the protein table:

> psql malariamine
malariamine#  select count(*) from protein;

And see the first few rows of data:

malariamine#  select * from protein limit 5;

9.3 Object relational mapping

InterMine works with objects, objects are loaded into the production system and queries return lists of objects. These objects are persisted to a relational database. Internal InterMine code (the ObjectStore) handles the storage and retrieval of objects from the database automatically. By using an object model InterMine queries benefit from inheritance, for example the Gene and Exon classes are both subclasses of SequenceFeature. When querying for SequenceFeatures (representing any genome feature) both Genes and Exons will be returned automatically.

We can see how see how inheritance is represented in the database:

• One table is created for each class in the data model.
• Where one class inherits from another, entries are written to both tables. For example:

  malariamine#  select * from gene limit 5;
  

  The same rows appear in the sequencefeature table:

  malariamine#  select * from sequencefeature limit 5;
  

• All classes in the object model inherit from InterMineObject. Querying the intermineobject table in the database is a useful way to find the total number of objects in a Mine:

   malariamine#  select count(*) from intermineobject;
  

• All tables include an id column for unique ids and a class column with the actual class of that object. Querying the class column of intermineobject you can find the counts of different objects in a Mine:

   malariamine#  select class, count(*) from intermineobject group by class;
  

• A technical detail: for speed when retrieving objects and to deal with inheritance correctly (e.g. to ensure a Gene object with all of its fields is returned even if the query was on the SequenceFeature class) a serialised copy of each object is stored in the intermineobject table. When queries are run by the ObjectStore they actually return the ids of objects - these objects are may already be in a cache, if not the are retrieved from the intermineobject table.

10 Loading Genome Data from GFF3 and FASTA

We will load genome annotation data for P. falciparum from  PlasmoDB:

• genes, mRNAs, exons and their chromosome locations - in GFF3 format
• chromosome sequences - in FASTA format

10.1 Data integration

Note that genes from the gff3 file will have the same primaryIdentifier as those already loaded from UniProt. These will merge in the database such that there is only one copy of each gene with information from both data sources. We will load the genome data then look at how data integration in InterMine works.

First, look at the information currently loaded for gene PFL1385c from UniProt:

malariamine=#  select * from gene where primaryIdentifier = 'PFL1385c';

10.2 GFF3 files

GFF3 is a standard format use to represent genome features and their locations. It is flexible and expressive and defined by a clear standard -  more details here. An example of the file will load can be used to explain the format, each line represents one feature and has nine tab-delimited columns:

MAL1    ApiDB   gene    183057  184457  .       -       .       ID=gene.46311;description=hypothetical%20protein;Name=PFA0210c
MAL1    ApiDB   mRNA    183057  184457  .       +       .       ID=mRNA.46312;Parent=gene.46311
MAL1    ApiDB   exon    183057  184457  .       -       0       ID=exon.46313;Parent=mRNA.46312

• col 1: "seqid"
  • an identifier for a 'landmark' on which the current feature is locatated, in this case 'MAL1', a P. falciparum chromosome.
• col 2: "source"
  • the database or algorithm that provided the feature
• col 3: "type"
  • a valid SO term defining the feature type - here gene or mRNA
• col 4 & 5: "start" and "end"
  • coordinates of the feature on the landmark in col 1
• col 6: "score"
  • an optional score, used if the feature has been generated by an algorithm
• col 7: "strand"
  • '+' or '-' to indicate the strand the feature is on
• col 8: "phase"
  • for CDS features to show where the feature begins with reference to the reading frame
• col 9: "attributes"
  • custom attributes to describe the feature, these are name/value pairs separated by ';'. Some attributes have predefined meanings, relevant here:
    • ID - identifier of feature, unique in scope of the GFF3 file
    • Name - a display name for the feature
    • Parent - the ID of another feature in the file that is a parent of this one. In our example the gene is a Parent of the mRNA.

A dot means there is no value provided for the column.

The files we are loading are from PlasmoDB and contain gene, exon and mRNA features, there is one file per chromosome. Look at an example:

> less ~/malaria/genome/gff/MAL1.gff3

10.3 The GFF3 source

InterMine includes a parser to load valid GFF3 files. The creation of features, sequence features (usually chromosomes), locations and standard attributes is taken care of automatically.

To add the GFF3 source to your Mine:

1. Select the "Add source" option on the Sources menu.
2. Fill out the form as below:

3. Enter the name of the source, "malaria-gff":

• The properties set for malaria-gff are:
  • gff3.seqClsName = Chromosome
    • the ids in the first column represent Chromosome objects, e.g. MAL1
  • gff3.taxonId = 36329
    • taxon id of malaria
  • gff3.dataSourceName = PlasmoDB
    • the data source for features and their identifiers, this is used for the DataSet (evidence) and synonyms.
  • gff3.seqDataSourceName = PlasmoDB
    • the source of the seqids (chromosomes) is sometimes different to the features described
  • gff3.dataSetTitle = PlasmoDB P. falciparum genome
    • a DataSet object is created as evidence for the features, it is linked to a DataSource (PlasmoDB)

• In some cases specific code is required to deal with attributes in the gff file and any special cases. A specific source can be created to contain the code to do this and any additions to the data model necessary. For malaria gff we need a handler to switch which fields from the file are set as primaryIdentifier and symbol/secondaryIdentifier in the features created. This is to match the identifiers from UniProt, it is quite a common issue when integrating from multiple data sources.
  • From the example above, by default: ID=gene.46311;description=hypothetical%20protein;Name=PFA0210c would make Gene.primaryIdentifier be gene.46311 and Gene.symbol be PFA0210c. We need PFA0210c to be the primaryIdentifier.
• The malaria-gff source is held in the bio/sources/malaria-gff directory. Look at the project.properties file in this directory, there are two properties of interest:

  # set the source type to be gff
  have.file.gff=true
  
  # specify a Java class to be called on each row of the gff file to cope with attributes
  gff3.handlerClassName = org.intermine.bio.dataconversion.MalariaGFF3RecordHandler
  

• Look at the MalariaGFF3RecordHandler class in bio/sources/malaria-gff/main/src/org/intermine/bio/dataconversion. This code changes which fields the ID and Name attributes from the GFF file have been assigned to.

  > less ~/dev/bio/sources/malaria-gff/main/src/org/intermine/bio/dataconversion/MalariaGFF3RecordHandler.java
  

10.5 FASTA files

FASTA is a minimal format for representing sequence data. Files comprise a header with some identifier information preceded by '>' and a sequence. At present the InterMine FASTA parser loads just the first entry in header after > and assigns it to be an attribute of the feature created. Here we will load one FASTA file for each malaria chromosome. Look at an example of the files we will load:

> less DATA_DIR/malaria/genome/fasta/MAL1.fasta

Add a fasta source to your Mine by following these steps:

1. Select "Add source" option from the Sources menu
2. Select the "fasta" type and name your source "malaria-chromosome-fasta"
3. Click the "Add" button and save your changes.

The following properties are defined for malaria-chromosome-fasta:

• fasta.className = org.intermine.model.bio.Chromosome
  • the type of feature that each sequence is for
• fasta.dataSourceName = PlasmoDB
  • the source of identifiers to be created
• fasta.dataSetTitle = PlasmoDB chromosome sequence
  • a DataSet object is created as evidence
• fasta.taxonId = 36329
  • the organism id for malaria

This will create features of the class Chromosome with primaryIdentifier set and the Chromosome.sequence reference set to a Sequence object. Also created is DataSet and DataSource as evidence.

11 Data Integration

For an introduction to data integration in InterMine please read: DataIntegration

Click on the "Build Mine" button at the bottom of the page to build all three sources: UniProt, gff, and fasta. This should take about 5 minutes.

11.1 Data integration in MalariaMine

The sources uniprot and malaria-gff have both loaded information about the same genes. Before loading genome data we ran a query to look at the information UniProt provided about the gene PFL1385c:

malariamine=# select id, primaryidentifier, secondaryidentifier, symbol, length , chromosomeid, chromosomelocationid, organismid from gene where primaryIdentifier = 'PFL1385c';
    id    | primaryidentifier | secondaryidentifier | symbol | length | chromosomeid | chromosomelocationid | organismid 
----------+-------------------+---------------------+--------+--------+--------------+----------------------+------------
 83000626 | PFL1385c          |                     | ABRA   |        |              |                      |   83000003
(1 row)

Which showed that UniProt provided primaryIdentifier and symbol attributes and set the organism reference. The id was set automatically by the ObjectStore and will be different each time you build your Mine.

Running the same query after malaria-gff is added shows that more fields have been filled in for same gene and that it has kept the same id:

malariamine=# select id, primaryidentifier, secondaryidentifier, symbol, length , chromosomeid, chromosomelocationid, organismid from gene where primaryIdentifier = 'PFL1385c';
    id    | primaryidentifier | secondaryidentifier | symbol | length | chromosomeid | chromosomelocationid | organismid 
----------+-------------------+---------------------+--------+--------+--------------+----------------------+------------
 83000626 | PFL1385c          | gene.33449          | ABRA   |   2232 |     84017653 |             84018828 |   83000003
(1 row)

This means that when the second source was loaded the integration code was able to identify that an equivalent gene already existed and merged the values for each source, the equivalence was based on primaryIdentifier as this was the field that the two sources had in common.

Note that malaria-gff does not include a value for symbol but it did not write over the symbol provided by UniProt, actual values always take precedence over null values (unless configured otherwise).

Now look at the organism table:

malariamine=# select * from organism;
 genus | taxonid | species | abbreviation |    id    | shortname | name |               class                
-------+---------+---------+--------------+----------+-----------+------+------------------------------------
       |   36329 |         |              | 83000003 |           |      | org.flymine.model.genomic.Organism
(1 row)

Three sources have been loaded so far that all included the organism with taxonId 36329, and more importantly they included objects that reference the organism. There is still only one row in the organism table so the data from three sources has merged, in this case taxonId was the field used to define equivalence.

11.2 How data integration works

Data integration works by defining keys for each class of object to describe fields that can be used to define equivalence for objects of that class. For the examples above:

• primaryIdentifier was used as a key for Gene
• taxonId was used as a key for Organism

For each Gene object loaded by malaria-gff a query was performed in the malariamine database to find any existing Gene objects with the same primaryIdentifier. If any were found fields from both objects were merged and the resulting object stored.

• Note that many performance optimisation steps are applied to this process. We don't actually run a query for each object loaded, requests are batched and queries can be avoided completely if the system can work out no integration will be needed.

We may also load data from some other source that provides information about genes but doesn't use the identifier scheme we have chosen for primaryIdentifier (in our example PFL1385c). Instead it only knows about the symbol (ABRA), in that case we would want that source to use the symbol to define equivalence for Gene.

Important points:

• A primary key defines a field or fields of a class that can be used to search for equivalent objects
• Multiple primary keys can be defined for a class, sources can use different keys for a class if they provide different identifiers
• One source can use multiple primary keys for a class if the objects of that class don't consistently have the same identifier type
• null - if a source has no value for a field that is defined as a primary key then the key is not used and the data is loaded without being integrated.

11.3 Primary keys in MalariaMine

The keys used by each source are configured in the corresponding bio/sources/ directory.

For uniprot:

> less ../bio/sources/uniprot/resources/uniprot_keys.properties

And malaria-gff:

> less ../bio/sources/malaria-gff/resources/malaria-gff_keys.properties

NOTE - the key on Gene.primaryIdentifier is defined in both sources, that means that the same final result would have been achieved regardless of the order in the two sources were loaded.

These _keys.properties files define keys in the format:

Class.name_of_key = field1, field2

The name_of_key can be any string but you must use different names if defining more than one key for the same class, for example in uniprot_keys.properties there are two different keys defined for Gene:

Gene.key_primaryidentifier = primaryIdentifier
Gene.key_secondaryidentifier = secondaryIdentifier

It is better to use common names for identical keys between sources as this will help avoid duplicating database indexes.

Each key should list one or more fields that can be a combination of attributes of the class specified or references to other classes, in this cases there should usually be a key defined for the referenced class as well.

Note - it is still possible to use a legacy method of configuring keys, where keys are defined centrally in dbmodel/resources/genomic_keyDefs.properties and referenced in source _keys.properties files.

12.4 Dealing with conflicts

It is possible that two different sources could have a value for the same field, in this case we need to define which source should take precedence.

Try loading data from the interpro source which will cause a conflict with uniprot. UniProt loads a list of protein domains for each protein but doesn't include the full names and descriptions of ProteinDomain objects, the interpro source adds this information.

You will see a message that includes the text:

Conflicting values for field ProteinDomain.shortName between uniprot (value "CO_DHase_flav_C", in database with ID 10008203) and interpro (value "CO_DH_flav_C", being stored). This field needs configuring in the genomic_priorities.properties file

Slightly different values have been provided for the shortName of a particular ProteinDomain. In this case we want to take the value from interpro. Edit dbmodel/resources/genomic_priorities.properties and add the line:

ProteinDomain.shortName = interpro, uniprot

Now run try to load interpro again.

See here for a full description of configuring priorities: PriorityConfig

12.5 The tracker table

A special tracker table is created in the target database by the data integration system. This tracks which sources have loaded data for each field of each object. The data is used along with priorities configuration when merging objects but is also useful to view where objects have come from.

• Look at the columns in the tracker table, objectid references an object from some other table

• Query tracker information for the objects in the examples above:

  select distinct sourcename from tracker, gene where objectid = id and primaryidentifier = 'PFL1385c';
  
  select objectid, sourcename, fieldname, version from tracker, gene where objectid = id and primaryidentifier = 'PFL1385c';
  
  select distinct sourcename from tracker, organism where objectid = id;
  

13 Add GO and GO-annotation sources

Gene Ontology (GO) provides a controlled vocabulary of terms for describing gene product characteristics and gene product annotation data in any organism. See  http://geneontology.org for more information.

Add the go and go-annotation sources to your Mine.

• OBO file for GO is in `malaria/go/' directory
• go-annotation data is in malaria/go-annotation

14 Updating Organism and Publication Information

Organisms and publications in InterMine are loaded by their taxon id and PubMed id respectively. The entrez-organism and update-publications sources can be run at the end of the build to examine the ids loaded, fetch details via the NCBI Entrez web service and add those details to the Mine.

Add these two sources to your mine:

• update-publications
• entrez-organism

14.1 Fetching organism details

You will have noticed that in previous sources and in project.xml we have referred to organisms by their  NCBI Taxonomy id. These are numerical ids assigned to each species and strain. We use these for convenience in integrating data, the taxon id is a good unique identifier for organisms whereas names can come in many different formats: for example in fly data sources we see: Drosophila melanogaster, D. melanogaster, Dmel, DM, etc.

• Looking at the organism table in the database you will see that the only column filled in is taxonid:

  > psql malariamine
  malariamine#  select * from organism;
  

• Run "Build Mine"
• Now run the same query in the production database, you should see details for P. falciparum added:

  > psql malariamine
  malariamine#  select * from organism;
  

NOTE: As this source depends on organism data previously loaded it should be one of the last sources run and should appear at the end of <sources> in project.xml.

14.2 Fetching publication details

Publications are even more likely to be cited in different formats and are prone to errors in their description. We will often load data referring to the same publication from multiple sources and need to ensure those publications are integrated correctly. Hence we load only the PubMed id and fetch the details from the NCBI Entrez web service as above.

• Several sources InterMine sources load publications:

  malariamine#  select count(*) from publication;
  malariamine#  select * from publication limit 5;
  

• Run "Build Mine"
• Now details will have been added to the publication table:

  malariamine#  select * from publication where title is not null limit 5;
  

NOTE: Sometimes, especially with very large numbers of publications, this source will fail to fetch details correctly. Usually running it again will work correctly.

Occasionally erroneous PubMed ids are included from some sources and their details will not be updated, there is no good way to deal with this situation.

15 Post Processing

Post-processing steps are run after all data is loaded, they are accessed via the "Post-processes" option on the "Sources" menu.

• Some of these can only be run after data from multiple sources are loaded. For example, for the Malaria genome information we load features and their locations on chromosomes from malaria-gff but the sequences of chromosomes from malaria-chromosome-fasta. These are loaded independently and the Chromosome objects from each are integrated, neither of these on their own could set the sequence of each Exon. However, now they are both loaded the transfer-sequences post-process can calculate and set the sequences for all features located on a Chromosome for which the sequence is known.

• Some post-process steps are used to homogenize data from different sources or fill in shortcuts in the data model to improve usability - e.g. create-references.

• Finally, there are post-process operations that create summary information to be used by the web application: summarise-objectstore, create-search-index and create-autocomplete-indexes.

A list of common post-process targets with some indication whether you will need to use them is give here: PostProcessing.

15.1 MalariaMine Post Processing

The following <post-process> targets are included in the MalariaMine project.xml. Run queries listed here before and after running the post-processing to see examples of what each step does.

15.1.1 create-references

This fills in some shortcut references in the data model to make querying easier. For example, Gene has a collection of transcripts and Transcript has a collection of exons. create-references will follow these collections and create a gene reference in Exon and the corresponding exons collection in Gene.

malariamine#  select * from exon limit 5;

The empty geneid column will be filled in representing the reference to gene.

15.1.2 transfer-sequences

The sequence for chromosomes is loaded by malaria-chromosome-fasta but no sequence is set for the features located on them. This step reads the locations of features, calculates and stores their sequence and sets the sequenceid column. The sequenceid column for this exon is empty:

malariamine# select * from exon where primaryidentifier = 'exon.32017';

After running transfer-sequences the sequenceid column is filled in and we can see the sequence of this exon:

malariamine# select primaryidentifier, exon.length, residues from exon, intermine_sequence where sequenceid = intermine_sequence.id and exon.primaryidentifier = 'exon.32017';

15.1.3 do-sources

Each source can also provide code to execute post-process steps if required. This command loops through all of the sources and checks whether there are any post-processing steps configured. There aren't any for the sources we are using for MalariaMine but you should always include the do-sources element.

15.1.4 summarise-objectstore, create-search-index & create-autocomplete-index

These generate summary data and indexes used by the web application, see WebappConfig.

Save your changes and click on the "Build Mine" button to build your database.

16 Creating your own source

You can create custom sources to load your own data, there is a tutorial for doing this and two workshop exercises for trying it out in Java or Perl:

• SourceHowto
• WorkshopCreateSourcePerl

17 Webapp

You can now release a webapp against your production database, see:

WorkshopWeb