How to integrate your own data with a local copy of FlyMine

This page explains how to set up a local copy of the latest FlyMine release and integrate your own data. You will be able to query the combined data in a local web application or with the query API. The instructions here assume you need to add classes and fields to the data model. If the data you want to load is already described by the model you can follow the standard mirror instructions and load your data in at the end.

1 Prequisites

Ensure you have the required software installed and configured. See: prerequisites.

2 Get the InterMine software

You should check out the latest FlyMine release. See SVNCheckout for information on how to download the InterMine code.

3 Create PostgreSQL databases

These correspond to database names in a flymine.properties file in the .intermine directory.

createdb flymine
createdb items

If you intend to use the web application:

createdb userprofile-flymine

4 Create a new datasource

Create and modify necessary files for new source(s).

See: creating a new source.

5 Create a blank database schema

The schema will include your additions to the data model. In flymine/dbmodel run:

ant clean build-db -v

You should see table/columns corresponding to the new classes you added to the model in the production database.

6 Dump the metadata table from the database

This is the table that stores an XML version of the object model. It is needed by any client program that queries the FlyMine. As we have added to the model we need to preserve the copy we just created to restore after the FlyMine data dump has been loaded:

pg_dump -c -f updated-metadata-table -t intermine_metadata flymine

7 Create tables

Some tables and indexes need to be added to the production database. You need to create these two tables because the data dump you will be loading in the next step has data for these tables.

CREATE TABLE precompute_index (
       name character varying(255),
       "statement" bytea,
      category character varying(255)
);

CREATE TABLE tracker (
      objectid integer,
      fieldname text,
      sourcename text,
      version integer
);

CREATE INDEX tracker_objectid ON tracker(objectid);

8 Get the latest data

From  http://www.flymine.org/download/flymine (file has -data-only suffix). Load up this dump:

zcat dump_file | psql flymine

9 Update model

This will have overwritten the updated model we created with the previous FlyMine model. Now we need to load back in the updated one:

cat updated-metadata-table | psql flymine

10 Check data

Check you still have all the updated tables/columns in the database and some data. It would be a good idea to make a new dump of the database so you don't have to repeat this procedure again.

pg_dump -c -f updated-flymine-version database_name

11 Analyse

Analyse the database to update postgres' statistics.

psql flymine
flymine# set default_statistics_target = 500;
flymine# analyse;

12 Configure

• Copy the example flymine.properties files to your home directory
• Edit them to include your own database and tomcat locations and usernames/passwords.

13 Load data

You should now be able to load data from your own sources into this database. In flymine integrate run:

ant -v -Dsource=new_source integrate

14 Search index

InterMine's search is powered by Lucene. To create the Lucene search index, run this command in flymine/postprocess:

>ant -Daction=create-search-index

If this step isn't completed, the webapp will work correctly but the keyword search will never return any results.

See: KeywordSearch

15 Release the web application

In flymine/webapp:

# the first time only
ant build-db-userprofile
ant default release-webapp

You only need to run 'build-db-userprofile' the first time you set up a webapp. Use ant remove-webapp if there is already a web application deployed at the specified path:

# to re-release the webapp
ant default remove-webapp release-webapp