Last modified 3 months ago Last modified on 10/11/11 11:40:40

InterMine 0.98

UserProfile Porting

This release of InterMine contains a major shift in the UserProfile porting process (moving the userprofile data from one production database to another). Previously InterMine developers had to write and read the userprofile XML manually, during which the code "upgraded" lists to reflect the data in the new production database. This was transparent to the user and time consuming for the devs - FlyMine's userprofile upgrade process took over 3 hours - during which time FlyMine.org had to be unavailable.

The new process is much simpler. When changing production databases, the lists don't "upgrade" to the new database until the user logs in. If there are no conflicts, the lists are upgraded automatically and the user does not need to take any action. If there are problems and the object in their list cannot be resolved in the new database, eg. a gene has been split etc, the user is taken through the list upgrade process where they are asked to resolve the various issues.

NOTE - to successfully upgrade to InterMine 0.98, you must run an ant task using the same production database that created the users's saved lists.

See UpgradeInterMine for details.

Human readable labels

Prior to InterMine 0.98, the field and table names from the database were what was used in the webapp. Now the data fields are labeled using human readable labels. For example:

name in databasename displayed in webapp
primaryIdentifierDB Identifier
secondaryIdentifierSecondary Identifier
trnatRNA

These labels are fully customisable for each Mine.

See ClassAndFieldLabels

Themes

RGD's RatMine theme is now maintained and tested by InterMine. If you have a custom theme, please contact us an we can integrate into the core InterMine code.

Webapp Tables & Lists

Tables in the webapp now use a generic frameworks that allows us to deliver a consistent look & feel across many mines while keeping the maintenance cost to the minimum. This CSS and JS framework is accessible from anywhere in the webapp, so if you would like to create a table that looks similar to the ones on the Report page, refer to WebappTables for instructions and sample code.

See WebappTables

InterMine JavaScript library

This library is now included on all pages of the webapp. You can see it has been loaded by opening up Firebug (in Firefox or Chrome) as it logs a status message. This library makes it easier for you to debug the JavaScript side of the webapp by, for example, allowing you to see stack traces or the path to an element that you are currently dealing with (jQuery).

See InterMineJavaScript

OpenID

Users can now log in to your Mine using their GMail, Yahoo, or MyID. See the wiki page for more details.

Region Search

  • Additional parameter allows users to extend regions at both sides
  • Can create lists by feature types for all results or for an individual region
  • Results show gene symbols as well as DB identifier
  • Ability to exclude feature types from feature selection, see upgrade instructions for details.
  • Fixed by when exporting a long list of objects to galaxy caused a "Request-URI Too Large" error
  • Now works correctly with subclasses

Web services

  • NEW: region search The ability to search for genomic features by interval (the Regions tab of the web-app) has been exposed as as a web-service resource. Region data can be converted to lists, or downloaded directly as GFF3, FASTA or BED.
  • NEW: token authentication A new authentication mechanism has been introduced which has advantages over the current system of sending username and password details. The use of tokens means that users can authenticate requests without having to send their username and password, thus minimising chances of these being compromised. Tokens can be replaced without having to change login details, or indeed disabled entirely, meaning web-service access to user data can be forbidden on a user-by-user basis. This authentication mechanism is to be preferred to the current mechanism. One important benefit is the ability to make authenticated requests (for dealing with lists for instance) from the browser.
  • NEW: Ruby client Our commitment to supporting the range of languages used by bio-informatians has been extended to Ruby, which has a growing community based around the Bio-Ruby project. The Ruby client libraries are available from  rubygems.org. For documentation please visit http://www.intermine.org/docs/ruby-docs.
  • NEW: Exporting BED from webservice queries In addition to GFF3 and FASTA, the BED biological standard format is supported from web-service queries.
  • NEW: Column summary data The column summary data (visible when clicking on the Σ icon on a results table) is now available via web-services when passing the summaryPath parameter to a query or template results request.
  • NEW: Code generation The ability to generate code to run queries is now available programmatically via a web-service. Please see the WebService wiki page for details.
  • NEW: Summary Service For each class in the data-model, a user may also fetch the list of fields needed summarise that class. This is useful when automatically building queries from classes passed in as variables.

Export

  • New export format -  BED
  • Fixed bugs that caused export to fail when:
    • feature didn't have location info
    • sequence feature was not the root class, eg. Pathways.genes
    • all columns of an outer join are removed

Data

  • New EnsemblCompara data converter
  • UniProt converter updated to new XML format
  • Strains are handled in config files

Displayers

  • Publications displayer orders by number of genes mentioned
  • New Pathways displayer shows pathways from other Mines for orthologues of current gene
  • New RGD disease displayer shows disease ontology terms from RatMine for orthologues of current gene
  • Mines without Gene Ontology terms in the model saw a compilation error caused by the GO displayer, this has been fixed
  • New viewer shows interactions, GO terms etc on publication report page

Templates

  • Radio buttons stay checked when selecting/deselecting list constraint
  • Clicking on the "query >" breadcrumb to get back to the template form works correctly
  • Editing constraints for nested classes (eg. Gene > homologue > organism) doesn't break model browser
  • In query builder, "Show results" button greyed out and not clickable anymore
  • After editing a template, if the user clicked on a template on the home page they would be directed to the previous template they had been editing.
  • Can now import template queries containing "ONE OF" constraints
  • Added option to delete trackers to the template import form. This allows superusers to export/import templates from other Mines, but save the usage information.
  • Fixed error when running a template with a bag constraint in an attribute and a lookup constraint in its parent

Network Displayer

  • Added help text for different file types
  • Large datasets are now handled correctly. If there are too many interactions (> 2000) a link to download the data is shown instead of the displayer: 
    The network contains more than 2000 elements, the interaction can become sluggish in the displayer.
Please [Download the network data] and import it to [Cytoscape desktop] and use Force-Directed Layout
to view or [view interaction data] in a table.

Scripts

  • TemplateComparison
    • syntax has changed
    • can run with single mine now
  • New data download scripts

Other changes

  • NEW - You no longer need configure fields to summarise into dropdowns, all fields are summarised automatically.
  • FIX - keyword search now handles booleans correctly
  • FIX - pathway widget correctly filters for KEGG pathways
  • UPDATE - widget data easier to download (request from user testing)
  • NEW - filter linkouts by dataset, see upgrade instructions for details
  • FIX - black background with black font bug fixed
  • NEW - struts upgraded to 1.1
  • UPDATE - list upload form has default organism selected, leading to less issues when uploading lists
  • FIX - links to other mines is cleverer, so results appear faster
  • NEW - order lists on home page, see Tagging
  • FIX - report pages are now cached globally instead of per session

InterMine 0.97

New Report Pages

Report pages have had a major design and technical overhaul. Based on a lot of feedback we've redesigned the header section of the page, added section headers for easy navigation in the page and added a new information section on the right hand side. The main changes:

  • New report pages displayer architecture makes it much easier to customise data display. There are standard displayers for GO, homologues, GBrowse, gene models, sequence feature summaries and interactions. There are also specific displayers in FlyMine and modMine that can be used as example - for example FlyAtlas expression charts in FlyMine. See the [guide to configuring displayers].
  • Customisable page headers - you can configure one or more fields to as the top level page heading, and format them, e.g. italics for organism names.
  • Inline lists are a compact way of displaying collections of strings for which a table would waste space, e.g. for a list of synonyms. Any collection can be configured to be an inline list.
  • New section headers appear as tab bar that docks to the top of the screen as you scroll, the relevant header is highlighted when the section is in view.
  • Templates and collections are greyed out if they do not have any results.
  • The various tables and lists of data have expanders, togglers and collapsers.
  • Related collections have been grouped together, for example now you would find CRMs and TFBindingSites under Regulatory Regions.
  • You can quickly move across categories of content through a sticky menu. This is a compromise that lets us show you all of the content we have for a given object at once, while allowing you to skip to the things you really want to see.

For information on how to configure the various parts of the report page, please head to [ReportPage].

Usage statistics

The Usage page shows some user activities recorded during browsing of the webapp:

  • Template usage
  • Custom queries executed
  • Logins
  • Keyword Searches
  • List Analysis page views and list creation.

You can access to the Usage page, via the MyMine page, only if you are a superuser.
For more details see: trackers

List Upload page redesign

  • List analysis is visually split into the stages of list upload, verifying identifier matches and the analysis itself.
  • A default list name (based on type of items, date and time) is generated
  • List upload verification stage now shows you how many items and additional matches you have in a list, allowing you to add groups of them while informing you if you have more matches to add to the list.

Web Services

In addition the changes below, all our client libraries now have corresponding code generators accessible from the web interface to help you start writing your code.

New Web-Services

We have added:

  • List functionality:
    • Retrieval of details and inspection. Logged in users can retrieve their own list details.
    • Queries against lists (including private lists, if logged in).
    • Renaming
    • Deletion
    • Creation, from lists of identifiers, and queries.
    • Appending.
    • Union/Intersection/Symmetric? Difference
    • Subtraction (Asymmetric Difference)
    • List membership (which lists is an object in) Many services support tagging.
  • Biological webservices:
    • Getting results as GFF3 and Fasta is now supported.
  • Possible Path Values:
    • Users can get summaries of the values a path may represent, along with frequency details.
  • Template and Query upload services:
    • templates and queries can be saved to a user's profile.
  • Resource reporting - users are able to check if an optional service has been enabled.
  • Schemata for result formats (xsd for XML and json-schema for JSON)

See WebService for details.

Python Library

There is a new client library for our webservices API written in Python. This offers access to a very broad range of webservice functions, including the new list webservices. There are complete API docs available at our docs section: http://www.intermine.org/docs/python-docs

See PythonClient

JavaScript library

This has been made more flexible and extensible, and includes helpers for accessing services beyond query results. The main changes regard content presentation in the default table display, with a number of rendering bug fixes and UI improvements. The latest version is 0.2.

See JavaScriptClient

Perl Library

Now with access to all webservice features, including lists and possible path values. The major addition here however is character safe result parsing using structured message formats. Also there have been a number of method additions and improvements to make common jobs easy, such as shortcuts for adding all attributes on a class, or iterating over a query's result-set. Lists and queries are broadly interchangable (both can be used as sources of results, or used in list operations, or in list constraints). All constraint types are now supported, including list and loop constraints.

In addition, a separate optional module, Webservice::InterMine::Bio provides access to the new biological webservices.

See PerlWebServiceAPI

Java Library

The Java webservice client now has paging support in its requests, along with better, more thorough documentation.

See JavaClient

Genomic Region Search

See GenomeRegionSearch for details.

Layout changes

  • Removed breadcrumbs on right side
  • Tab behaviour: no tab is active on results and report pages
  • In the query builder page, changed links so they look like the ones on template form and added the green button Show Result in the top too.
  • Moved Contact Us link, removed contextual help.
  • Display different content on homepage if new user.

Data

  • Default config for GO converter
  • Introns collection added to Gene
  • New human, RGD and MGI identifier sources
  • New Ensembl compara and identifier data sources

Keyword search

  • Keyword search correctly handles special characters
  • You can now use a wildcard as the first character of a search string
  • Option to exclude specific fields from search index

Other changes

  • UPDATE - Add LIKE and NOT LIKE to input boxes of constraints
  • UPDATE - The dropdowns of lists in the template form are in alphabetical order
  • FIX - Query Builder constraint logic is case insensitive and displays A and B instead of A,B
  • FIX - Template export works when select single or multiple templates on MyMine and Templates tab
  • FIX - Forgotten password link sends a message with the url
  • FIX - Input a filter value or select a category in Templates (or List) page , select a template (or a list), click 'Back': the page still displays the filter edited and/or the category selected
  • FIX - When copy one or more lists the name is automatically generated
  • UPDATE - Javascript and CSS are now minified
  • UPDATE - Removed Excel export
  • NEW - Display different content on homepage if new user
  • NEW - Users (non-superusers) can change column header descriptions
  • FIX - Error generated when clicking on summary button twice
  • FIX - Bug which caused templates to not appear on the correct report page
  • FIX - Search box toggle on homepage displays correct help text
  • FIX - Nav trail now displayed in the query builder
  • FIX - Multi-Value template parameters now comma-safe

InterMine 0.96

Linking in to InterMine

There are several new ways to link into your Mine. See Links for details.

Web service JSON formats

There are several new webservice results formats. Joining jsonobjects which appeared in 0.95, we now have JSON formats for pretty much everything else:

Results:

  • jsonrows - tabular representation of data with extra meta-data
  • "jsontable" - container format for building embedded templates (see: IMBedding tutorial).
  • "count" and "jsoncount" - formats for just receiving the count of a query

Webservice meta-data (Model and Templates):

  • "json" - easily parsable versions of the data model and available template list

Authentication Now Working in Perl Webservice Client

Users of the  Perl Webservice client now have programmatic access to their lists and private templates. Simply provide your login details to the client to access your account (your details are transmitted in encrypted format). This feature will work with all mines with webservices, not just those at 0.96 and above.

Object Results with Webservice Clients

The Perl and Java clients are able to get results back as objects (JSON Objects, or in the Perl, inflated up to full result objects). This simplifies some complex data processing issues, and dealing with outer joins and collections.

Embedded Tables

You can now natively embed data tables from any mine with a supporting webservice into your webpages with JavaScript and CSS. See the IMBedding tutorial for a demonstration and an introduction to this technique. Be sure to play around with the new JavaScript library available at http://www.intermine.org/lib/imbedding.

Registry

We have launched a central registration service for InterMine implementations. This will not only serve as a "Yellow-pages", enabling visitors to discover new data-sources, and facilitating interoperation between the mines themselves, but it also supports the webservices by providing a registry of webservice urls and versions. Visit the registry at http://www.intermine.org/registry, and for help in registering your own mine, visit RegisteringYourMine.

Link Redirects

You can add a property to have links in the results pages redirect to any URL. See LinkRedirects for more information.

Home page content

The home page for InterMine can now be configured to display different information based on whether or not the user is new. See HomepageConfig for instructions on how to configure this behaviour.

New tag

We've added a new tag: im:admin. If you tag a template with im:admin, it will not show up in your webapp. This is useful for hiding templates needed by InterMine's webservices from your users. See Tagging for a list of all available tags.

List upload

The list upload process has been improved.

Other changes

  • UPDATE – more rigorous multiple hypothesis testing correction on enrichment widgets. "Number of tests" is now the number of terms associated with the reference list.
  • UPDATE – more helpful filter options on enrichment widgets
  • NEW -  Holm-Bonferroni option added to enrichment widgets
  • UPDATE – new "delete" button on lists page
  • UPDATE – interactions displayer loaded via AJAX so report page loads faster
  • FIX – "score" images displaying correctly on search results
  • UPDATE – QueryBuilder more interactive, selecting an item in the list will take you back to the original item after page refresh/reload
  • UPDATE – link added to column summary in query results, making is easier to view distinct values
  • FIX – login bug concerning mixed case usernames
  • FIX – widget bug when viewing after exporting widget data
  • NEW – template queries are ordered by popularity, unless you are logged in, in which case they are ordered by the date each query was last run
  • FIX – data isn't repeated in the "miscellaneous" section on report pages
  • FIX – templates displayed correctly on report page even for subclasses
  • FIX – templates correctly rendering on aspect pages
  • UPDATE – updated BioGRID converter to handle changes in newest data files
  • FIX – various "gold" theme CSS and images improved and updated
  • FIX – thread safety issue addressed in template tracking
  • UPDATE – table widgets can group by an attribute as well as a class

InterMine 0.95

New style home page

The start page of the webapp is now clearer and puts more emphasis on keyword search and lists. There is an optional section for popular in data category tabs. See  FlyMine as an example.

It's easy to configure the text and examples, see instructions here.

Popular templates

Template usage by users is now tracked and used to rank templates by popularity. In the templates tab the most used templates are shown first, in the data categories on the home page the most popular in each category are shown.

The superuser can see the number of times each template has been executed and the rank for each template in the saved templates page of MyMine. The rank of each template is determined by the sum for all users of log(number of times executed). So a template becomes more popular if many people use than if a single user executes it many times.

A new templatetrack table will be created automatically in your userprofile database when you deploy the new webapp.

Permalinks on report pages

Report pages still use internal ids of the object in the URL, these change in each release. Now you can always get a link that will work between database builds with a link button a the top of each report page.

API tab and web service code generation

At the bottom of each template query form and the QueryBuilder interface is a link to obtain automatically generated Java or Perl code to run that query. This is a great way for people to get started with the web service query API. The new API tab includes instructions for installing Perl modules, downloading the client jar and running the examples. The Java client jar file is now deployed with the webapp so users will always be able to download the correct version.

Web service JSON objects

There is a new web service results format which returns JSON representations of objects. This is designed for use in external programs that aim to write libraries of code to handle results of many queries. The returned JSON contains an array of objects representing the root class in the query with attributes populated, with nested JSON objects and arrays for references/collections that were included in the query view. The JSON can easily parsed by standard libraries in many languages.

More details

MineManager model viewer and editor

The MineManager is a graphical user interface to make setting up a new InterMine systems simple. It helps set up configuration files, add sources to the project.xml file and run database builds. It now includes a model viewer that provides a graphical view of the data model and shows which sources provided particular fields. There is also an interface for creating and editing additions files to avoid mistakes when writing XML by hand.

To view your data model:

  • in /intermine/MineManager do ./run
  • select your mine from the list on the right hand side
  • from the menu select Model > View Model

For more information see MineManager and ModelViewer.

Other changes

  • NEW - The behaviour of list upload is now configurable. By default any identifiers in the list being uploaded will be associated with the first (best) match. You can now set a matchOnFirst=false in the bag-queries.xml file to find all matches for each identifier and display any duplicates in the list upload confirmation page.
  • FIX - The create-attribute-indexes post process was dropping indexes that already existed in the database. This caused problems for bioseg (genomic range queries) and made the userprofile upgrade slow. This is now fixed.
  • FIX - inline references and collections on report pages no longer appear in Miscellaneous if they are included in the summary at the top.
  • FIX - templates included on data category pages now display and link correctly.
  • UPDATE - the algorithm for test correction on enrichment widgets has been altered to lower the instance of false positives. Please see the enrichment documentation for details.
  • NEW - the CytoscapeWeb interaction viewer now includes links to export the interaction data and to create a list of genes involved in the network.
  • UPDATE - The version of the Sequence Ontology InterMine uses is updated to 2.4.4.
  • FIX - building the keyword search multiple times on the same database was slow due to binary objects left in the database, these are now deleted each time.
  • FIX - renaming a template now correctly keeps the tags associated with the template.
  • FIX - in the QueryBuilder removing a view column from an outer joined class now works.

InterMine 0.94.1

InterMine 0.94.1 contains several bug fixes identified when upgrading to 0.94 and a number of minor improvements/fixes.

Data model

  • GO annotation has an "evidence" collection
    • Includes a reference to an evidence code
    • collection of publications
    • Currently GO annotation only has a collection of codes and publications.
  • Remove mRNA
  • Made Gene and Chromosome extend BioEntity in core model.
  • Postprocessing code now doesn't require non-core classes to be in the model to compile.
  • fasta source now don't require non-core classes to be in the model to compile.

QueryBuilder/Template

  • If you click 'export' from the template list, the template is loaded from the template manager; if you click 'export' from the template detail page, the template is loaded from the session (with the values edited by the user)
  • Removed an hidden attribute fixing the following bug: when you try to edit a constraint (with possible value) in the edit template page and in the template there is another constraint of lookup type, when you change the operator the querybuilderconstraint page doesn't display the right html object (e.g if you select NONE OF the multiselect is not displayed )
  • If the constraint has null value, the radio button is selected and the input field disabled contains no values (and not IS NULL or IS NOT NULL)

bio/sources

  • Removed synonyms from intron post-process.
  • GO annotation converter updated to GAF 2.0
  • GO converter - "alt_id" entries are added as synonyms
  • In PubMed/Gene source, fixed NullPointerException reading human genes.
  • In update-publications, only set Publication/Authors? collection in one direction: faster and prevents potential bug.
  • In update-publications, prevent exception if journal name is empty.

Galaxy export

  • Logged in/anonymous users can export queries that use lists to Galaxy

List Upload

  • fixed the bug that disabled file input if paste input was clicked on

Template XML import fixed for templates

  • Fix error when importing template XML containing a subclass constraint
  • Fix error when importing template XML containing a loop constraint

Build system

  • Fix 'too many open files' error that occurred on some platforms

Update script fixes

  • Doesn't try to update non-existent files
  • Now will not operate on anything in the testmodel directory

Compare templates script

  • modified to run with one pre and one post 0.94 webservice (so suitable for anyone upgrading from 0.93 to 0.94)

Ensembl API script

  • updated to work with changes to Perl modules

Perl API

  • Items now produced with their classnames set, rather than their implementations
  • Added a version number so it can be installed to the system
  • Fixed empty sort-order bug - updated queries had no sortOrder
  • logging file names when processed, as well as better logging of events and errors
  • Made multi-bit translation heuristic more generous when updating pathstrings
  • Check for definedness of pathstring bits (debugging an error from A. Vallejos)
  • Added ant task to build.xml replacement list
  • Stopped replacing postgres driver in mine.properties files
  • Improved error message for failed directory open * MineManager
  • bug when adding custom source fixed
  • reorder sources graphically

InterMine 0.94

InterMine 0.94 represents a significant change from the previous version. We've updated the data model, improved queries and templates, and added a lot of new functionality.

Web Interface

  • Template form easier to read and use (see next section).
  • List upload delimiters can now be configured.
  • Images now display in results.
  • Search has been improved, see ComingSoonSearch
  • Cytoscape Web - a plugin showing interaction networks in Cytoscape Web (a flash based plugin) on the gene and protein report pages.
  • Occasional lag on list analysis page fixed.
  • Removed confusing message on query history page.

Queries and Template Queries

  • XML format has been simplified but is backwards compatible, see XMLFormatChanges0.94
  • New constraint type, Multivalue - Users can select multiple values in forms.
  • IN/NOT IN renamed to ONE OF/NONE OF
  • Template form
    • Allow optional constraints - user can choose not to restrict a particular constraint
      • some mechanism to 'switch off' a constraint
      • when creating the template you should be able to choose whether constraints are 'on' or 'off'
    • Improve layout of template form
    • Set which order constraints appear in with drag/drop javascript on the Template Preview section in the query builder.
  • QueryBuilder
    • Reverse references obvious & prevent following links e.g. Genes.proteins.gene.proteins.gene...
    • Made subclasses easier to discover and constrain.
    • Added vertical lines to make the model browser easier to navigate.
    • Improved the layout of the constraint popup.
    • "Remove all" button
    • Can now change column heading

Data Model

  • Sequence Ontology updated
  • Several classes renamed, deleted
  • so_additions.xml is generated automatically from Sequence Ontology
  • Unnecessary inheritance means writing to more tables, making data loads slower and databases larger

ObjectStore

  • Chromosome sequences are CLOBs in the databases and feature sequences will be fetched on the fly.
  • Build speeds improved
    • FlyMine's build completed in 10 hours compared to 18 hours
    • The improvement is down to several things:
      • fetch from databases in batches so individual lookups for objects are eliminated
      • more caches throughout the code, avoiding reflection
      • the updated data model removes a lot of superclass tables that were unnecessary but took time to write to (now 334 tables, down from 400)
      • we've removed about 9m synonyms that were used for the old quicksearch, this is now replaced by the Lucene search.
  • Updated postgres driver

Data Converters

  • TreeFam only processes selected organisms
  • ID resolver in Perl

Model Comparison

  • Can compare models with other InterMines

Export

  • Export is much faster as it uses batching intelligently.
  • Galaxy export - export query results to Galaxy in plain text format.

Model Viewer

  • InterMine now has a model viewer in which you can navigate through the model.

 Perl Web Service Client

  • New Perl Distribution: Webservice::InterMine
  • Complete rewrite of Perl Webservice Client - now more modular, more extensible and more powerful
  • Templates can be fetched via a web service and thus run by name.
  • Less boilerplate, and as a result more readable code.
  • Reflects changes to queries and template queries, handling all new constraint types
  • Pre-parsing of results into lines, arrays or hashes.
  • Extension of API through Moose Roles, allowing 'plug-in' style functionality.
  • Easy manual install of dependencies with Module::Build 0.36+
  • Now on CPAN!

Perl Items API

  • Renamed InterMine::ItemFactory to InterMine::Item::Document.
  • Backwards compatible - just need to change the import statement.
  • Less boilerplate, more readable code.
  • Now handles the stuff you don't want to: can handle item storage and writing.
  • Now on CPAN!
  • See here for full change list.

InterMine 0.93

In InterMine 0.93 we have done a large amount of internal refactoring of code to make the core functionality more robust. In addition there are several bug fixes, validation improvements and a new 'hints' help system.

Web interface

  • New hints system displaying short help text at the top of pages. Hints are included by default but you can easily add your own, see Hints.
  • Configuration errors in webconfig-model.xml are logged instead of throwing exception on deployment.
  • The QueryBuilder now uses inner joins as the default when adding new fields to the view list
  • Fixes to help text in the QueryBuilder
  • Display informative error message if the superuser in the database doesn't match the superuser in the properties file.
  • Removed the option to filter lists by tag on the template form, this made the interface clumsy and wasn't very useful
  • Enrichment widgets - fixed rounding error, adding link to documentation on each widget.
  • Use better MIME types for GFF3 and FASTA export - previously export links opened in a browser window for many users
    • now uses text/tab-separated-values for GFF3 and chemical/x-fasta for FASTA.
  • Fix bug that allowed lists to be saved with empty names if only whitespace characters were entered
  • Fixed bad interaction with saved list names and anti-spam JavaScript for the contact form.
  • Link outs can post identifiers as multipart/form encrypted data, required for Reactome skypainter
  • Option to include a link to the contact form just below the header ...

Bio/sources

new

See BioSources for details on how to add these sources to your mine.

  • KEGG homologues
  • Miranda - mirBase targets from the Sanger Institute.

updated

  • UniProt converter
    • Improved memory management so converter can load larger files and works faster.
    • Restored the option to load the whole UniProt XML files.
  • BioGRID
    • If organisms are specified in project.xml, process interactions with ONLY those organisms.
    • If organism is not specified in project.xml, process all files in data directory.
  • OrthologueEvidence now has an evidence code and a publication collection.
  • BioPAX
    • Doesn't throw an exception if a bad file names are found.
    • Synonyms now merging correctly.

Performance

  • Fixes for several bugs that were causing slow export of very large data sets, export should now create temporary precomputed tables correctly to stage the data to be exported. Speed of export should be significantly improved.
  • Better JavaScript and reduced HTML markup have made the list analysis page a bit faster to load.

Web services

  • Embedded templates now use custom column descriptions as column headers if they have been configured.
  • Option to fetch the release version of a Mine from the web service see web service documentation.

Web properties config

  • OrthologueConverter? - Add links to other intermines to list analysis pages. See OrthologueConverter? for details.
  • Meta tag properties can now be configured in web.properties. See WebProperties for more information. Example configuration: 
    # in web.properties, populates meta tags 
meta.keywords = microarray, bioinformatics, drosophila, genomics 
meta.description = Integrated queryable database for Drosophila and Anopheles genomics
  • Mines can now link to Reactome's SkyPainter, an example entry from WebProperties: 
    attributelink.reactomeListProt.Protein.7227.primaryIdentifier.list.url=http://www.reactome.org/cgi-bin/skypainter2?QUERY=<<attributeValue>>&SUBMIT=1&DB=gk_current
attributelink.reactomeListProt.Protein.7227.primaryIdentifier.list.text=Reactome
attributelink.reactomeListProt.Protein.7227.primaryIdentifier.list.usePost=true
attributelink.reactomeListProt.Protein.7227.primaryIdentifier.list.enctype=multipart/form-data
attributelink.reactomeListProt.Protein.7227.primaryIdentifier.list.delimiter=NEWLINE

InterMine 0.92

InterMine 0.92 contains the following changes. The upgrade from 0.91 should require no changes to existing mines and sources, see UpgradeInterMine.

Data loading performance

  • Fixed a bug in query caching while loading data, should improve speed of loading data.
  • Run database analyse less frequently, can be very slow in large databases.

Error messages

  • We have tried to make some of the most commonly encountered error messages better describe the problem and possible solutions. If you spot an error message we could make better, please let us know!

Mine configuration

  • Improved validation of priorities file:
    • validation is done before data loading starts
    • checks that specific classes and field names exist
    • checks that configured source names exist in project.xml
  • Simplified system for configuring integration keys - you can now specify data integration keys entirely in the sourcename_keys.properties file in a source, without first defining the key in genome_keyDefs.properties.
    • More information available here: PrimaryKeys.
    • Note that the old system still works so you don't need to update anything you have already set up.

New widgets

  • A feature length widget that can be used for any genome feature. This draws a normal distribution of lengths for featues in the list and for all features of that type for the selected organism.
  • Pathway enrichment widget - enrichment of KEGG/Reactome pathways.
  • Enrichment widgets now have an option to view all entries analysed by widget in a results table

Bio parsers

  • Added a new source bio/sources/biopax to parse BioPAX format files.
  • FileConverter now has option to parse pipe delimited files.
  • Fetch GO annotation directly from UniProt XML files, set <property name="creatego" value="true"/> in `project.xml

Web application

  • Fix concurrency issue when saving queries to a user profile - previously had a small chance of entering an infinite loop.
  • A new option to export data as a gzipped file.
  • Performance:
    • Consider existing precomputed tables when attempting to precompute to boost export speed.
    • Fix bug that prevented clean up of temporary precomputed tables
  • Fix problem in QueryBuilder when using subclasses and outer joins in a query.
  • When linking out with a list of ids add option to configure delimiter to use.
  • Improved results table headers, narrower columns with abbreviated header, tooltip to get full header
  • Better wrapping of long text on report pages, no longer breaks within words.
  • Can configure results table column summaries to treat some numbers as strings - e.g. for Publication.year it is better to see the counts per year rather than mean and standard deviation.

InterMine 0.91

InterMine 0.91 contains the following changes. The upgrade from 0.90 should require no changes to existing mines and sources, see UpgradeInterMine.

QueryBuilder

There are several improvements to advanced features of the QueryBuilder

  • The Query Overview in the top right pane is now more consistently ordered.
  • Constraint Logic: queries with multiple sets of OR constraints now work correctly, subsequent OR constraints were being ignored.
  • Constraint Logic: removing constraints no longer resets all operations from OR to AND, and no longer removes brackets.
  • ‘loop queries’ and outer joins now work correctly together, loop constraints are not permitted across outer joins.
  • New constraints added to the query always default to inner joins.
  • Paths are no longer added to the Query Overview when you cancel adding a constraint
  • A space is no longer a valid separator in LOOKUP constraints (in the QueryBuilder and templates), e.g. 'Drosophila melanogaster' is now treated as a single value. Comma is still used as a delimiter.

Export

Some bugs have been fixed for some edge cases in export.

  • Fixed a bug where export failed in some cases with complex queries.
  • Exported column headings now use the description if this has been specified in a Template Query – to match the results table behaviour.
  • The column description for the root node is no longer lost when exporting.
  • Rows are no longer duplicated when exporting a query with multiple outer joined collections that each contain only one row.
  • Removing the column the original query was sorted by no longer gives an error when exporting.

User passwords

  • Passwords are now hashed in the userprofile database and are not sent as reminders in e-mail. There is a new reset password system that sends a link containing a unique token to the user, this link can be used to reset the password.
  • Passwords in existing userprofile databases will become hashed when upgrading between releases with write-userprofile-xml and read-userprofile-xml.
  • When a new userprofile database is created the superuser password will now be hashed, therefore there is a new superuser.initialPassword in the xxxmine.properties file to set the password to some initial known value. The password should then be changed.

Ontology model change

  • There is now a parents collection on the OntologyTerm class (and its subclasses) containing all parents of the term and the term itself. The collection is automatically filled in for any obo source. This makes it easier to create queries that search for genes that have a particular term applied or any child of that term, e.g. by searching for Gene.goAnnotation.ontologyTerm.parents.name = 'DNA binding'.

bio/sources

  • Added a new parser for TreeFam orthologue data in bio/sources/treefam.
  • Updated bio/sources/go-annotation with a new, faster parser. The go-annotation source now uses less memory and no longer requires that the gene association file is ordered by the gene/protein identifier.
  • The make_source script can now create sources to load .obo files.

Data integration priorities

It is now possible to specify data integration priorities in a more flexible manner. Previous you could set a priority only for the class which declared the field (e.g. BioEntity.primaryIdentifier), so it wasn't possible to have different priority configurations for different subclasses (i.e. Gene.primaryIdentifier and Protein.primaryIdentifier had to be the same).

This is now fixed. You can specify priorities for Gene.primaryIdentifier and Protein.primaryIdentfier separately. The only restriction is that you can't override a priority set by a superclass, i.e. you can't set LocatedSequenceFeature.primaryIdentifier and Exon.primaryIdentifier because Exon is a subclass of LocatedSequenceFeature.

GFF3 export

  • Fixed a bug where exporting as GFF3 would set features on the negative strand if no strand was set in the feature location, a '.' is now added according to the standard.

Dataloading performace

  • dataloading speed should be slightly improved, we have fixed a bug where the Model object which has a complex hashCode() method was used as a HashMap key when the model name was sufficient.

Attachments