Migrating from a Prior Version

For version 2.1, upgrading is possible from v2.0 or v2.0.1 only. If you are using v2.0 and have not upgraded to v2.0.1, you will not need to upgrade to v2.0.1 before moving to v2.1. If you are using v1.7 or v1.8 of LucidWorks, and would like to move to v2.1, you will first need to move to v2.0. That can only be done in consultation with Lucid Imagination Support. In version 2.0 of LucidWorks, we used a process for migrating from v1.7 or v1.8 to v2.0 that used the LucidWorks REST API to copy configuration information from the older version of LucidWorks and import it into the new version of LucidWorks. That process has been completely replaced in v2.1. The current process upgrades the older version in place. A parallel installation of LucidWorks is made to get the new code, then a script is run to update the existing configuration files. Before you begin, review the list of Changes from LucidWorks v2.0 to v2.1 and the Release Notes for v2.1.

This functionality is available in LucidWorks Enterprise but not LucidWorks Cloud.

Upgrading LucidWorks requires the following steps:

Step 1: Stop Existing Installation and Create a Backup

LucidWorks can be stopped with the scripts available in $LWE_HOME/app/bin. Use either stop.sh or stop.bat depending on your operating system.

A backup of your existing installation can be made by copying the entire $LWE_HOME directory to a parallel location. If your installation has been made in a directory called /usr/local/LucidWorksEnterprise, then in Unix, for example, the command would be: cp -r /usr/local/LucidWorksEnterprise /tmp/lwe-old-backup. You can name the backup whatever you'd like, as long as you remember it later if you need it.

Step 2: Install the New Version of LucidWorks

Follow the steps outlined in the sections on installing LucidWorks to install the new version of LucidWorks. This can be done in any location on your server, but you should change the default path if your existing LucidWorks is already installed there. You can choose the same ports as are used in your existing installation: this installation will only be used to get an updated app directory, the MetadataUpgradeTool, and the IndexUpgradeTool (See Step 3 and Step 4 below).

When the installer asks if you would like to start the new version of LucidWorks, say no (uncheck the box).

The instructions from here on will refer to the top level of this installation as $LWE_NEW.

Step 3: Run the MetadataUpgradeTool

The MetadataUpgradeTool is found in the $LWE_NEW/app/migration_tools directory. It will update v2.0/v2.0.1 configuration files within LucidWorks to the proper format for v2.1 and will also move the user database from the app dir to the data dir if it still exists in the old location.

To run it, issue this command:

$ java -jar $LWE_NEW/app/migration_tools/MetadataUpgradeTool.jar -verbose -lwe_conf_dir <existing conf dir> 
-lwe_app_dir <existing app dir> -lwe_data_dir <existing data dir>

The parameters -lwe_conf_dir, -lwe_app_dir, and -lwe_data_dir are required and should point to the installation in v2.0 or v2.0.1 of the $LWE_HOME/conf, $LWE_HOME/app and $LWE_HOME/data directories, respectively.

The parameter -verbose is optional, but is highly recommended. It allows the full output of the tool to show on the screen in case there are problems or failures. If support is needed after using the MetadataUpgradeTool, Lucid Imagination will request this output. The output will look something like:

### Upgrading format of collections.yml ...
Writing upgraded collections.yml
### Upgrading similarity configuration in Solr schema ...
Upgrading similarity configuration in /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/schema.xml
Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/schema.xml
Upgrading similarity configuration in /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/schema.xml
Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/schema.xml
### Upgrading field mapping configuration in solrconfig ...
Processing /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/solrconfig.xml ...
Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/collection1_0/conf/solrconfig.xml
Processing /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/solrconfig.xml ...
Writing upgraded /Applications/LucidImagination/lwe2.0.1-upgrade/conf/solr/cores/LucidWorksLogs/conf/solrconfig.xml
### Removing old autocomplete data files...
### Upgrading UI DB location...
Moved /Applications/LucidImagination/lwe2.0.1-upgrade/app/webapps/lwe-ui/WEB-INF/db/production.sqlite3 
to /Applications/LucidImagination/lwe2.0.1-upgrade/data/lwe-ui
### Upgrading UI DB...
Executing UI DB upgrade
==  AddExternalProtocolToSettings: migrating ==================================
-- change_table(:settings)
   -> 0.0060s
==  AddExternalProtocolToSettings: migrated (0.0070s) =========================

If you have created any collection templates they will not be upgraded with the MetadataUpgradeTool. It's recommended that all templates created with previous versions be recreated from scratch using a new or upgraded collection.

One of the changes from v2.0x to v2.1 was to change the names of the data source types "S3" and "S3N". The "S3N" type, for Amazon over S3, is now known as "S3". The former "S3" type, for a Hadoop Filesystem on S3, is now known as "S3H". During the upgrade process, these types will be converted automatically. However, there is a new requirement for these types of data sources that the value for the URL must include a trailing slash if the URL points to a directory of files and not to a single resource. This is not done automatically by the upgrade. In order to successfully crawl an existing "S3" or "S3H" data source after upgrading, the trailing slash must be added manually if the path specified is not to a single file or resource.

Step 4: Upgrade the Indexes or Delete Them

The MetadataUpgradeTool only upgrades configuration and other system files. However, the indexes also need to be upgraded as a separate step. If, however, the indexed content is not needed, the indexes can be simply deleted instead of upgraded.

Upgrading the Indexes

In order to upgrade an index from an earlier version of LucidWorks Enterprise to v2.1, the Index Upgrade Tool should be run. This tool is found under $LWE_HOME/app/migration-tools/ in a .jar file called IndexUpgradeTool.jar.

It is recommended to run this Upgrade Tool only after consultation with Lucid Imagination Support to be sure you understand the full ramifications of running this tool on your local, customized, index.

While we have provided this index upgrade tool to help you avoid re-indexing your data, it is recommended to do so with every migration to a new version.

Before running the tool, you should make sure LucidWorks Enterprise is not running to ensure there are no indexing processes taking place while performing the upgrade.

To run the tool, open a command line interface, navigate to the $LWE_NEW/app/migration_tools directory and issue the command:

$ java -jar IndexUpgradeTool.jar [options] <sourcedir> <destinationdir>

The <sourcedir> parameter is the existing index that will be upgraded and moved to the <destinationdir>. Unlike the MetadataUpgradeTool, which updates configuration files in place, the IndexUpgradeTool makes a copy of the index while upgrading it. The <sourcedir> is $LWE_HOME/data/solr/cores/collection/data/index (where collection is the name of the collection to be upgraded. The destinationdir should be a temporary location (preferably a directory, such as /tmp/index-upgrade, as the output is a number of raw index files).

The IndexUpgradeTool can upgrade the index for one collection at a time. If there are multiple collection in the origin LucidWorks Enterprise, these would each need to be converted separately. By default, there are at least two indexes: the one that contains your data (called collection1 at initial installation) and one that indexes log data called LucidWorksLogs. Don't forget to upgrade the LucidWorksLogs collection or delete the index - skipping that collection will cause LucidWorks to fail.

There is currently one option that can be used with the tool, which is -checkindex. This will run Lucene's checkindex tool to validate that the index is correct. This is a good idea to do, especially for systems already in production, but will add time to the upgrade process.

The output of the tool is a number of index files in the location you specified with the <destinationdir> parameter. Once the tool has finished, delete the existing index files for each collection (found under $LWE_HOME/data/solr/cores/collection/data/index) and copy the new index files for each collection to the index dir.

Deleting the Indexes

If the index files are not particularly needed, which may be the case for the LucidWorksLogs collection particularly, they can be deleted. To do so, delete all of the subdirectories found under $LWE_HOME/data/solr/cores/collection/ for each collection.

Step 5: Update the $LWE_HOME/app dir

Neither the MetadataUpgradeTool nor the IndexUpgradeTool update the program files that make up the LucidWorks application. These need to be updated manually by replacing the app directory in the original v2.0 or v2.0.1 installation with the app directory from the new v2.1 installation. To do this, delete the original $LWE_HOME/app and copy the $LWE_NEW/app directory to $LWE_HOME. Assuming default locations of the installations, the sequence would be:

$ rm -r /LucidImagination/LucidWorksEnterprise/app
$ cp -r /LucidImagination/LucidWorksEnterprise2.1/app /LucidImagination/LucidWorksEnterprise

Step 6: Update master.conf

The upgrade process does not add two parameters to the master.conf file found in $LWE_HOME/conf/. These changes are recomended for optimal performance in typical situations. They are not added automatically by the MetadataUpgradeTool because these or similar changes may have already been made in your implementation depending on your specific situation. Both parameters are entered in the section of the file for JVM Settings for LWE-UI:

• Add -XX:MaxPermSize=256M (this may already be there if you started with v2.0.1)
• Add -Dcom.sun.management.jmxremote

Step 7: Start LucidWorks

Start LucidWorks using the scripts available in $LWE_HOME/app/bin. Use either start.sh or start.bat depending on your operating system.

After the upgrade, the indexes that drive autocomplete and spell check will have been removed. As described in the section on Spell Check, the spell check indexes will be automatically regenerated by default. The autocomplete indexes are not automatically generated unless there is already an autocomplete activity scheduled (and it will happen on its schedule). If using autocomplete in your search application, and it is not scheduled to run periodically, use the Activity API or the Admin UI to manually kick it off or schedule it.

After checking to be sure the upgrade was successful, the $LWE_NEW installation and/or the backup can be deleted. In order to be sure the upgrade was successful, try to index some content and perform some queries, while checking for errors in the "core" log found in $LWE_HOME/data/logs.

Related Topics

• Changes from LucidWorks v2.0 to v2.1
• Release Notes for v2.1

  Installation

  Installation and Upgrade Guide

  Working With LucidWorks Platform Components