DTEK

Web Strategy for Progressive Causes and Big Ideas

Drupal 6 to Drupal 7 via Migrate 2

Benj's picture
Fri, 04/27/2012 - 12:02pm -- Benj

Upgrading a complex Drupal site can be a tricky proposition. This post will explore using the Migrate module to bring users and their role assignments from a D6 site to a fresh new D7 site. We created similar migration classes for taxonomy terms, nodes, and comments, which will not be covered (see the 'Resources' section below, or leave a comment if you need some direction). This is a followup to my last post about upgrading from Drupal 6 to Drupal 7, and presents a potential alternative approach. Before the migration, we built out the new Drupal 7 site structure - meaning content types and fields, views, contexts, and features. Theme development is happening in parallel with my data migration work below.

Why Migrate? What is a migration?

From the Migrate module page:

"The migrate module provides a flexible framework for migrating content into Drupal from other sources (e.g., when converting a web site from another CMS to Drupal). Out-of-the-box, support for creating core Drupal objects such as nodes, users, and comments is included - it can easily be extended for migrating other kinds of content."

And, from the main documentation page:

"It is important to recognize that the Migrate module is a developer's tool - if you do not have object-oriented programming skills in PHP, this module is not for you. The primary purpose is to provide the infrastructure for managing complex, large-scale site migrations."

So, why use it for a Drupal upgrade? In certain situations it may make sense to start with a cleaner and more stable Drupal 7 site, and avoid any contrib module "lock-in" or upgrade path bugs. Migrate allows us to re-name content types or fields, and massage any content on its way into the new D7 site. Another compelling feature is the ability to roll back and re-run migrations; this feels much more efficient than starting over with the D6 site and running a full upgrade each time we need to change or try something. And somewhat related, you can re-run a migration without rolling back first and Migrate will only import the new content - awesome! Though I've barely scratched the surface with this exercise, Migrate has definitely proven to be powerful!

Setting up a new Migration

Since the Migrate V2 handbook pages are excellent, I'm just going to walk through the actual steps of creating a migration. This example will import Drupal users, keeping any role assignments (and even their user id's). This is a real-world example, which I then modified slightly to import Blog content, Page content, and Taxonomy Terms. In each case, the destination "container" must already exist. So for this example, I have already created the roles in the new database (for terms, the vocabularies must exist). It's also important to run the migrations in the correct order - I don't want to import Blog posts before I have any users or terms to associate with them.

First, we need to create a custom migration module, to tell Drupal about our migration classes (the included migrate_example sub-module is a handy reference). Following the handbook page linked above, we'll start with the .info file and a skeletal .module file:

our_migration.info:

name = "Example migration"
description = "The DTEK drupal 6 to 7 migration."
package = "DTEK"
core = 7.x
dependencies[] = taxonomy
dependencies[] = image
dependencies[] = comment
dependencies[] = migrate
dependencies[] = list
dependencies[] = number
 
files[] = our_migration.module
files[] = our_users.inc

our_migration.module skeleton:

/*
 * You must implement hook_migrate_api(), setting the API level to 2, for
 * your migration classes to be recognized by the Migrate module.
 */
function our_migration_migrate_api() {
  $api = array(
    'api' => 2,
  );
  return $api;
}

Next, we're going to define our d6 database at the top of the .module file:

// define the source database
define("SOURCE_DATABASE", 'our_d6_database');

For this to work, you need to alter your $databases array in settings.php:

$databases['default']['default'] = array(
  'driver' => 'mysql',
  'database' => 'default_d7_db',
  'username' => 'root', // this is just an example, don't do this!
  'password' => 'root', // or especially this!!
  'host' => 'localhost',
  'prefix' => '',
);
// the name 'legacy' is not used anywhere
$databases['legacy']['default'] = array(
  'driver' => 'mysql',
  'database' => 'our_d6_database', // this is the name used to define the SOURCE_DATABASE constant above
  'username' => 'root',
  'password' => 'root',
  'host' => 'localhost',
  'prefix' => '',
);

Now head over to the modules page and enable the new migration module (it won't do anything yet).

Migration Classes

As stated on the Migration classes handbook page, we need to create the following objects, either by extending the classes provided by the Migrate module, or using them as-is:

  1. Source - define where the data is coming from
  2. Destination - define the Drupal 7 destinations for the data
  3. Mappings - link the Source and Destination so Migrate knows where to put the data

We'll start by creating our_users.inc, to extend the generic Migrate class, override its constructor, and set up its group and description:

// User Migration class
class ourUserMigration extends Migration {
  // class constructor
  public function __construct() {
    parent::__construct(MigrateGroup::getInstance('examplegroup'));
    $this->description = t('Migrate Drupal 6 users');
    // ... 

The parameter to the parent constructor is optional, and will define a migration group for this migration to belong to. From the handbook, "This may be useful if you have many migrations which fall into logical groups - say, a group for importing from a MySQL database, and another group operating off XML feeds."

Source

Continuing on, we define the Source class, using an SQL query and the conveniently included MigrateSourceSQL class (dig around in migrate/plugins/sources/ for other included sources):

    // ... 
    // some field definitions; roles will be added later in prepareRow()
    $source_fields = array(
      'uid' => t('User ID'),
      'roles' => t('The set of roles assigned to a user.'),
    );
 
    // only import active users from the old db
    $query = db_select(SOURCE_DATABASE .'.users', 'u')
      ->fields('u', array('uid', 'name', 'pass', 'mail', 'created', 'access', 'login', 'status', 'init'))
      ->condition('u.status', 1, '=')
      ->condition('u.uid', 1, '>')
      ->orderBy('u.uid', 'ASC');
    $this->source = new MigrateSourceSQL($query, $source_fields);
    // ...

Destination

The Migrate module includes a number of pre-defined destination classes (in migrate/plugins/destinations/) - and Migrate Extras adds even more. For this example, Migrate Extras is not required.

    // ...
    $this->destination = new MigrateDestinationUser(array('md5_passwords' => TRUE));
    // ...

Phew, that was easy! The md5_passwords parameter, "Indicates whether incoming passwords are md5-encrypted - if so, we will rehash them similarly to the D6->D7 upgrade path." Awesome, no need to force every user to change their password!

Mappings

I doubt I can improve on the handbook description:

To track migration status, and allow rollback, the migrate module needs to remember what source record resulted in the creation of which destination object. It does this using a MigrateSQLMap class - this class creates database tables which map source and destination keys. To do this, it needs to know what data types the respective keys are. Thus, we pass in schema arrays declaring the respective source and destination keys. For the source key, since the migrate module knows nothing about the source table, you need to explicitly define the schema array (making sure that the key of the array, uid in this case, matches the field name used in the query passed to MigrateSourceSQL above). On the other hand, since we're using the canned class MigrateDestinationUser, it knows what the correct schema is already and can provide it through the static method getKeySchema().

The 'machineName' is used to create the migrate_map_ and migrate_message_ tables; passing in our migration class machine name should ensure that the table names are unique and easy to identify.

    // ...
    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'uid' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'D6 Unique User ID',
          'alias' => 'u',
        )
      ),
      MigrateDestinationUser::getKeySchema()
    );
    // ...

Next, we get to the actual data mappings (note that $destination is the first parameter, and $source is the second, though many of the field names are identical):

    // Create the field mappings
    $this->addFieldMapping('is_new')->defaultValue(TRUE);
    $this->addFieldMapping('uid', 'uid');
    $this->addFieldMapping('name', 'name')->dedupe('users', 'name');
    $this->addFieldMapping('pass', 'pass');
    $this->addFieldMapping('mail', 'mail')->dedupe('users', 'mail');
    $this->addFieldMapping('language')->defaultValue('');
    $this->addFieldMapping('theme')->defaultValue('');
    $this->addFieldMapping('signature')->defaultValue('');
    $this->addFieldMapping('signature_format')->defaultValue('filtered_html');
    $this->addFieldMapping('created', 'created');
    $this->addFieldMapping('access', 'access');
    $this->addFieldMapping('login', 'login');
    $this->addFieldMapping('status', 'status');
    $this->addFieldMapping('picture')->defaultValue(0);
    $this->addFieldMapping('init', 'init');
    $this->addFieldMapping('timezone')->defaultValue(NULL);
    $this->addFieldMapping('path')->issueGroup(t('DNM')); // DNM = Do Not Map
    $this->addFieldMapping('roles', 'roles');
  }
  // ...

This ends our constructor function, but we still need to do a little work for the user roles. Enter prepareRow(). More details can be found on the Commonly implemented Migration methods handbook page, but the gist is that we can use prepareRow to modify data before it gets saved, or to conditionally skip importing a record (by returning FALSE). Keep in mind that records excluded in this manner will not be reflected on the Migrate admin page.

  // ...
  // massage the user roles before saving
  public function prepareRow($current_row) {
    $source_id = $current_row->uid;
    $query = db_select(SOURCE_DATABASE .'.users_roles', 'r')
      ->fields('r', array('uid', 'rid'))
      ->condition('r.uid', $source_id, '=');
    $results = $query->execute();
    // add the Authenticated role for everyone
    $roles = array('2' => '2');
    foreach ($results as $row) {
      // adjust the old to new role id's
      // Editors (rid 3) need role id 4
      if ($row->rid == 3) {
        $roles['4'] = '4';
      }
      // Admins (rid 4) need role id 3
      if ($row->rid == 4) {
        $roles['3'] = '3';
      }
      // Collaboraters are role id 5 in both old and new dbs
      $roles[$row->rid] = $row->rid;
    }
    $current_row->roles = $roles;
    return TRUE;
    // return FALSE if you wish to skip a particular row
  }
} // close the ourUserMigration class

Running the Migration

There are two ways to interact with migrations: the Migrate UI or drush. In either case, you can do the following:

  • Import - Perform one or more migration processes.
  • Rollback - Roll back the destination objects from a given migration
  • Rollback and Import - UI only, combines the two actions above
  • Stop - Stop an active migration operation
  • Reset - Reset a active migration's status to idle
  • Wipe - drush only, Delete all nodes from specified content types

There are also drush commands and UI pages to view source fields, destination fields, and field mappings. This is helpful for debugging and visualization, but you cannot assign any mappings through the UI or drush; they need to be in the php class.

Migrate Resources

The Migrate handbook pages are great, but sometimes nothing replaces some working example code. Luckily, Drupal 6 to Drupal 7 migrations are a pretty common use case, and I found the resources below to be super helpful!

Just a Start

As mentioned above, this is a real world example from a Drupal 6 to 7 migration which included Users, Terms, Pages, Blogs, and a custom Project content type. To minimize unforeseen issues and mapping work, we decided to keep all user, term, and node id's the same (this may not be practical or desirable in all cases). This "upgrade" process using the Migrate module definitely feels more solid to me than a standard, "in-place" Drupal upgrade (for reasonably complex sites). And as a nice side-affect, you can import new content at the end of the dev cycle by simply running the migrations again, before the final launch.

Hopefully this helps you get started with more complex Drupal migrations using the Migrate module. Please leave a comment if anything needs clarification!

Comments

Submitted by Mike Purvis (not verified) on

This post really helped demystify how to use Migrate. Just a quick note, in the sample module you provided under Migrate Resources, the .module file has a hook name
tms_migration_migrate_api

which should be our_migration_migrate_api

thanks again

Submitted by Some Guy (not verified) on

In .info, files[] only needs to include the names of files that have classes in them. It's used for autoloading classes.

Benj's picture
Submitted by Benj on

Thanks for the feedback, guys. I appreciate you taking the time to clarify things! {drupal community}++

Submitted by Mika Andrianarijaona (not verified) on

Thanks it's really interesting. Unfortunatly I have developed a custom module to migrate a d6 to d7 website because I didn't know about this module. Now I know :D
Thanks!!

Submitted by Avi (not verified) on

Thanks a lot for this great article Benj. I'm stuck with this one weird issue. The table prefix of my default database is being added to my SOURCE_DATABASE. Any help will be appreciated !. Thanks :)

@nedatsea @zoekhollister still *is* for another week, dammit :) 4 days 13 hours ago
And welcome to f1seattle, @amandaestone ! /cc @ForumOne 4 days 18 hours ago
So @zoekhollister is pretty much the best @ForumOne colleague ever. Fruit crisp. Happy Friday http://t.co/8vTjYyBeB8 4 days 18 hours ago
RT @drupalsummit: Deadline for PNWDS 2014 session proposals is quickly approaching. Submit your proposal soon! http://t.co/69XC3cLqIt #drup… 6 days 4 hours ago
Just signed a few petitions like this one in support of the "Stop Militarizing Law Enforcement Act": http://t.co/PV1UNljGhc #Ferguson 6 days 4 hours ago

You are all consummate professionals and very clear and on the ball. I value your expertise, thoroughness, and timeliness!

- Brooke W.