In the last post, we laid the technical foundation necessary to create migrations. We installed the Migrate Plus and Migrate Tools modules to support our migrations. We imported our Drupal 7 database locally, and configured settings.php with connection credentials. Finally, we created a migration group to use those connection credentials.
But we still haven't migrated any content as of yet. Let's change that.
Where to start?
Now that we're ready to start creating custom migrations, we immediately run into our first problem: where do we start? You might immediately jump to the answer, "Nodes!", but this would be like eating a sandwich by first removing the bread.
For each migration, you need to do some dependency mapping. If you image a single node as a object, you'll be able to visualize that it has data that it contains. For nodes, most of that data is in fields. Nodes also refer to other, independent objects in your site. The node's author refers to a separarate, and independent user object.
So, we have to migrate users first, right? Well, no, you have to do the dependency mapping again for users. Users have fields, but like nodes, those are contained by the user object so we don't need to think about those for now. Users are, however, a member of one or more roles. "Aha!" you might say, "So what do roles depend on?" For the purposes of our custom migration, roles do not rely on any external objects. Roles are to permissions as nodes are to fields, they contain the data, to refer to something external.
Now, if you dig into how Drupal 8 does a automated migration, the above seems drastically simplified with numerous gotchas. While that's true, it's a good enough model for custom migrations.
So now we know we need to start by migrating our roles. How do we do that? When we created our migration group, we used a web UI to create it, then used drush cex to export it. From there, we got the migration group *.yml in our sync directory. That made it much easier to edit. You might think you can just go to Admin > Structure > Migrations, select List Migrations under your migration group, and then click a button "Add migration". When you go there, however, there's nothing. Just an empty list and no button.
One thing you quickly discover in Drupal 8 is that not everything is suited for a web UI. Sometimes you need to break out a text editor, crack your knuckles, and get down to some code. Fortunately, Drupal 8 also doesn't make it too hard to figure out, provided you have the correct framing.
If you've written a migration in Drupal 7 before, you might expect at this point you'll need to dig around the examples in Migrate Plus and find a migration class. That class not only represented the migration, it also provided key points where you can customize the source query, transforming individual rows, and inserting the result into the target site. This called an ETL, or Extract, Transform, Load process. The thing is, when you look at the example migrations Migrate Plus provides, you don't really find a migration class.
In Drupal 8, migrations are written as a *.yml file. The reason is that the ETL process itself rarely changes. Instead, each part is broken out into separate objects:
- Source plugins extract data. This can be a *.csv or *.xml file, a SOAP or JSON web service, or a previous installation of Drupal.
- Process plugins transform the data. Each field can have a different process plugin, as well as more than one.
- Destination plugins load the data into the Drupal 8 site.
The Migrate Plus examples do give you some idea on how a migration *.yml is supposed to look, but not for a Drupal to Drupal migration. For that, we need to look elsewhere.
We know we want to start with a role migration. Furthermore, it's a Drupal to Drupal migration. We know that Drupal 8 provides autogenerated migrations, so it's got to be in core somewhere, right? That's true, but if you look in core/modules/migrate, or even core/modules/migrate_drupal you still won't find anything like what we're looking for. Instead, you need to ask yourself a different question.
"Who in core owns the thing I'm trying to migrate?" We're trying to migrate user roles, something owned by the User module. If we use a file browser or IDE to look at the contents of the core/modules/user directory, we'll see this:
path/to/yoursite/core/modules/user ├── config ├── css ├── migration_templates ├── src ├── templates ├── tests │ ... └── user.views.inc
The directory that sticks out is migration_templates. Inside you'll find a treasure trove of *.yml all rather clearly named. Since we're migrating a user role from a Drupal 7 (D7) site, The one that is the most interest to us is d7_user_role.yml. We don't want to edit the d7_user_role.yml file core provides, but we do want to copy the contents of it to a text editor for us to work with.
Examining the migration *.yml
Finding a migration template saved us a considerable amount of time (and hair!) in writing our migration. For roles, there's actually not much more we need to do with the *.yml before we can use it. Examining the file, we'll see that it's broken down into the following structure:
- id and label, which uniquely identify the migration.
- migration tags which can be used to organize our migrations into sets.
- source, which describes the extract portion of our ETL.
- process, which does two things: Allows us to map fields from the source to fields in the destination, and transform the data in between.
- destination, which describes where in Drupal 8 the data is to be saved.
- migration_dependencies, which allow us to specify if this migration depends on another migration.
For now, let's ignore the process section. We'll get to it later. Notice the source and destination sections, both of these specify a plugin:
source: plugin: d7_user_role destination: plugin: entity:user_role
We don't need to change either of these, since we copied just the right migration template for our needs. They do give us some insight into how the migration system works. Migrate module heavily relies on the Drupal 8 plugin system to find source, process, and destination code. If we search around, we can find the d7_user_role plugin ID refers to a class in core/modules/user/src/Plugin/migrate/source/d7/Role.php. The annotation of which looks like this:
/** * Drupal 7 role source from database. * * @MigrateSource( * id = "d7_user_role" * ) */
The destination plugin is a little more complicated. The plugin name has the form of plugin:derivative. In most cases, that will be entity:entity_type, where entity_type is an entity type ID like node, user, or user_role.
Customising the template
Now that we know our way around the template, we can customize it to our needs.
- Copy the contents of d7_user_role.yml into a new file in your text editor. Do NOT edit the file in core/modules/user!
- Change the id and label to something unique, such as yoursite_role, and YourSite Role.
- We don't really need the migration_depenendcies section, so edit it to be empty:
That takes care of the basics, but you might have noticed that there's nothing in our role migration that refers to the group. That's because migration groups aren't part of core -- from where we copied our template. Instead, we need to add the following right below the label:
Where yoursite_group_name is the name of the migration group you created earlier. This links our migration to the group, and since the group provides the database key to use in settings.php, our role migration now knows what database to use.
Importing the migration
At this point, there's several ways we can import our migration into Drupal 8. We could save the file to our sync directory, but it's essential to get the filename correct for Drupal to ensure consistency. For this reason, I prefer to let Drupal create the file for me:
- Login to your Drupal 8 site.
- Navigate to Admin > Configuration > Configuration Synchronization.
- Open the Import tab, and click the Single item subtab.
- For the Configuration type, select Migration. Be careful not to select other types!
- Paste your edited migration *.yml into the provided field.
- Click Import.
- You will be asked to confirm the import, do so.
At this point, the configuration is inside Drupal, but we still don't have a file we can add to our repo. To do that, go back to the command line and use drush cex to do a configuration export. You should see a new file being created named migrate_plus.migration.yoursite_role.yml, where yoursite_role is the migration id you used earlier.
You only need to visit the webUI the first time you import the *.yml for a migration. Once you have a file in your sync directory for it, you can edit it, then use drush cim to import your changes.
Verifying and running the role migration
With the configuration imported, we need to check we did everything properly. Since we enabled Migrate UI, we would do this by going to Admin > Structure > Migrations, but now's a good time to get used to the command line. To list our migrations, use drush migrate-status, or ms:
$ drush ms Group: YourSite group (yoursite) Status Total Imported Unprocessed Last imported yoursite_role Idle 4 0 4 N/A
This tells us a number of important things. First of all, our migration was registered correctly with our group as it's listed under the group. Notice the Total and Unprocessed columns lines are populated. This means that the Drupal 7 database credientials and the legacy key were configured correctly. If not, we would only see N/A for both Total and Unprocessed.
To run our migration, we can use drush migrate-import or mi, specifying the name of your migration:
$ drush mi yoursite_role
This will import all the items in the yoursite_role migration. Since we only have a handful of items, the migration will not run for very long even if there are problems. If you only want to migrate an limited number of items, we can use the --limit switch:
$ drush mi yoursite_role --limit=n
Where n is the number of items you want to import. You can also import all the items in your group, but this can be a bit risky. It's far safer at this point to run each migration individually.
If we've done everything right, we can now visit Admin > People > Roles and see a list of our imported roles. Note that this includes the system provided Anonymous, Authenticated, and Administrator roles. Now we can move on to users.
Export your configuration again
Some migrations -- like roles -- import configuration and not content. This can make things difficult if you need to update a migration and run drush cim again. When that happens, you'll potentially face a conflict. There are changes in the sync directory, but the active configuration known to Drupal has changes elsewhere.
To avoid any configuration conflicts, it's best to run drush cex again before editing any files in sync. Once that's done, commit it to git, and move on to your editing. This will avoid any of your changes from being overwritten and lost.
Creating and running the migration for users is going to be very similar to creating one for roles:
- Find the template for Drupal 7 users under core/modules/user/migration_templates.
- Copy the contents of the template into your editor, updating the id and label fields.
- Add the migration_group item as you did before.
At this point we could import it into Drupal, but we should specify the migration_dependencies first. We want our user migration to be dependent on our role migration so that the roles migration is always run first. To do that, we add a required item:
migration_dependencies: required: - yoursite_role
With that done, we can do the rest:
- Navigate to Admin > Configuration > Configuration Synchronization, and import your user migration.
- Use drush cex to export the migration *.yml to the sync directory.
- Verify that the migration is registered using drush ms.
- Import users using drush mi.
Whew! We've done a lot in this post. We've mapped out our migration dependencies. We've created our first migrations for roles and users and imported them into our Drupal 8 site. It's a small start, but we've also picked up some valuable knowledge and skills along the way. We now understand the workflow for creating, updating, and running our migrations. Next time, we finally get to nodes.
Thanks to our sponsors!
This post was created with the support of my wonderful supporters on Patreon:
- Alina Mackenzie
- Chris Weber
If you like this post, consider becoming a supporter at patreon.com/socketwench.