Drupal 8: The Valley of Dearth

Friday was an interesting day. I was -- and still am -- suffering a particularly bad head cold. I called in sick hoping to have a low key day where I could poke at some Drupal code and attempt a nap. By the afternoon, I was in the middle of a argument about Drupal 8.

Background

I've been working to port Flag module to Drupal 8 for almost a year now. I've leaned heavily on API documentation, my wits, and conversations on IRC to make as much progress as I have. Often I need to pattern my code on that I find elsewhere in core. I've been able to avoid being discouraged (for the most part) because this isn't a job for me. I have no deadlines, no income set against it, no managers asking when it will be finished. Instead, this is something I work on in my free time. It helps to think of it as an interesting puzzle that would be useful to others when complete. 

While I've gotten pretty far with Flag module, but the code is undocumented and in no way set up to be demonstrative. Simply pointing other contrib developers at Flag wouldn't be ask helpful as it could be. There are pieces that aren't neatly isolated from each other (such as config entities and plugins) in Flag that most certainly could be in Drupal 8. This can lead to a lot of false positives and frustrations if you try to learn code by reading. I've tried in my own little ways to provide the D8 knowledge I've gathered in ways that could be useful to others, but they are all "off the beaten track" as it were. 

Friday afternoon one of the other maintainers of Flag module discovered just how much progress I had made on the 8.x version. He simply thought I was off "doing other things" given my absence in the Flag 7.x issue queues. A perfectly understandable line of thinking in a gratis, open source project. (It also put in stark relief how much hell I give myself on these things.) While I regularly posted on Twitter about Flag 8.x, not everyone checks that as regularly as I. His opinion is that no one should start D8 work until there are enough resources and stability for module maintainers to make the effort worth it. 

It didn't come across as discouraging for me. It seemed a sensible opinion. The thing is that there are several factors that made me pursue my current course:

• Drupal 8 simply makes more sense to me given my background. The PHP 4-ness of Drupal 7 throws me each time I look at it.
• I didn't understand Flag module all that well until I started porting it. This made it frustrating to work on anything more than cosmetic code issues.
• I really, really want to contribute to Drupal 8, but I can't keep up the momentum to contribute directly.
• I have an innate need to help an educate people about technology. 
• Efforts to encourage module developers have really stalled with Drupal 8.

While many people who have helped me out on IRC jumped to my defense, I found myself in the position to see both sides of the problem. Drupal 8 is terrifying for module developers. It's very, very very different. We're not yet doing enough as a community to make it less scary. We do have excellent API documentation, but that's only a piece of the puzzle. 

API Docs vs. Code Samples vs. Educational Prose

API Documentation is an essential tool for developers. It provides the what's-this-part-and-what-does-it-do of source code. The problem is that API documentation is basically a software project's dictionary. It defines the words, but doesn't tell you how to use them beyond their meaning. With a dictionary alone an no other tools, a budding writer may have an impressive vocabulary, only to make basic grammar errors every other sentence. API docs are the easiest for Drupal core developers to maintain. After all, they need to use API docs too! The problem for contrib developers is that they are looking at the dictionary alone, without all the other "inside" knowledge about how to piece the words together.

Code samples are like stories. One of the best pieces of advice you can give a writer is: "Read a lot." This includes huge tomes to the labels on the back of a tube of lip balm. Reading stories provides a writer with the source code of other writer's ideas. Like canonical code samples, it is a completed product that you need to pick apart over the course of hours or days. It takes effort and some expert knowledge to analyze and understand. While a talented writer can write wonderful stories after reading extensively, most of us need more help. Even heavily commented code samples do little to help attract inexperienced developers that need a guiding hand.

Educational prose is that guiding hand. For writers, this is often the first creative writing course they attend in high school or college. The goal is to provide new writers with the expert knowledge they need to build better, and more engaging stories. This is more than source code, this is how to code. Writing code for an existing software project adds a further layer of complexity given preexisting structures, concepts, and standards in that project. Think of it as a writer not just needing to write a story, but a story for an existing fictional universe such as Star Trek. Educational prose helps the uninitiated, the inexperienced, and the time-constrained to get started sooner and with less effort on their part. Drupal 8 has little to no educational prose available. This is expected given the project is so new, but it does create a problem.

The Contrib Catch 22

Drupal 8, as I've written above, is very different from Drupal 7. Writing modules is going to likewise be very different. Flag 8.x has a lot of the same functional pieces of code, but the structure is vastly different. It's as if you had taken the occupants and possessions of one apartment building, and put them in a new one. Sure, it's the same people and the same stuff, but it's all in different places. 

In striving to make Drupal 8 a modern base for a great content manager, we have leveled many module developers back to zero. Go back to the beginning of the dungeon young adventurer and start again. Core developers are asking -- even begging -- for people to port their modules. Contrib developers are waiting until "there is enough documentation". Core devs will shout, "There is!" pointing at the API docs. Contrib devs say, "That's not what I mean," and on and on.

And therein lies the catch 22.

The problem is contrib devs have an excellent dictionary, but they have little reading material that isn't core. Core code can help, but it's muddied by actually being practical code rather than demonstrative. Furthermore, any educational prose is the sparse and often outdated blog entry or video that causes a lot of unnecessary handwringing. There's a lot of basic, getting started stuff, and lots of advanced, well documented stuff but there's a Valley of Dearth (of knowledge) in the middle.

"But Examples Module..."

We are supposed to have canonical code samples in the form of Examples module. As a contrib developer, it makes perfect sense to me that Examples should be in core so that it can keep apace with all the quick changes and encourage updated via failing tests. We essentially tie Example module with core at the human process level. As a core developer, it doesn't make any sense to include Examples in core as putting examples in core negates the essential barrier between core code and contrib module code. The result would be that examples module no longer walks like a module or quacks like a module, it walks and quacks like core.

And we're back at the catch 22.

A few frustrated yet intrepid souls are trying to write canonical code examples in between maintaining their own modules and projects. Until Friday, I had assumed that this was going well until the argument occurred. Only then did I discover that some patches were waiting weeks or months to be reviewed. In that time, changes to core have already made those patches obsolete. Demonstrative code is also deceptively easy to write, and thus, very easy to do badly. Combined with core's swift pace makes keeping example code updated a very difficult task indeed. 

Breaking the Catch

How do we fix this? First of all we need to admit to ourselves the problem is people and not technology. If technology would have solved this problem, I'm sure the Drupal community would have already. To me, there's a role here that needs to be filled. What we need is a "Contrib Initiative Lead" who can bridge the gap between the rapid changes in Drupal core, while providing vital, well designed, and timely code samples and educational prose. Wishful thinking that "the community will step up and fix it" will only create more fear as Drupal 8's release date approaches. We need someone that can devote a considerable portion of their day job to this activity. In addition to producing material, the Contrib Lead would act as a liaison to contrib developers with popular modules. This would not only involve helping to plan, design, and execute the porting process, but to assist with human issues such as time management and project funding.

Why is this important? Drupal changes a lot between versions; why is this different? We've managed to get through it by the skin of our teeth before, as well as the tireless work of core maintainers like Angie Byron (webchick). Drupal 8 is bigger, differenter, and more intimidating. We need to do more than ask existing core maintainers to do a little more. We need to do more than hoping contrib developers will step up.

We need someone -- recalling Disney film Tron  -- to fight for the contrib devs. Not to prevent change, but so that we might all finish the battle together.