I started a small GitHub project called Wisp after some initial failed experiments with the Felix OSGi container, IntelliJ and Java 10. Once complete Wisp will host ultra-lightweight, Netty-based Websocket services in much the same way as Tomcat, Jetty and other Web containers host servlets, and to do this it uses the Jigsaw modularity mechanisms first introduced in Java 9 to allow you to extend the container with new services. It also aims to dispel the myth that Java 9 modularity is all about static linking, while OSGi and other, earlier ClassLoader-based extension mechanisms provide dynamic loading. Jigsaw is instead an interesting hybrid of both, while still offering much stricter controls at compile- and runtime.

This article is not meant to introduce Java 9 modules or Jigsaw in detail; I recommend Alex Buckley's JavaOne talk Modular Development with JDK 9 as a starting point if you are not familiar with them, and JEP 261 and The State of the Module System for detailed treatments of the topic.

Introduction

Java 9 introduced a new concept beyond classes and packages for organizing code: the module. A module exports packages, requires other modules, uses service interfaces and provides service implementations. This metadata is contained in a module-info.java source file at the root of the module's JAR or directory. Taking the boot module from Wisp as an example:

module wisp.boot {
   requires wisp.api;

   requires com.google.common;
   requires jsr305;
   requires typesafe.config;

   uses wisp.api.ServiceModule;
}

It uses interfaces exported by the wisp-api module, including the ServiceModule service interface. ServiceModule is a simple lifecycle service interface: you can link it with other services by doing lookups with ServiceLoader -- covered below -- you can configure it, start, stop and destroy. WispBoot's job is basically to locate all ServiceModules in the modulepath and start them up. This lets us have a completely generic entrypoint which knows nothing of the types of services it is hosting: you could embed a webserver, CXF SOAP services, or anything else you like.

public interface ServiceModule extends Linkable, Configurable, Destroyable {
    @Override
    default void link(ServiceLocator locator) { }

    @Override
    default void configure(Config config) { }

    default void start() { }

    default void stop() { }

    @Override
    default void destroy() { }
}

But how can we introduce new modules at runtime and get access to their ServiceModules?

First attempt

The first attempt at dynamic service location was more closely based on the JerryMouse example (see citation below). We start by scanning the modules directory:

        Path smlibPath = Paths.get(baseDir, "modules").toAbsolutePath().normalize();
        try (DirectoryStream<Path> stream = Files.newDirectoryStream(smlibPath, 
                 file -> Files.isDirectory(file))) {
            for (Path path : stream) {
                var smName = path.normalize().getFileName().toString();
                var rootModuleName = path.getFileName().toString();

                var modSearchPath = Paths.get(path.toString());
                var finder = ModuleFinder.of(modSearchPath);
                var result = finder.find(rootModuleName);

ModuleFinder takes a set of Path objects, and we pass it each directory found under the modules directory. We will make an assumption here: the root module name and the directory name are the same. (This helps avoid requiring any special manifest or configuration.)

We then find the root module, and if it's present resolve that module: this is what triggers the runtime linking:

                if (!result.isPresent()) {
                    System.err.println("[" + smName + "] Error: Root module " + rootModuleName + " not found.");
                } else {
                    // Create Configuration based on the root module
                    var cf = ModuleLayer.boot().configuration().resolve
                            (finder, ModuleFinder.of(), Set.of(rootModuleName));

                    // Create new Jigsaw Layer with configuration and ClassLoader
                    var layer = ModuleLayer.boot().defineModulesWithOneLoader(cf, ClassLoader.getSystemClassLoader());

                    System.out.println("[" + smName + "] Created layer containing the following modules:");
                    for (var module : layer.modules()) {
                        System.out.println("         " + module.getName());
                    }
                }

The last step is where the magic happens in terms of dynamic loading:

                    for (var smod : ServiceLoader.load(layer, ServiceModule.class)) {
                        smod.start();
                    }

ServiceLoader will locate every provided ServiceModule implementation, and we can then -- with no Reflection, casting, etc. -- just use it and call start() to boot the ServiceModule. And this works beautifully so long as you don't want to have one module depend upon another; if you want total plugin isolation, this is the approach you want. Unfortunately for Wisp I wanted services to be able to use one another.

Second attempt

The second attempt at a solution first locates all the root modules and then resolves them with a single ModuleLayer composed of all the roots and paths. This might at first seem contrary to the whole modularization project: everything ends up with a common ClassLoader, and every module has to be loaded once and only once. But this is where Jigsaw truly shows its value: despite this the boundaries enforced by modularization still apply, because they go deeper than the ClassLoader relationships, unlike in OSGi. And by creating a unified module path you don't have to mess with the dependency graph of modules: they get linked at runtime by Jigsaw so long as every dependency has been satisfied by the set of modules loaded. We start by collecting the module directory metadata without resolving it:

        var moduleDirs = new ArrayList<ServiceModuleDir>();
        var smlibPath = Paths.get(baseDir, "modules").toAbsolutePath().normalize();
        try (DirectoryStream<Path> stream = Files.newDirectoryStream(smlibPath, 
                     file -> Files.isDirectory(file))) {
            for (Path path : stream) {
                moduleDirs.add(new ServiceModuleDir(path));
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

where ServiceModuleDir is just a simple value class:

    private class ServiceModuleDir {
        private final Path path;
        private final String rootModuleName;

        private ServiceModuleDir(Path path) {
            this.path = path;
            this.rootModuleName = path.normalize().getFileName().toString();
        }

        public Path getPath() {
            return path;
        }

        private String getRootModuleName() {
            return rootModuleName;
        }
    }

We then collect all the names and paths, and once again create a ModuleFinder: this time with multiple paths, one for each of the ServiceModule directories.

        var rootModuleNames = new TreeSet<String>();
        var paths = new Path[moduleDirs.size()];
        for (int i = 0; i < paths.length; i++) {
            paths[i] = moduleDirs.get(i).getPath();
            rootModuleNames.add(moduleDirs.get(i).getRootModuleName());
        }
        var finder = ModuleFinder.of(paths);

Finally, we resolve all roots simultaneously using our composite ModuleFinder, and create a ModuleLayer which contains all modules:

        var cf = ModuleLayer.boot().configuration().resolve(finder, ModuleFinder.of(), rootModuleNames);

        // Create new Jigsaw Layer with configuration and ClassLoader
        var layer = ModuleLayer.boot().defineModulesWithOneLoader(cf, ClassLoader.getSystemClassLoader());

References

This article builds on the JerryMouse example in the Java 9 modules example suite, O'Reilly's Java 9 Modularity, and the Gradle Java 9 modules guide. I also recommend the JavaOne talk Modules and Services by Alex Buckley, which introduces the modules-enhanced ServiceLoader mechanism used in Wisp.