Plugin Development¶

Force BDSS is extensible through plugins. A plugin can be (and generally is) provided as a separate python package that makes available some new classes. Force BDSS will find these classes from the plugin at startup.

A single Plugin can provide one or more of the following entities: MCO, DataSources, NotificationListeners, UIHooks. It can optionally provide DataViews to be used by the force_wfmanager GUI.

An example plugin implementation is available at:

https://github.com/force-h2020/force-bdss-plugin-enthought-example

To implement a new plugin, you must define at least four classes:

The Plugin class itself.
One of the entities you want to implement: a DataSource, NotificationListener, MCO, or UIHook.
A Factory class for the entity above: it is responsible for creating the specific entity, for example, a DataSource
A Model class which contains configuration options for the entity. For example, it can contain login and password information so that its data source knows how to connect to a service. The Model is also shown visually in the force_wfmanager UI, so some visual aspects need to be configured as well.

The plugin is made available by having it defined in the setup.py file entry_points section, under the namespace force.bdss.extensions. For example:

entry_points={
    "force.bdss.extensions": [
        "enthought_example = "
        "enthought_example.example_plugin:ExamplePlugin",
    ]
}

The plugin¶

The plugin class must be

Inheriting from force_bdss.api.BaseExtensionPlugin
Implement a id class member, that must be set to the result of calling the function plugin_id(). For example:
```
id = plugin_id("enthought", "example", 0)
```
Implement a method get_factory_classes() returning a list of all the classes (NOT the instances) of the entities you want to export.
Implement the methods get_name(), get_version() and get_description() to return appropriate values. The get_version() method in particular should return the same value as in the id (in this case zero). It is advised to extract this value in a global, module level constant.

The Factory¶

The factory must inherit from the appropriate factory for the given type. For example, to create a DataSource, the factory must inherit from BaseDataSourceFactory. It then needs these methods to be redefined

get_identifier(): must returns a unique string, e.g. a uuid or a memorable string that must be unique across your plugins, present and future.
get_name(): a memorable, user presentable name for the data source.
get_description(): a user presentable description.
get_model_class(): Must return the Model class.
get_data_source_class(): Must return the data source class.

The Model class¶

The model class must inherit from the appropriate Base model class, depending on the entity, for example BaseDataSourceModel in case of a data source.

This class then must be treated as a Traits class, where you can use traits to define the type of data it holds. Pay particular attention to those data that can modify the slots. For those, add a changes_slots=True metadata tag to the trait. This will notify the UI that the new slots need to be recomputed and presented to the user. Failing to do so will have unexpected consequences. Example:

class MyModel(BaseDataSourceModel):
    normal_option = String()
    option_changing_slots = String(changes_slots=True)

Typically, options that change slots are those options that modify the behavior of the computational engine, thus requiring more or less input (input slots) or producing more or less output (output slots).

Many BaseModel subclasses also include a verify method, which is called before an MCO run starts to ensure that the execution will be successful. This verification step can also be triggered in the WfManager UI even before an MCO run is submitted. For BaseDataSourceModel subclasses it is automatically performed whenever the slots objects are updated, however developers can also include the verify=True metadata on any additional trait that requires verification. Including this in example above:

class MyModel(BaseDataSourceModel):
    normal_option = String(verify=True)
    option_changing_slots = String(changes_slots=True)

You can also define a UI view with traitsui (import traitsui.api). This is recommended as the default view arranges the options in random order. To do so, have a default_traits_view() method:

def default_traits_view():
    return View(
        Item("normal_option"),
        Item("option_changing_slots")
    )

The DataSource class¶

This is the “business end” of the data source, and where things are done. The class must be derived from BaseDataSource), and reimplement the appropriate methods:

run(): where the actual computation takes place, given the configuration options specified in the model (which is received as an argument). It is strongly advised that the run() method is stateless.
slots(): must return a 2-tuple of tuples. Each tuple contains instances of the Slot class. Slots are the input and output entities of the data source. Given that this information depends on the configuration options, slots() accepts the model and must return the appropriate values according to the model options.

The MCO class¶

Like the data source, the MCO needs a model (derived from BaseMCOModel), a factory (derived from BaseMCOFactory) and a MCO class (derived from BaseMCO). Additional entities must be also provided:

MCOCommunicator: this class is responsible for handling communication between the MCO and the spawned process when the MCO is using a “subprocess” model, that is, the MCO invokes the force_bdss in evaluation mode to compute a single point.
parameters: We assume that different MCOs can support different parameter types for the generated variables. Currently, only the “range” type is commonly handled.

The factory then must be added to the plugin get_factory_classes() list.

The factory must define the following methods:

def get_identifier(self):
def get_name(self):
def get_description(self):
def get_model_class(self):

as in data source factory. The following:

def get_optimizer_class(self):
def get_communicator_class(self):

Must return classes of the MCO and the MCOCommunicator. Finally:

def get_parameter_factory_classes(self):

Must return a list of classes of the parameter factories.

Optimizer Engines¶

The force_bdss.api package offers the BaseOptimizerEngine and SpaceSampler abstract classes, both of which are designed as utility objects for backend developers.

The BaseOptimizerEngine class provides a schema that can easily be reimplemented to act as an interface between the BDSS and an external optimization library. Although it is not strictly required to run an MCO, it is expected that a developer would import the object into a BaseMCO.run implementation, whilst providing any relevant pre and post processing of information for the specific used case that the MCO is solving. The base class must simply define the following method:

def optimize(self):

Which is expected to act as a generator, yielding values corresponding to optimised input parameters and their corresponding KPIs. A concrete implementation of this base class, the WeightedOptimizerEngine, is provided that uses the SciPy library as a backend.

The SpaceSampler abstract class also acts as a utility class in order to sample vectors of values from a given distribution. Implementations of this class could be used to either provide trial parameter sets to feed into an optimiser as initial points, or importance weights to apply to each KPI. The base class must define the following methods:

def _get_sample_point(self):
def generate_space_sample(self, *args, **kwargs):

Two concrete implementations of this class are provided: UniformSpaceSampler, which performs a grid search and DirichletSpaceSampler, which samples random points from the Dirichlet distribution.

MCO Communicator¶

The MCO Communicator must reimplement BaseMCOCommunicator and two methods: receive_from_mco() and send_to_mco(). These two methods can use files, stdin/stdout or any other trick to send and receive data between the MCO and the BDSS running as a subprocess of the MCO to evaluate a single point.

Parameter factories¶

MCO parameter types also require a model and a factory per each type. Right now, the only typo encountered is Range, but others may be provided in the future, by MCOs that support them.

The parameter factory must inherit from BaseMCOParameterFactory and reimplement:

def get_identifier(self):
def get_name(self):
def get_description(self):

as in the case of data source. Then:

def get_model_class(self):

must return a model class for the given parameter, inheriting from BaseMCOParameter. This model contains the data the user can set, and is relevant to the given parameter. For example, in the case of a Range, it might specify the min and max value, as well as the starting value.

Notification Listeners¶

Notification listeners are used to notify the state of the MCO to external listeners, including the data that is obtained by the MCO as it performs the evaluation. Communication to databases (for writing) and CSV/HDF5 writers are notification listeners.

The notification listener requires a model (inherit from BaseNotificationListenerModel), a factory (from BaseNotificationListenerFactory) and a notification listener (from BaseNotificationListener). The factory requires, in addition to:

def get_identifier(self):
def get_name(self):
def get_description(self):
def get_model_class(self):

the method:

get_listener_class()
 return the notification listener object class.

The NotificationListener class must reimplement the following methods, that are invoked in specific lifetime events of the BDSS:

def initialize(self):
    Called once, when the BDSS is initialized. For example, to setup the
    connection to a database, or open a file.

def finalize(self):
    Called once, when the BDSS is finalized. For example, to close the
    connection to a database, or close a file.

def deliver(self, event):
    Called every time the MCO generates an event. The event will be passed
    as an argument. Depending on the argument, the listener implements
    appropriate action. The available events are in the api module.

UI Hooks¶

UI Hooks are callbacks that are triggered at some events during the lifetime of the UI. It has no model. The factory must inherit from BaseUIHooksFactory, and must reimplement get_ui_hooks_manager_class() to return a class inheriting from BaseUIHooksManager. This class has specific methods to be reimplemented to perform operations before and after some UI operations.

Any BaseDriverEvents that are required to be delivered to a UI can be indicated using the UIEventMixin class. The MCOStartEvent, MCOProgressEvent and MCOFinishEvent are all examples of such objects.

Envisage Service Offers¶

A plugin can also define one or more custom visualization classes for the GUI application force-wfmanager, typically to either display data or provide a tailor-made UI for a specific user. In which case, the plugin class must inherit from force_bdss.core_plugins.service_offer_plugin.ServiceOfferExtensionPlugin , which is a child class of BaseExtensionPlugin. Any UI subclasses can then be made discoverable by force-wfmanager using the envisage ServiceOffer protocol through the get_service_offer_factories method:

def get_service_offer_factories(self):
    """A method returning a list user-made objects to be provided by this
    plugin as envisage ServiceOffer objects. Each item in the outer list is
    a tuple containing an Interface trait to be used as the ServiceOffer
    protocol and an inner list of subclass factories to be instantiated
    from said protocol.

    Returns
    -------
    service_offer_factories: list of tuples
        List of objects to load, where each tuple takes the form
        (Interface, [HasTraits1, HasTraits2..]), defining a Traits
        Interface subclass and a list of HasTraits subclasses to be
        instantiated as an envisage ServiceOffer.
    """

Make sure to import the module containing the data view class from inside get_service_offer_factories: this ensures that running BDSS without a GUI application doesn’t import the graphical stack.

Custom UI classes¶

There are currently two types of custom UI object that may be contributed by a plugin: IBasePlot and IContributedUI. These interfaces represent requirements for any UI feature that can be used to display MCO data or a present a simplified workflow builder respectively.

Also, multiple types of plugin contributed UI objects can be imported in the same call. For instance:

def get_service_offer_factories(self):
    from force_wfmanager.ui import IBasePlot, IContributedUI
    from .example_custom_uis import PlotUI, ExperimentUI, AnalysisUI

    return [
        (IBasePlot, [PlotUI]),
        (IContributedUI, [ExperimentUI, AnalysisUI])
    ]