Introduction

In addition to the built-in nodes, you can implement your own nodes as plugins to process items (images, point clouds, inference results, etc.).

What can you do with plugins?

You can implement any logic in form of a plugin that can be represented using Python. Here are some use cases:

  • Load images and other data from individual sources (e.g., via specific SDKs)

  • Add custom processing steps to modify images, point clouds, or inference results

  • Implement additional algorithms (e.g., for image processing)

  • Post-process the inference results (e.g., applying coordinate transformations)

  • Filter items before forwarding them to an output (e.g., forward only items containing a label “not ok“)

  • Initiate actions (e.g., initiate robot picking)

  • Implement custom logic

  • Forward inference results via an output plugin to a database

There are two types of plugins: the class SimplePlugin is intended for basic processing steps that take one input item and expect one output. The Plugin class can be used to implement arbitrary nodes.

Plugins are written in Python and must inherit from one of the above classes (SimplePlugin or Plugin). Plugins (in the form of .py files) must be placed in the directory plugins relative to the location where the Inference DS Core is executed. That is the working directory from which the command inds (or python -m inds.app) is called. For example, you have the plugin custom_output_plugin.py that you would like to use as part of Inference DS. Place this plugin into the directory plugins and start inds from the same directory where plugins are located.

Each plugin is imported during the start of the Core. Once it is imported, it appears as a plugin type in the user interface in “config” → “plugins” → “add plugin”. In case the plugin could not be imported, for instance, due to a missing dependency, you will see a warning in the console output of the Core. In this case, the plugin won’t appear in the user interface.

Plugins can contain configuration entries that also appear in the user interface. This is described in the section “Configuration“ below.

Furthermore, it is possible to display plugin outputs in the user interface. To use this feature, you must specify which data of a producer shall appear. This is done via so-called properties. More details can be found in the section “Properties“.

It is also possible to add so-called actions to plugins that are displayed as buttons in the tab “control“ of the user interface. Go to the section “Actions“ to learn more about that.

Concepts and Lifecycle

Plugin instances follow a simple lifecycle:

  • Init

  • Start

  • Stop

  • Destroy

The initialization of a plugin instance happens during the start of the Inference DS Core and when a plugin is created using the user interface (“config“ → “plugins“ → “add plugin“). The method __init__ is called in these cases. Hence, this method is only called during the creation of the plugin. Producers and consumers must be registered here (the SimplePlugin takes care of this) in order to be available for routing.

After the plugin is initialized, it can be connected to other nodes of the system (in case it has any producer or consumer) and you can start it in the user interface (“control“ → start button of the node). Each plugin and each node is started in its own thread allowing their parallel execution. The method start is called in these threads that set the attribute running to True and calls the methods run. The method run contains the custom plugin logic. Nodes and plugins communicate via producers and consumers as explained in the Introduction of Inference DS.

Although each plugin (node) is started in its own thread, the global interpreter lock (GIL) guarantees that only one Python thread accesses one Python object at the same time. This ensures thread-safety and prevents race conditions at the cost of no full parallelism on multi-core systems. Libraries such as Numpy and OpenCV implement their own data management and are execute alongside the Python bytecode so that you can still achieve parallel execution using Python threads. We recommend making use of such libraries in your own plugins. If this is not possible, it can be beneficial to use the multiprocessing functionalities of Python which introduce efforts for data (de-)serialization but offer full parallel execution.

On startup of the Inference DS Core, all plugins and nodes are started automatically. This is done in two stages. At first, all nodes are initialized based on their configuration. Afterward, the threads are created and started.

You can stop nodes via the user interface. In this case, the method stop is called. This method is implemented and simply sets the attribute running to False. In case some other behavior is intended, you need to overwrite this method.

Stopped nodes can be started again. This will create a new thread for the node and call the method start in this thread again.

Nodes are destroyed once the Core is shut down or when you remove nodes via the user interface (“config“ → “remove”). In this case, the node is stopped and its connections are removed before it is handed over to the garbage collector.

Plugin

The class Plugin is the base class for implementing a custom plugin. You need to implement the following methods:

  • build Parse the configuration and return an instance of the plugin.

  • __init__ Constructor of the plugin that must call super().__init__(name=name). Consumers and producers shall be created and registered (self.add_producer, self.add_consumer) here.

  • run Shall contain the main processing loop. This method must return when the plugin is stopped.

  • get_config_key Return the configuration key for the plugin class that must be unique. This key is used in the configuration file.

  • get_config_entries Return configuration entries that appear in the user interface when configuring a node.

In the following, you can find an exemplary plugin that receives items, prints a configurable message to the console output of the Inference DS Core, and sets a black image with a size of 100x100 that can be visualized in the user interface.

# example_plugin.py

import logging
from typing import Dict, Optional

import numpy as np

from inds.core.config import ConfigFieldTypes as Types
from inds.core.consumer import Consumer
from inds.core.exceptions import ConfigError
from inds.core.plugin import Plugin
from inds.core.producer import Producer, Property, PropertyType

logger = logging.getLogger(__name__)


class ExamplePlugin(Plugin):

    def __init__(self, name: str, message: str) -> None:
        super().__init__(name=name)

        self.message = message

        # initialize one consumer for node inputs
        self.consumer: Consumer = Consumer(name)
        self.add_consumer(self.consumer)

        # initialize one producer for node outputs
        properties = [Property('image', PropertyType.IMAGE)]
        self.producer: Producer = Producer(name, properties=properties)
        self.add_producer(self.producer)

    def run(self):
        # run the processing as long as the running is True
        while self.running:
            item = self.consumer.receive()
            if item is not None:
                print(self.message)

                # set the image of the item (or override it) to a black image
                item['image'] = np.zeros((100, 100, 3), dtype=np.uint8)

                # forward item to other nodes
                self.producer.send(item)

    @staticmethod
    def get_config_entries() -> Dict:
        return {
            'display_name': 'Example Plugin',
            'description': 'A description can be added here.',
            'fields': [
                {
                    'name': 'message',
                    'type': Types.STRING_INPUT,
                    'display_name': 'message',
                    'description': 'Message that will be printed after an item was received.'
                }
            ]
        }

    @staticmethod
    def get_config_key() -> str:
        return 'example_plugin'

    @staticmethod
    def build(name: str, config: Dict) -> Optional['Node']:
        message = config.get('message')
        if message is None or type(message) != str:
            raise ConfigError('message is required and must contain a string value')

        return ExamplePlugin(name, message)
PY

Simple Plugin

The class SimplePlugin implements a basic plugin that contains one consumer and one producer and is intended for tasks that receive items, manipulate their content, and forward the manipulated items. Instead of registering the consumer and producer as well as implementing the method run, you only need to implement the method process_item that takes one item as input and must return the modified item.

The following code snippet shows how the example plugin from above could be written as a SimplePlugin:

# example_simple_plugin.py

import logging
from typing import Dict, Optional

import numpy as np

from inds.core.config import ConfigFieldTypes as Types
from inds.core.exceptions import ConfigError
from inds.core.simple_plugin import SimplePlugin

logger = logging.getLogger(__name__)


class ExampleSimplePlugin(SimplePlugin):

    def __init__(self, name: str, message: str) -> None:
        super().__init__(name=name)

        self.message = message

    def process_item(self, item: Dict) -> Dict:
        # run the processing as long as the running is True
        print(self.message)

        # set the image of the item (or override it) to a black image
        item['image'] = np.zeros((100, 100, 3), dtype=np.uint8)

        # forward item to other nodes
        return item

    @staticmethod
    def get_config_entries() -> Dict:
        return {
            'display_name': 'Example Plugin',
            'description': 'A description can be added here.',
            'fields': [
                {
                    'name': 'message',
                    'type': Types.STRING_INPUT,
                    'display_name': 'message',
                    'description': 'Message that will be printed after an item was received.'
                }
            ]
        }

    @staticmethod
    def get_config_key() -> str:
        return 'example_simple_plugin'

    @staticmethod
    def build(name: str, config: Dict) -> Optional['Node']:
        message = config.get('message')
        if message is None or type(message) != str:
            raise ConfigError('message is required and must contain a string value')

        return ExampleSimplePlugin(name, message)
PY

Configuration

As you could see in the exemplary plugin above, you can add configuration entries in the method get_config_entries that appear in the user interface when you create an instance of the plugin. The configuration entries should contain:

  • display_name: Name of the plugin class.

  • description: Description of the plugin class.

  • fields: Configuration fields that allow user input.

Configuration Field Types

The entry fields can contain fields of the following field types.

GROUP

GROUP_SELECTION

GROUP_SWITCHABLE

BOOL_INPUT

DOUBLE_INPUT

INTEGER_INPUT

STRING_INPUT

JSON_TEXT_INPUT

SELECTION_DROPDOWN

Properties

In case your plugin contains a producer and the output of the producer should be displayed in the tab “view“ in the user interface, you must specify the possible content as the so-called properties of the consumer. The example plugin (example_plugin.py) contains an example where the image can be displayed in the user interface:

properties = [Property('image', PropertyType.IMAGE)]
self.producer: Producer = Producer(name, properties=properties)
PY

At the moment, the following three properties can be visualized:

  • Image Property('image', PropertyType.IMAGE
    Images are rendered on the canvas.

  • Point cloud Property('point_cloud', PropertyType.POINT_CLOUD)
    Point clouds are displayed in 3D.

  • Inference results Property('inference', PropertyType.INFERENCE)
    Inference results are rendered depending on their type as text (classification), 2D boxes (detection in images), 3D boxes (detection in point clouds)

Actions

Actions are displayed as buttons in the user interface in the tab “control” and allow the user to interact with nodes and plugins. For instance, you can create an action to trigger the recording of an item. In the following, you find a modified example of the example_simple_plugin.py that implements the action “save next item“.

# example_simple_plugin.py

import logging
from typing import Dict, Optional

import cv2
import numpy as np

from inds.core.config import ConfigFieldTypes as Types
from inds.core.decorators import action
from inds.core.exceptions import ConfigError
from inds.core.simple_plugin import SimplePlugin

logger = logging.getLogger(__name__)


class ExampleSimplePlugin(SimplePlugin):

    def __init__(self, name: str, message: str) -> None:
        super().__init__(name=name)

        self.message = message
        self.save_next = False
    
    @action
    def save_next_item(self):
        self.save_next = True

    def process_item(self, item: Dict) -> Dict:
        # run the processing as long as the running is True
        print(self.message)

        if self.save_next and 'image' in item:
            self.save_next = False
            cv2.imwrite('image.png', item['image'])

        # set the image of the item (or override it) to a black image
        item['image'] = np.zeros((100, 100, 3), dtype=np.uint8)

        # forward item to other nodes
        return item

    @staticmethod
    def get_config_entries() -> Dict:
        return {
            'display_name': 'Example Plugin',
            'description': 'A description can be added here.',
            'fields': [
                {
                    'name': 'message',
                    'type': Types.STRING_INPUT,
                    'display_name': 'message',
                    'description': 'Message that will be printed after an item was received.'
                }
            ]
        }

    @staticmethod
    def get_config_key() -> str:
        return 'example_simple_plugin'

    @staticmethod
    def build(name: str, config: Dict) -> Optional['Node']:
        message = config.get('message')
        if message is None or type(message) != str:
            raise ConfigError('message is required and must contain a string value')

        return ExampleSimplePlugin(name, message)
PY