General Plugin Concepts

Architecture

The YedMQ plugin system is designed with a language-agnostic and process-isolated architecture to ensure maximum flexibility and stability.

• Language Agnostic: You can develop plugins in any programming language that supports network socket communication.
• Process Isolation: Each plugin runs in its own child process, managed by the main YedMQ broker. This ensures that a crash or error in a plugin does not affect the stability of the broker or other plugins.
• Cross-Platform: The communication mechanism is built on standard technologies available on most operating systems (Unix Domain Sockets on Linux/Unix, Named Pipes on Windows).

Communication

The YedMQ broker and its plugins communicate via Inter-Process Communication (IPC).

• Transport:
  • Linux/Unix: Unix Domain Sockets
  • Windows: Named Pipes
• Protocol: Communication follows a custom binary protocol. The payload of the protocol messages is serialized using Protocol Buffers (Protobuf), providing a well-defined, efficient, and language-agnostic way to structure data.

Protocol Frame

The basic structure of a communication frame is as follows:

* +-----------+--------+----------------------+----------------------+
* | Magic     | Version| Length               | Payload              |
* | Number    |        | (4 bytes)            | (Protobuf)           |
* | (4 bytes) |(1 byte)| (Big Endian)         |                      |
* +-----------+--------+----------------------+----------------------+
* 0           4        5                      9                      9+N

• Magic Number: 0x5514
• Version: 0x01
• Length: A 4-byte integer indicating the size of the payload.
• Payload: The Protobuf-serialized ProtocolMessage.

All communication, whether it's a request, response, or notification, is wrapped in a ProtocolMessage. For detailed message definitions, please refer to the Protocol Definition page or the yedmq_plugin_protocol.proto file.

Plugin Configuration

Each plugin requires a plugin.toml file that describes its metadata and runtime configuration. This file tells YedMQ how to start and manage the plugin process.

Example of plugin.toml:

[plugin]
name = "my_auth_plugin"
version = "0.1.0"
description = "An authentication plugin"
author = "My Company"
license = "MIT"
homepage = "http://example.com"
repository = "https://github.com/example/my-plugin"

[runtime]
type = "process"
executable = "./my_auth_plugin_executable" # Path to the plugin's executable
args = ["--config", "/etc/yedmq/my_plugin.json"] # Optional arguments
env = { LOG_LEVEL = "info" } # Optional environment variables
working_dir = "." # Optional working directory
timeout_secs = 12 # Startup timeout in seconds

• [plugin] section: Contains general metadata about the plugin.
• [runtime] section:
  • type: Must be "process".
  • executable: The path to the plugin's executable file.
  • args: An array of command-line arguments to pass when starting the plugin.
  • env: A map of environment variables to set for the plugin process.
  • working_dir: The working directory for the plugin process.
  • timeout_secs: Present in the manifest, but the current runtime still relies on built-in initialization, request, and heartbeat timeouts.

Plugin Lifecycle

A plugin goes through several states during its lifecycle, managed by the YedMQ broker.

graph TD
    Discovered -->|Load| Starting
    Starting -->|Initialization Complete| Running
    Starting -->|Startup Error| Failed
    Running -->|Stop Command| Stopping
    Stopping -->|Resources Freed| Stopped
    Stopped -->|Restart| Starting
    Failed -->|Retry| Starting

1. Discovered: YedMQ has found the plugin's configuration file but has not yet started it.
2. Starting: The broker has forked a new process for the plugin and is waiting for it to initialize.
3. Running: The plugin has successfully initialized and is actively handling hooks.
4. Stopping: The plugin has received a shutdown signal and is cleaning up.
5. Stopped: The plugin process has terminated gracefully.
6. Failed: The plugin failed to start or crashed during operation.

Health Checks

The broker periodically sends a Ping message to a running plugin. The plugin must respond with a Pong message. If the broker fails to receive a response after repeated retries, it marks the plugin as failed. Automatic restart is not part of the current default runtime behavior.

Security

To prevent unauthorized processes from connecting to the broker, YedMQ uses a security token mechanism.

1. When the broker starts a plugin process, it passes a unique --auth-code as a command-line argument.
2. The plugin must include this auth-code in its InitializeResponse message back to the broker.
3. If the auth-code is missing or incorrect, the broker will immediately terminate the connection.

Execution Priority

You can define a priority in the [plugin] section of plugin.toml. This integer value determines the order of execution for hooks that are implemented by multiple plugins.

• Lower numbers mean higher priority. A plugin with priority = 100 will run before a plugin with priority = 1000.
• For hooks that can terminate the execution chain (e.g., authentication), the first plugin to handle the request may prevent others from running.
• For hooks that modify data in a chain, the execution order is critical.