Setting Up Apache Pulsar with MQTT Support and Testing with a Dart Client

Apache Pulsar is a versatile, cloud-native messaging platform designed for high-throughput and low-latency data streaming. By integrating MQTT support, you can leverage Pulsar's powerful features with the lightweight MQTT protocol, making it ideal for IoT applications and real-time data exchange. In this guide, we'll walk you through setting up Apache Pulsar with MQTT support using Docker Compose and testing the setup with a Dart-based MQTT client.

Prerequisites

Before diving into the setup, ensure you have the following installed on your machine:

• Docker: Download and install Docker

• Docker Compose: Usually bundled with Docker Desktop, but verify installation.

• Dart SDK: Download and install Dart

• Git: For cloning repositories if needed.

Setting Up Apache Pulsar with Docker Compose

We'll use Docker Compose to simplify the deployment of Apache Pulsar with MQTT support. Below is the provided docker-compose.yml file, which defines the Pulsar service.

Understanding the docker-compose.yml File

version: '3.8'

services:
  pulsar:
    image: apachepulsar/pulsar:3.3.1
    container_name: pulsar-broker
    environment:
      - PULSAR_MEM=-Xms2g -Xmx2g
    ports:
      - "6650:6650"  # Pulsar binary protocol port
      - "8080:8080"  # Pulsar web admin port
      - "1883:1883"  # MQTT protocol port
    volumes:
      - ./protocols:/pulsar/protocols
      - ./narfile.nar:/pulsar/protocols
      - ./settings.txt:/tmp/settings.txt  # Mount settings.txt
    command: >
      sh -c "cat /tmp/settings.txt >> /pulsar/conf/broker.conf && bin/pulsar standalone"

Key Components:

• Image: Uses the Apache Pulsar Docker image version 3.3.1.

• Container Name: Named pulsar-broker for easy identification.

• Environment Variables: Allocates 2GB of memory to Pulsar.

• Ports:

  • 6650: Pulsar's binary protocol port.

  • 8080: Pulsar's web admin interface.

  • 1883: Standard MQTT protocol port.

• Volumes:

  • Mounts a local ./protocols directory to /pulsar/protocols in the container.

  • Mounts a local narfile.nar (the MQTT protocol handler) to the same directory.

  • Mounts a local settings.txt file to /tmp/settings.txt.

• Command: Appends the content in settings.txt to the broker.conf file in /pulsar/conf/broker.conf and then runs Pulsar in standalone mode, suitable for development and testing.

Downloading the MQTT Protocol Handler .nar File

The MQTT protocol handler is essential for enabling MQTT support in Pulsar. Here's how to download and set it up:

1. Download the .nar File:

   Visit the StreamNative GitHub Releases to download the MQTT protocol handler .nar file. Alternatively, use wget or curl:

    wget https://github.com/streamnative/mop/releases/download/v0.2.0/pulsar-protocol-handler-mqtt-0.2.0-SNAPSHOT.nar -O pulsar-protocol-handler-mqtt.nar
   

2. Rename for Simplicity (Optional):

   For consistency with the docker-compose.yml, rename the file:

    mv pulsar-protocol-handler-mqtt.nar narfile.nar
   

Creating a Configuration file for the Pulsar Client

1. Create a new text file in the same directory as the docker-compose.yaml wsl/bash:

    vi settings.txt
   

2. Paste the below contents in the settings.txt file:

    # MQTT over WebSocket Settings
   
    messagingProtocols=mqtt
    protocolHandlerDirectory=./protocols
    mqttListeners=tcp://0.0.0.0:1883
   
    webServicePort=8080
    webSocketServiceEnabled=true
    webSocketServicePort=8080
   

Configuring Volumes and Directories

1. Create Project Directory:

   Choose a directory for your project and navigate to it:

    mkdir pulsar-mqtt-setup
    cd pulsar-mqtt-setup
   

2. Create Necessary Directories:

    mkdir protocols
   

3. Place the .nar File:

   Ensure the narfile.nar is in the project root. The docker-compose.yml mounts both ./protocols and ./narfile.nar to /pulsar/protocols, ensuring Pulsar loads the MQTT handler.

4. Add the docker-compose.yml File:

   Create a docker-compose.yml file with the provided content.

Running Docker Compose

With everything in place, start the Pulsar broker:

1. Start the Service:

    docker-compose up -d
   

   The -d flag runs the container in detached mode.

2. Verify the Containers:

    docker ps
   

   You should see the pulsar-broker container running with the specified ports mapped.

3. Access Pulsar Web UI (Optional):

   Open your browser and navigate to http://localhost:8080 to access the Pulsar web admin interface.

Testing the Setup with a Dart MQTT Client

To ensure that the Pulsar broker with MQTT support is functioning correctly, we'll use a Dart-based MQTT client. The provided Dart script connects to the Pulsar MQTT endpoint, subscribes to topics, and publishes messages.

Understanding the Dart MQTT Client Code

Here's the provided Dart code with annotations:

import 'dart:async';
import 'dart:io';
import 'package:mqtt_client/mqtt_client.dart';
import 'package:mqtt_client/mqtt_server_client.dart';

// Initialize the MQTT client to connect to localhost on port 1883
final client = MqttServerClient('localhost', '1883');

/*
If needed, we can have the MQTT Client talk over WebSockets i.e., on port 8080.
Replace the above client variable as described below:
final client = MqttServerClient('localhost', '8080')
*/

Future<int> main() async {
  client.logging(on: false); // Disable logging for cleaner output
  client.keepAlivePeriod = 60; // Set keep-alive period to 60 seconds
  client.onDisconnected = onDisconnected;
  client.onConnected = onConnected;
  client.onSubscribed = onSubscribed;
  client.pongCallback = pong;

  // Define the connection message with client identifier and will message
  final connMess = MqttConnectMessage()
      .withClientIdentifier('dart_client')
      .withWillTopic('willtopic') 
      .withWillMessage('My Will message')
      .startClean() 
      .withWillQos(MqttQos.atLeastOnce);
  print('Client connecting....');
  client.connectionMessage = connMess;

  try {
    await client.connect();
  } on NoConnectionException catch (e) {
    print('Client exception: $e');
    client.disconnect();
  } on SocketException catch (e) {
    print('Socket exception: $e');
    client.disconnect();
  }

  if (client.connectionStatus!.state == MqttConnectionState.connected) {
    print('Client connected');
  } else {
    print('Client connection failed - disconnecting, status is ${client.connectionStatus}');
    client.disconnect();
    exit(-1);
  }

  // Subscribe to a test topic
  const subTopic = 'topic/sub_test';
  print('Subscribing to the $subTopic topic');
  client.subscribe(subTopic, MqttQos.atMostOnce);

  // Listen for messages on subscribed topics
  client.updates!.listen((List<MqttReceivedMessage<MqttMessage?>>? c) {
    final recMess = c![0].payload as MqttPublishMessage;
    final pt = MqttPublishPayload.bytesToStringAsString(recMess.payload.message);
    print('Received message: topic is ${c[0].topic}, payload is $pt');
  });

  // Listen for published messages
  client.published!.listen((MqttPublishMessage message) {
    print('Published topic: topic is ${message.variableHeader!.topicName}, with Qos ${message.header!.qos}');
  });

  // Publish a message to a test topic
  const pubTopic = 'topic/pub_test';
  final builder = MqttClientPayloadBuilder();
  builder.addString('Hello from mqtt_client');

  print('Subscribing to the $pubTopic topic');
  client.subscribe(pubTopic, MqttQos.exactlyOnce);

  print('Publishing our topic');
  client.publishMessage(pubTopic, MqttQos.exactlyOnce, builder.payload!);

  print('Sleeping....');
  await MqttUtilities.asyncSleep(80);

  // Unsubscribe from topics
  print('Unsubscribing');
  client.unsubscribe(subTopic);
  client.unsubscribe(pubTopic);

  await MqttUtilities.asyncSleep(2);
  print('Disconnecting');
  client.disconnect();

  return 0;
}

/// Callback when subscribed to a topic
void onSubscribed(String topic) {
  print('Subscription confirmed for topic $topic');
}

/// Callback when disconnected
void onDisconnected() {
  print('OnDisconnected client callback - Client disconnection');
  if (client.connectionStatus!.disconnectionOrigin ==
      MqttDisconnectionOrigin.solicited) {
    print('OnDisconnected callback is solicited, this is correct');
  }
  exit(-1);
}

/// Callback when connected
void onConnected() {
  print('OnConnected client callback - Client connection was successful');
}

/// Callback for Pong responses
void pong() {
  print('Ping response client callback invoked');
}

Key Functionalities:

• Connection Setup: Connects to the MQTT broker at localhost:1883 with a client identifier dart_client.

• Subscriptions: Subscribes to topic/sub_test and topic/pub_test.

• Publishing: Publishes a message to topic/pub_test.

• Callbacks: Handles connection, disconnection, subscription confirmations, and pong responses.

• Listening: Listens for incoming messages on subscribed topics and prints them.

Running the Dart Client

Follow these steps to run the Dart MQTT client:

1. Set Up Dart Project:

   Create a new Dart project or navigate to your existing project directory.

    mkdir dart_mqtt_client
    cd dart_mqtt_client
    dart create .
   

2. Add Dependencies:

   Update your pubspec.yaml to include the mqtt_client package:

    dependencies:
      mqtt_client: ^9.6.1
   

   Then, fetch the dependencies:

    dart pub get
   

3. Create the Dart Script:

   Replace the bin/main.dart file with the provided Dart code or create a new Dart file, e.g., mqtt_client.dart, and paste the code.

4. Run the Dart Client:

   Execute the Dart script:

    dart run bin/mqtt_client.dart
   

   Expected Output:

    Client connecting....
    Client connected
    Subscribing to the topic/sub_test topic
    Subscription confirmed for topic topic/sub_test
    Subscribing to the topic/pub_test topic
    Subscription confirmed for topic topic/pub_test
    Publishing our topic
    Published topic: topic is topic/pub_test, with Qos MqttQos.exactlyOnce
    Sleeping....
    Received message: topic is topic/pub_test, payload is Hello from mqtt_client
    Unsubscribing
    Disconnecting
   

   This output indicates that the client successfully connected to the Pulsar MQTT broker, subscribed to topics, published a message, received the published message, and gracefully disconnected.

Conclusion

By following this guide, you've successfully set up Apache Pulsar with MQTT support using Docker Compose and tested the setup with a Dart-based MQTT client. This configuration is ideal for developing and testing real-time applications, IoT solutions, or any system that benefits from high-throughput, low-latency messaging.

Next Steps:

• Explore Pulsar Features: Delve deeper into Pulsar's capabilities, such as topic partitioning, message retention, and multi-tenancy.

• Enhance the MQTT Client: Implement more sophisticated MQTT functionalities, like handling multiple subscriptions, QoS levels, and persistent sessions.

• Deploy to Production: Consider scaling the Pulsar cluster, securing communications with TLS, and integrating with other systems as needed.

Apache Pulsar's flexibility combined with MQTT's simplicity opens up a world of possibilities for modern messaging solutions. Happy coding!