Setting Up Apache Pulsar with MQTT Support and Testing with a Dart Client
Table of contents
- Prerequisites
- Setting Up Apache Pulsar with Docker Compose
- Understanding the docker-compose.yml File
- Downloading the MQTT Protocol Handler .nar File
- Creating a Configuration file for the Pulsar Client
- Configuring Volumes and Directories
- Running Docker Compose
- Testing the Setup with a Dart MQTT Client
- Understanding the Dart MQTT Client Code
- Running the Dart Client
- Conclusion
Apache Pulsar is a versatile, cloud-native messaging platform designed for high-throughput and low-latency data streaming. By integrating MQTT support, you can leverage Pulsar's powerful features with the lightweight MQTT protocol, making it ideal for IoT applications and real-time data exchange. In this guide, we'll walk you through setting up Apache Pulsar with MQTT support using Docker Compose and testing the setup with a Dart-based MQTT client.
Prerequisites
Before diving into the setup, ensure you have the following installed on your machine:
Docker: Download and install Docker
Docker Compose: Usually bundled with Docker Desktop, but verify installation.
Dart SDK: Download and install Dart
Git: For cloning repositories if needed.
Setting Up Apache Pulsar with Docker Compose
We'll use Docker Compose to simplify the deployment of Apache Pulsar with MQTT support. Below is the provided docker-compose.yml
file, which defines the Pulsar service.
Understanding the docker-compose.yml
File
version: '3.8'
services:
pulsar:
image: apachepulsar/pulsar:3.3.1
container_name: pulsar-broker
environment:
- PULSAR_MEM=-Xms2g -Xmx2g
ports:
- "6650:6650" # Pulsar binary protocol port
- "8080:8080" # Pulsar web admin port
- "1883:1883" # MQTT protocol port
volumes:
- ./protocols:/pulsar/protocols
- ./narfile.nar:/pulsar/protocols
- ./settings.txt:/tmp/settings.txt # Mount settings.txt
command: >
sh -c "cat /tmp/settings.txt >> /pulsar/conf/broker.conf && bin/pulsar standalone"
Key Components:
Image: Uses the Apache Pulsar Docker image version
3.3.1
.Container Name: Named
pulsar-broker
for easy identification.Environment Variables: Allocates 2GB of memory to Pulsar.
Ports:
6650
: Pulsar's binary protocol port.8080
: Pulsar's web admin interface.1883
: Standard MQTT protocol port.
Volumes:
Mounts a local
./protocols
directory to/pulsar/protocols
in the container.Mounts a local
narfile.nar
(the MQTT protocol handler) to the same directory.Mounts a local
settings.txt
file to/tmp/settings.txt
.
Command: Appends the content in
settings.txt
to thebroker.conf
file in/pulsar/conf/broker.conf
and then runs Pulsar in standalone mode, suitable for development and testing.
Downloading the MQTT Protocol Handler .nar File
The MQTT protocol handler is essential for enabling MQTT support in Pulsar. Here's how to download and set it up:
Download the
.nar
File:Visit the StreamNative GitHub Releases to download the MQTT protocol handler
.nar
file. Alternatively, usewget
orcurl
:wget https://github.com/streamnative/mop/releases/download/v0.2.0/pulsar-protocol-handler-mqtt-0.2.0-SNAPSHOT.nar -O pulsar-protocol-handler-mqtt.nar
Rename for Simplicity (Optional):
For consistency with the
docker-compose.yml
, rename the file:mv pulsar-protocol-handler-mqtt.nar narfile.nar
Creating a Configuration file for the Pulsar Client
Create a new text file in the same directory as the
docker-compose.yaml
wsl/bash:vi settings.txt
Paste the below contents in the
settings.txt
file:# MQTT over WebSocket Settings messagingProtocols=mqtt protocolHandlerDirectory=./protocols mqttListeners=tcp://0.0.0.0:1883 webServicePort=8080 webSocketServiceEnabled=true webSocketServicePort=8080
Configuring Volumes and Directories
Create Project Directory:
Choose a directory for your project and navigate to it:
mkdir pulsar-mqtt-setup cd pulsar-mqtt-setup
Create Necessary Directories:
mkdir protocols
Place the
.nar
File:Ensure the
narfile.nar
is in the project root. Thedocker-compose.yml
mounts both./protocols
and./narfile.nar
to/pulsar/protocols
, ensuring Pulsar loads the MQTT handler.Add the
docker-compose.yml
File:Create a
docker-compose.yml
file with the provided content.
Running Docker Compose
With everything in place, start the Pulsar broker:
Start the Service:
docker-compose up -d
The
-d
flag runs the container in detached mode.Verify the Containers:
docker ps
You should see the
pulsar-broker
container running with the specified ports mapped.Access Pulsar Web UI (Optional):
Open your browser and navigate to
http://localhost:8080
to access the Pulsar web admin interface.
Testing the Setup with a Dart MQTT Client
To ensure that the Pulsar broker with MQTT support is functioning correctly, we'll use a Dart-based MQTT client. The provided Dart script connects to the Pulsar MQTT endpoint, subscribes to topics, and publishes messages.
Understanding the Dart MQTT Client Code
Here's the provided Dart code with annotations:
import 'dart:async';
import 'dart:io';
import 'package:mqtt_client/mqtt_client.dart';
import 'package:mqtt_client/mqtt_server_client.dart';
// Initialize the MQTT client to connect to localhost on port 1883
final client = MqttServerClient('localhost', '1883');
/*
If needed, we can have the MQTT Client talk over WebSockets i.e., on port 8080.
Replace the above client variable as described below:
final client = MqttServerClient('localhost', '8080')
*/
Future<int> main() async {
client.logging(on: false); // Disable logging for cleaner output
client.keepAlivePeriod = 60; // Set keep-alive period to 60 seconds
client.onDisconnected = onDisconnected;
client.onConnected = onConnected;
client.onSubscribed = onSubscribed;
client.pongCallback = pong;
// Define the connection message with client identifier and will message
final connMess = MqttConnectMessage()
.withClientIdentifier('dart_client')
.withWillTopic('willtopic')
.withWillMessage('My Will message')
.startClean()
.withWillQos(MqttQos.atLeastOnce);
print('Client connecting....');
client.connectionMessage = connMess;
try {
await client.connect();
} on NoConnectionException catch (e) {
print('Client exception: $e');
client.disconnect();
} on SocketException catch (e) {
print('Socket exception: $e');
client.disconnect();
}
if (client.connectionStatus!.state == MqttConnectionState.connected) {
print('Client connected');
} else {
print('Client connection failed - disconnecting, status is ${client.connectionStatus}');
client.disconnect();
exit(-1);
}
// Subscribe to a test topic
const subTopic = 'topic/sub_test';
print('Subscribing to the $subTopic topic');
client.subscribe(subTopic, MqttQos.atMostOnce);
// Listen for messages on subscribed topics
client.updates!.listen((List<MqttReceivedMessage<MqttMessage?>>? c) {
final recMess = c![0].payload as MqttPublishMessage;
final pt = MqttPublishPayload.bytesToStringAsString(recMess.payload.message);
print('Received message: topic is ${c[0].topic}, payload is $pt');
});
// Listen for published messages
client.published!.listen((MqttPublishMessage message) {
print('Published topic: topic is ${message.variableHeader!.topicName}, with Qos ${message.header!.qos}');
});
// Publish a message to a test topic
const pubTopic = 'topic/pub_test';
final builder = MqttClientPayloadBuilder();
builder.addString('Hello from mqtt_client');
print('Subscribing to the $pubTopic topic');
client.subscribe(pubTopic, MqttQos.exactlyOnce);
print('Publishing our topic');
client.publishMessage(pubTopic, MqttQos.exactlyOnce, builder.payload!);
print('Sleeping....');
await MqttUtilities.asyncSleep(80);
// Unsubscribe from topics
print('Unsubscribing');
client.unsubscribe(subTopic);
client.unsubscribe(pubTopic);
await MqttUtilities.asyncSleep(2);
print('Disconnecting');
client.disconnect();
return 0;
}
/// Callback when subscribed to a topic
void onSubscribed(String topic) {
print('Subscription confirmed for topic $topic');
}
/// Callback when disconnected
void onDisconnected() {
print('OnDisconnected client callback - Client disconnection');
if (client.connectionStatus!.disconnectionOrigin ==
MqttDisconnectionOrigin.solicited) {
print('OnDisconnected callback is solicited, this is correct');
}
exit(-1);
}
/// Callback when connected
void onConnected() {
print('OnConnected client callback - Client connection was successful');
}
/// Callback for Pong responses
void pong() {
print('Ping response client callback invoked');
}
Key Functionalities:
Connection Setup: Connects to the MQTT broker at
localhost:1883
with a client identifierdart_client
.Subscriptions: Subscribes to
topic/sub_test
andtopic/pub_test
.Publishing: Publishes a message to
topic/pub_test
.Callbacks: Handles connection, disconnection, subscription confirmations, and pong responses.
Listening: Listens for incoming messages on subscribed topics and prints them.
Running the Dart Client
Follow these steps to run the Dart MQTT client:
Set Up Dart Project:
Create a new Dart project or navigate to your existing project directory.
mkdir dart_mqtt_client cd dart_mqtt_client dart create .
Add Dependencies:
Update your
pubspec.yaml
to include themqtt_client
package:dependencies: mqtt_client: ^9.6.1
Then, fetch the dependencies:
dart pub get
Create the Dart Script:
Replace the
bin/main.dart
file with the provided Dart code or create a new Dart file, e.g.,mqtt_client.dart
, and paste the code.Run the Dart Client:
Execute the Dart script:
dart run bin/mqtt_client.dart
Expected Output:
Client connecting.... Client connected Subscribing to the topic/sub_test topic Subscription confirmed for topic topic/sub_test Subscribing to the topic/pub_test topic Subscription confirmed for topic topic/pub_test Publishing our topic Published topic: topic is topic/pub_test, with Qos MqttQos.exactlyOnce Sleeping.... Received message: topic is topic/pub_test, payload is Hello from mqtt_client Unsubscribing Disconnecting
This output indicates that the client successfully connected to the Pulsar MQTT broker, subscribed to topics, published a message, received the published message, and gracefully disconnected.
Conclusion
By following this guide, you've successfully set up Apache Pulsar with MQTT support using Docker Compose and tested the setup with a Dart-based MQTT client. This configuration is ideal for developing and testing real-time applications, IoT solutions, or any system that benefits from high-throughput, low-latency messaging.
Next Steps:
Explore Pulsar Features: Delve deeper into Pulsar's capabilities, such as topic partitioning, message retention, and multi-tenancy.
Enhance the MQTT Client: Implement more sophisticated MQTT functionalities, like handling multiple subscriptions, QoS levels, and persistent sessions.
Deploy to Production: Consider scaling the Pulsar cluster, securing communications with TLS, and integrating with other systems as needed.
Apache Pulsar's flexibility combined with MQTT's simplicity opens up a world of possibilities for modern messaging solutions. Happy coding!