Running Apache Pulsar with MQTT Support Inside Docker

Apache Pulsar is a powerful, distributed messaging system that supports various protocols, including MQTT (Message Queuing Telemetry Transport). This blog will guide you through the steps to run Pulsar in a Docker container with MQTT enabled, using a docker-compose.yaml file, along with the necessary configuration files.

PermalinkPrerequisites

Before you begin, ensure that you have:

• Docker and Docker Compose installed on your machine (Docker Desktop works well for this).

• Basic familiarity with terminal commands.

• A directory to store your configuration files.

PermalinkStep 1: Create the Required Files

Permalink1. Create the Project Directory

First, create a directory for your Pulsar setup:

mkdir pulsar-docker
cd pulsar-docker

Permalink2. Create the docker-compose.yaml File

Create a file named docker-compose.yaml in your project directory and paste the following content:

version: '3.8'

services:
  pulsar:
    image: apachepulsar/pulsar:3.3.1
    container_name: pulsar-broker
    environment:
      - PULSAR_MEM=-Xms2g -Xmx2g
    ports:
      - "6650:6650"  # Pulsar binary protocol port
      - "8083:8080"  # Pulsar web admin port
      - "1883:1883"  # MQTT protocol port
    volumes:
      - ./protocols:/pulsar/protocols
      - ./narfile.nar:/pulsar/protocols
      - ./settings.txt:/tmp/settings.txt  # Mount settings.txt
    command: >
      sh -c "cat /tmp/settings.txt >> /pulsar/conf/broker.conf && bin/pulsar standalone"

Permalink3. Create the settings.txt File

Now, create a file named settings.txt and add the following configuration:

# MQTT over WebSocket Settings

messagingProtocols=mqtt
protocolHandlerDirectory=./protocols
mqttListeners=tcp://0.0.0.0:1883

webServicePort=8080
webSocketServiceEnabled=true

webSocketServicePort=8080

log_type=warning

You can create this file using a text editor like vim or nano:

nano settings.txt

Permalink4. Download the NAR File

Next, download the MQTT protocol handler for Pulsar. Run the following command to download the NAR file:

wget https://github.com/streamnative/mop/releases/download/v0.2.0/pulsar-protocol-handler-mqtt-0.2.0-SNAPSHOT.nar -O pulsar-protocol-handler-mqtt.nar

PermalinkRename for Simplicity (Optional)

To keep things consistent with the docker-compose.yaml file, rename the downloaded file:

mv pulsar-protocol-handler-mqtt.nar narfile.nar

Permalink5. Directory Structure

At this point, your directory should look like this:

pulsar-docker/
├── docker-compose.yaml
├── narfile.nar
└── settings.txt

PermalinkStep 2: Running Pulsar with Docker Compose

Now that all your files are set up, it’s time to run Pulsar.

Permalink1. Start the Pulsar Container

From your pulsar-docker directory, run the following command to start the Pulsar broker:

docker-compose up -d

This command will build the necessary Docker image and start the Pulsar container in detached mode. You should see logs in your terminal indicating that Pulsar is starting up.

Permalink2. Verify the Setup

You can check the status of your containers with:

docker-compose ps

If everything is running correctly, you should see the pulsar-broker listed and its status as “Up.”

PermalinkStep 3: Testing MQTT Connection

Now that your Pulsar broker is running, let’s test the MQTT connection.

Permalink1. Install MQTT Client

If you don’t have an MQTT client installed, you can use mosquitto_pub and mosquitto_sub for testing. Install it with:

sudo apt install mosquitto-clients

Permalink2. Publish a Test Message

Open a new terminal and run the following command to publish a test message to a topic:

mosquitto_pub -h localhost -p 1883 -t "test/topic" -m "Hello, Pulsar with MQTT!"

Permalink3. Subscribe to the Topic

In another terminal, subscribe to the same topic:

mosquitto_sub -h localhost -p 1883 -t "test/topic"

You should see the message “Hello, Pulsar with MQTT!” printed in the terminal where you subscribed.

PermalinkConclusion

Congratulations! You’ve successfully set up Apache Pulsar with MQTT support inside a Docker container. You can now use Pulsar as a powerful messaging broker that supports MQTT, allowing for easier integration with IoT devices and other applications.

Feel free to explore more features of Pulsar and experiment with different messaging patterns. Happy messaging!