image block
1310 Views 0 Comments
Hi, all! Nice to see you here!
I know that almost everyone heard about these tools and lot used them, but I want to make clear the question why messaging middleware systems are an important part of environments with a high load.

Overview

So what is the messaging system?
System's, which has the responsibility for safely delivering of messages are calling messaging system. The main point of this tools is ensuring that the receiver will get the correct message and it will not disappear somewhere.
Advantages :
  • Direct connection to receiver is not needed
  • If receiver is not available or have full threads and can't getting more, messages still will be in safe
  • Validation of messages
  • Distribution of messages and delivering them to different receivers or sending same messages to more than one system
But before we will start the review of tools, let's clear some definitions, producers and consumers.
Producers - tools or scripts which publish data. Also, the producer is responsible for choosing which record to assign to which partition within the topic.
Consumers - tools or scripts which are getting data. They can be in separate processes or on separate machines.

Apache Kafka

Apache Kafka is one of most popular a distributed streaming platform.
The main advantage of Kafka over other that it's open-source and very easy in scalability.
Apache Kafka has a row of additional functions, which are give more comfortable usage of it, like monitoring of data, stream processing, event sourcing and logging of actions.
The installation process is very easy and can be done in few quick steps. At first, we need to have the installed JAVA on our machine.
# yum install java -y
It will install the latest version. You can check this via this command
# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
Download Kafka package and extract it
# wget https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
# tar -zxvf kafka_2.11-0.10.2.0.tgz ; cd kafka_2.11-0.10.2.0
Start Zookeeper (service for maintaining configuration information)
# ./bin/zookeeper-server-start.sh config/zookeeper.properties
After running of Zookeeper you must see the next line
[2017-06-16 15:15:50,610] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
which is mean, that the Zookeeper is available on port 2181.
Before start the Kafka need to edit configuration, because will receive an issue due to hostname. So edit the config file config/server.properties and start the tool. On configuration file need to change line advertised.listeners
# ./bin/kafka-server-start.sh config/server.properties
Next steps are creation and testing of topics.
# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic cmdb
# ./bin/kafka-topics.sh --list --zookeeper localhost:2181
cmdb
# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic cmdb
test message1
test message2
# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic cmdb --from-beginning
test message1
test message2
From this, we see how we can easily manage of messages via Kafka. And in the end, let's make a little example with the script on Python to automate this process.
On this will help Python module "kafka". To install it use this command
# pip install kafka-python
No more action needed and we can start work with it.
# python3.5
Python 3.5.3 (default, Jan 17 2017, 14:34:36)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kafka import KafkaConsumer
>>> consumer = KafkaConsumer('cmdb')
>>> for msg in consumer:
        print(msg)
ConsumerRecord(topic='cmdb', partition=0, offset=4, timestamp=1497629033994, timestamp_type=0, key=None, value=b'test message1', checksum=234866335, serialized_key_size=-1, serialized_value_size=0)
ConsumerRecord(topic='cmdb', partition=0, offset=5, timestamp=1497629034614, timestamp_type=0, key=None, value=b'test message2', checksum=1539232508, serialized_key_size=-1, serialized_value_size=4)

RabbitMQ

RabbitMQ is also Open Source tool for working with messages, but with little differences. At the first, RabbitMQ must be installed on each node.
But also it has few important advantages, like the possibility of removing the item when it will be delivered to the receiver or routing between exchanges. Also RabbitMQ is supporting JSON message format, which is give additional validation of messages.
For install it need to made the next steps.
Get the link on latest version on official site https://www.rabbitmq.com/install-rpm.html and download it
# wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.10/rabbitmq-server-3.6.10-1.el7.noarch.rpm
After this we will get the .rpm package and we need to install it (in case if we don't want install dependency packages use rpm -ivh command)
# yum install rabbitmq-server-3.6.10-1.el7.noarch.rpm
Unlike the Kafka, RabbitMQ needs more permissions to work. To do this we have two ways. In the first case, we can just stop all security tools in the machine. It's the easiest way, just put in command line the next commands.
# setenforce 0
# systemctl stop firewalld
In case if we need keep working security tools, we need to create few rules, that the RMQ (RabbitMQ) will have permissions to startup on needed ports and work with them. At the first let's reconfigure the Selinux
# setsebool -P nis_enabled 1
This command will make the rule on Selinux and it will not deny this process. Next need to add rules on Firewall to open ports.
# firewall-cmd --permanent --add-port=4369/tcp
# firewall-cmd --permanent --add-port=25672/tcp
# firewall-cmd --permanent --add-port=5671-5672/tcp
# firewall-cmd --permanent --add-port=15672/tcp
# firewall-cmd --permanent --add-port=61613-61614/tcp
# firewall-cmd --permanent --add-port=8883/tcp
# firewall-cmd --reload
And here we go, right now we can run RMQ.
# systemctl start rabbitmq-server
After this, we will able to reach the web interface of RMQ, but before, need to create the user to login on it.
# rabbitmqctl add_user admin admin
# rabbitmqctl set_user_tags admin administrator
# rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
It's example of interface of RabbitMQ
In this web interface, we can easily to create our working queue with needed parameters. Just got to "Queues" bookmark and create one. For RabbitMQ, Python has module also. To install it just type this
# pip install pika
Also in the official page of RabbitMQ, exist example how to use it, so we can just use it.
Link on the page : RabbitMQ Pika
Just don't forget to change credentials and target host.
import pika
import time

credentials = pika.PlainCredentials("admin", "admin")
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost', credentials=credentials))
channel = connection.channel()

channel.queue_declare(queue='cmdb', durable=False)
print(' [*] Waiting for messages. To exit press CTRL+C')

def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    time.sleep(body.count(b'.'))
    print(" [x] Done")
    ch.basic_ack(delivery_tag = method.delivery_tag)

channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
                      queue='callback_queue')

channel.start_consuming()
To check it, let's add test message to our queue. You can do this via WebUI also. Go to "Queues" bookmark, the name of the queue and select "Publish message" option.
{
    "message": "test"
}
And we will receive this message.
# ./pika.py
 [*] Waiting for messages. To exit press CTRL+C
 [x] Received {"message": "test"}
 [x] Done

Conclusion

After reviewing two most popular messaging middleware systems, we saw why they are so important part of modern IT world and why large companies are often using them. And you could probably use them in your infrastructure to avoid issues with direct connection, loading on system etc.

Thank you very much for your attention. I hope this information was useful for you, because I did this with pleasure. Good Luck to you in all your endeavors!
- Kostia

0 Comments


    Leave a Comment