Metaq installation deployment document

One ．MetaQ Installation and deployment ：

place	IP	Broker ID	Master/Slave	Slave ID:Group
Hefei	192.168.52.23	1	Slave	1:meta-slave-group-hf
Beijing	192.168.51.33	1	Master	/
Guangzhou	192.168.70.157	1	Slave	2:meta-slave-group-gz

Two ．MetaQ install

1. Premise

Already installed ZooKeeper Machine , And three places MetaQ You can access , Three IP Fine ping Through installation ZooKeeper Machine .ZooKeeper It's used to register MetaQ Of broker、topic、pub、sub Information , Used for load balancing , Play a coordinating role .

I have installed in all three places ZooKeeper, Beijing is currently used 192.168.51.33:2181 As ZooKeeper The server .

Zookeeper Use Ubuntu Of deb Bag dpkg Way to install .

start-up ZooKeeper：

Service zookeeper start

2. download MetaQserver 1.4.6.2

Download link ：https://meta-queue.googlecode.com/files/metaq-server-1.4.6.2.tar.gz, download metaq-server-1.4.6.2.tar.gz Copy to three places /usr/lib Under the table of contents .

3. Deploy

decompression tar zxvfmetaq-server-1.4.6.2.tar.gz, unzipped taobao/metamorphosis-server-wrapper Folder move to /usr/lib Next , And rename to metaq.

Metaq File structure ：

l bin： metaq Command folder for ./meteServer.sh perform metaq command

l conf：metaq Configuration folder ,async_slave.properties Asynchronous replication configuration ,server.ini To configure broker,samsa_master.properties Synchronous replication configuration .

l lib：metaq Rely on the jar package

l provided: Developers develop their own programs

4. master/slave colony To configure

Use asynchronous replication architecture .

Modify Beijing 192.168.51.33 Mechanical /usr/lib/metaq/conf/server.ini File for ：

[system]

;; brokerId mustbe unique, the slave should be the same with master

brokerId=1

;; topicpartitions default set to 1, I changed it to 16, it should not be too large

numPartitions=16

serverPort=8123

dashboardHttpPort=8120

unflushThreshold=0

unflushInterval=10000

maxSegmentSize=1073741824

;default 1048576

maxTransferSize=104857600

deletePolicy=delete,168

deleteWhen=0 0 6,18 * * ?

flushTxLogAtCommit=1

stat=true

putProcessThreadCount=100

;; Update consumers offsets tocurrent max offsets when consumers offsets are out of range of current broker'smessages.

;; It must be false inproduction.But recommend to be true in development or test.

updateConsumerOffsets=true

;; your zookeeperserver

[zookeeper]

zk.zkConnect=192.168.51.33:2181

zk.zkSessionTimeoutMs=30000

zk.zkConnectionTimeoutMs=30000

zk.zkSyncTimeMs=5000

;; Topics section

[topic=test]

[topic=meta-test]

;; add your topicshere before use it

[topic=oct29]

Because use master/slave Pattern , Therefore, Hefei must be maintained at all times 192.168.52.23 And Guangzhou 192.168.70.157 Of server.ini Document keeping and Beijing 192.168.51.33 identical , Even if new topic, It is also necessary to manually server.ini Register in .

Modify the three places conf/async_slave.properties, Beijing's remains unchanged , Hefei is revised as

#slave Number , Greater than or equal to 0 To act as slave start-up , The same master Under the slave The number should be set to a different value .

slaveId=1

# As slave Start up to master Subscribe to the message group, If it is not configured, it defaults to meta-slave-group

# Different slaveId Please use different group

slaveGroup=meta-slave-group

#slave Maximum delay of data synchronization , Unit millisecond

slaveMaxDelayInMills=500

# Automatically from master Sync server.ini, 1.4.2 New options

# You still need to copy it yourself for the first time server.ini, You can set this option to true To automatically synchronize

autoSyncMasterConfig=true

Guangzhou conf/async_slave.properties It is amended as follows ：

#slave Number , Greater than or equal to 0 To act as slave start-up , The same master Under the slave The number should be set to a different value .

Copy

slaveId=2

# As slave Start up to master Subscribe to the message group, If it is not configured, it defaults to meta-slave-group

Copy

# Different slaveId Please use different group

Copy

slaveGroup=meta-slave-group-gz

#slave Maximum delay of data synchronization , Unit millisecond 

Copy

slaveMaxDelayInMills=500

Copy

# Automatically from master Sync server.ini, 1.4.2 New options 

Copy

# You still need to copy it yourself for the first time server.ini, You can set this option to true To automatically synchronize 

Copy

autoSyncMasterConfig=true

Copy

5. start-up MetaQ colony

Start two first slave：

Hefei and Guangzhou will switch directories to /usr/lib/metaq/bin Next , Use slave mode MetaQ：

./metaServer.sh start slave &

Beijing launch master：

./metaServer.sh start &

3、 ... and ． problem

1. Topic How much is the default partition ？ namely server.ini in numPartitions Configuration of , At present for 16, When using 1024 Many times, the disk efficiency will be too low ,message The probability of sending failure increases .

2. Asynchronously replicated slave Will participate in consumer activities , Message consumers can choose from slave Get information from and consume , Consumers will randomly choose from master and slaves Choose one of them for consumption broker. How to specify consumer from slave Middle consumption message Instead of spending master What about ？

3. You can add one topic It doesn't restart MetaQ Do you ？

Four ． Reference resources

MetaQ HA: https://github.com/killme2008/Metamorphosis/wiki/HA

MetaQ Source code analysis ：https://github.com/killme2008/Metamorphosis/wiki/%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90

MetaQ Wiki：https://github.com/killme2008/Metamorphosis/wiki

MetaQ User guide ：http://www.linuxidc.com/Linux/2013-11/92750.htm

Distributed Publish subscribe message system Kafka Architecture design http://www.linuxidc.com/Linux/2013-11/92751.htm