AliKevin 写道
S4的的运行模式有两种Static和Dynamic,期之前我们所做的学习完全基于s4的static模式。static模式不能对s4的集群进行智能的负载均衡,这对于分布式的大数量计算能力是一个不好的消息哦:),幸运的是S4可以借助现有的ZooKeeper智能的管理S4的集群:),接下来我们看看如何利用zookeeper和s4结合部署。
一、ZooKeeper的简单部署
我们这里只是简单介绍s4与ZooKeeper的结合,不涉及到ZooKeeper的集群,这里试只是单节点启动ZooKeeper。
a.[下载ZooKeeper|http://labs.renren.com/apache-mirror//zookeeper/zookeeper-3.3.3/zookeeper-3.3.3.tar.gz]
b.安装ZooKeeper
将下载的ZooKeeper解压到你的安装目录,这里目录是/kevin/zookeeper-3.3.3,执行如下命令:
root@master:/# cd /kevin/zookeeper-3.3.3/conf
root@master:/kevin/zookeeper-3.3.3/conf# cp zoo.cfg.bak zoo.cfg
root@master:/kevin/zookeeper-3.3.3/conf# vi zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/kevin/zookeeper-3.3.3/data ---> 修改到你的指定目录
# the port at which the clients will connect
clientPort=2181--->设置你要监听的端口(大于1024)
c.启动ZooKeeper实例
root@master:/kevin/zookeeper-3.3.3/conf# cd /kevin/zookeeper-3.3.3/bin/
root@master:/kevin/zookeeper-3.3.3/bin# ./zkServer.sh start
root@master:/kevin/zookeeper-3.3.3/bin# ./zkServer.sh start
JMX enabled by default
Using config: /kevin/zookeeper-3.3.3/bin/../conf/zoo.cfg
Starting zookeeper ...
STARTED
....
....
d.验证ZooKeeper的实例启动情况
root@master:/kevin/zookeeper-3.3.3/bin# cd /
root@master:/# cd /kevin/zookeeper-3.3.3/bin
root@master:/kevin/zookeeper-3.3.3/bin# ./zkCli.sh -server localhost:2181
Connecting to localhost:2181
...
...
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
[zk: localhost:2181(CONNECTED) 29] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 30]
如上显示证明我们已经启动的ZooKeeper实例,当前实例说管理的节点是跟目录下有个zookeeper节点。
二、将S4节点告知ZooKeeper
a.将S4集群信息告知ZooKeeper
root@master:~# $S4_IMAGE/scripts/task-setup.sh localhost:2181 clean setup $S4_IMAGE/s4-core/conf/dynamic/clusters.xml
RUNNING /usr/jdk1.6.0_26/bin/java -classpath /kevin/s4/build/s4-image/s4-core/lib/gson-1.6.jar:/kevin/s4/build/s4-image/s4-core/lib/junit-3.8.1.jar:/kevin/s4/build/s4-image/s4-core/lib/spring-2.5.6.jar:/kevin/s4/build/s4-image/s4-core/lib/S4WordCount.jar:/kevin/s4/build/s4-image/s4-core/lib/jakarta-regexp-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/bcel-5.2.jar:/kevin/s4/build/s4-image/s4-core/lib/log4j-1.2.15.jar:/kevin/s4/build/s4-image/s4-core/lib/kryo-1.01.jar:/kevin/s4/build/s4-image/s4-core/lib/json-20090211.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-driver-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/mail-1.4.jar:/kevin/s4/build/s4-image/s4-core/lib/activation-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/flexjson-2.1.jar:/kevin/s4/build/s4-image/s4-core/lib/minlog-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/zookeeper-3.3.1.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-example-driver-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/asm-3.2.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-cli-1.2.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-jexl-1.1.jar:/kevin/s4/build/s4-image/s4-core/lib/jline-0.9.94.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-core-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/s4-comm-0.3.0.jar:/kevin/s4/build/s4-image/s4-core/lib/reflectasm-0.8.jar:/kevin/s4/build/s4-image/s4-core/lib/commons-logging-1.1.1.jar: io.s4.comm.tools.TaskSetupApp localhost:2181 clean setup /kevin/s4/build/s4-image/s4-core/conf/dynamic/clusters.xml
log4j:WARN Continuable parsing error 1 and column 70
log4j:WARN Document root element "log4j:configuration", must match DOCTYPE root "null".
log4j:WARN Continuable parsing error 1 and column 70
log4j:WARN Document is invalid: no grammar found.
log4j:WARN No appenders could be found for logger (io.s4.comm.core.DefaultWatcher).
log4j:WARN Please initialize the log4j system properly.
sTimeout=null
sTimeout=null
root@master:~#
b.检查ZooKeeper接管S4后的节点情况
root@master:/kevin/zookeeper-3.3.3/bin# ./zkCli.sh -server localhost:2181
Connecting to localhost:2181
...
...
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[client-adapter, zookeeper, s4]
[zk: localhost:2181(CONNECTED) 1] ls /s4/s4
[process, task]
[zk: localhost:2181(CONNECTED) 5] ls /client-adapter/s4
[process, task]
[zk: localhost:2181(CONNECTED) 7]
如上信息证明我们已经将S4的集群信息告知给ZooKeeper了:)
三、S4实例验证
与ZooKeeper结合要以Dynamic方式启动S4,故要修改S4的启动sh.当然修改方式有好多种(命令参数或硬性修改)这里只强调是要Dynamic模式:)
a.启动S4和Adapdter
root@slave:~# $S4_IMAGE/scripts/start-s4.sh dynamic -r client-adapter
...
appName=s4
dequeuer number: 6
sTimeout=5000
[]
[/kevin/s4/build/s4-image/s4-apps/s4-example-speech02/s4-example-speech02-conf.xml]
sTimeout=5000
sTimeout=5000
Adding processing element with bean name eventCatcher, id eventCatcher
adding pe: io.s4.example.speech01.SentenceReceiverPE@d58939
Using ConMapPersister ..
Adding processing element with bean name rerouteSentencePE, id rerouteSentencePE
adding pe: io.s4.processor.ReroutePE@93df2c
Using ConMapPersister ..
Adding processing element with bean name rerouteSpeechPE, id rerouteSpeechPE
adding pe: io.s4.processor.ReroutePE@3a1834
Using ConMapPersister ..
Adding processing element with bean name sentenceJoinPE, id sentenceJoinPE
adding pe: io.s4.processor.JoinPE@190a0d6
Using ConMapPersister ..
root@slave:~# $S4_IMAGE/scripts/run-client-adapter.sh -s client-adapter -g s4 -d $S4_IMAGE/s4-core/conf/dynamic/client-stub-conf.xml dynamic
...
appName=client-adapter
dequeuer number: 12
sTimeout=null
Adding InputStub genericStub
Adding OutputStub genericStub
sTimeout=null
b.执行测试和查看服务端输出
测试类:
root@slave:~# head -10 $S4_IMAGE/s4-example-testinput/speeches.txt | sh $S4_IMAGE/s4-tools-loadgenerator/scripts/generate-load.sh -r 2 -a localhost:2334 -d 1500 -
[: 27: ==: unexpected operator
[: 57: xlocalhost:2334: unexpected operator
[: 61: x2: unexpected operator
[: 65: x1500: unexpected operator
[: 69: x: unexpected operator
{0=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@60420f, 1=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@19106c7, 2=io.s4.tools.loadgenerator.LoadGenerator$EventTypeInfo@540408}
Emitted 9 events
服务端:
Using ConMapPersister ..
Sentence is 'Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal.', location gettysburg, pa, us
Sentence is 'Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal.', location gettysburg, pa, us
Sentence is 'Now we are engaged in a great civil war, testing whether that nation or any nation so conceived and so dedicated can long endure.', location gettysburg, pa, us
Sentence is 'Now we are engaged in a great civil war, testing whether that nation or any nation so conceived and so dedicated can long endure.', location gettysburg, pa, us
Sentence is 'We are met on a great battlefield of that war.', location gettysburg, pa, us
Sentence is 'We are met on a great battlefield of that war.', location gettysburg, pa, us
Sentence is 'We have come to dedicate a portion of that field as a final resting-place for those who here gave their lives that that nation might live.', location gettysburg, pa, us
Sentence is 'We have come to dedicate a portion of that field as a final resting-place for those who here gave their lives that that nation might live.', location gettysburg, pa, us
OK,如上信息证明我们联合ZooKeeper部署成功了,在使用的角度对客户是透明的,我们感性上感觉不到ZooKeeper的存在,而在S4内部则发生了根本的变化,当static模式时候S4的事件流处理由StaticTaskManager处理,而当我们以
dynamic 模式结合ZooKeeper运行S4时候,S4内部会应用ZkTaskManager管理事件流的处理。ZkTaskManager会利用ZooKeeper强大的集群管理能力,实现负载均衡等高效率的集群管理!:)
分享到:
相关推荐
apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper-3.7.1 apache-zookeeper...
apache-zookeeper分布式框架,压缩包内容:(apache-zookeeper-3.7.1-bin.tar.gz、apache-zookeeper-3.7.1.tar.gz、apache-zookeeper-3.6.4-bin.tar.gz、apache-zookeeper-3.6.4.tar.gz、apache-zookeeper-3.5.10-...
Zookeeper安装包下载----Zookeeper节点可视化工具(ZooInspector)下载 解压后即可使用,初次启动需修改配置文件 conf--->zoo_sample.cfg 复制一份出来 重命名为 zoo.cfg ...zookeeper-dev-ZooInspector.jar
apache-zookeeper-3.6.0.tar.gz
基于ZooKeeper的集群节点管理方案的设计与实现,李文韵,崔毅东,随着集群技术的普及,集群技术在节点管理方面的缺陷也日益凸显。无法及时了解集群中各节点的资源状态,无法对节点管理给予有效的
利用ShardingSphere(sharding-proxy)实现分库分表,通过整合ZooKeeper,进行简单配置文件修改轻松实现跨库跨表相关操作,轻量级零侵入整合项目开发。
Kafka集群安装部署-自带zookeeper 1、kafka简介 2、环境准备 3、安裝配置工作
赠送jar包:twill-zookeeper-0.6.0-incubating.jar; 赠送原API文档:twill-zookeeper-0.6.0-incubating-javadoc.jar; 赠送源代码:twill-zookeeper-0.6.0-incubating-sources.jar; 赠送Maven依赖信息文件:twill-...
netty-redis-zookeeper高并发实战学习-netty-redis-zookeeper
赠送jar包:twill-zookeeper-0.6.0-incubating.jar; 赠送原API文档:twill-zookeeper-0.6.0-incubating-javadoc.jar; 赠送源代码:twill-zookeeper-0.6.0-incubating-sources.jar; 赠送Maven依赖信息文件:twill-...
zookeeper-3.4.8zookeeper-3.4.8zookeeper-3.4.8zookeeper-3.4.8
06.hadoop-HA机制整体解析--引入zookeeper.mp4
11.hbase集群搭建--及其各种机制的观察--hdfs中的目录--zookeeper中的状态数据.mp4
apache-zookeeper-3.5.7-bin.tar.gz 。
apache-zookeeper-3.5.10-bin 环境搭配 ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件,...
apache-zookeeper-3.5.7-bin.tar.gz 及 安装方法。apache-zookeeper-3.5.7-bin.tar.gz 及 安装方法。apache-zookeeper-3.5.7-bin.tar.gz 及 安装方法。
apache-zookeeper-3.7.0-bin.tar.gz
NULL 博文链接:https://fxzcollege6.iteye.com/blog/2008563
hadoop-2.8.1 zookeeper-3.4.9 hbase-1.3.1分布式环境搭建整理
apache-zookeeper-3.5.5.tar.gz,zookeeper的安装文件,解压后配置集群即可