一、DolphinScheduler 简介
1.1、DolphinScheduler 概述
Apache DolphinScheduler 是一个分布式、易扩展的可视化 DAG 工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。
1.2、DolphinScheduler 核心架构
DolphinScheduler 的主要角色如下：
- MasterServer 采用分布式无中心设计理念，MasterServer 主要负责 DAG 任务切分、任务提交、任务监控，并同时监听其它 MasterServer 和 WorkerServer 的健康状态。
- WorkerServer 也采用分布式无中心设计理念，WorkerServer 主要负责任务的执行和提供日志服务。
- ZooKeeper 服务，系统中的 MasterServer 和 WorkerServer 节点都通过 ZooKeeper 来进行集群管理和容错。
- Alert 服务，提供告警相关服务。
- API 接口层，主要负责处理前端 UI 层的请求。
- UI，系统的前端页面，提供系统的各种可视化操作界面。
二、DolphinScheduler 部署说明
2.1、软硬件环境要求
2.1.1、操作系统版本要求
操作系统 版本
Red Hat Enterprise Linux 7.0 及以上
CentOS 7.0 及以上
Oracle Enterprise Linux 7.0 及以上
Ubuntu LTS 16.04 及以上
2.1.2 服务器硬件要求
CPU 内存 硬盘类型 网络 实例数量
4 核+ 8 GB+ SAS 千兆网卡 1+
2.2 部署模式
DolphinScheduler 支持多种部署模式，包括单机模式（Standalone）、伪集群模式（Pseudo Cluster）、集群模式（Cluster）等。
2.2.1 单机模式
单机模式（standalone）模式下，所有服务均集中于一个 StandaloneServer 进程中，并且其中内置了注册中心 Zookeeper 和数据库 H2。只需配置 JDK 环境，就可一键启动DolphinScheduler，快速体验其功能。
2.2.2 伪集群模式
伪集群模式（Pseudo-Cluster）是在单台机器部署 DolphinScheduler 各项服务，该模式下 master、worker、api server、logger server 等服务都只在同一台机器上。Zookeeper 和数据库需单独安装并进行相应配置。
2.2.3 集群模式
集群模式（Cluster）与伪集群模式的区别就是在多台机器部署 DolphinScheduler 各项服务，并且 Master、Worker 等服务可配置多个。
三、DolphinScheduler 集群模式部署
3.1 集群规划
集群模式下，可配置多个 Master 及多个 Worker。通常可配置 2~3 个 Master，若干个Worker。由于集群资源有限，此处配置一个 Master，三个 Worker，集群规划如下。
hadoop102 master、worker
hadoop103 worker
hadoop104 worker
3.2 前置准备工作
- （1）三台节点均需部署 JDK（1.8+），并配置相关环境变量。
- （2）需部署数据库，支持 MySQL（5.7+）或者 PostgreSQL（8.2.15+）。
- （3）需部署 Zookeeper（3.4.6+）。
- （4）三台节点均需安装进程树分析工具 psmisc。
[hadoop102 ~]$ sudo yum install -y psmisc
[hadoop103 ~]$ sudo yum install -y psmisc
[hadoop104 ~]$ sudo yum install -y psmisc
3.3 解压 DolphinScheduler 安装包
- （1）上传 DolphinScheduler 安装包到 hadoop102 节点的/opt/software 目录
- （2）解压安装包到当前目录
注：解压目录并非最终的安装目录
[hadoop102 software]$ tar -zxvf apache-dolphinscheduler-2.0.5-bin
3.4 创建元数据库及用户
DolphinScheduler 元数据存储在关系型数据库中，故需创建相应的数据库和用户。
（1）创建数据库
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
（2）创建用户
mysql> CREATE USER 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';
注： 若出现以下错误信息，表明新建用户的密码过于简单。
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
可提高密码复杂度或者执行以下命令降低 MySQL 密码强度级别。
mysql> set global validate_password_policy=0; mysql> set global validate_password_length=4;
（3）赋予用户相应权限
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%';
mysql> flush privileges;
3.5 配置一键部署脚本
修改解压目录下的 conf/config 目录下的 install_config.conf 文件。
[hadoop102 apache-dolphinscheduler-2.0.5-bin]$ vim conf/config/install_config.conf
修改内容如下。
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version
2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed
DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudodistributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs:
ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="hadoop102,hadoop103,hadoop104"
# 将要部署任一 DolphinScheduler 服务的服务器主机名或 ip 列表
# Port of SSH protocol, default value is 22. For now we only support
same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed
Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs:
masters="192.168.8.1,192.168.8.2"
masters="hadoop102"
# master 所在主机名列表，必须是 ips 的子集
# A comma separated list of machine <hostname>:<workerGroup> or
<IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as
`default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default",
Example for IPs:
workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="hadoop102:default,hadoop103:default,hadoop104:default"
# worker 主机名及队列，此处的 ip 必须在 ips 列表中
# A comma separated list of machine hostname or IP would be installed
Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP:
alertServer="192.168.8.3"
alertServer="hadoop102"
# 告警服务所在服务器主机名
# A comma separated list of machine hostname or IP would be installed
API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP:
apiServers="192.168.8.1"
apiServers="hadoop102"
# api 服务所在服务器主机名
# A comma separated list of machine hostname or IP would be installed
Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP:
pythonGatewayServers="192.168.8.1"
# pythonGatewayServers="ds1"
# 不需要的配置项，可以保留默认值，也可以用 # 注释
# The directory to install DolphinScheduler for all machine we config
above. It will automatically be created by `install.sh` script if not
exists.
# Do not set this configuration same as the current path (pwd)
installPath="/opt/module/dolphinscheduler"
# DS 安装路径，如果不存在会创建
# The user to deploy DolphinScheduler for all machine we config above.
For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to
operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="atguigu"
# 部署用户，任务执行服务是以 sudo -u {linux-user} 切换不同 Linux 用户的方式来
实现多租户运行作业，因此该用户必须有免密的 sudo 权限。
# The directory to store local data for all machine we config above.
Make sure user `deployUser` have permissions to read and write this
directory.
dataBasedirPath="/tmp/dolphinscheduler"
# 前文配置的所有节点的本地数据存储路径，需要确保部署用户拥有该目录的读写权限
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going
to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/module/jdk1.8.0_212"
# JAVA_HOME 路径
# DolphinScheduler API service port, also this is your DolphinScheduler
UI component's URL port, default value is 12345
apiServerPort="12345"
# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as
`.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# 注意：数据库相关配置的 value 必须加引号，否则配置无法生效
DATABASE_TYPE="mysql"
# 数据库类型
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter>
format, If you using mysql, you could use jdbc
# string
jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characte
rEncoding=UTF-8 as example
# SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-
"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-
1;DATABASE_TO_LOWER=true"}
SPRING_DATASOURCE_URL="jdbc:mysql://hadoop102:3306/dolphinscheduler?u
seUnicode=true&characterEncoding=UTF-8"
# 数据库 URL
# Spring datasource username
# SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"sa"}
SPRING_DATASOURCE_USERNAME="dolphinscheduler"
# 数据库用户名
# Spring datasource password
# SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-""}
SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
# 数据库密码
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of
`registryPluginDir`, DolphinScheduler use this for verifying
configuration consistency
registryPluginName="zookeeper"
# 注册中心插件名称，DS 通过注册中心来确保集群配置的一致性
# Registry Server address.
registryServers="hadoop102:2181,hadoop103:2181,hadoop104:2181"
# 注册中心地址，即 Zookeeper 集群的地址
# Registry Namespace
registryNamespace="dolphinscheduler"
# DS 在 Zookeeper 的结点名称
# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load
the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="HDFS"
# 资源存储类型
# resource store on HDFS/S3 path, resource file will store to this
hdfs path, self configuration, please make sure the directory exists
on hdfs and has read write permissions. "/dolphinscheduler" is
recommended
resourceUploadPath="/dolphinscheduler"
# 资源上传路径
# if resourceStorageType is HDFS，defaultFS write namenode address，HA,
you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://hadoop102:8020"
# 默认文件系统
# if resourceStorageType is S3, the following three configuration is
required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# yarn RM http 访问端口
# if resourcemanager HA is enabled, please set the HA IPs; if
resourcemanager is single node, keep this value empty
yarnHaIps=
# Yarn RM 高可用 ip，若未启用 RM 高可用，则将该值置空
# if resourcemanager HA is enabled or not use resourcemanager, please
keep the default value; If resourcemanager is single node, you only
need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="hadoop103"
# Yarn RM 主机名，若启用了 HA 或未启用 RM，保留默认值
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="atguigu"
# 拥有 HDFS 根目录操作权限的用户
# kerberos config
# whether kerberos starts, if kerberos starts, following four items
need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# worker tenant auto create
workerTenantAutoCreate="false"
3.6、初始化数据库
（1）拷贝 MySQL 驱动到 DolphinScheduler 的解压目录下的 lib 中，要求使用 MySQL JDBC Driver 8.0.16。
[hadoop102 apache-dolphinscheduler-2.0.5-bin]$ cp /opt/software/mysql-connector-java-8.0.16.jar lib/
（2）执行数据库初始化脚本
数据库初始化脚本位于 DolphinScheduler 解 压 目 录 下 的 script 目 录 中 ， 即/opt/software/ds/apache-dolphinscheduler-2.0.5-bin/script/。
[hadoop102 apache-dolphinscheduler-2.0.5-bin]$ script/createdolphinscheduler.sh
3.7、一键部署 DolphinScheduler
（1）启动 Zookeeper 集群
[hadoop102 apache-dolphinscheduler-2.0.5-bin]$ zk.sh start
（2）一键部署并启动 DolphinScheduler
[hadoop102 apache-dolphinscheduler-2.0.5-bin]$ ./install.sh
（3）查看 DolphinScheduler 进程
--------- hadoop102 ----------
29139 ApiApplicationServer
28963 WorkerServer
3332 QuorumPeerMain
2100 DataNode
28902 MasterServer
29081 AlertServer
1978 NameNode
29018 LoggerServer
2493 NodeManager
29551 Jps
--------- hadoop103 ----------
29568 Jps
29315 WorkerServer
2149 NodeManager
1977 ResourceManager
2969 QuorumPeerMain
29372 LoggerServer
1903 DataNode
--------- hadoop104 ----------
1905 SecondaryNameNode
27074 WorkerServer
2050 NodeManager
2630 QuorumPeerMain
1817 DataNode
27354 Jps
27133 LoggerServer
（4）访问 DolphinScheduler UI
DolphinScheduler UI 地址为 http://hadoop102:12345/dolphinscheduler
初始用户的用户名为：admin，密码为 dolphinscheduler123
3.8、DolphinScheduler 启停命令
DolphinScheduler 的启停脚本均位于其安装目录的 bin 目录下。
1）一键启停所有服务
./bin/start-all.sh
./bin/stop-all.sh
注意同 Hadoop 的启停脚本进行区分。
2）启停 Master
./bin/dolphinscheduler-daemon.sh start master-server
./bin/dolphinscheduler-daemon.sh stop master-server
3）启停 Worker
./bin/dolphinscheduler-daemon.sh start worker-server
./bin/dolphinscheduler-daemon.sh stop worker-server
4）启停 Api
./bin/dolphinscheduler-daemon.sh start api-server
./bin/dolphinscheduler-daemon.sh stop api-server
5）启停 Logger
./bin/dolphinscheduler-daemon.sh start logger-server
./bin/dolphinscheduler-daemon.sh stop logger-server
6）启停 Alert
./bin/dolphinscheduler-daemon.sh start alert-server
./bin/dolphinscheduler-daemon.sh stop alert-server
评论