Skip to content

Commit c6585b4

Browse files
yaooqinnyaooqinn
committed
[KYUUBI #471] Separate Embedded Zookeeper server from ha
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #471](https://badgen.net/badge/Preview/Closes%20%23471/blue)](https://github.com/yaooqinn/kyuubi/pull/471) ![386](https://badgen.net/badge/%2B/386/red) ![152](https://badgen.net/badge/-/152/green) ![8](https://badgen.net/badge/commits/8/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one 2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage. 3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #471 from yaooqinn/qp. Closes #471 1c24d02 [Kent Yao] address comments 9666a40 [Kent Yao] refine f069fc0 [Kent Yao] refine 8bb2f3a [Kent Yao] refine c20593a [Kent Yao] refine 8fa9a40 [Kent Yao] Merge branch 'master' into qp 1e2e1a3 [Kent Yao] tmp 7d66d05 [Kent Yao] tmp Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]>
1 parent 0aafc55 commit c6585b4

File tree

21 files changed

+386
-152
lines changed

21 files changed

+386
-152
lines changed

‎dev/kyuubi-codecov/pom.xml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,12 @@
4444
<version>${project.version}</version>
4545
</dependency>
4646

47+
<dependency>
48+
<groupId>org.apache.kyuubi</groupId>
49+
<artifactId>kyuubi-zookeeper</artifactId>
50+
<version>${project.version}</version>
51+
</dependency>
52+
4753
<dependency>
4854
<groupId>org.apache.kyuubi</groupId>
4955
<artifactId>kyuubi-ha</artifactId>

‎docs/deployment/settings.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,8 +213,19 @@ kyuubi\.session<br>\.timeout|<div style='width: 65pt;word-wrap: break-word;white
213213

214214
Key | Default | Meaning | Type | Since
215215
--- | --- | --- | --- | ---
216+
kyuubi\.zookeeper<br>\.embedded\.client\.port|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>2181</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>clientPort for the embedded zookeeper server to listen for client connections, a client here could be Kyuubi server, engine and JDBC client</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
217+
kyuubi\.zookeeper<br>\.embedded\.client\.port<br>\.address|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>clientPortAddress for the embedded zookeeper server to</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
218+
kyuubi\.zookeeper<br>\.embedded\.data\.dir|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>embedded_zookeeper</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>dataDir for the embedded zookeeper server where stores the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
219+
kyuubi\.zookeeper<br>\.embedded\.data\.log\.dir|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>embedded_zookeeper</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>dataLogDir for the embedded zookeeper server where writes the transaction log .</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
216220
kyuubi\.zookeeper<br>\.embedded\.directory|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>embedded_zookeeper</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The temporary directory for the embedded zookeeper server</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.0.0</div>
221+
kyuubi\.zookeeper<br>\.embedded\.election<br>\.port|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>0</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>electionPort for the embedded zookeeper server</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
222+
kyuubi\.zookeeper<br>\.embedded\.max\.client<br>\.connections|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>120</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>maxClientCnxns for the embedded zookeeper server to limits the number of concurrent connections of a single client identified by IP address</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
223+
kyuubi\.zookeeper<br>\.embedded\.max\.session<br>\.timeout|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>maxSessionTimeout in milliseconds for the embedded zookeeper server will allow the client to negotiate. Defaults to 20 times the tickTime</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
224+
kyuubi\.zookeeper<br>\.embedded\.min\.session<br>\.timeout|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>minSessionTimeout in milliseconds for the embedded zookeeper server will allow the client to negotiate. Defaults to 2 times the tickTime</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
217225
kyuubi\.zookeeper<br>\.embedded\.port|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>2181</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The port of the embedded zookeeper server</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
226+
kyuubi\.zookeeper<br>\.embedded\.quorum\.port|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>0</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>quorumPort for the embedded zookeeper server</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
227+
kyuubi\.zookeeper<br>\.embedded\.server\.id|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>-1</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>serverId for the embedded zookeeper server</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
228+
kyuubi\.zookeeper<br>\.embedded\.tick\.time|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>3000</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>tickTime in milliseconds for the embedded zookeeper server</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.2.0</div>
218229

219230
## Spark Configurations
220231

‎externals/kyuubi-spark-sql-engine/pom.xml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,13 @@
145145
<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
146146
<scope>test</scope>
147147
</dependency>
148+
149+
<dependency>
150+
<groupId>org.apache.kyuubi</groupId>
151+
<artifactId>kyuubi-zookeeper</artifactId>
152+
<version>${project.version}</version>
153+
<scope>test</scope>
154+
</dependency>
148155
</dependencies>
149156

150157
<build>

‎externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/WithDiscoverySparkSQLEngine.scala‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,12 @@ import org.apache.curator.framework.CuratorFramework
2121

2222
import org.apache.kyuubi.Utils
2323
import org.apache.kyuubi.config.KyuubiConf
24-
import org.apache.kyuubi.config.KyuubiConf.{EMBEDDED_ZK_PORT, EMBEDDED_ZK_TEMP_DIR}
2524
import org.apache.kyuubi.ha.HighAvailabilityConf.{HA_ZK_ACL_ENABLED, HA_ZK_NAMESPACE, HA_ZK_QUORUM}
2625
import org.apache.kyuubi.ha.client.ServiceDiscovery
27-
import org.apache.kyuubi.ha.server.EmbeddedZkServer
26+
import org.apache.kyuubi.zookeeper.{EmbeddedZookeeper, ZookeeperConf}
2827

2928
trait WithDiscoverySparkSQLEngine extends WithSparkSQLEngine {
30-
private var zkServer: EmbeddedZkServer = _
29+
private var zkServer: EmbeddedZookeeper = _
3130
def namespace: String
3231
override def withKyuubiConf: Map[String, String] = {
3332
assert(zkServer != null)
@@ -37,11 +36,11 @@ trait WithDiscoverySparkSQLEngine extends WithSparkSQLEngine {
3736
}
3837

3938
override def beforeAll(): Unit = {
40-
zkServer = new EmbeddedZkServer()
39+
zkServer = new EmbeddedZookeeper()
4140
val zkData = Utils.createTempDir()
4241
val tmpConf = KyuubiConf()
43-
tmpConf.set(EMBEDDED_ZK_PORT, -1)
44-
tmpConf.set(EMBEDDED_ZK_TEMP_DIR, zkData.toString)
42+
tmpConf.set(ZookeeperConf.ZK_CLIENT_PORT, -1)
43+
tmpConf.set(ZookeeperConf.ZK_DATA_DIR, zkData.toString)
4544
zkServer.initialize(tmpConf)
4645
zkServer.start()
4746
}

‎kyuubi-common/pom.xml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@
4646
<dependency>
4747
<groupId>org.slf4j</groupId>
4848
<artifactId>slf4j-log4j12</artifactId>
49-
<scope>provided</scope>
5049
</dependency>
5150

5251
<dependency>

‎kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,6 @@ case class KyuubiConf(loadSysDefault: Boolean = true) extends Logging {
122122
}
123123

124124
private val serverOnlyConfEntries: Set[ConfigEntry[_]] = Set(
125-
EMBEDDED_ZK_PORT,
126-
EMBEDDED_ZK_TEMP_DIR,
127125
FRONTEND_BIND_HOST,
128126
FRONTEND_BIND_PORT,
129127
AUTHENTICATION_METHOD,
@@ -192,18 +190,6 @@ object KyuubiConf {
192190
ConfigBuilder("kyuubi." + key).onCreate(register)
193191
}
194192

195-
val EMBEDDED_ZK_PORT: ConfigEntry[Int] = buildConf("zookeeper.embedded.port")
196-
.doc("The port of the embedded zookeeper server")
197-
.version("1.0.0")
198-
.intConf
199-
.createWithDefault(2181)
200-
201-
val EMBEDDED_ZK_TEMP_DIR: ConfigEntry[String] = buildConf("zookeeper.embedded.directory")
202-
.doc("The temporary directory for the embedded zookeeper server")
203-
.version("1.0.0")
204-
.stringConf
205-
.createWithDefault("embedded_zookeeper")
206-
207193
val SERVER_PRINCIPAL: OptionalConfigEntry[String] = buildConf("kinit.principal")
208194
.doc("Name of the Kerberos principal.")
209195
.version("1.0.0")

‎kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@ class KyuubiConfSuite extends KyuubiFunSuite {
2727

2828
test("kyuubi conf defaults") {
2929
val conf = new KyuubiConf()
30-
assert(conf.get(EMBEDDED_ZK_PORT) === 2181)
31-
assert(conf.get(EMBEDDED_ZK_TEMP_DIR).endsWith("embedded_zookeeper"))
30+
assert(conf.get(SERVER_PRINCIPAL) === None)
31+
assert(conf.get(KINIT_MAX_ATTEMPTS) === 10)
3232
assert(conf.get(OPERATION_IDLE_TIMEOUT) === Duration.ofHours(3).toMillis)
3333
}
3434

@@ -45,7 +45,6 @@ class KyuubiConfSuite extends KyuubiFunSuite {
4545
assert(conf.getOption("spark.kyuubi.yes").get === "no")
4646
}
4747

48-
4948
test("set and unset conf") {
5049
val conf = new KyuubiConf()
5150

@@ -62,11 +61,11 @@ class KyuubiConfSuite extends KyuubiFunSuite {
6261
conf.setIfMissing(OPERATION_IDLE_TIMEOUT, 60L)
6362
assert(conf.get(OPERATION_IDLE_TIMEOUT) === 5)
6463

65-
conf.setIfMissing(EMBEDDED_ZK_PORT, 2188)
66-
assert(conf.get(EMBEDDED_ZK_PORT) === 2188)
64+
conf.setIfMissing(FRONTEND_MIN_WORKER_THREADS, 2188)
65+
assert(conf.get(FRONTEND_MIN_WORKER_THREADS) === 2188)
6766

68-
conf.unset(EMBEDDED_ZK_PORT)
69-
assert(conf.get(EMBEDDED_ZK_PORT) === 2181)
67+
conf.unset(FRONTEND_MIN_WORKER_THREADS)
68+
assert(conf.get(FRONTEND_MIN_WORKER_THREADS) === 9)
7069

7170
conf.unset(key)
7271
assert(conf.getOption(key).isEmpty)

‎kyuubi-ha/pom.xml‎

Lines changed: 7 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -38,18 +38,6 @@
3838
<version>${project.version}</version>
3939
</dependency>
4040

41-
<dependency>
42-
<groupId>org.apache.hadoop</groupId>
43-
<artifactId>hadoop-client-api</artifactId>
44-
<scope>provided</scope>
45-
</dependency>
46-
47-
<dependency>
48-
<groupId>org.apache.hadoop</groupId>
49-
<artifactId>hadoop-client-runtime</artifactId>
50-
<scope>provided</scope>
51-
</dependency>
52-
5341
<dependency>
5442
<groupId>org.apache.curator</groupId>
5543
<artifactId>curator-framework</artifactId>
@@ -65,14 +53,6 @@
6553
<artifactId>zookeeper</artifactId>
6654
</dependency>
6755

68-
<!-- Begin: for EmbeddedZkServer -->
69-
<dependency>
70-
<groupId>org.apache.curator</groupId>
71-
<artifactId>curator-test</artifactId>
72-
<scope>provided</scope>
73-
</dependency>
74-
<!-- End: for EmbeddedZkServer -->
75-
7656
<dependency>
7757
<groupId>org.apache.kyuubi</groupId>
7858
<artifactId>kyuubi-common</artifactId>
@@ -92,6 +72,13 @@
9272
<artifactId>apacheds-service</artifactId>
9373
<scope>test</scope>
9474
</dependency>
75+
76+
<dependency>
77+
<groupId>org.apache.kyuubi</groupId>
78+
<artifactId>kyuubi-zookeeper</artifactId>
79+
<version>${project.version}</version>
80+
<scope>test</scope>
81+
</dependency>
9582
</dependencies>
9683

9784
<build>

‎kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/server/EmbeddedZkServer.scala‎

Lines changed: 0 additions & 80 deletions
This file was deleted.

‎kyuubi-ha/src/test/scala/org/apache/kyuubi/ha/client/ServiceDiscoverySuite.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,15 @@ import org.apache.kyuubi.{KerberizedTestHelper, KYUUBI_VERSION}
2929
import org.apache.kyuubi.config.KyuubiConf
3030
import org.apache.kyuubi.ha.HighAvailabilityConf
3131
import org.apache.kyuubi.ha.HighAvailabilityConf._
32-
import org.apache.kyuubi.ha.server.EmbeddedZkServer
3332
import org.apache.kyuubi.service.{NoopServer, Serverable, ServiceState}
33+
import org.apache.kyuubi.zookeeper.{EmbeddedZookeeper, ZookeeperConf}
3434

3535
class ServiceDiscoverySuite extends KerberizedTestHelper {
36-
val zkServer = new EmbeddedZkServer()
36+
val zkServer = new EmbeddedZookeeper()
3737
val conf: KyuubiConf = KyuubiConf()
3838

3939
override def beforeAll(): Unit = {
40-
conf.set(KyuubiConf.EMBEDDED_ZK_PORT, 0)
40+
conf.set(ZookeeperConf.ZK_CLIENT_PORT, 0)
4141
zkServer.initialize(conf)
4242
zkServer.start()
4343
super.beforeAll()

0 commit comments

Comments
 (0)