diff --git a/.gitignore b/.gitignore index 49950ab..b8d69e5 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,8 @@ .env .style *.lint +scripts/downscaling/crate +scripts/downscaling/dist/* +scripts/downscaling/data/* +scripts/downscaling/repo/* +scripts/downscaling/logs.csv diff --git a/docs/downscaling/README.rst b/docs/downscaling/README.rst new file mode 100644 index 0000000..1958d44 --- /dev/null +++ b/docs/downscaling/README.rst @@ -0,0 +1,215 @@ +=========== +Downscaling +=========== + +In this tutorial we: + +- Create a Vanilla cluster. +- Add some data to it. +- Downscale it to a single node cluster. + + +Starting a Vanilla cluster +-------------------------- + +A Vanilla cluster is a three node cluster that runs on a single host. Each of the +nodes share the file system and operating system scheduler of the host, to form a +cluster. This rig provides parallel processing power on large scale data, when you +only have one host, and this comes at the cost increased latency in writes. + +Proceed: + +1. *~/.../crate-tutorials/scripts/downscaling* should contain: + + - *update-dist*: script to install **CrateDB**. + - *dist*: **CrateDB** distributions. + - *crate*: a symlink to a particular distribution in the *dist* folder (not + the **CrateDB**, executable script), where you will also find a *crate-clone* + git repository. + - *conf*: **CrateDB** configurations, each node in the cluster has a folder + in there, with the *crate.yml* and *log4j2.properties*. + - *data*: **CrateDB** the nodes will persist their data under *data/ni/nodes/0*. + - *repo*: **CrateDB** repo, for snapshotting. + - *start-node*: script to start **CrateDB** with a given configuration. + - *detach-node*: script to detach a node from the cluster. + - *bootstrap-node*: script to bootstrap a node to form a new cluster. Which + means, recreating its cluster state so that it may be started on its own + forming a new cluster that has access to the data in the previous. + - *data.py*: script produce dummy data. + +2. Run *./update-dist* + + - This will install the latest, unreleased, **CrateDB** under *dist/*, creating + a link *./crate -> dist/crate..*. + - Assumed **git**, **java 11** or later, **python3** and a **terminal** are + available to you, and you have an account in GitHub_. + +3. The configuration for the Vanilla cluster: + + - *~/.../crate-tutorials/scripts/downscaling/conf/n1/crate.yml* + - *~/.../crate-tutorials/scripts/downscaling/conf/n2/crate.yml* + - *~/.../crate-tutorials/scripts/downscaling/conf/n3/crate.yml* + - *~/.../crate-tutorials/scripts/downscaling/conf/log4j2.properties* + +4. Run *./startnode* in three different terminals + + - *./startnode n1* + - *./startnode n2* + - *./startnode n3* + + Which will form the Vanilla cluster, electing a master. You can + interact with the Vanilla cluster by opening a browser and pointing + it to *http://localhost:4200*, *CrateDB*'s `Admin UI`_. + + +Adding some data to the cluster +------------------------------- + +Proceed: + +1. Produce a CSV_ file containing 3600 rows of log data (1 hour's worth of logs @1Hz): + + :: + + ./data.py > logs.csv + +2. In the `Admin UI`_: + + :: + + CREATE TABLE logs (log_time timestamp NOT NULL, + client_ip ip NOT NULL, + request string NOT NULL, + status_code short NOT NULL, + object_size long NOT NULL); + + COPY logs FROM 'file:///.../crate-tutorials/scripts/downscaling/logs.csv'; + REFRESH TABLE logs; + select * from logs order by log_time limit 10800; + + The three nodes perform the copy, so we are expecting to see 3600 * 3 rows, with + what looks like "repeated" data. Because we did not define a primary key, **CrateDB** + created the default *_id* primary key, which is a unique hash (varchar), so in effect, + because each row has a unique id, they are all inserted. + + +Exploring the Data +------------------ + +Using the `Admin UI`_, shards view on the left: + +.. image:: imgs/shards-view.png + +We can see the three nodes, each having a number of shards, specifically: + + +-------+---+---+---+---+---+---+ + | Shard | 0 | 1 | 2 | 3 | 4 | 5 | + +=======+===+===+===+===+===+===+ + | n1 | . | . | . | | . | | + +-------+---+---+---+---+---+---+ + | n2 | . | . | | . | | . | + +-------+---+---+---+---+---+---+ + | n3 | | | . | . | . | . | + +-------+---+---+---+---+---+---+ + +Thus in this cluster setup, one node can crash, yet the data in the cluster +will still remain fully available because any two nodes have access to all +the shards, when they work together to fulfill query requests. A SQL table +is a composite of shards, six in our case. When a query is executed, the +planner will define steps for accessing all the shards of the table. +By adding nodes to the cluster, the data is spread over more nodes, so that +the computing is parallelized. + +Having a look at the setup for table *logs*: + +:: + + SHOW CREATE TABLE logs; + +Will return: + +:: + + CREATE TABLE IF NOT EXISTS "doc"."logs" ( + "log_time" TIMESTAMP WITH TIME ZONE NOT NULL, + "client_ip" IP NOT NULL, + "request" TEXT NOT NULL, + "status_code" SMALLINT NOT NULL, + "object_size" BIGINT NOT NULL + ) + CLUSTERED INTO 6 SHARDS + WITH ( + ... + number_of_replicas = '0-1', + ... + ) + +We have a default min number of replicas of zero, and a max of one for each +of our six shards. A replica is simply a copy or a shard. + + +Downscaling (by means of replicas) +---------------------------------- + +Downscaling by means of replicas is achieved by making sure the surviving nodes +of the cluster have access to all the shards, even when the other nodes are missing. + +1. We need to ensure that the number of replicas matches the number of nodes: + +:: + + ALTER TABLE logs SET (number_of_replicas = '1-all'); + +In the `Admin UI`_, we can follow the progress of replication. + +2. After replication is completed, we can take down all the nodes in the cluster + (*ctrl^C* in the terminal). + +3. Run *./detach-node ni*, where i in [2,3], to detach **n2** and **n3** from the cluster. + We will let **n1** form a new cluster all by itself, with access to the original data. + +4. Change **n1**'s configuration *crate.yml*. The best practice is to select the node + that was master, as then we know it has the latest version of the cluster state. For + our tutorial, we are running in a single host so cluster state is more or less + guaranteed to be consistent across nodes, but, in principle, the cluster could be + running across multiple hosts, and then we would want the master node to become the + new single node cluster: + + :: + + cluster.name: simple # don't need to change this + node.name: n1 + stats.service.interval: 0 + network.host: _local_ + node.max_local_storage_nodes: 1 + + http.cors.enabled: true + http.cors.allow-origin: "*" + + transport.tcp.port: 4301 + #gateway.expected_nodes: 3 + #gateway.recover_after_nodes: 2 + #discovery.seed_hosts: + # - 127.0.0.1:4301 + # - 127.0.0.1:4302 + #cluster.initial_master_nodes: + # - 127.0.0.1:4301 + # - 127.0.0.1:4302 + +5. Run *./bootstrap-node n1* to let **n1** join a new cluster when it starts. + +6. Run *./start-node n1*. + Panic not, the cluster state is *[YELLOW]*, we sort that out with: + + :: + + ALTER TABLE logs SET (number_of_replicas = '0-1'); + +Further reading: crate-node_. + + +.. _GitHub: https://github.com/crate/crate.git +.. _`Admin UI`: http://localhost:4200 +.. _crate-node: https://crate.io/docs/crate/reference/en/latest/cli-tools.html#cli-crate-node +.. _CSV: https://en.wikipedia.org/wiki/Comma-separated_values +.. _crate-node: https://crate.io/docs/crate/guide/en/latest/best-practices/crate-node.html \ No newline at end of file diff --git a/docs/downscaling/imgs/shards-view.png b/docs/downscaling/imgs/shards-view.png new file mode 100644 index 0000000..e4245a1 Binary files /dev/null and b/docs/downscaling/imgs/shards-view.png differ diff --git a/scripts/downscaling/bootstrap-node b/scripts/downscaling/bootstrap-node new file mode 100755 index 0000000..1f110d3 --- /dev/null +++ b/scripts/downscaling/bootstrap-node @@ -0,0 +1,9 @@ +#!/bin/sh + +source common.sh + +./crate/bin/crate-node unsafe-bootstrap \ + -Cpath.home=$path_home \ + -Cpath.conf=$path_conf \ + -Cpath.data=$path_data \ + -Cpath.repo=$path_repo diff --git a/scripts/downscaling/common.sh b/scripts/downscaling/common.sh new file mode 100755 index 0000000..60b19c7 --- /dev/null +++ b/scripts/downscaling/common.sh @@ -0,0 +1,30 @@ +#!/bin/sh + +display_usage_and_exit() { + echo "usage: $0 node_name" + exit 1 +} + +if [ $# -lt 1 ]; then + display_usage_and_exit +fi + +node_name=$1 +path_conf="$(pwd)/conf/$node_name" +if [ ! -d $path_conf ]; then + echo "No configuration available in [conf/$node_name]." + exit 1 +fi +path_home=$(pwd)/crate +path_data=$(pwd)/data/$node_name +path_repo=$(pwd)/repo + +if [ -z "$CRATE_HEAP_SIZE" ]; then + export CRATE_HEAP_SIZE="2G" +fi +echo 'setup: ' +echo " - path_home: $path_home" +echo " - node_name: $node_name" +echo " - path_data: $path_data" +echo " - path_repo: $path_repo" +echo " - CRATE_HEAP_SIZE: $CRATE_HEAP_SIZE" diff --git a/scripts/downscaling/conf/log4j2.properties b/scripts/downscaling/conf/log4j2.properties new file mode 100644 index 0000000..c8daed4 --- /dev/null +++ b/scripts/downscaling/conf/log4j2.properties @@ -0,0 +1,13 @@ +status = error + +rootLogger.level = info +rootLogger.appenderRef.console.ref = console + +# log action execution errors for easier debugging +logger.action.name = org.crate.action.sql +logger.action.level = debug + +appender.console.type = Console +appender.console.name = console +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name] %marker%m%n diff --git a/scripts/downscaling/conf/n1/crate.yml b/scripts/downscaling/conf/n1/crate.yml new file mode 100644 index 0000000..9b01240 --- /dev/null +++ b/scripts/downscaling/conf/n1/crate.yml @@ -0,0 +1,18 @@ +cluster.name: vanilla +node.name: n1 +stats.service.interval: 0 +network.host: _local_ +node.max_local_storage_nodes: 1 + +http.cors.enabled: true +http.cors.allow-origin: "*" + +transport.tcp.port: 4301 +gateway.expected_nodes: 3 +gateway.recover_after_nodes: 2 +discovery.seed_hosts: +- 127.0.0.1:4301 +- 127.0.0.1:4302 +cluster.initial_master_nodes: +- 127.0.0.1:4301 +- 127.0.0.1:4302 diff --git a/scripts/downscaling/conf/n1/crate.yml.short b/scripts/downscaling/conf/n1/crate.yml.short new file mode 100644 index 0000000..14ca126 --- /dev/null +++ b/scripts/downscaling/conf/n1/crate.yml.short @@ -0,0 +1,10 @@ +cluster.name: vanilla +node.name: n1 +stats.service.interval: 0 +network.host: _local_ +node.max_local_storage_nodes: 1 + +http.cors.enabled: true +http.cors.allow-origin: "*" + +transport.tcp.port: 4301 diff --git a/scripts/downscaling/conf/n1/log4j2.properties b/scripts/downscaling/conf/n1/log4j2.properties new file mode 120000 index 0000000..2a55cac --- /dev/null +++ b/scripts/downscaling/conf/n1/log4j2.properties @@ -0,0 +1 @@ +../log4j2.properties \ No newline at end of file diff --git a/scripts/downscaling/conf/n2/crate.yml b/scripts/downscaling/conf/n2/crate.yml new file mode 100644 index 0000000..bfa6f39 --- /dev/null +++ b/scripts/downscaling/conf/n2/crate.yml @@ -0,0 +1,18 @@ +cluster.name: vanilla +node.name: n2 +stats.service.interval: 0 +network.host: _local_ +node.max_local_storage_nodes: 1 + +http.cors.enabled: true +http.cors.allow-origin: "*" + +transport.tcp.port: 4302 +gateway.expected_nodes: 3 +gateway.recover_after_nodes: 2 +discovery.seed_hosts: +- 127.0.0.1:4301 +- 127.0.0.1:4302 +cluster.initial_master_nodes: +- 127.0.0.1:4301 +- 127.0.0.1:4302 diff --git a/scripts/downscaling/conf/n2/log4j2.properties b/scripts/downscaling/conf/n2/log4j2.properties new file mode 120000 index 0000000..2a55cac --- /dev/null +++ b/scripts/downscaling/conf/n2/log4j2.properties @@ -0,0 +1 @@ +../log4j2.properties \ No newline at end of file diff --git a/scripts/downscaling/conf/n3/crate.yml b/scripts/downscaling/conf/n3/crate.yml new file mode 100644 index 0000000..24bf3d7 --- /dev/null +++ b/scripts/downscaling/conf/n3/crate.yml @@ -0,0 +1,18 @@ +cluster.name: vanilla +node.name: n3 +stats.service.interval: 0 +network.host: _local_ +node.max_local_storage_nodes: 1 + +http.cors.enabled: true +http.cors.allow-origin: "*" + +transport.tcp.port: 4303 +gateway.expected_nodes: 3 +gateway.recover_after_nodes: 2 +discovery.seed_hosts: +- 127.0.0.1:4301 +- 127.0.0.1:4302 +cluster.initial_master_nodes: +- 127.0.0.1:4301 +- 127.0.0.1:4302 diff --git a/scripts/downscaling/conf/n3/log4j2.properties b/scripts/downscaling/conf/n3/log4j2.properties new file mode 120000 index 0000000..2a55cac --- /dev/null +++ b/scripts/downscaling/conf/n3/log4j2.properties @@ -0,0 +1 @@ +../log4j2.properties \ No newline at end of file diff --git a/scripts/downscaling/data.py b/scripts/downscaling/data.py new file mode 100755 index 0000000..65ed31a --- /dev/null +++ b/scripts/downscaling/data.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 + +import random +import string +import ipaddress +import time + + +# to achieve log lines as in: +# 2012-01-01T00:00:00Z,25.152.171.147,/crate/Five_Easy_Pieces.html,200,280278 +# -> timestamp, +# -> random ip address, +# -> random request (a path), +# -> random status code, +# -> random object size, + + +def timestamp_range(start, end, format): + st = int(time.mktime(time.strptime(start, format))) + et = int(time.mktime(time.strptime(end, format))) + dt = 1 # 1 sec + fmt = lambda x: time.strftime(format, time.localtime(x)) + return (fmt(x) for x in range(st, et, dt)) + + +def rand_ip(): + return str(ipaddress.IPv4Address(random.getrandbits(32))) + + +def rand_request(): + rand = lambda src: src[random.randint(0, len(src) - 1)] + path = lambda: "/".join((rand(("usr", "bin", "workspace", "temp", "home", "crate"))) for _ in range(4)) + name = lambda: ''.join(random.sample(string.ascii_lowercase, 7)) + ext = lambda: rand(("html", "pdf", "log", "gif", "jpeg", "js")) + return "{}/{}.{}".format(path(), name(), ext()) + + +def rand_object_size(): + return str(random.randint(0, 1024)) + + +def rand_status_code(): + return str(random.randint(100, 500)) + + +if __name__ == "__main__": + print("log_time,client_ip,request,status_code,object_size") + for ts in timestamp_range("2019-01-01T00:00:00Z", "2019-01-01T01:00:00Z", '%Y-%m-%dT%H:%M:%SZ'): + print(",".join([ts, rand_ip(), rand_request(), rand_status_code(), rand_object_size()])) diff --git a/scripts/downscaling/detach-node b/scripts/downscaling/detach-node new file mode 100755 index 0000000..3e58941 --- /dev/null +++ b/scripts/downscaling/detach-node @@ -0,0 +1,9 @@ +#!/bin/sh + +source common.sh + +./crate/bin/crate-node detach-cluster \ + -Cpath.home=$path_home \ + -Cpath.conf=$path_conf \ + -Cpath.data=$path_data \ + -Cpath.repo=$path_repo diff --git a/scripts/downscaling/start-node b/scripts/downscaling/start-node new file mode 100755 index 0000000..65e4b69 --- /dev/null +++ b/scripts/downscaling/start-node @@ -0,0 +1,9 @@ +#!/bin/sh + +source common.sh + +./crate/bin/crate \ + -Cpath.home=$(pwd)/crate \ + -Cpath.conf=$path_conf \ + -Cpath.data=$path_data \ + -Cpath.repo=$path_repo diff --git a/scripts/downscaling/update-dist b/scripts/downscaling/update-dist new file mode 100755 index 0000000..38f84d3 --- /dev/null +++ b/scripts/downscaling/update-dist @@ -0,0 +1,26 @@ +#!/bin/sh + +if [ ! -d dist ]; then + mkdir dist +fi + +if [ ! -d dist/crate-clone ]; then + git clone https://github.com/crate/crate.git dist/crate-clone +fi + +cd dist/crate-clone +git pull +./gradlew clean +./gradlew disTar + +latest_tar_ball=$(find ./app/build/distributions -name 'crate-*.tar.gz') +cp $latest_tar_ball .. +cd .. +name=$(basename $latest_tar_ball) +tar -xzvf $name +rm -f $name +name=${name/.tar.gz/} +cd .. +rm -f crate +ln -s dist/$name crate +echo "Crate $name has been installed."