Opensource ktq

.dockerignore

@@ -0,0 +1,2 @@
+.git
+frontend/node_modules

.gitignore

@@ -0,0 +1,93 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# IPython Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# dotenv
+.env
+
+# virtualenv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+
+# Rope project settings
+.ropeproject
+
+.remote-sync.json
+dev_environment.sh
+frontend/node_modules

Dockerfile

@@ -0,0 +1,21 @@
+FROM ubuntu:16.04
+
+RUN apt-get update && apt-get install -y apt-transport-https ca-certificates
+
+RUN echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial main restricted universe multiverse" > /etc/apt/sources.list && \
+    echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
+    echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
+    echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security main restricted universe multiverse" >> /etc/apt/sources.list
+
+RUN apt-get update
+RUN apt-get install -y wget python3.5 python3-pip git
+
+RUN python3.5 -m pip install kubernetes tornado aiohttp pymongo --ignore-installed -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+ADD . /ktqueue
+WORKDIR /ktqueue
+
+RUN python3.5 -m pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
+CMD python3 /ktqueue/server.py
+
+EXPOSE 8080

README.md

@@ -1 +1,27 @@
-# ktq
+# KTQueue
+
+kubernetes task queue with GPU support
+
+# Features
+
+- support GPU tasks
+- support assigning the task to node manually
+- realtime logs on webpage & log version management
+- mount host-path to Pod manually
+- Tensorboard manage & proxy
+- git clone repository with ssh-key or username & password or Github OAuth
+- CPU & Memory limit supported
+
+# screenshoot
+
+![screenshoot](https://user-images.githubusercontent.com/1068203/28708229-10e6e19e-73ae-11e7-882f-f4fb6bff877a.png)
+
+# How to deploy
+
+deployment guide under [deploy](./docs/deploy) directory
+
+# How to build images for KTQueue
+
+You can use any framework you want as long as you have the correct docker image, here are examples to build docker image for KTQueue
+
+- [tensorflow](./docs/docker_image_example/tensorflow)

docs/deploy/.gitignore

@@ -0,0 +1 @@
+dep*.yaml

docs/deploy/NODE.md

@@ -0,0 +1,42 @@
+# How to setup a KTQueue node
+
+## Brief
+
+1. join the kubernetes cluster, and ensure `kubelet`'s GPU support is enabled.
+2. mount the cephfs on `/mnt/cephfs`
+3. install `nvidia-docker` and run it once
+
+## GPU support of kubelet
+
+Kubernetes (> v1.6.0-beta.1) is supporting multi-GPU now, please follow the newest guide to enable GPU support.
+
+## Nvidia-Docker
+
+[nvidia-docker](https://github.com/NVIDIA/nvidia-docker) homepage
+
+CUDA in container need NVIDIA drivers such as `libcuda.so.1` to work. And KTQueue assume that drivers  located at `/var/lib/nvidia-docker/volumes/nvidia_driver/`. So you need to install nvidia-docker and use nvidia-docker to run a cuda image to ensure that the drivers are located in the right place.
+
+1. follow the installation instruction to install nvidia-docker
+2. run the test nvidia-smi with nvidia-docker
+3. make sure that there is a driver like `367.57` are located at `/var/lib/nvidia-docker/volumes/nvidia_driver/`
+
+## Trouble shoot
+
+- nvidia-smi print the right output, but there are no drivers found at `/var/lib/nvidia-docker/volumes/nvidia_driver/`
+
+try to update the nvidia-docker(>1.0.0), and check docker volumes
+
+> docker volume ls
+
+you may get a line like  
+
+```
+DRIVER              VOLUME NAME
+nvidia-docker       nvidia_driver_367.57
+```
+
+try to remove that volume and run `nvidia-docker run --rm nvidia/cuda nvidia-smi` again.
+
+- docker: Error response from daemon: create nvidia_driver_367.57: VolumeDriver.Create: internal error, check logs for details.
+
+> sudo chown nvidia-docker:nvidia-docker /var/lib/nvidia-docker/volumes/nvidia_driver

docs/deploy/README.md

@@ -0,0 +1,115 @@
+# Deployment
+
+# Brief
+
+1. setup kubernetes and ceph cluster
+2. add ceph-secret to kubernetes
+3. deploy mongodb based on kubernetes and ceph
+4. build ktqueue docker image
+5. deploy ktqueue
+
+To setup a KTQueue node, please read [How to setup a KTQueue node](./NODE.md)
+
+
+# File list
+
+- ktqueue.yaml
+- mongodb-service.yaml
+- mongodb-dev.yaml
+- mongodb-production.yaml
+
+# Dependancy
+
+ktqueue requires kubernetes and cephfs, make sure you already have them.
+
+# Prepare ceph
+
+kubernetes and other components needs permission (aka, a secret) to access ceph.
+
+if you want to know mons in your ceph cluster, just type:
+
+> ceph mon stat
+
+then, get your ceph-secret to kubernetes
+
+> sudo ceph auth get-key client.admin | base64
+
+Note: though `ceph get-key`'s response is encoded by base64, you should encode it again.
+
+then, update your ceph-secret.yaml, add your secret after `key:`
+
+> cp ceph-secret.yaml dep-ceph-secret.yaml && vi dep-ceph-secret.yaml  
+
+and import `ceph-secret.yaml`
+
+> kubectl create -f dep-ceph-secret.yaml
+
+# Deploy mongodb
+
+## create rbd
+
+mongodb needs a `persistent volume`(just like a disk) to store data. so you should create one.
+
+> rbd create rbd/ktqueue-mongodb -s 10240
+
+> rbd info rbd/ktqueue-mongodb
+
+try:
+
+> rbd map ktqueue-mongodb
+
+current linux kernal doesn't support all the features. if you get error, refer [this](http://tonybai.com/2016/11/07/integrate-kubernetes-with-ceph-rbd/)
+
+> rbd feature disable ktqueue-mongodb exclusive-lock, object-map, fast-diff, deep-flatten
+
+## Create mongodb services
+
+> cp mongodb-production.yaml dep-mongodb-production.yaml
+
+create mongodb service
+
+> kubectl create -f mongodb-service.yaml
+
+create mongodb server
+
+> kubectl create -f dep-mongodb-production.yaml
+
+# Deploy ktqueue
+
+## mount cephfs
+ktqueue dameon needs to access ceph to clone code, store log, etc. and ktqueue job needs to access ceph to store output.
+
+so you should ensure that cephfs has been mounted at `/mnt/cephfs` on ensure every single node you want to run ktqueue jobs or ktqueue dameon.
+
+you should modify `fstab` and add cephfs mount.
+
+## build image
+
+build front-end
+
+> cd frontend  
+> npm install  
+> npm run build  
+
+build docker image
+
+> docker build -t ktqueue .
+
+you can modify `dep-ktqueue.yaml`(see next step)and change image name if you want.
+
+
+## deploy
+
+> cp ktqueue.yaml dep-ktqueue.yaml
+
+if you want to change IP/port of ktqueue or assign ktqueue to a specific node, you should modify `dep-ktqueue.yaml`
+
+to select node you want to run ktqueue, change `host_name_you_want` after `kubernetes.io/hostname` and uncomment this line.
+
+to select IP/port, change `ip_you_want_to_access_form_outside` under `externalIPs`
+
+finish it:
+
+> kubectl create -f dep-ktqueue.yaml
+
+enjoy!

docs/deploy/ceph-secret.yaml

@@ -0,0 +1,6 @@
+Version: v1
+kind: Secret
+metadata:
+  name: ceph-secret
+data:
+  key:

docs/deploy/ktqueue-rbac.yaml

@@ -0,0 +1,40 @@
+# Create the clusterrole:
+# $ kubectl create -f ktqueue-rbac.yaml
+# Bind the ktqueue serviceaccount to the ktqueue clusterrole:
+# $ kubectl create clusterrolebinding ktqueue --clusterrole=ktqueue --serviceaccount=default:ktqueue
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: ktqueue
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/log
+    verbs: ["*"]
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - list
+  - apiGroups:
+      - "batch"
+    resources:
+      - jobs
+    verbs: ["*"]
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: ktqueue
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: ktqueue
+subjects:
+- kind: ServiceAccount
+  name: ktqueue
+  namespace: default