hadoop-spark-perf

Deprecated, this project is no longer maintained.

Hadoop and Spark CPU performance Benchmark (Separately)

This benchmark focuses on collecting CPU performance and scaling down input size while keeping the same characteristics. It does not try to monitor nor compare overall performance.

• Environment Setup Local cluster and single setup using Vagrant
• Web UI Dashboard linking to other Admin UI
• Process manager Control process, perf etc.
• Specification What is measured
• Implementation How we do it

Roadmap

Deployment

• deploy Hadoop cluster #6
• deploy Spark cluster #10

Control

• may use consul for configuration
• start cluster based on configuration use fixed size cluster
• let perf attach to process when job start

Perf

• use perf without agent
• use perf with agent
• convert perf data to more common format

Monitor

• Web UI
• vm for collecting monitor metrics #9
• collect perf data, generate report of each experiment