diff --git a/TESTING.asciidoc b/TESTING.asciidoc
index 1cb05792b8298..54a447316f6f9 100644
--- a/TESTING.asciidoc
+++ b/TESTING.asciidoc
@@ -302,15 +302,22 @@ comma separated list of nodes to connect to (e.g. localhost:9300). A transport c
 be created based on that and used for all the before|after test operations, and to extract
 the http addresses of the nodes so that REST requests can be sent to them.
 
-== Testing scripts
+== Testing distributions and packaging
 
-The simplest way to test scripts and the packaged distributions is to use
-Vagrant. You can get started by following there five easy steps:
+The packaging tests use Vagrant virtual machines to verify that installing
+and running elasticsearch distributions works correctly on supported operating systems.
+These tests should really only be run in vagrant vms because they're destructive.
+
+The existing packaging tests are written with https://github.com/sstephenson/bats[Bats]
+and run on Linux boxes. These tests are being ported to a Groovy project that will
+run on all boxes.
+
+To set up your environment and run the packaging tests, follow these steps:
 
 . Install Virtual Box and Vagrant.
 
-. (Optional) Install vagrant-cachier to squeeze a bit more performance out of
-the process:
+. (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze
+a bit more performance out of the process:
 
 --------------------------------------
 vagrant plugin install vagrant-cachier
@@ -322,10 +329,9 @@ vagrant plugin install vagrant-cachier
 gradle :qa:vagrant:vagrantCheckVersion
 -------------------------------------
 
-. Download and smoke test the VMs with `gradle vagrantSmokeTest` or
-`gradle -Pvagrant.boxes=all vagrantSmokeTest`. The first time you run this it will
-download the base images and provision the boxes and immediately quit. If you
-you this again it'll skip the download step.
+. Download and smoke test the VMs with `gradle vagrantSmokeTest`. The first time
+you run this it will download the base images and provision the boxes and immediately
+quit. If run again it will use the boxes already downloaded.
 
 . Run the tests with `gradle packagingTest`. This will cause gradle to build
 the tar, zip, and deb packages and all the plugins. It will then run the tests
@@ -343,7 +349,7 @@ All the regular vagrant commands should just work so you can get a shell in a
 VM running trusty by running
 `vagrant up ubuntu-1404 --provider virtualbox && vagrant ssh ubuntu-1404`.
 
-These are the linux flavors the Vagrantfile currently supports:
+These are the linux flavors supported, all of which have boxes provided
 
 * ubuntu-1404 aka trusty
 * ubuntu-1604 aka xenial
@@ -363,12 +369,42 @@ quality boxes available in vagrant atlas:
 
 * sles-11
 
-We're missing the following because our tests are very linux/bash centric:
+=== Testing packaging on Windows
+
+The packaging tests also support Windows Server 2012R2 and Windows Server 2016.
+Unfortunately we're not able to provide boxes for them in open source use
+because of licensing issues. Any Virtualbox image that has WinRM enabled
+for remote management and supports Powershell for remote users should work.
+
+Testing on Windows requires the https://github.com/criteo/vagrant-winrm[vagrant-winrm] plugin.
+
+------------------------------------
+vagrant plugin install vagrant-winrm
+------------------------------------
+
+To specify the Windows boxes to use, pass the project properties to gradle:
+
+* `-Pvagrant.windows.2012r2.box`
+* `-Pvagrant.windows.2016.box`
+
+These properties are required for Windows support in all gradle tasks that
+handle packaging tests. Either or both may be specified. Remember that to run tests
+on these boxes, they still need to be included in the value of `-Pvagrant.boxes`.
+When these properties are present, passing `-Pvagrant.boxes=all` will include the
+Windows boxes. You can also use `-Pvagrant.boxes=windows-all` to only test on Windows boxes.
+
+If running vagrant directly, pass the box names in the environment variables:
 
-* Windows Server 2012
+* `VAGRANT_WINDOWS_2012R2_BOX`
+* `VAGRANT_WINDOWS_2016_BOX`
 
-It's important to think of VMs like cattle. If they become lame you just shoot
-them and let vagrant reprovision them. Say you've hosed your precise VM:
+=== Testing VMs are disposable
+
+If your testing VM gets into a bad state, just destroy it and let Vagrant
+create a new one. It only takes a minute or two and Vagrant handles all
+the provisioning for you.
+
+Let's say you've hosed your Ubuntu Precise VM:
 
 ----------------------------------------------------
 vagrant ssh ubuntu-1404 -c 'sudo rm -rf /bin'; echo oops
@@ -380,9 +416,6 @@ All you've got to do to get another one is
 vagrant destroy -f ubuntu-1404 && vagrant up ubuntu-1404 --provider virtualbox
 ----------------------------------------------
 
-The whole process takes a minute and a half on a modern laptop, two and a half
-without vagrant-cachier.
-
 Its possible that some downloads will fail and it'll be impossible to restart
 them. This is a bug in vagrant. See the instructions here for how to work
 around it:
@@ -398,58 +431,72 @@ vagrant destroy -f
 `vagrant up` would normally start all the VMs but we've prevented that because
 that'd consume a ton of ram.
 
-== Testing scripts more directly
+=== Running packaging tests more directly
 
-In general its best to stick to testing in vagrant because the bats scripts are
-destructive. When working with a single package it's generally faster to run its
-tests in a tighter loop than gradle provides. In one window:
+Gradle's full packaging test build takes a while, but you can iterate faster
+by managing the VM and running the tests yourself.
+
+To get the build ready for the packaging tests, run (on the host)
 
 --------------------------------
-gradle :distribution:rpm:assemble
+gradle :qa:vagrant:setupPackaging :qa:vagrant:prepareGradleBuild
 --------------------------------
 
-and in another window:
+and in another window, run the gradle task to bring up the VM. In this
+example let's test on Centos 7
+
+-------------------------------------------------
+gradle :qa:vagrant:vagrantCentos7#up
+-------------------------------------------------
+
+This task is available for all boxes specified with `-Pvagrant.boxes` -
+to see what's available, run `gradle :qa:vagrant:tasks --all`.
+
+Then in another terminal, ssh into the VM
+
+----------------------------------------------------
+vagrant ssh centos-7
+----------------------------------------------------
+
+To run the Bats tests for just the RPM package (on the guest)
 
 ----------------------------------------------------
-vagrant up centos-7 --provider virtualbox && vagrant ssh centos-7
 cd $BATS_ARCHIVES
 sudo -E bats $BATS_TESTS/*rpm*.bats
 ----------------------------------------------------
 
-If you wanted to retest all the release artifacts on a single VM you could:
+Or to retest all the release artifacts on a single VM (on the guest)
 
 -------------------------------------------------
-gradle setupBats
-cd qa/vagrant; vagrant up ubuntu-1404 --provider virtualbox && vagrant ssh ubuntu-1404
 cd $BATS_ARCHIVES
 sudo -E bats $BATS_TESTS/*.bats
 -------------------------------------------------
 
-You can also use Gradle to prepare the test environment and then starts a single VM:
-
--------------------------------------------------
-gradle vagrantFedora27#up
--------------------------------------------------
-
-Or any of vagrantCentos6#up, vagrantCentos7#up, vagrantDebian8#up,
-vagrantDebian9#up, vagrantFedora26#up, vagrantFedora27#up, vagrantOel6#up, vagrantOel7#up,
-vagrantOpensuse42#up,vagrantSles12#up, vagrantUbuntu1404#up, vagrantUbuntu1604#up.
-
-Once up, you can then connect to the VM using SSH from the elasticsearch directory:
+To run the Groovy packaging tests, run (on the guest)
 
--------------------------------------------------
-vagrant ssh fedora-27
--------------------------------------------------
-
-Or from another directory:
-
--------------------------------------------------
-VAGRANT_CWD=/path/to/elasticsearch vagrant ssh fedora-27
--------------------------------------------------
+------------------------------------------------
+cd ~/.elasticsearch && gradle :qa:packaging:run
+------------------------------------------------
 
-Note: Starting vagrant VM outside of the elasticsearch folder requires to
-indicates the folder that contains the Vagrantfile using the VAGRANT_CWD
-environment variable.
+=== Testing plugins in packaging tests
+
+If you're building plugins in an `elasticsearch-extra` directory, the Vagrantfile
+will mount that directory to the VM so it's included in the VM's gradle build. If
+your plugin's build needs some additional setup steps before it runs, you can add
+task dependencies to the `#prepareGradleBuild` tasks. For example
+
+[source, groovy]
+----------------------------------------
+project(':qa:vagrant).tasks.findAll { t -> t.name.endsWith('prepareGradleBuild') }.each { task ->
+  Task myPluginSetupTask = project.tasks.create(task.replace('prepareGradleBuild, 'mySetup')) {
+    doLast {
+      String vagrantBox = task.ext.box
+      // vagrant command to setup box
+    }
+  }
+  task.dependsOn(myPluginSetupTask)
+}
+----------------------------------------
 
 == Testing backwards compatibility
 
diff --git a/Vagrantfile b/Vagrantfile
index 021b4d630a1e6..531a376960914 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -21,167 +21,207 @@
 # specific language governing permissions and limitations
 # under the License.
 
+GRADLE_VERSION = '3.3'
+
+define_opts = {
+  autostart: false
+}.freeze
+
 Vagrant.configure(2) do |config|
-  config.vm.define "ubuntu-1404" do |config|
-    config.vm.box = "elastic/ubuntu-14.04-x86_64"
-    ubuntu_common config
+
+  config.vm.provider 'virtualbox' do |vbox|
+    # Give the box more memory and cpu because our tests are beasts!
+    vbox.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192)
+    vbox.cpus = Integer(ENV['VAGRANT_CPUS'] || 4)
   end
-  config.vm.define "ubuntu-1604" do |config|
-    config.vm.box = "elastic/ubuntu-16.04-x86_64"
-    ubuntu_common config, extra: <<-SHELL
-      # Install Jayatana so we can work around it being present.
-      [ -f /usr/share/java/jayatanaag.jar ] || install jayatana
-    SHELL
+
+  # Vagrant and ruby stdlib expand '.' differently, so look for the build
+  # in the directory containing the Vagrantfile
+  vagrantfile_dir = File.expand_path('..', __FILE__)
+
+  # Switch the default share for the project root from /vagrant to
+  # /elasticsearch because /vagrant is confusing when there is a project inside
+  # the elasticsearch project called vagrant....
+  config.vm.synced_folder vagrantfile_dir, '/vagrant', disabled: true
+  config.vm.synced_folder vagrantfile_dir, '/elasticsearch'
+
+  # If building with extra plugins, mount that too
+  extra_projects = "#{vagrantfile_dir}-extra"
+  if Dir.exists?(extra_projects)
+    config.vm.synced_folder extra_projects, '/elasticsearch-extra'
+  end
+
+  # Expose project directory. Note that VAGRANT_CWD may not be the same as Dir.pwd
+  PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd
+  config.vm.synced_folder PROJECT_DIR, '/project'
+
+  'ubuntu-1404'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/ubuntu-14.04-x86_64'
+      deb_common config, box
+    end
+  end
+  'ubuntu-1604'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/ubuntu-16.04-x86_64'
+      deb_common config, box, extra: <<-SHELL
+        # Install Jayatana so we can work around it being present.
+        [ -f /usr/share/java/jayatanaag.jar ] || install jayatana
+      SHELL
+    end
   end
   # Wheezy's backports don't contain Openjdk 8 and the backflips
   # required to get the sun jdk on there just aren't worth it. We have
   # jessie and stretch for testing debian and it works fine.
-  config.vm.define "debian-8" do |config|
-    config.vm.box = "elastic/debian-8-x86_64"
-    deb_common config
-  end
-  config.vm.define "debian-9" do |config|
-    config.vm.box = "elastic/debian-9-x86_64"
-    deb_common config
-  end
-  config.vm.define "centos-6" do |config|
-    config.vm.box = "elastic/centos-6-x86_64"
-    rpm_common config
+  'debian-8'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/debian-8-x86_64'
+      deb_common config, box
+    end
   end
-  config.vm.define "centos-7" do |config|
-    config.vm.box = "elastic/centos-7-x86_64"
-    rpm_common config
+  'debian-9'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/debian-9-x86_64'
+      deb_common config, box
+    end
   end
-  config.vm.define "oel-6" do |config|
-    config.vm.box = "elastic/oraclelinux-6-x86_64"
-    rpm_common config
+  'centos-6'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/centos-6-x86_64'
+      rpm_common config, box
+    end
   end
-  config.vm.define "oel-7" do |config|
-    config.vm.box = "elastic/oraclelinux-7-x86_64"
-    rpm_common config
+  'centos-7'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/centos-7-x86_64'
+      rpm_common config, box
+    end
   end
-  config.vm.define "fedora-26" do |config|
-    config.vm.box = "elastic/fedora-26-x86_64"
-    dnf_common config
+  'oel-6'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/oraclelinux-6-x86_64'
+      rpm_common config, box
+    end
   end
-  config.vm.define "fedora-27" do |config|
-    config.vm.box = "elastic/fedora-27-x86_64"
-    dnf_common config
+  'oel-7'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/oraclelinux-7-x86_64'
+      rpm_common config, box
+    end
   end
-  config.vm.define "opensuse-42" do |config|
-    config.vm.box = "elastic/opensuse-42-x86_64"
-    opensuse_common config
+  'fedora-26'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/fedora-26-x86_64'
+      dnf_common config, box
+    end
   end
-  config.vm.define "sles-12" do |config|
-    config.vm.box = "elastic/sles-12-x86_64"
-    sles_common config
+  'fedora-27'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/fedora-27-x86_64'
+      dnf_common config, box
+    end
   end
-  # Switch the default share for the project root from /vagrant to
-  # /elasticsearch because /vagrant is confusing when there is a project inside
-  # the elasticsearch project called vagrant....
-  config.vm.synced_folder ".", "/vagrant", disabled: true
-  config.vm.synced_folder ".", "/elasticsearch"
-  # Expose project directory
-  PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd
-  config.vm.synced_folder PROJECT_DIR, "/project"
-  config.vm.provider "virtualbox" do |v|
-    # Give the boxes 3GB because Elasticsearch defaults to using 2GB
-    v.memory = 3072
+  'opensuse-42'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/opensuse-42-x86_64'
+      suse_common config, box
+    end
   end
-  if Vagrant.has_plugin?("vagrant-cachier")
-    config.cache.scope = :box
+  'sles-12'.tap do |box|
+    config.vm.define box, define_opts do |config|
+      config.vm.box = 'elastic/sles-12-x86_64'
+      sles_common config, box
+    end
   end
-  config.vm.defined_vms.each do |name, config|
-    config.options[:autostart] = false
-    set_prompt = lambda do |config|
-      # Sets up a consistent prompt for all users. Or tries to. The VM might
-      # contain overrides for root and vagrant but this attempts to work around
-      # them by re-source-ing the standard prompt file.
-      config.vm.provision "prompt", type: "shell", inline: <<-SHELL
-        cat \<\<PROMPT > /etc/profile.d/elasticsearch_prompt.sh
-export PS1='#{name}:\\w$ '
-PROMPT
-        grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc |
-          cat \<\<SOURCE_PROMPT >> ~/.bashrc
-# Replace the standard prompt with a consistent one
-source /etc/profile.d/elasticsearch_prompt.sh
-SOURCE_PROMPT
-        grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc |
-          cat \<\<SOURCE_PROMPT >> ~vagrant/.bashrc
-# Replace the standard prompt with a consistent one
-source /etc/profile.d/elasticsearch_prompt.sh
-SOURCE_PROMPT
-      SHELL
-      # Creates a file to mark the machine as created by vagrant. Tests check
-      # for this file and refuse to run if it is not present so that they can't
-      # be run unexpectedly.
-      config.vm.provision "markerfile", type: "shell", inline: <<-SHELL
-        touch /etc/is_vagrant_vm
-      SHELL
+
+  windows_2012r2_box = ENV['VAGRANT_WINDOWS_2012R2_BOX']
+  if windows_2012r2_box && windows_2012r2_box.empty? == false
+    'windows-2012r2'.tap do |box|
+      config.vm.define box, define_opts do |config|
+        config.vm.box = windows_2012r2_box
+        windows_common config, box
+      end
     end
-    config.config_procs.push ['2', set_prompt]
   end
-end
 
-def ubuntu_common(config, extra: '')
-  deb_common config, extra: extra
+  windows_2016_box = ENV['VAGRANT_WINDOWS_2016_BOX']
+  if windows_2016_box && windows_2016_box.empty? == false
+    'windows-2016'.tap do |box|
+      config.vm.define box, define_opts do |config|
+        config.vm.box = windows_2016_box
+        config.vm.provision 'enable long paths', type: 'shell', inline: <<-SHELL
+          Set-ItemProperty -Path "HKLM:/SYSTEM/CurrentControlSet/Control/Filesystem/" -Name "LongPathsEnabled" -Value 1
+        SHELL
+        windows_common config, box
+      end
+    end
+  end
 end
 
-def deb_common(config, extra: '')
+def deb_common(config, name, extra: '')
   # http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html
-  config.vm.provision "fix-no-tty", type: "shell" do |s|
+  config.vm.provision 'fix-no-tty', type: 'shell' do |s|
       s.privileged = false
       s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile"
   end
-  provision(config,
-    update_command: "apt-get update",
-    update_tracking_file: "/var/cache/apt/archives/last_update",
-    install_command: "apt-get install -y",
-    extra: extra)
+  linux_common(
+    config,
+    name,
+    update_command: 'apt-get update',
+    update_tracking_file: '/var/cache/apt/archives/last_update',
+    install_command: 'apt-get install -y',
+    extra: extra
+  )
 end
 
-def rpm_common(config)
-  provision(config,
-    update_command: "yum check-update",
-    update_tracking_file: "/var/cache/yum/last_update",
-    install_command: "yum install -y")
+def rpm_common(config, name)
+  linux_common(
+    config,
+    name,
+    update_command: 'yum check-update',
+    update_tracking_file: '/var/cache/yum/last_update',
+    install_command: 'yum install -y'
+  )
 end
 
-def dnf_common(config)
-  provision(config,
-    update_command: "dnf check-update",
-    update_tracking_file: "/var/cache/dnf/last_update",
-    install_command: "dnf install -y",
-    install_command_retries: 5)
-  if Vagrant.has_plugin?("vagrant-cachier")
-    # Autodetect doesn't work....
+def dnf_common(config, name)
+  # Autodetect doesn't work....
+  if Vagrant.has_plugin?('vagrant-cachier')
     config.cache.auto_detect = false
-    config.cache.enable :generic, { :cache_dir => "/var/cache/dnf" }
+    config.cache.enable :generic, { :cache_dir => '/var/cache/dnf' }
   end
+  linux_common(
+    config,
+    name,
+    update_command: 'dnf check-update',
+    update_tracking_file: '/var/cache/dnf/last_update',
+    install_command: 'dnf install -y',
+    install_command_retries: 5
+  )
 end
 
-def opensuse_common(config)
-  suse_common config, ''
-end
-
-def suse_common(config, extra)
-  provision(config,
-    update_command: "zypper --non-interactive list-updates",
-    update_tracking_file: "/var/cache/zypp/packages/last_update",
-    install_command: "zypper --non-interactive --quiet install --no-recommends",
-    extra: extra)
+def suse_common(config, name, extra: '')
+  linux_common(
+    config,
+    name,
+    update_command: 'zypper --non-interactive list-updates',
+    update_tracking_file: '/var/cache/zypp/packages/last_update',
+    install_command: 'zypper --non-interactive --quiet install --no-recommends',
+    extra: extra
+  )
 end
 
-def sles_common(config)
+def sles_common(config, name)
   extra = <<-SHELL
     zypper rr systemsmanagement_puppet puppetlabs-pc1
     zypper --non-interactive install git-core
-SHELL
-  suse_common config, extra
+  SHELL
+  suse_common config, name, extra: extra
 end
 
-# Register the main box provisioning script.
+# Configuration needed for all linux boxes
 # @param config Vagrant's config object. Required.
+# @param name [String] The box name. Required.
 # @param update_command [String] The command used to update the package
 #   manager. Required. Think `apt-get update`.
 # @param update_tracking_file [String] The location of the file tracking the
@@ -189,27 +229,77 @@ end
 #   is cached by vagrant-cachier.
 # @param install_command [String] The command used to install a package.
 #   Required. Think `apt-get install #{package}`.
-# @param extra [String] Extra provisioning commands run before anything else.
-#   Optional. Used for things like setting up the ppa for Java 8.
-def provision(config,
-    update_command: 'required',
-    update_tracking_file: 'required',
-    install_command: 'required',
-    install_command_retries: 0,
-    extra: '')
-  # Vagrant run ruby 2.0.0 which doesn't have required named parameters....
-  raise ArgumentError.new('update_command is required') if update_command == 'required'
-  raise ArgumentError.new('update_tracking_file is required') if update_tracking_file == 'required'
-  raise ArgumentError.new('install_command is required') if install_command == 'required'
-  config.vm.provider "virtualbox" do |v|
-    # Give the box more memory and cpu because our tests are beasts!
-    v.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192)
-    v.cpus = Integer(ENV['VAGRANT_CPUS'] || 4)
+# @param install_command_retries [Integer] Number of times to retry
+#   a failed install command
+# @param extra [String] Additional script to run before installing
+#   dependencies
+#
+def linux_common(config,
+                 name,
+                 update_command: 'required',
+                 update_tracking_file: 'required',
+                 install_command: 'required',
+                 install_command_retries: 0,
+                 extra: '')
+
+  raise ArgumentError, 'update_command is required' if update_command == 'required'
+  raise ArgumentError, 'update_tracking_file is required' if update_tracking_file == 'required'
+  raise ArgumentError, 'install_command is required' if install_command == 'required'
+
+  if Vagrant.has_plugin?('vagrant-cachier')
+    config.cache.scope = :box
   end
-  config.vm.synced_folder "#{Dir.home}/.gradle/caches", "/home/vagrant/.gradle/caches",
+
+  gradle_cache config, '/home/vagrant/.gradle/caches'
+
+  config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
+    touch /etc/is_vagrant_vm
+  SHELL
+  sh_set_prompt config, name
+  sh_install_deps(
+    config,
+    update_command,
+    update_tracking_file,
+    install_command,
+    install_command_retries,
+    extra
+  )
+end
+
+def gradle_cache(config, guest_path)
+  config.vm.synced_folder "#{Dir.home}/.gradle/caches", guest_path,
     create: true,
-    owner: "vagrant"
-  config.vm.provision "dependencies", type: "shell", inline: <<-SHELL
+    owner: 'vagrant'
+end
+
+# Sets up a consistent prompt for all users. Or tries to. The VM might
+# contain overrides for root and vagrant but this attempts to work around
+# them by re-source-ing the standard prompt file.
+def sh_set_prompt(config, name)
+  config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL
+      cat \<\<PROMPT > /etc/profile.d/elasticsearch_prompt.sh
+export PS1='#{name}:\\w$ '
+PROMPT
+      grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc |
+        cat \<\<SOURCE_PROMPT >> ~/.bashrc
+# Replace the standard prompt with a consistent one
+source /etc/profile.d/elasticsearch_prompt.sh
+SOURCE_PROMPT
+      grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc |
+        cat \<\<SOURCE_PROMPT >> ~vagrant/.bashrc
+# Replace the standard prompt with a consistent one
+source /etc/profile.d/elasticsearch_prompt.sh
+SOURCE_PROMPT
+  SHELL
+end
+
+def sh_install_deps(config,
+                    update_command,
+                    update_tracking_file,
+                    install_command,
+                    install_command_retries,
+                    extra)
+  config.vm.provision 'install dependencies', type: 'shell', inline:  <<-SHELL
     set -e
     set -o pipefail
 
@@ -243,9 +333,9 @@ def provision(config,
       echo "==> Installing $1"
       if [ #{install_command_retries} -eq 0 ]
       then
-            #{install_command} $1
+        #{install_command} $1
       else
-            retry_installcommand $1 #{install_command_retries}
+        retry_installcommand $1 #{install_command_retries}
       fi
     }
 
@@ -256,12 +346,13 @@ def provision(config,
     #{extra}
 
     installed java || {
-      echo "==> Java is not installed on vagrant box ${config.vm.box}"
+      echo "==> Java is not installed"
       return 1
     }
     ensure tar
     ensure curl
     ensure unzip
+    ensure rsync
 
     installed bats || {
       # Bats lives in a git repository....
@@ -275,31 +366,22 @@ def provision(config,
 
     installed gradle || {
       echo "==> Installing Gradle"
-      curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-3.3-bin.zip
+      curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip
       unzip -q /tmp/gradle.zip -d /opt
       rm -rf /tmp/gradle.zip
-      ln -s /opt/gradle-3.3/bin/gradle /usr/bin/gradle
+      ln -s /opt/gradle-#{GRADLE_VERSION}/bin/gradle /usr/bin/gradle
       # make nfs mounted gradle home dir writeable
       chown vagrant:vagrant /home/vagrant/.gradle
     }
 
-
     cat \<\<VARS > /etc/profile.d/elasticsearch_vars.sh
-export ZIP=/elasticsearch/distribution/zip/build/distributions
-export TAR=/elasticsearch/distribution/tar/build/distributions
-export RPM=/elasticsearch/distribution/rpm/build/distributions
-export DEB=/elasticsearch/distribution/deb/build/distributions
-export BATS=/project/build/bats
-export BATS_UTILS=/project/build/bats/utils
-export BATS_TESTS=/project/build/bats/tests
-export BATS_ARCHIVES=/project/build/bats/archives
-export GRADLE_HOME=/opt/gradle-3.3
+export BATS=/project/build/packaging/bats
+export BATS_UTILS=/project/build/packaging/bats/utils
+export BATS_TESTS=/project/build/packaging/bats/tests
+export BATS_ARCHIVES=/project/build/packaging/archives
+export GRADLE_HOME=/opt/gradle-#{GRADLE_VERSION}
 VARS
     cat \<\<SUDOERS_VARS > /etc/sudoers.d/elasticsearch_vars
-Defaults   env_keep += "ZIP"
-Defaults   env_keep += "TAR"
-Defaults   env_keep += "RPM"
-Defaults   env_keep += "DEB"
 Defaults   env_keep += "BATS"
 Defaults   env_keep += "BATS_UTILS"
 Defaults   env_keep += "BATS_TESTS"
@@ -308,3 +390,125 @@ SUDOERS_VARS
     chmod 0440 /etc/sudoers.d/elasticsearch_vars
   SHELL
 end
+
+def windows_common(config, name)
+  gradle_cache config, '/Users/vagrant/.gradle/caches'
+
+  config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
+    $ErrorActionPreference = "Stop"
+    New-Item C:/is_vagrant_vm -ItemType file -Force | Out-Null
+  SHELL
+
+  config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL
+    $ErrorActionPreference = "Stop"
+    $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }'
+    $ps_prompt | Out-File $PsHome/Microsoft.PowerShell_profile.ps1
+  SHELL
+
+  # Windows' system APIs limit paths to 260 characters. In server 2016 we can raise this limit,
+  # (see LongPathsEnabled) but not in server 2012r2. This adds a powershell module that wraps
+  # robocopy, which can handle paths longer than the system limit. The wrapper converts
+  # robocopy's unusual exit code and error handling to the behavior we'd normally expect
+  #
+  # https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx
+  # https://ss64.com/nt/robocopy-exit.html
+  config.vm.provision 'long path shim module', type: 'shell' do |s|
+    s.privileged = false
+    s.inline = <<-SHELL
+      $ErrorActionPreference = "Stop"
+      $longPathScript = @'
+#{powershell_long_path_module}
+'@
+      $ModuleDir = "C:/Users/vagrant/Documents/WindowsPowerShell/Modules/LongPathShims"
+      $ModuleFile = "$ModuleDir/LongPathShims.psm1"
+      if (-Not (Test-Path "$ModuleDir")) {
+        New-Item "$ModuleDir" -ItemType Directory | Out-Null
+      }
+      $longPathScript | Out-File "$ModuleFile"
+    SHELL
+  end
+
+  powershell_install_deps config
+end
+
+def powershell_long_path_module
+  <<-SHELL
+    function Copy-Long-Path {
+      param(
+        [string]$Source,
+        [string]$Destination
+      )
+
+      robocopy "$Source" "$Destination" /E /COPY:DT /DCOPY:T /NFL /NDL /NJH /NJS /NC /NS /NP
+      Handle-Robocopy-Exit-Code $LASTEXITCODE
+    }
+
+    function Remove-Long-Path {
+      param(
+        [string]$Target
+      )
+
+      $EmptyDir = "C:\\intentionally_empty"
+
+      try {
+        New-Item "$EmptyDir" -ItemType Directory | Out-Null
+        # There doesn't appear to be a way to silence output about files that are purged,
+        # so just sent it to null as writing it to the terminal is very slow
+        robocopy "$EmptyDir" "$Target" /PURGE /NFL /NDL /NJH /NJS /NC /NS /NP | Out-Null
+        $RobocopyExitCode = $LASTEXITCODE
+        Remove-Item "$Target" -Recurse
+        Handle-Robocopy-Exit-Code $RobocopyExitCode
+      } finally {
+        Remove-Item "$EmptyDir" -Recurse
+      }
+    }
+
+    function Handle-Robocopy-Exit-Code {
+      param(
+        [int]$ExitCode
+      )
+
+      Write-Verbose "robocopy returned exit code $ExitCode"
+      if ($ExitCode -ge 7) {
+        Write-Error "robocopy encountered an error and returned exit code $ExitCode"
+      }
+    }
+  SHELL
+end
+
+def powershell_install_deps(config)
+  config.vm.provision 'install deps', type: 'shell', inline: <<-SHELL
+    $ErrorActionPreference = "Stop"
+
+    function Installed {
+      Param(
+        [string]$command
+      )
+
+      try {
+        Get-Command $command
+        return $true
+      } catch {
+        return $false
+      }
+    }
+
+    if (-Not (Installed java)) {
+      Write-Error "java is not installed"
+    }
+
+    if (-Not (Installed gradle)) {
+      Write-Host "==> Installing gradle"
+      $Source="https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip"
+      $Zip="C:\\tmp\\gradle.zip"
+      $Destination="C:\\gradle"
+      New-Item (Split-Path $Zip) -ItemType Directory -ErrorAction Ignore | Out-Null
+      (New-Object Net.WebClient).DownloadFile($Source, $Zip)
+      Add-Type -assembly "System.IO.Compression.Filesystem"
+      [IO.Compression.ZipFile]::ExtractToDirectory($Zip, $Destination)
+      Remove-Item $Zip
+      [Environment]::SetEnvironmentVariable("Path", $env:Path + ";$Destination\\gradle-#{GRADLE_VERSION}\\bin", "Machine")
+      [Environment]::SetEnvironmentVariable("GRADLE_HOME", "$Destination\\gradle-#{GRADLE_VERSION}", "Machine")
+    }
+  SHELL
+end
diff --git a/Vagrantfile.alternate.rb b/Vagrantfile.alternate.rb
new file mode 100644
index 0000000000000..3723b4505907e
--- /dev/null
+++ b/Vagrantfile.alternate.rb
@@ -0,0 +1,462 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# This Vagrantfile exists to test packaging. Read more about its use in the
+# vagrant section in TESTING.asciidoc.
+
+# Licensed to Elasticsearch under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+GRADLE_VERSION = '3.3'
+
+boxes = {}
+linux_boxes = {}
+
+ubuntu_boxes = {
+  'ubuntu-1404' => 'elastic/ubuntu-14.04-x86_64',
+  'ubuntu-1604' => 'elastic/ubuntu-16.04-x86_64'
+}.freeze
+linux_boxes.merge!(ubuntu_boxes)
+
+# Wheezy's backports don't contain Openjdk 8 and the backflips
+# required to get the sun jdk on there just aren't worth it. We have
+# jessie and stretch for testing debian and it works fine.
+debian_boxes = {
+  'debian-8' => 'elastic/debian-8-x86_64',
+  'debian-9' => 'elastic/debian-9-x86_64'
+}.freeze
+linux_boxes.merge!(debian_boxes)
+
+centos_boxes = {
+  'centos-6' => 'elastic/centos-6-x86_64',
+  'centos-7' => 'elastic/centos-7-x86_64'
+}.freeze
+linux_boxes.merge!(centos_boxes)
+
+oel_boxes = {
+  'oel-6' => 'elastic/oraclelinux-6-x86_64',
+  'oel-7' => 'elastic/oraclelinux-7-x86_64'
+}.freeze
+linux_boxes.merge!(oel_boxes)
+
+fedora_boxes = {
+  'fedora-25' => 'elastic/fedora-25-x86_64',
+  'fedora-26' => 'elastic/fedora-26-x86_64'
+}.freeze
+linux_boxes.merge!(fedora_boxes)
+
+opensuse_boxes = {
+  'opensuse-42' => 'elastic/opensuse-42-x86_64'
+}.freeze
+linux_boxes.merge!(opensuse_boxes)
+
+sles_boxes = {
+  'sles-12' => 'elastic/sles-12-x86_64'
+}.freeze
+linux_boxes.merge!(sles_boxes)
+
+linux_boxes.freeze
+boxes.merge!(linux_boxes)
+
+windows_boxes = {}
+
+windows_2012r2_box = ENV['VAGRANT_WINDOWS_2012R2_BOX']
+if windows_2012r2_box && windows_2012r2_box.empty? == false
+  windows_boxes['windows-2012r2'] = windows_2012r2_box
+end
+
+windows_2016_box = ENV['VAGRANT_WINDOWS_2016_BOX']
+if windows_2016_box && windows_2016_box.empty? == false
+  windows_boxes['windows-2016'] = windows_2016_box
+end
+
+windows_boxes.freeze
+
+boxes = linux_boxes.merge(windows_boxes).freeze
+
+def gradle_cache(guest_path)
+  lambda do |config|
+    config.vm.synced_folder "#{Dir.home}/.gradle/caches", guest_path,
+      create: true,
+      owner: 'vagrant'
+  end
+end
+
+def sh_set_prompt(name)
+  # Sets up a consistent prompt for all users. Or tries to. The VM might
+  # contain overrides for root and vagrant but this attempts to work around
+  # them by re-source-ing the standard prompt file.
+  lambda do |config|
+    config.vm.provision 'prompt sh', type: 'shell', inline: <<-SHELL
+        cat \<\<PROMPT > /etc/profile.d/elasticsearch_prompt.sh
+export PS1='#{name}:\\w$ '
+PROMPT
+        grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc |
+          cat \<\<SOURCE_PROMPT >> ~/.bashrc
+# Replace the standard prompt with a consistent one
+source /etc/profile.d/elasticsearch_prompt.sh
+SOURCE_PROMPT
+        grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc |
+          cat \<\<SOURCE_PROMPT >> ~vagrant/.bashrc
+# Replace the standard prompt with a consistent one
+source /etc/profile.d/elasticsearch_prompt.sh
+SOURCE_PROMPT
+    SHELL
+  end
+end
+
+configuration_procs = Hash.new { |hash, key| hash[key] = Array.new }
+
+# Share gradle cache
+linux_boxes.keys.each do |name|
+  configuration_procs[name] << gradle_cache('/home/vagrant/.gradle/caches')
+end
+
+windows_boxes.keys.each do |name|
+  configuration_procs[name] << gradle_cache('/Users/vagrant/.gradle/caches')
+end
+
+# Workaround provisoning bug
+# http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html
+(debian_boxes.keys + ubuntu_boxes.keys).each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'fix-no-tty', type: 'shell' do |s|
+        s.privileged = false
+        s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile"
+    end
+  end
+end
+
+# Create vagrant vm markerfile
+linux_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
+      touch /etc/is_vagrant_vm
+    SHELL
+  end
+end
+
+windows_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
+      New-Item \\is_vagrant_vm -ItemType file -Force | Out-Null
+    SHELL
+  end
+end
+
+# Set prompt to be machine name
+linux_boxes.keys.each do |name|
+  configuration_procs[name] << sh_set_prompt(name)
+
+  configuration_procs[name] << lambda do |config|
+    if Vagrant.has_plugin?('vagrant-cachier')
+      config.cache.scope = :box
+    end
+  end
+end
+
+windows_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'prompt powershell', type: 'shell', inline: <<-SHELL
+      $ErrorActionPreference = "Stop"
+      # set powershell prompt for all users
+      $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }'
+      $ps_prompt | Out-File $PsHome\\Microsoft.PowerShell_profile.ps1
+    SHELL
+  end
+end
+
+
+# Install Jayatana so we can work around it being present.
+configuration_procs['ubuntu-1604'] << lambda do |config|
+  config.vm.provision 'install jayatana', type: 'shell', inline: sh_install_apt(
+    '[ -f /usr/share/java/jayatanaag.jar ] || install jayatana'
+  )
+end
+(debian_boxes.keys + ubuntu_boxes.keys).each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install common deps (apt)', type: 'shell', inline: sh_install_apt(linux_deps)
+  end
+end
+
+(centos_boxes.keys + oel_boxes.keys).each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install common deps (rpm)', type: 'shell', inline: sh_install_rpm(linux_deps)
+  end
+end
+
+fedora_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install common deps (dnf)', type: 'shell', inline: sh_install_dnf(linux_deps)
+    if Vagrant.has_plugin?("vagrant-cachier")
+      # Autodetect doesn't work....
+      config.cache.auto_detect = false
+      config.cache.enable :generic, { :cache_dir => "/var/cache/dnf" }
+    end
+  end
+end
+
+sles_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install puppet and git for sles', type: 'shell', inline: <<-SHELL
+      zypper rr systemsmanagement_puppet puppetlabs-pc1
+      zypper --non-interactive install git-core
+    SHELL
+  end
+end
+
+(sles_boxes.keys + opensuse_boxes.keys).each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install common deps (zypper)', type: 'shell', inline: sh_install_zypper(linux_deps)
+  end
+end
+
+
+windows_boxes.keys.each do |name|
+  configuration_procs[name] << lambda do |config|
+    config.vm.provision 'install common deps', type: 'shell', inline: windows_deps
+  end
+end
+
+Vagrant.configure(2) do |config|
+
+  config.vm.provider 'virtualbox' do |vbox|
+    # Give the box more memory and cpu because our tests are beasts!
+    vbox.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192)
+    vbox.cpus = Integer(ENV['VAGRANT_CPUS'] || 4)
+  end
+
+  # Vagrant and ruby stdlib expand '.' differently, so look for the build
+  # in the directory containing the Vagrantfile
+  vagrantfile_dir = File.expand_path('..', __FILE__)
+
+  # Switch the h share for the project root from /vagrant to
+  # /elasticsearch because /vagrant is confusing when there is a project inside
+  # the elasticsearch project called vagrant....
+  config.vm.synced_folder vagrantfile_dir, '/vagrant', disabled: true
+  config.vm.synced_folder vagrantfile_dir, '/elasticsearch'
+
+  extra_projects = "#{vagrantfile_dir}-extra"
+  if Dir.exists?(extra_projects)
+    config.vm.synced_folder extra_projects, '/elasticsearch-extra'
+  end
+
+  # Expose project directory
+  PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd
+  config.vm.synced_folder PROJECT_DIR, '/project'
+
+  boxes.each do |name, image|
+    config.vm.define name, autostart: false do |config|
+      config.vm.box = image
+      configuration_procs[name].each do |configuration_proc|
+        configuration_proc.call(config)
+      end
+    end
+  end
+end
+
+
+def sh_install_apt(script)
+  sh_install(
+    script,
+    update_command: 'apt-get update',
+    update_tracking_file: '/var/cache/apt/archives/last_update',
+    install_command: 'apt-get install -y'
+  )
+end
+
+def sh_install_rpm(script)
+  sh_install(
+    script,
+    update_command: 'yum check-update',
+    update_tracking_file: '/var/cache/yum/last_update',
+    install_command: 'yum install -y'
+  )
+end
+
+def sh_install_dnf(script)
+  sh_install(
+    script,
+    update_command: 'dnf check-update',
+    update_tracking_file: '/var/cache/dnf/last_update',
+    install_command: 'dnf install -y',
+    install_command_retries: 5
+  )
+end
+
+def sh_install_zypper(script)
+  sh_install(
+    script,
+    update_command: 'zypper --non-interactive list-updates',
+    update_tracking_file: '/var/cache/zypp/packages/last_update',
+    install_command: 'zypper --non-interactive --quiet install --no-recommends'
+  )
+end
+
+def linux_deps
+  <<-SHELL
+    installed java || {
+      echo "==> Java is not installed"
+      return 1
+    }
+    ensure tar
+    ensure curl
+    ensure unzip
+    ensure rsync
+
+    installed bats || {
+      # Bats lives in a git repository....
+      ensure git
+      echo "==> Installing bats"
+      git clone https://github.com/sstephenson/bats /tmp/bats
+      # Centos doesn't add /usr/local/bin to the path....
+      /tmp/bats/install.sh /usr
+      rm -rf /tmp/bats
+    }
+
+    installed gradle || {
+      echo "==> Installing Gradle"
+      curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip
+      unzip -q /tmp/gradle.zip -d /opt
+      rm -rf /tmp/gradle.zip
+      ln -s /opt/gradle-#{GRADLE_VERSION}/bin/gradle /usr/bin/gradle
+      # make nfs mounted gradle home dir writeable
+      chown vagrant:vagrant /home/vagrant/.gradle
+    }
+
+    cat \<\<VARS > /etc/profile.d/elasticsearch_vars.sh
+export BATS=/project/build/packaging/bats
+export BATS_UTILS=/project/build/packaging/bats/utils
+export BATS_TESTS=/project/build/packaging/bats/tests
+export BATS_ARCHIVES=/project/build/packaging/archives
+export GRADLE_HOME=/opt/gradle-#{GRADLE_VERSION}
+VARS
+    cat \<\<SUDOERS_VARS > /etc/sudoers.d/elasticsearch_vars
+Defaults   env_keep += "BATS"
+Defaults   env_keep += "BATS_UTILS"
+Defaults   env_keep += "BATS_TESTS"
+Defaults   env_keep += "BATS_ARCHIVES"
+SUDOERS_VARS
+    chmod 0440 /etc/sudoers.d/elasticsearch_vars
+  SHELL
+end
+
+def windows_deps
+  <<-SHELL
+    $ErrorActionPreference = "Stop"
+
+    function Installed {
+      Param(
+        [string]$command
+      )
+
+      try {
+        Get-Command $command
+        return $true
+      } catch {
+        return $false
+      }
+    }
+
+    if (-Not (Installed java)) {
+      Write-Error "java is not installed"
+    }
+
+    if (-Not (Installed gradle)) {
+      Write-Host "==> Installing gradle"
+			$Source="https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip"
+      $Zip="\\tmp\\gradle.zip"
+      $Destination="\\gradle"
+      New-Item (Split-Path $Zip) -Type Directory -ErrorAction Ignore | Out-Null
+      (New-Object Net.WebClient).DownloadFile($Source, $Zip)
+      Add-Type -assembly "System.IO.Compression.Filesystem"
+      [IO.Compression.ZipFile]::ExtractToDirectory($Zip, $Destination)
+      Remove-Item $Zip
+      [Environment]::SetEnvironmentVariable("Path", $env:Path + ";$Destination\\gradle-#{GRADLE_VERSION}\\bin", "Machine")
+      [Environment]::SetEnvironmentVariable("GRADLE_HOME", "$Destination\\gradle-#{GRADLE_VERSION}", "Machine")
+    }
+  SHELL
+end
+
+# Build the script for installing build dependencies
+# @param script [String] the script to run with install statements
+# @param update_command [String] The command used to update the package
+#   manager. Required. Think `apt-get update`.
+# @param update_tracking_file [String] The location of the file tracking the
+#   last time the update command was run. Required. Should be in a place that
+#   is cached by vagrant-cachier.
+# @param install_command [String] The command used to install a package.
+#   Required. Think `apt-get install #{package}`.
+#
+def sh_install(script,
+                 update_command: 'required',
+                 update_tracking_file: 'required',
+                 install_command: 'required',
+                 install_command_retries: 0)
+
+  raise ArgumentError, 'update_command is required' if update_command == 'required'
+  raise ArgumentError, 'update_tracking_file is required' if update_tracking_file == 'required'
+  raise ArgumentError, 'install_command is required' if install_command == 'required'
+
+  command = ''
+  command << <<-SHELL
+    set -e
+    set -o pipefail
+
+    # Retry install command up to $2 times, if failed
+    retry_installcommand() {
+      n=0
+      while true; do
+        #{install_command} $1 && break
+        let n=n+1
+        if [ $n -ge $2 ]; then
+          echo "==> Exhausted retries to install $1"
+          return 1
+        fi
+        echo "==> Retrying installing $1, attempt $((n+1))"
+        # Add a small delay to increase chance of metalink providing updated list of mirrors
+        sleep 5
+      done
+    }
+
+    installed() {
+      command -v $1 2>&1 >/dev/null
+    }
+
+    install() {
+      # Only apt-get update if we haven't in the last day
+      if [ ! -f #{update_tracking_file} ] || [ "x$(find #{update_tracking_file} -mtime +0)" == "x#{update_tracking_file}" ]; then
+        echo "==> Updating repository"
+        #{update_command} || true
+        touch #{update_tracking_file}
+      fi
+      echo "==> Installing $1"
+      if [ #{install_command_retries} -eq 0 ]
+      then
+        #{install_command} $1
+      else
+        retry_installcommand $1 #{install_command_retries}
+      fi
+    }
+
+    ensure() {
+      installed $1 || install $1
+    }
+  SHELL
+  command << script
+  command
+end
diff --git a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy
index e6e7fca62f97e..4c7fab93b5d86 100644
--- a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy
+++ b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy
@@ -22,9 +22,16 @@ import org.gradle.api.tasks.Input
 
 class VagrantPropertiesExtension {
 
+    /** The boxes that we will actually run the tests on - not all boxes **/
     @Input
     List<String> boxes
 
+    @Input
+    Map<String, String> vagrantEnvVars
+
+    @Input
+    String testTask
+
     @Input
     String upgradeFromVersion
 
@@ -43,8 +50,7 @@ class VagrantPropertiesExtension {
     @Input
     Boolean inheritTestUtils
 
-    VagrantPropertiesExtension(List<String> availableBoxes) {
-        this.boxes = availableBoxes
+    VagrantPropertiesExtension() {
         this.batsDir = 'src/test/resources/packaging'
     }
 
@@ -52,6 +58,10 @@ class VagrantPropertiesExtension {
         this.boxes = Arrays.asList(boxes)
     }
 
+    void setTestTask(String testTask) {
+        this.testTask = testTask
+    }
+
     void setBatsDir(String batsDir) {
         this.batsDir = batsDir
     }
diff --git a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy
index 2510620ac712b..663752688b724 100644
--- a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy
+++ b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy
@@ -1,6 +1,5 @@
 package org.elasticsearch.gradle.vagrant
 
-import com.carrotsearch.gradle.junit4.RandomizedTestingPlugin
 import org.elasticsearch.gradle.FileContentsTask
 import org.gradle.api.*
 import org.gradle.api.artifacts.dsl.RepositoryHandler
@@ -13,8 +12,8 @@ import org.gradle.api.tasks.TaskState
 
 class VagrantTestPlugin implements Plugin<Project> {
 
-    /** All available boxes **/
-    static List<String> BOXES = [
+    /** All Linux boxes **/
+    static List<String> LINUX_BOXES = [
             'centos-6',
             'centos-7',
             'debian-8',
@@ -29,6 +28,12 @@ class VagrantTestPlugin implements Plugin<Project> {
             'ubuntu-1604'
     ]
 
+    /** Windows boxes we have images for and can build with **/
+    static Map<String, String> AVAILABLE_WINDOWS_BOXES = [:]
+
+    /** All available boxes **/
+    static List<String> BOXES
+
     /** Boxes used when sampling the tests **/
     static List<String> SAMPLE = [
             'centos-7',
@@ -41,21 +46,29 @@ class VagrantTestPlugin implements Plugin<Project> {
     /** Packages onboarded for upgrade tests **/
     static List<String> UPGRADE_FROM_ARCHIVES = ['rpm', 'deb']
 
+    static final PACKAGING_CONFIGURATION = 'packaging'
+
     private static final BATS = 'bats'
     private static final String BATS_TEST_COMMAND ="cd \$BATS_ARCHIVES && sudo bats --tap \$BATS_TESTS/*.$BATS"
-    private static final String PLATFORM_TEST_COMMAND ="rm -rf ~/elasticsearch && rsync -r /elasticsearch/ ~/elasticsearch && cd ~/elasticsearch && \$GRADLE_HOME/bin/gradle test integTest"
 
     @Override
     void apply(Project project) {
 
+        collectAvailableBoxes(project)
+
         // Creates the Vagrant extension for the project
-        project.extensions.create('esvagrant', VagrantPropertiesExtension, listVagrantBoxes(project))
+        project.extensions.create('esvagrant', VagrantPropertiesExtension)
+        List<String> boxesToRun = listSelectedVagrantBoxes(project)
+        assert BOXES.containsAll(boxesToRun)
+        project.extensions.esvagrant.boxes = boxesToRun
+
+        project.extensions.esvagrant.vagrantEnvVars = collectVagrantEnvVars(project)
 
         // Add required repositories for Bats tests
         configureBatsRepositories(project)
 
         // Creates custom configurations for Bats testing files (and associated scripts and archives)
-        createBatsConfiguration(project)
+        createPackagingConfiguration(project)
 
         // Creates all the main Vagrant tasks
         createVagrantTasks(project)
@@ -74,14 +87,54 @@ class VagrantTestPlugin implements Plugin<Project> {
         createVagrantBoxesTasks(project)
     }
 
-    private List<String> listVagrantBoxes(Project project) {
+    private static void collectAvailableBoxes(Project project) {
+        String windows_2012r2_box = project.properties.get('vagrant.windows.2012r2.box', null)
+        if (windows_2012r2_box != null && windows_2012r2_box.isEmpty() == false) {
+            AVAILABLE_WINDOWS_BOXES['windows-2012r2'] = windows_2012r2_box
+        }
+
+        String windows_2016_box = project.properties.get('vagrant.windows.2016.box', null)
+        if (windows_2016_box != null && windows_2016_box.isEmpty() == false) {
+            AVAILABLE_WINDOWS_BOXES['windows-2016'] = windows_2016_box
+        }
+
+        BOXES = LINUX_BOXES + AVAILABLE_WINDOWS_BOXES.keySet()
+        project.ext.boxes = BOXES
+    }
+
+    private static Map<String, String> collectVagrantEnvVars(Project project) {
+        /*
+         * We always use the main project.rootDir as Vagrant's current working directory (VAGRANT_CWD)
+         * so that boxes are not duplicated for every Gradle project that use this VagrantTestPlugin.
+         */
+        def vagrantEnvVars = [
+                'VAGRANT_CWD'           : "${project.rootDir.absolutePath}",
+                'VAGRANT_VAGRANTFILE'   : 'Vagrantfile',
+                'VAGRANT_PROJECT_DIR'   : "${project.projectDir.absolutePath}"
+        ]
+        if ('windows-2012r2' in AVAILABLE_WINDOWS_BOXES) {
+            vagrantEnvVars['VAGRANT_WINDOWS_2012R2_BOX'] = AVAILABLE_WINDOWS_BOXES['windows-2012r2']
+        }
+        if ('windows-2016' in AVAILABLE_WINDOWS_BOXES) {
+            vagrantEnvVars['VAGRANT_WINDOWS_2016_BOX'] = AVAILABLE_WINDOWS_BOXES['windows-2016']
+        }
+
+        return vagrantEnvVars
+    }
+
+    private static List<String> listSelectedVagrantBoxes(Project project) {
         String vagrantBoxes = project.getProperties().get('vagrant.boxes', 'sample')
-        if (vagrantBoxes == 'sample') {
-            return SAMPLE
-        } else if (vagrantBoxes == 'all') {
-            return BOXES
-        } else {
-            return vagrantBoxes.split(',')
+        switch (vagrantBoxes) {
+            case 'sample':
+                return SAMPLE
+            case 'linux-all':
+                return LINUX_BOXES
+            case 'windows-all':
+                return AVAILABLE_WINDOWS_BOXES.keySet().toList()
+            case 'all':
+                return BOXES
+            default:
+                return vagrantBoxes.split(',')
         }
     }
 
@@ -100,10 +153,10 @@ class VagrantTestPlugin implements Plugin<Project> {
         }
     }
 
-    private static void createBatsConfiguration(Project project) {
-        project.configurations.create(BATS)
+    private static void createPackagingConfiguration(Project project) {
+        project.configurations.create(PACKAGING_CONFIGURATION)
 
-        String upgradeFromVersion = System.getProperty("tests.packaging.upgradeVersion");
+        String upgradeFromVersion = System.getProperty("tests.packaging.upgradeVersion")
         if (upgradeFromVersion == null) {
             String firstPartOfSeed = project.rootProject.testSeed.tokenize(':').get(0)
             final long seed = Long.parseUnsignedLong(firstPartOfSeed, 16)
@@ -113,12 +166,14 @@ class VagrantTestPlugin implements Plugin<Project> {
 
         DISTRIBUTION_ARCHIVES.each {
             // Adds a dependency for the current version
-            project.dependencies.add(BATS, project.dependencies.project(path: ":distribution:${it}", configuration: 'archives'))
+            project.dependencies.add(PACKAGING_CONFIGURATION,
+                    project.dependencies.project(path: ":distribution:${it}", configuration: 'archives'))
         }
 
         UPGRADE_FROM_ARCHIVES.each {
             // The version of elasticsearch that we upgrade *from*
-            project.dependencies.add(BATS, "org.elasticsearch.distribution.${it}:elasticsearch:${upgradeFromVersion}@${it}")
+            project.dependencies.add(PACKAGING_CONFIGURATION,
+                    "org.elasticsearch.distribution.${it}:elasticsearch:${upgradeFromVersion}@${it}")
         }
 
         project.extensions.esvagrant.upgradeFromVersion = upgradeFromVersion
@@ -147,22 +202,23 @@ class VagrantTestPlugin implements Plugin<Project> {
     }
 
     private static void createPrepareVagrantTestEnvTask(Project project) {
-        File batsDir = new File("${project.buildDir}/${BATS}")
+        File packagingDir = new File("${project.buildDir}/${PACKAGING_CONFIGURATION}")
+        File batsDir = new File(packagingDir, BATS)
 
-        Task createBatsDirsTask = project.tasks.create('createBatsDirs')
-        createBatsDirsTask.outputs.dir batsDir
-        createBatsDirsTask.doLast {
+        Task createPackagingDirsTask = project.tasks.create('createBatsDirs')
+        createPackagingDirsTask.outputs.dir batsDir
+        createPackagingDirsTask.doLast {
             batsDir.mkdirs()
         }
 
-        Copy copyBatsArchives = project.tasks.create('copyBatsArchives', Copy) {
-            dependsOn createBatsDirsTask
-            into "${batsDir}/archives"
-            from project.configurations[BATS]
+        Copy copyArchives = project.tasks.create('copyPackagingArchives', Copy) {
+            dependsOn createPackagingDirsTask
+            into "${packagingDir}/archives"
+            from project.configurations[PACKAGING_CONFIGURATION]
         }
 
         Copy copyBatsTests = project.tasks.create('copyBatsTests', Copy) {
-            dependsOn createBatsDirsTask
+            dependsOn createPackagingDirsTask
             into "${batsDir}/tests"
             from {
                 "${project.extensions.esvagrant.batsDir}/tests"
@@ -170,7 +226,7 @@ class VagrantTestPlugin implements Plugin<Project> {
         }
 
         Copy copyBatsUtils = project.tasks.create('copyBatsUtils', Copy) {
-            dependsOn createBatsDirsTask
+            dependsOn createPackagingDirsTask
             into "${batsDir}/utils"
             from {
                 "${project.extensions.esvagrant.batsDir}/utils"
@@ -180,7 +236,7 @@ class VagrantTestPlugin implements Plugin<Project> {
         // Now we iterate over dependencies of the bats configuration. When a project dependency is found,
         // we bring back its own archives, test files or test utils.
         project.afterEvaluate {
-            project.configurations.bats.dependencies.findAll {it.targetConfiguration == BATS }.each { d ->
+            project.configurations.packaging.dependencies.findAll {it.targetConfiguration == PACKAGING_CONFIGURATION }.each { d ->
                 if (d instanceof DefaultProjectDependency) {
                     DefaultProjectDependency externalBatsDependency = (DefaultProjectDependency) d
                     Project externalBatsProject = externalBatsDependency.dependencyProject
@@ -189,9 +245,6 @@ class VagrantTestPlugin implements Plugin<Project> {
                     if (project.extensions.esvagrant.inheritTests) {
                         copyBatsTests.from(externalBatsProject.files("${externalBatsDir}/tests"))
                     }
-                    if (project.extensions.esvagrant.inheritTestArchives) {
-                        copyBatsArchives.from(externalBatsDependency.projectConfiguration.files)
-                    }
                     if (project.extensions.esvagrant.inheritTestUtils) {
                         copyBatsUtils.from(externalBatsProject.files("${externalBatsDir}/utils"))
                     }
@@ -200,20 +253,20 @@ class VagrantTestPlugin implements Plugin<Project> {
         }
 
         Task createVersionFile = project.tasks.create('createVersionFile', FileContentsTask) {
-            dependsOn createBatsDirsTask
-            file "${batsDir}/archives/version"
+            dependsOn createPackagingDirsTask
+            file "${packagingDir}/archives/version"
             contents project.version
         }
 
         Task createUpgradeFromFile = project.tasks.create('createUpgradeFromFile', FileContentsTask) {
-            dependsOn createBatsDirsTask
-            file "${batsDir}/archives/upgrade_from_version"
+            dependsOn createPackagingDirsTask
+            file "${packagingDir}/archives/upgrade_from_version"
             contents project.extensions.esvagrant.upgradeFromVersion
         }
 
-        Task vagrantSetUpTask = project.tasks.create('setupBats')
+        Task vagrantSetUpTask = project.tasks.create('setupPackaging')
         vagrantSetUpTask.dependsOn 'vagrantCheckVersion'
-        vagrantSetUpTask.dependsOn copyBatsTests, copyBatsUtils, copyBatsArchives, createVersionFile, createUpgradeFromFile
+        vagrantSetUpTask.dependsOn copyBatsTests, copyBatsUtils, copyArchives, createVersionFile, createUpgradeFromFile
     }
 
     private static void createPackagingTestTask(Project project) {
@@ -251,6 +304,9 @@ class VagrantTestPlugin implements Plugin<Project> {
     private static void createVagrantBoxesTasks(Project project) {
         assert project.extensions.esvagrant.boxes != null
 
+        assert project.extensions.esvagrant.vagrantEnvVars != null
+        Map<String, String> vagrantEnvVars = project.extensions.esvagrant.vagrantEnvVars
+
         assert project.tasks.stop != null
         Task stop = project.tasks.stop
 
@@ -263,8 +319,8 @@ class VagrantTestPlugin implements Plugin<Project> {
         assert project.tasks.virtualboxCheckVersion != null
         Task virtualboxCheckVersion = project.tasks.virtualboxCheckVersion
 
-        assert project.tasks.setupBats != null
-        Task setupBats = project.tasks.setupBats
+        assert project.tasks.setupPackaging != null
+        Task setupPackaging = project.tasks.setupPackaging
 
         assert project.tasks.packagingTest != null
         Task packagingTest = project.tasks.packagingTest
@@ -272,25 +328,16 @@ class VagrantTestPlugin implements Plugin<Project> {
         assert project.tasks.platformTest != null
         Task platformTest = project.tasks.platformTest
 
-        /*
-         * We always use the main project.rootDir as Vagrant's current working directory (VAGRANT_CWD)
-         * so that boxes are not duplicated for every Gradle project that use this VagrantTestPlugin.
-         */
-        def vagrantEnvVars = [
-                'VAGRANT_CWD'           : "${project.rootDir.absolutePath}",
-                'VAGRANT_VAGRANTFILE'   : 'Vagrantfile',
-                'VAGRANT_PROJECT_DIR'   : "${project.projectDir.absolutePath}"
-        ]
-
         // Each box gets it own set of tasks
         for (String box : BOXES) {
-            String boxTask = box.capitalize().replace('-', '')
 
+            String boxTask = box.capitalize().replace('-', '')
             // always add a halt task for all boxes, so clean makes sure they are all shutdown
             Task halt = project.tasks.create("vagrant${boxTask}#halt", VagrantCommandTask) {
                 command 'halt'
                 boxName box
                 environmentVars vagrantEnvVars
+                description "Runs 'vagrant halt' for box ${box}"
             }
             stop.dependsOn(halt)
 
@@ -300,8 +347,9 @@ class VagrantTestPlugin implements Plugin<Project> {
                 boxName box
                 environmentVars vagrantEnvVars
                 dependsOn vagrantCheckVersion, virtualboxCheckVersion
+                description "Runs 'vagrant update' for box ${box}"
             }
-            update.mustRunAfter(setupBats)
+            update.mustRunAfter(setupPackaging)
 
             Task up = project.tasks.create("vagrant${boxTask}#up", VagrantCommandTask) {
                 command 'up'
@@ -321,58 +369,169 @@ class VagrantTestPlugin implements Plugin<Project> {
                 /* It'd be possible to check if the box is already up here and output
                   SKIPPED but that would require running vagrant status which is slow! */
                 dependsOn update
+                description "Runs 'vagrant up' for box ${box} (with provisioning)"
             }
 
+            // We use an Exec task here (instead of VagrantCommandTask) because we want the full output displayed
             Task smoke = project.tasks.create("vagrant${boxTask}#smoketest", Exec) {
                 environment vagrantEnvVars
                 dependsOn up
                 finalizedBy halt
-                commandLine 'vagrant', 'ssh', box, '--command',
-                        "set -o pipefail && echo 'Hello from ${project.path}' | sed -ue 's/^/    ${box}: /'"
+                description "Tries to run a 'hello world' shell command on box ${box}"
+            }
+            if (box in LINUX_BOXES) {
+                smoke.commandLine = ['vagrant', 'ssh', box, '--command', """
+                    set -euo pipefail
+                    echo 'Hello from ${project.path}' | sed -ue 's/^/    ${box}: /'
+                """]
+            } else {
+                smoke.commandLine = ['vagrant', 'winrm', box, '--command', wrapPowershell("""
+                    Write-Host '    ${box}: Hello from ${project.path}'
+                """)]
             }
             vagrantSmokeTest.dependsOn(smoke)
 
-            Task packaging = project.tasks.create("vagrant${boxTask}#packagingTest", BatsOverVagrantTask) {
-                remoteCommand BATS_TEST_COMMAND
+            Task relocateProject = project.tasks.create("vagrant${boxTask}#relocateProject", VagrantCommandTask) {
                 boxName box
                 environmentVars vagrantEnvVars
-                dependsOn up, setupBats
+                dependsOn up
                 finalizedBy halt
+                description "Run a command on box ${box} to move the project to vagrant's home directory"
             }
+            if (box in LINUX_BOXES) {
+                relocateProject.command = 'ssh'
+                relocateProject.args = ['--command', """
+                    set -euo pipefail
+                    rm -rf ~/elasticsearch ~/elasticsearch-extra
+                    rsync -rlt /elasticsearch/ ~/elasticsearch
+                    if [ -d /elasticsearch-extra ]; then
+                        rsync -rlt /elasticsearch-extra/ ~/elasticsearch-extra
+                    fi
+                """]
+            } else {
+                relocateProject.command = 'winrm'
+                relocateProject.args = ['--command', wrapPowershell("""
+                    if (Test-Path "~/elasticsearch") {
+                        Remove-Long-Path C:\\Users\\vagrant\\elasticsearch
+                    }
+                    if (Test-Path "~/elasticsearch-extra") {
+                        Remove-Long-Path C:\\Users\\vagrant\\elasticsearch-extra
+                    }
 
-            TaskExecutionAdapter packagingReproListener = new TaskExecutionAdapter() {
+                    Copy-Long-Path C:\\elasticsearch C:\\Users\\vagrant\\elasticsearch
+
+                    if (Test-Path "C:/elasticsearch-extra") {
+                        Copy-Long-Path C:\\elasticsearch-extra C:\\Users\\vagrant\\elasticsearch-extra
+                    }
+                """)]
+            }
+
+            /*
+             * Other projects can hook dependencies onto this task to get the vm ready to run the gradle build
+             */
+            Task prepareGradle = project.tasks.create("vagrant${boxTask}#prepareGradleBuild") {
+                dependsOn relocateProject
+            }
+            prepareGradle.ext.box = box
+
+            if (box in LINUX_BOXES) {
+                Task batsPackagingTest = project.tasks.create("vagrant${boxTask}#batsPackagingTest", BatsOverVagrantTask) {
+                    remoteCommand BATS_TEST_COMMAND
+                    boxName box
+                    environmentVars vagrantEnvVars
+                    dependsOn up, setupPackaging
+                    finalizedBy halt
+                    description "Runs the BATS packaging test scripts on box ${box}"
+                }
+
+                TaskExecutionAdapter batsPackagingReproListener = new TaskExecutionAdapter() {
+                    @Override
+                    void afterExecute(Task task, TaskState state) {
+                        if (state.failure != null) {
+                            println "REPRODUCE WITH: gradle ${batsPackagingTest.path} " +
+                                    "-Dtests.seed=${project.testSeed} "
+                        }
+                    }
+                }
+                batsPackagingTest.doFirst {
+                    project.gradle.addListener(batsPackagingReproListener)
+                }
+                batsPackagingTest.doLast {
+                    project.gradle.removeListener(batsPackagingReproListener)
+                }
+                if (project.extensions.esvagrant.boxes.contains(box)) {
+                    packagingTest.dependsOn(batsPackagingTest)
+                }
+            }
+
+            Task groovyPackagingTest = project.tasks.create("vagrant${boxTask}#groovyPackagingTest", VagrantCommandTask) {
+                boxName box
+                environmentVars vagrantEnvVars
+                dependsOn up, setupPackaging, prepareGradle
+                finalizedBy halt
+                description "Runs the Groovy portable packaging tests on box ${box}"
+            }
+            if (box in LINUX_BOXES) {
+                groovyPackagingTest.command = 'ssh'
+                groovyPackagingTest.args = ['--command', """
+                    set -euo pipefail
+                    cd ~/elasticsearch
+                    \$GRADLE_HOME/bin/gradle ${-> project.extensions.esvagrant.testTask} -Dtests.seed=${-> project.testSeed}
+                """]
+            } else {
+                groovyPackagingTest.command = 'winrm'
+                groovyPackagingTest.args = ['--command', wrapPowershell("""
+                    cd ~/elasticsearch
+                    & "\$Env:GRADLE_HOME/bin/gradle.bat" "${-> project.extensions.esvagrant.testTask}" "-Dtests.seed=${-> project.testSeed}"
+                """)]
+            }
+
+            TaskExecutionAdapter groovyPackagingReproListener = new TaskExecutionAdapter() {
                 @Override
                 void afterExecute(Task task, TaskState state) {
                     if (state.failure != null) {
-                        println "REPRODUCE WITH: gradle ${packaging.path} " +
-                            "-Dtests.seed=${project.testSeed} "
+                        println "REPRODUCE WITH: gradle ${groovyPackagingTest.path} -Dtests.seed=${project.testSeed}"
                     }
                 }
             }
-            packaging.doFirst {
-                project.gradle.addListener(packagingReproListener)
+
+            groovyPackagingTest.doFirst {
+                project.gradle.addListener(groovyPackagingReproListener)
             }
-            packaging.doLast {
-                project.gradle.removeListener(packagingReproListener)
+            groovyPackagingTest.doLast {
+                project.gradle.removeListener(groovyPackagingReproListener)
             }
             if (project.extensions.esvagrant.boxes.contains(box)) {
-                packagingTest.dependsOn(packaging)
+                packagingTest.dependsOn(groovyPackagingTest)
             }
 
+
             Task platform = project.tasks.create("vagrant${boxTask}#platformTest", VagrantCommandTask) {
-                command 'ssh'
                 boxName box
                 environmentVars vagrantEnvVars
-                dependsOn up
+                dependsOn up, prepareGradle
                 finalizedBy halt
-                args '--command', PLATFORM_TEST_COMMAND + " -Dtests.seed=${-> project.testSeed}"
             }
+            if (box in LINUX_BOXES) {
+                platform.command = 'ssh'
+                platform.args = ['--command', """
+                    set -euo pipefail
+                    cd ~/elasticsearch
+                    \$GRADLE_HOME/bin/gradle test integTest -Dtests.seed=${-> project.testSeed}
+                """]
+            } else {
+                platform.command = 'winrm'
+                platform.args = ['--command', wrapPowershell("""
+                    cd ~/elasticsearch
+                    & "\$Env:GRADLE_HOME/bin/gradle.bat" "test" "integTest" "-Dtests.seed=${-> project.testSeed}"
+                """)]
+            }
+
             TaskExecutionAdapter platformReproListener = new TaskExecutionAdapter() {
                 @Override
                 void afterExecute(Task task, TaskState state) {
                     if (state.failure != null) {
-                        println "REPRODUCE WITH: gradle ${platform.path} " +
-                            "-Dtests.seed=${project.testSeed} "
+                        println "REPRODUCE WITH: gradle ${platform.path} -Dtests.seed=${project.testSeed}"
                     }
                 }
             }
@@ -387,4 +546,23 @@ class VagrantTestPlugin implements Plugin<Project> {
             }
         }
     }
+
+    /*
+     * The library that vagrant uses to talk to WinRM [1] executes commands in such a way that causes commands to return a success exit
+     * code when $ErrorActionPreference = "Stop" is set and the script is terminated by an error. If we wrap our command inside a single
+     * call to the powershell executable, then failures cause that call to return a failure exit code, which is picked up by WinRB's
+     * wrapper script
+     *
+     * This returns GString and not String because several of the scripts that use closures that are lazily evaluated
+     * (of the form "foo ${-> bar}") to pull in configuration from the plugin's extension after the plugin has already been applied
+     *
+     * [1] https://github.com/WinRb/WinRM/blob/52918d73590449466332aaf06f69b0cf77d91dc7/lib/winrm/shells/power_shell.rb#L99-L115
+     */
+    static GString wrapPowershell(GString script) {
+        "powershell -Command { \$ErrorActionPreference = 'Stop'; ${-> script} }"
+    }
+
+    static GString wrapPowershell(String script) {
+        wrapPowershell("${script}")
+    }
 }
diff --git a/qa/packaging-common/build.gradle b/qa/packaging-common/build.gradle
new file mode 100644
index 0000000000000..104a0741c6a7e
--- /dev/null
+++ b/qa/packaging-common/build.gradle
@@ -0,0 +1,22 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+apply plugin: 'groovy'
+
+description = 'Shared utilities for packaging tests'
diff --git a/qa/packaging/build.gradle b/qa/packaging/build.gradle
new file mode 100644
index 0000000000000..3e0e3d402c582
--- /dev/null
+++ b/qa/packaging/build.gradle
@@ -0,0 +1,30 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+apply plugin: 'groovy'
+apply plugin: 'application'
+
+description = 'Tests that packaging (distributions) and plugins can be installed and run'
+
+mainClassName = 'org.elasticsearch.packaging.PackagingMain'
+
+dependencies {
+    compile localGroovy()
+    compile project(':qa:packaging-common')
+}
diff --git a/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy b/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy
new file mode 100644
index 0000000000000..4c3b39fdafb1f
--- /dev/null
+++ b/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy
@@ -0,0 +1,27 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.packaging
+
+class PackagingMain {
+
+    static void main(String[] args) {
+        println("hello world groovy packaging test")
+    }
+}
diff --git a/qa/vagrant/build.gradle b/qa/vagrant/build.gradle
index 40fc3998185c8..abd19e2293325 100644
--- a/qa/vagrant/build.gradle
+++ b/qa/vagrant/build.gradle
@@ -20,19 +20,23 @@
 apply plugin: 'elasticsearch.vagrantsupport'
 apply plugin: 'elasticsearch.vagrant'
 
+esvagrant {
+  testTask ':qa:packaging:run'
+}
+
 List<String> plugins = []
 for (Project subproj : project.rootProject.subprojects) {
   if (subproj.path.startsWith(':plugins:')) {
     // add plugin as a dep
     dependencies {
-      bats project(path: "${subproj.path}", configuration: 'zip')
+      packaging project(path: "${subproj.path}", configuration: 'zip')
     }
     plugins.add(subproj.name)
   }
 }
 plugins = plugins.toSorted()
 
-setupBats {
+setupPackaging {
   doFirst {
     File expectedPlugins = file('build/plugins/expected')
     expectedPlugins.parentFile.mkdirs()
diff --git a/settings.gradle b/settings.gradle
index cdac374198b5e..863b3f9ce5f62 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -70,6 +70,8 @@ List projects = [
   'qa:mixed-cluster',
   'qa:multi-cluster-search',
   'qa:no-bootstrap-tests',
+  'qa:packaging',
+  'qa:packaging-common',
   'qa:reindex-from-old',
   'qa:rolling-upgrade',
   'qa:smoke-test-client',