Deprecating Appveyor as tests CIS

[OmarTawfik]

As part of the continuous work of refactoring/cleaning up our infrastructure related tooling/scripts, and after merging the recent refactoring work, here is what I think is needed for us to be in Jenkins and drop appveyor for good:

• Move all the remaining tests (like FSharpQA suite) to NUnit projects under VisualFSharp.sln ([WIP] Migrate "fsharpqa" suite to use NUnit #872)
• Delete ell PERL related tools/dependencies
• Remove need for administrative access in build scripts (like NGENing test scripts)
• Rename all tests to make sure they don't have dots in names (underscores?)
• Add running the tests to jenkins script ([WIP] Add tests back to the jenkins PR test script #888)
• Remove appveyor scripts, verify jenkins scripts

Build workflow:

1. Restore Nuget packages
2. MSBuild VisualFSharp.sln
3. Nunit VisualFSharp.sln

This would have the benefit of:

1. Running all tests in CI (not just smoke tests), which improves verification time, and would allow us to make use of more powerful jenkins machines/add parallel jobs to test different suites in the same time.
2. Improve dev story greatly, as everything will be loadable in VS, and runnable/debuggable through the test explorer window.

/cc @Microsoft/fsharp-compiler @dsyme @enricosada thoughts?

[OmarTawfik]

There is currently one issue regarding exploring/running F# tests completely from within VS, and it is a bug in the NUnit VS adapter. I filed the following bug to them to track it:
nunit/nunit3-vs-adapter#128
Update: it is a VS issue, and we would have to find a workaround for now.

[OmarTawfik]

/cc @davkean

[enricosada]

@otawfik-ms can you link pr near the bullet points? It's easier to track current wip

That's a good meta issue, but i think there are distinct issues:

1. run current build script and run tests in jenkins
2. run tests inside visual studio, that's ok, fsharpqa test suite should be migrated to nunit
3. improve build script

The 1. (run all tests in ci) depends only on removing admin permission.
If the pr about that #888 can be completed, we can just call build.cmd from jenkins.bat, build and run the whole test suites

After that, we can port perl => nunit, that's nicer developer experience ( wip in #872 )
And improve the build script build.cmd

But having a real build script, make easier to evolve ci and workflow, for example migrate perl => nunit it's only a matter of change runtest.cmd => nunit-console.

The build script is a lot more than msbuild a solution. It should be the complete build workflow, from clone repo to vsix/nuget packages

I think the jenkins.cmd should call the build.cmd (the real build script) passing some argument or env var to parametrize the build (like now does appveyor.yml)

The build script can build, run tests and create packages, and should be the real deal (i mean used by you to create nighly and rtm packages/vsix) that mean it should have parameters for package version, sign etc. Not an compile and run test for open source only.
The output of build script (under a parameter) are the final nuget packages, vsix etc.

If build.cmd is the real deal, we can run the same command locally. That really really important.
msbuild the solution doesnt create nuget packages or that can be done with msbuild target.
We should try to avoid special msbuild target, because are harder to maintain instead of call executables from a build script, like nuget pack to create a package.

So split in multiple phases in the build script (can use multiple .cmd inside a scripts diretory):

• restore deps: nuget restore (not from vs if possibile, it's really slow) or generate others deps (like assemblyinfo.fs)
• build (msbuild .fsproj projects, each target fw or one only)
• run test (run nunit.exe )
• create vsix or packages (from nuget.exe etc)

So each step can be run individually, and jenkins.cmd it's only a list of build.cmd calls (like appveyor).
And it's easier to add step useful for developers

Atm the build.cmd can build or run part of test suites, that's really good for dev workflow.

if we msbuild directly the solution, that's different from build script or developer experience with build.cmd, and we cannot be sure build.cmd works as expected (or at all).

[OmarTawfik]

@enricosada thanks for the comments:

1. I'll look into removing admin permissions as part of the script changes. AFAIK, we only depend on this for NGening (perf improvement), which we can discard.
2. I added the PR numbers you mentioned to the checklist.
3. I agree that the build script should have the complete workflow, and VS scenario is only part of it. But it is the most dev-time consuming part. I'd gear our efforts into improving this scenario first.
4. Restoring nugets and running tests is doable via the script, and we should keep this functionality. It is only enabling it via VS as well.

Can you elaborate on your comment:
if we msbuild directly the solution, that's different from build script or developer experience with build.cmd, and we cannot be sure build.cmd works as expected (or at all).

[enricosada]

@otawfik-ms if you want to deprecate appveyor, you can do that without removing appveyor.yml (you can disable the project in appveyor website).
So we can use appveyor in our forks without adding the config file

[OmarTawfik]

@enricosada good point. Should start looking into that after we finalize #872

[ctaggart]

@otawfik-ms, AppVeyor is still very useful for those of us who create forks. It will help ensure higher quality pull requests. We are not going to stand up our own Jenkins servers. Keeping the appveyor.yml in the repo will still be useful. You can just disable the builds from the AppVeyor website or remove the webhooks in GitHub settings.

[forki]

yes appveyor build was super useful.

[OmarTawfik]

Closing this as the build was ported to Jenkins successfully.