Move analyzer test utilities to analyzer_testing package

[srawlins]

Quick background / example: I would like to move ResolveNameInScopeTest from the analyzer package to the linter package. But it depends on AbstractLinterContextTest (also in pkg/analyzer/test) which depends on PubPackageResolutionTest (also in pkg/analyzer/test). So I cannot move any tests until we have some shared base classes. Or I can copy swaths of code.

Creating shared testing code between the analyzer package, linter package, analysis_server and analysis_server_plugin packages is necessary for the plugins story. CC @bwilkerson @scheglov @pq

ContextResolutionTest

It would take significant investigation to determine how many test utility classes and mixins would be prudent to move. But as a starting place, I think it would be of huge benefit to move ContextResolutionTest into package:analyzer_utilities. How do we get there?

ResourceProviderMixin

ResourceProviderMixin (mixed into ContextResolutionTest) is declared in package:analyzer/src/test_utilities, so we're already in a good place there. Should be easier to move than other things, as it cannot depend on code in pkg/analyzer/test. This might be the best starting place.

ResolutionTest

ContextResolutionTest also mixes in ResolutionTest, so this is also a pre-requirement. Unfortunately it lives in pkg/analyzer/test, and is in a file which imports eight other files from pkg/analyzer/test. Yikes.

PubPackageResolutionTest

PubPackageResolutionTest is also an important base class, as it is the base for supporting analysis options files and package configs.

[bwilkerson]

It's quite possible that some of the base classes might include support that isn't needed by the tests that a plugin author would need to write. It might be worthwhile to start by thinking about what the requirements are and what those pieces of support actually depend on. Moving the rest of the support out to mixins might reduce the number of dependencies.

It's also the case that plugin authors will need to be able to do some things that we currently don't have support for.

For example, here are some initial thoughts.

Plugin authors need to be able to test both lints and fixes.

For both of those use cases the tests need to be able to create the code being tested and any supporting code in an in-memory disk layout. That includes being able to create analysis options files, package config files, and any (mock) packages that the test code needs to be able to depend on (including packages only available to the plugin author).

For lint tests they need to be able to specify which file(s) to analyze and the diagnostics that the lint is expected to produce. They'll also need to be able to register the lint rule being tested.

For fix tests they need to do every they need for testing lint rules as well as confirming the result of applying the fix to the appropriate files.

They probably don't need the printers that we use for printing resolution output, the findNode and findElement utilities, and a reasonable portion of the assert methods, among other things.

Also, are you thinking of analyzer_utilities as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publish analyzer_utilities? I'd prefer the former, but you might know of a reason why that isn't the best option.

[srawlins]

Also, are you thinking of analyzer_utilities as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publish analyzer_utilities?

Either. I hadn't thought of the differences.

Something that may make this task difficult is that much of ContextResolutionTest (and others I mentioned) depends on package:analyzer. It uses MemoryByteStore, AnalysisContextCollection, MockSdkLibrary, Folder, etc. I can't remember the cyclic dependency rules with certainty. But I think that two packages cannot depend on each other (directly or indirectly) through their dependencies. Two packages can only depend on each other if at least one of the dependencies is via dev_dependencies. And I think that also means that at least one of the dependency chains must involve a dev_dependency instead of a dependency, yes?

Anyways, I think that means that analyzer_utilities can have a dependency on analyzer, but then analyzer cannot have a dependency on analyzer_utilities, meaning that no code in pkg/analyzer/lib/src/test_utilities can depend on package:analyzer_utilities. That code would have to move first.

[srawlins]

Another anecdote pointing to the need to solve this:

I recently moved selection.dart from analysis_server to analysis_server_plugin. @bwilkerson pointed out I should move the test as well, selection_test.dart. When I tried doing this, I saw that this would require moving text_expectations.dart as well (or copying it). But text_expectations.dart (TextExpectationsCollector) is required by half a dozen other tests in analysis_server. So I could copy that over to analysis_server_plugin, in it's lib/ directory (something like lib/src/test_utilities) but then the analysis_server_plugin package would need a dependency (not a dev_dependency) on both package:test and packag:test_reflective_loader. ☹️ I think this is super smelly and not adhering to the purpose of the analysis_server_plugin package.

So I think we need an analysis_server_plugin_testing (or analyzer_testing or pick your better name) package. Above, @bwilkerson wrote:

Also, are you thinking of analyzer_utilities as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publish analyzer_utilities? I'd prefer the former [...]

I wrote, "Either. I hadn't thought of the differences." Now I'm thinking a fresh, test-oriented package (like, the name is also test-oriented) is warranted. We wouldn't need to publish this package for a while. Maybe not until plugin authors need it.