Enable Transformer LT search space for Dynamic Neural Architecture Search Toolkit

[macsz]

Type of Change

Feature. Extends current Dynamic Neural Architecture Search Toolkit set of supported search spaces with Transformer LT super-network for En-De language translation.

Description

DyNAS-T (Dynamic Neural Architecture Search Toolkit) is a SuperNet NAS optimization package (part of Intel Neural Compressor) designed for finding the optimal Pareto front during neural architecture search while minimizing the number of search validation measurements. It supports single-/multi-/many-objective problems for a variety of domains supported. The system currently heavily utilizes the pymoo optimization library. Some of the key DyNAS-T features are:

• Automatic handling of super-network parameters for search and predictor training
• Genetic Algorithm (e.g., NSGA-II) multi-objective subnetworks
• LINAS (Lightweight Iterative Neural Architecture Search) accelerated search using approximate predictors
• Warm-start (transfer) search
• Search population statistical analysis

This PR extends supported search spaces with Transformer-based Language Translation (transformer_lt_wmt_en_de) for English and German languages. Implementation of the supernet is based on Hardware Aware Transformers (HAT) by MIT HAN Lab.

How has this PR been tested?

To run an example, trained supernet weights and preprocessed WMT En-De dataset is needed. Both can be downloaded from Hardware Aware Transformers (HAT) repository.

• Script to download preprocessed dataset: link
• Download trained supernet weights: link

Example code to test new functionality:

config = NASConfig(approach='dynas', search_algorithm='nsga2')
config.dynas.supernet = 'transformer_lt_wmt_en_de'
config.seed = 42
config.dynas.metrics = ['acc', 'macs']

config.dynas.population = 50
config.dynas.num_evals = 500
config.dynas.batch_size = 64
config.dynas.results_csv_path = 'results.csv'
config.dynas.dataset_path = '/datasets/hat_dataset/data/binary/wmt16_en_de'
config.dynas.supernet_ckpt_path  ='/datasets/hat_dataset/HAT_wmt14ende_super_space0.pt'
agent = NAS(config)
results = agent.search()

Dependency Change?

• fairseq
• sacremoses
• torchprofile

Signed-off-by: Maciej Szankin [email protected]

[macsz]

Failing on timeout now. Locally the same test passes. Will re-run to see if anything has changed...

[chensuyue]

/Azurepipeline run

[ftian1]

@macsz we have root caused the timeout issue and fixed it. we also made rebase operation. now we are waiting for test report. if it passes, we will merge it today.

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[macsz]

@macsz we have root caused the timeout issue and fixed it. we also made rebase operation. now we are waiting for test report. if it passes, we will merge it today.

Thanks! Appreciate the help to rebase to speed things up! I will be monitoring the PR as well.

[chensuyue]

Failed with UT coverage regression. I think we can merge at first for code freeze.

[macsz]

Merged master & Removed cleanup in UTs that I added before you fixed the model cache problem. This triggered the CI to re-run the tests.

So we are ok with this PR once it passes? To avoid additional delays I won't be touching anything till you say so.

By the way - Where in the CI can I see visual test coverage, like the one you posted @chensuyue ? Tried decoding raw output but it was a little bit too much ...

[chensuyue]

1. Go to the artifacts.
   
2. Download the coverage report package.
   
3. Open the coverage compare html.
   

[macsz]

Should we merge now for code freeze, as @chensuyue suggested? I will work on adding unit tests as a follow up to this PR.