Publishing version 1.7.8

Author: ivan-khvostishkov

.gitlab-ci.yml

@@ -0,0 +1,50 @@
+# Official language image. Look for the different tagged releases at:
+# https://hub.docker.com/r/library/python/tags/
+image: python:3.8
+
+stages:
+  - scan_source_code
+  - test
+
+scan_source_code:
+  stage: scan_source_code
+  script:
+    - pip install bandit
+    - bandit -r ./ --skip B603,B404,B101,B607 2>&1 | tee bandit.txt
+  artifacts:
+    paths:
+      - bandit.txt
+
+run_tests:
+  stage: test
+  resource_group: ssh-helper-tests
+  script:
+    # Smoke test - Python version
+    - python --version
+    # Check all running processes on the agent
+    - hostname
+    - ps xfa
+    # Install AWS CLI
+    - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/tmp/awscliv2.zip"
+    - unzip -o -q -d /tmp/ /tmp/awscliv2.zip
+    - /tmp/aws/install
+    - aws --version
+    - aws sts get-caller-identity
+    - echo "AWS default region - $AWS_DEFAULT_REGION"
+    - echo "AWS region - $AWS_REGION"
+    - aws configure list
+    # Install SSM plugin
+    - curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o "session-manager-plugin.deb"
+    - dpkg -i session-manager-plugin.deb
+    - session-manager-plugin --version
+    # Install the package
+    - pip install '.[test]'
+    # Run tests
+    - echo "SageMaker role - $SAGEMAKER_ROLE"
+    - echo "Kernel Gateway name - $KERNEL_GATEWAY_NAME"
+    - echo "Extra args for pytest - $PYTEST_EXTRA_ARGS"
+    - cd tests; pytest --junitxml=pytest_report.xml -m 'not manual' -o sagemaker_role=$SAGEMAKER_ROLE -o kernel_gateway_name=$KERNEL_GATEWAY_NAME $PYTEST_EXTRA_ARGS
+  artifacts:
+    when: always
+    reports:
+      junit: tests/pytest_report.xml

.idea/.gitignore

@@ -1,4 +1,4 @@
 ## Code of Conduct
-This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
-For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
-[email protected] with any additional questions or comments.
+This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 
+For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 
+[email protected] with any additional questions or comments.

.idea/inspectionProfiles/profiles_settings.xml

@@ -6,6 +6,38 @@ documentation, we greatly value feedback and contributions from our community.
 Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
 information to effectively respond to your bug report or contribution.
 
+## Adding new features to the SageMaker SSH Helper
+
+SageMaker SSH helper uses Test Driven Development (TDD) methodology to implement its features.
+
+Before implementing new features, check [TODO](TODO) list. The contributors are either working already 
+on these features or planning to implement them in the future.
+
+To start development, install the library on your local machine:
+```shell
+pip install '.[test]'
+```
+
+Make sure all tests are working:
+```shell
+cd tests
+pytest -m 'not manual' \
+  -o sagemaker_role=arn:aws:iam::<<YOUR_ACCOUNT_ID>>:role/service-role/<<YOUR_AmazonSageMaker_ExecutionRole>> \
+  -o kernel_gateway_name=<<YOUR_KERNEL_GATEWAY_NAME>>
+```
+*Tip:* You can pass your parameters either in command line or set it in `tests/pytest.ini`.
+
+Check `tests/test_end_to_end.py` for automation of steps 
+required for training / processing and inference:
+
+1. `test_train_e2e()` - a simple training job
+2. `test_inference_e2e()` - a training job followed by deployment
+3. `test_inference_e2e_mms()` - a training job followed by deployment to a multi-model endpoint
+4. `test_processing_e2e()` - a Spark processing job
+5. `test_processing_framework_e2e()` - a PyTorch framework processing job
+
+Then write a failing test, put code to make it pass, and make sure other tests are still working to avoid any regression.
+
 
 ## Reporting Bugs/Feature Requests
 

.idea/misc.xml

@@ -0,0 +1,29 @@
+## Flow Diagrams
+
+#### Training job
+
+Also compare with the similar flow diagram for [IDE integration with SageMaker Studio](Flows_IDE.md).
+
+The following diagram describes the flow of events for the training job use case:
+
+![Screenshot](images/overall-flow.png)
+
+1. Data Scientists (DS) starts a training job, with SSH Helper Lib as a dependency.
+2. The SageMaker Python SDK starts the job, sending the train.py script, and SSH Helper lib as code dependencies.
+3. Amazon SageMaker control plain starts the host and the container.
+4. The user script (train.py) starts running, starting SSH helper, which fetches AWS SSM agent and other packages 
+from the Internet, and installs them.
+5. SSH Helper starts the SSM agent
+6. Through SSM, SSH Helper registers the container as an SSM `managed instance`, and tags it with the DS AWS user/role name.
+7. SSH Helper printouts the `managed instance` ID. The log is streamed to CloudWatch Logs.
+8. The DS manually/automatically tails the training job logs for the `managed instance` ID.
+
+9-12. Optionally: The DS starts a process to copy over his SSH Public key to the container, needed to set up port forwarding via SSH (e.g., for remote debugging)
+
+13. The DS uses the AWS SSM CLI to start a shell providing the managed instance ID as a parameter.
+Optionally, user starts SSM with SSH port forwarding with the helper command 
+`sm-local-ssh-training connnect <<training_job_name>>`.
+14. AWS SSM IAM rules verify that the user is allowed to take this action and that the instance is tagged with the DS's AWS user/role name. Once verified, a session is created with the SSM Agent running in the container.
+15. The SSM agent generates a shell by spinning off a new bash shell process.
+Optionally, SSH port forwarding starts over SSM connection to let user connect to remote processes over TCP
+in both directions.