upgrade to latest sagemaker-experiments #286

Removing intermediate container bdd44c2df758
 ---> 70ad5bf0207d
Step 26/31 : COPY $FRAMEWORK_SUPPORT_INSTALLABLE .
 ---> af15e3cad781
Step 27/31 : RUN pip3 --no-cache-dir install --upgrade     pip     setuptools
 ---> Running in 0bde5e70336c
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/70/b8/b23170ddda9f07c3444d49accde49f2b92f97bb2f2ebc312618ef12e4bd6/setuptools-46.0.0-py3-none-any.whl (582kB)
Installing collected packages: pip, setuptools
  Found existing installation: pip 9.0.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
  Found existing installation: setuptools 39.0.1
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-20.0.2 setuptools-46.0.0
Removing intermediate container 0bde5e70336c
 ---> d59ae273c3a0
Step 28/31 : RUN ln -s $(which python3) /usr/local/bin/python  && ln -s $(which pip3) /usr/bin/pip
 ---> Running in 8a70d17bab6f
Removing intermediate container 8a70d17bab6f
 ---> 39cad4ef9531
Step 29/31 : RUN pip install --no-cache-dir -U     numpy==1.17.4     scipy==1.2.2     scikit-learn==0.20.3     pandas==0.24.2     Pillow==6.2.1     h5py==2.9.0     keras_applications==1.0.8     keras_preprocessing==1.1.0     keras==2.3.1     requests==2.22.0     ${SMDEBUG_URL}     sagemaker==1.50.17     sagemaker-experiments==0.1.7     mpi4py==3.0.2     "cryptography>=2.3"     "sagemaker-tensorflow>=1.15,<1.16"  && pip install --force-reinstall --no-cache-dir -U     ${TF_URL}  && pip install --force-reinstall --no-cache-dir -U     horovod==0.18.2  && pip install --no-cache-dir -U     $FRAMEWORK_SUPPORT_INSTALLABLE     awscli && rm -f $FRAMEWORK_SUPPORT_INSTALLABLE
 ---> Running in a23aa88e9147
Collecting numpy==1.17.4
  Downloading numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl (20.0 MB)
Collecting scipy==1.2.2
  Downloading scipy-1.2.2-cp36-cp36m-manylinux1_x86_64.whl (24.8 MB)
Collecting scikit-learn==0.20.3
  Downloading scikit_learn-0.20.3-cp36-cp36m-manylinux1_x86_64.whl (5.4 MB)
Collecting pandas==0.24.2
  Downloading pandas-0.24.2-cp36-cp36m-manylinux1_x86_64.whl (10.1 MB)
Collecting Pillow==6.2.1
  Downloading Pillow-6.2.1-cp36-cp36m-manylinux1_x86_64.whl (2.1 MB)
Collecting h5py==2.9.0
  Downloading h5py-2.9.0-cp36-cp36m-manylinux1_x86_64.whl (2.8 MB)
Collecting keras_applications==1.0.8
  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
Collecting keras_preprocessing==1.1.0
  Downloading Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41 kB)
Collecting keras==2.3.1
  Downloading Keras-2.3.1-py2.py3-none-any.whl (377 kB)
Collecting requests==2.22.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting smdebug==0.7.0b20200304
�[91m  ERROR: HTTP error 403 while getting https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl
�[0m�[91m  ERROR: Could not install requirement smdebug==0.7.0b20200304 from https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl because of error 403 Client Error: Forbidden for url: https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl
�[0m�[91mERROR: Could not install requirement smdebug==0.7.0b20200304 from https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl because of HTTP error 403 Client Error: Forbidden for url: https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl for URL https://s3.amazonaws.com/kannanva-smdebug-0.7/smdebug-0.7.0b20200304-py2.py3-none-any.whl
�[0mThe command '/bin/sh -c pip install --no-cache-dir -U     numpy==1.17.4     scipy==1.2.2     scikit-learn==0.20.3     pandas==0.24.2     Pillow==6.2.1     h5py==2.9.0     keras_applications==1.0.8     keras_preprocessing==1.1.0     keras==2.3.1     requests==2.22.0     ${SMDEBUG_URL}     sagemaker==1.50.17     sagemaker-experiments==0.1.7     mpi4py==3.0.2     "cryptography>=2.3"     "sagemaker-tensorflow>=1.15,<1.16"  && pip install --force-reinstall --no-cache-dir -U     ${TF_URL}  && pip install --force-reinstall --no-cache-dir -U     horovod==0.18.2  && pip install --no-cache-dir -U     $FRAMEWORK_SUPPORT_INSTALLABLE     awscli && rm -f $FRAMEWORK_SUPPORT_INSTALLABLE' returned a non-zero code: 1

sagemaker-bot · 2020-03-10T19:59:02Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ghost · 2020-03-10T20:41:14Z

==================================== ERRORS ====================================
_______ ERROR collecting test/integration/sagemaker/test_experiments.py ________
ImportError while importing test module '/codebuild/output/src627080960/src/github.com/aws/sagemaker-tensorflow-container/test/integration/sagemaker/test_experiments.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/integration/sagemaker/test_experiments.py:21: in <module>
    from smexperiments.experiment import Experiment
E   ImportError: No module named smexperiments.experiment

danabens · 2020-03-10T20:47:00Z

the test has the "skip_py2_containers" mark on on it but is still being run with the py2 target:

No data to report.
___________________________________ summary ____________________________________
  py36: commands succeeded
  congratulations :)

[Container] 2020/03/10 19:49:05 Running command test_cmd="IGNORE_COVERAGE=- tox -e py27 -- test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $CPU_TAG_PY2 --py-version 2 --processor cpu --durations 10"

[Container] 2020/03/10 19:49:05 Running command execute-command-if-has-matching-changes "$test_cmd" "test/" "src/*.py" "setup.py" "docker/*" "buildspec.yml"
Changes Found. Executing command passed in as first parameter.
GLOB sdist-make: /codebuild/output/src627080960/src/github.com/aws/sagemaker-tensorflow-container/setup.py
py27 inst-nodeps: /codebuild/output/src627080960/src/github.com/aws/sagemaker-tensorflow-container/.tox/.tmp/package/1/sagemaker_tensorflow_training-2.2.4.dev0.zip
py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,absl-py==0.9.0,apipkg==1.5,appdirs==1.4.3,astor==0.8.1,atomicwrites==1.3.0,attrs==19.3.0,awscli==1.16.314,backports.shutil-get-terminal-size==1.0.0,backports.ssl-match-hostname==3.7.0.1,backports.weakref==1.0.post1,bcrypt==3.1.7,boto3==1.10.50,botocore==1.13.50,cached-property==1.5.1,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,click==7.1.1,colorama==0.4.1,configparser==4.0.2,contextlib2==0.6.0.post1,coverage==5.0.3,cryptography==2.8,distlib==0.3.0,docker==4.2.0,docker-compose==1.25.4,dockerpty==0.4.1,docopt==0.6.2,docutils==0.15.2,entrypoints==0.3,enum34==1.1.10,execnet==1.7.1,filelock==3.0.12,flake8==3.7.9,Flask==1.1.1,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.3.0,gast==0.2.2,gevent==1.4.0,google-pasta==0.1.8,greenlet==0.4.15,grpcio==1.27.2,gunicorn==19.10.0,h5py==2.10.0,idna==2.9,importlib-metadata==1.5.0,importlib-resources==1.3.1,inotify-simple==1.2.1,ipaddress==1.0.23,itsdangerous==1.1.0,Jinja2==2.11.1,jmespath==0.9.5,jsonschema==3.2.0,Keras-Applications==1.0.8,Keras-Preprocessing==1.1.0,Markdown==3.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,mock==3.0.5,more-itertools==5.0.0,numpy==1.16.6,opt-einsum==2.3.2,packaging==20.3,pandas==0.24.2,paramiko==2.4.2,pathlib2==2.3.5,Pillow==6.2.2,pluggy==0.13.1,protobuf==3.11.3,protobuf3-to-dict==0.1.5,psutil==5.4.8,py==1.8.1,pyasn1==0.4.8,pycodestyle==2.5.0,pycparser==2.20,pyflakes==2.1.1,PyNaCl==1.3.0,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==4.6.9,pytest-cov==2.8.1,pytest-forked==1.1.3,pytest-xdist==1.31.0,python-dateutil==2.8.0,pytz==2019.3,PyYAML==5.3,requests==2.23.0,requests-mock==1.7.0,retrying==1.3.3,rsa==3.4.2,s3transfer==0.2.1,sagemaker==1.50.1,sagemaker-containers==2.8.1,sagemaker-tensorflow-training==2.2.4.dev0,scandir==1.10.0,scikit-learn==0.20.4,scipy==1.2.3,singledispatch==3.4.0.3,six==1.13.0,sklearn==0.0,smdebug-rulesconfig==0.1.2,subprocess32==3.5.4,tensorboard==1.15.0,tensorflow==1.15.0,tensorflow-estimator==1.15.1,termcolor==1.1.0,texttable==1.6.2,toml==0.10.0,tox==3.14.5,typing==3.7.4.1,urllib3==1.25.8,virtualenv==20.0.10,wcwidth==0.1.8,websocket-client==0.57.0,Werkzeug==0.15.5,wrapt==1.12.1,zipp==1.2.0
py27 run-test-pre: PYTHONHASHSEED='4200421712'
py27 run-test: commands[0] | coverage run --rcfile .coveragerc_py27 --source sagemaker_tensorflow_container -m py.test test/integration/sagemaker -n 8 --region us-west-2 --docker-base-name sagemaker-test --account-id 142577830533 --tag 1.15.2-cpu-py2-sagemaker-tensorflow-container-pr-8789f131-d767-49bd-97a5-51c297ed6f8e --py-version 2 --processor cpu --durations 10
============================= test session starts ==============================
platform linux2 -- Python 2.7.17, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py27/.pytest_cache
rootdir: /codebuild/output/src627080960/src/github.com/aws/sagemaker-tensorflow-container, inifile: pytest.ini
plugins: xdist-1.31.0, cov-2.8.1, forked-1.1.3, requests-mock-1.7.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
gw0 [7] / gw1 [7] / gw2 [7] / gw3 [7] / gw4 [7] / gw5 [7] / gw6 [7] / gw7 [7]

.....F.                                                                  [100%]
==================================== ERRORS ====================================
_______ ERROR collecting test/integration/sagemaker/test_experiments.py ________
ImportError while importing test module '/codebuild/output/src627080960/src/github.com/aws/sagemaker-tensorflow-container/test/integration/sagemaker/test_experiments.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/integration/sagemaker/test_experiments.py:21: in <module>
    from smexperiments.experiment import Experiment
E   ImportError: No module named smexperiments.experiment
=================================== FAILURES ===================================

sagemaker-bot · 2020-03-10T22:07:02Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ghost · 2020-03-10T23:04:02Z

test/integration/sagemaker/test_experiments.py

+from smexperiments.experiment import Experiment
+from smexperiments.trial import Trial
+from smexperiments.trial_component import TrialComponent


Can you move these import lines to be inside the function? After line 36?

sagemaker-bot · 2020-03-11T00:37:34Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-11T02:18:11Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ghost · 2020-03-11T02:35:02Z

@danabens Build still failling. Can you try a pytest fixture like this: https://github.com/aws/sagemaker-mxnet-container/blob/6a1df9164b0c94764f4ace3ef3310ac4fc2e8fb9/test/conftest.py#L134 ?

sagemaker-bot · 2020-03-11T05:53:42Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-11T07:43:01Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ghost · 2020-03-11T07:50:16Z

=================================== FAILURES ===================================
________________________________ test_training _________________________________
[gw5] linux2 -- Python 2.7.17 /codebuild/output/src003255308/src/github.com/aws/sagemaker-tensorflow-container/.tox/py27/bin/python

sagemaker_session = <sagemaker.session.Session object at 0x7fd651564ad0>
ecr_image = '142577830533.dkr.ecr.us-west-2.amazonaws.com/sagemaker-test:1.15.2-cpu-py2-sagemaker-tensorflow-container-pr-f0586a4f-1e26-481c-8499-2e169a685adb'
instance_type = 'ml.p2.xlarge', framework_version = '2.0.0', py_version = ['2']

    @pytest.mark.skip_py2_containers
    def test_training(sagemaker_session, ecr_image, instance_type, framework_version, py_version):
    
        if py_version == '2' or py_version is None:
            pytest.skip('Skipping python2 {}'.format(py_version))
            return
    
>       from smexperiments.experiment import Experiment
E       ImportError: No module named smexperiments.experiment

test/integration/sagemaker/test_experiments.py:36: ImportError
=============================== warnings summary ===============================
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335
  /codebuild/output/src003255308/src/github.com/aws/sagemaker-tensorflow-container/.tox/py27/local/lib/python2.7/site-packages/_pytest/mark/structures.py:335: PytestUnknownMarkWarning: Unknown pytest.mark.skip_py2_containers - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    PytestUnknownMarkWarning,

-- Docs: https://docs.pytest.org/en/latest/warnings.html

ghost · 2020-03-11T18:22:16Z

setup.py

                 'requests-mock', 'awscli==1.16.314'],
-        'benchmark': ['click']
+        'benchmark': ['click'],
+        ':python_version=="3.6"': ['sagemaker-experiments==0.1.7']


Would this need to be fixed to be >3.5 too?

Just like in the MXNet/Pytorch containers?

it will work for now since the only 3 version is 3.6. should be updated it for future revisions though.

ill update tf once this pr is merged

sagemaker-bot · 2020-03-11T19:02:58Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

danabens force-pushed the experiments-upgrade branch from 3e697b8 to e9f5bd6 Compare February 18, 2020 00:24

danabens requested a review from a user February 19, 2020 22:12

ghost closed this Feb 27, 2020

ghost reopened this Feb 27, 2020

danabens force-pushed the experiments-upgrade branch from e9f5bd6 to c626229 Compare February 28, 2020 01:53

danabens force-pushed the experiments-upgrade branch from c626229 to a985957 Compare March 2, 2020 16:47

danabens force-pushed the experiments-upgrade branch from a985957 to f3d495e Compare March 2, 2020 17:04

danabens force-pushed the experiments-upgrade branch from f3d495e to aea0e8d Compare March 2, 2020 17:24

danabens force-pushed the experiments-upgrade branch from aea0e8d to fa40264 Compare March 2, 2020 22:03

danabens force-pushed the experiments-upgrade branch from fa40264 to c9474fc Compare March 2, 2020 22:33

danabens force-pushed the experiments-upgrade branch from c9474fc to ce0ff68 Compare March 3, 2020 01:07

YYStreet self-requested a review March 3, 2020 19:28

danabens force-pushed the experiments-upgrade branch from ce0ff68 to 961f166 Compare March 3, 2020 23:21

danabens mentioned this pull request Mar 3, 2020

add sagemaker-experiments #301

Merged

danabens force-pushed the experiments-upgrade branch from 5af0445 to 961f166 Compare March 5, 2020 00:45

danabens force-pushed the experiments-upgrade branch from 961f166 to bd993e6 Compare March 5, 2020 01:12

danabens force-pushed the experiments-upgrade branch from dd748ee to 3b610e7 Compare March 10, 2020 00:30

danabens force-pushed the experiments-upgrade branch from c560167 to 3082a9d Compare March 10, 2020 05:16

ghost reviewed Mar 10, 2020

View reviewed changes

danabens force-pushed the experiments-upgrade branch from 67d1caa to 5b6f759 Compare March 10, 2020 23:17

YYStreet requested a review from TusharKanekiDey March 11, 2020 00:51

danabens force-pushed the experiments-upgrade branch from 5b6f759 to 6019f94 Compare March 11, 2020 02:38

upgrade to latest sagemaker-experiments

8626260

danabens force-pushed the experiments-upgrade branch from 3874b9d to 8626260 Compare March 11, 2020 16:14

TusharKanekiDey and others added 2 commits March 11, 2020 09:55

Merge branch 'master' into experiments-upgrade

17bf225

Merge branch 'master' into experiments-upgrade

dfb6385

ghost approved these changes Mar 11, 2020

View reviewed changes

ghost self-requested a review March 11, 2020 18:23

ghost approved these changes Mar 11, 2020

View reviewed changes

YYStreet approved these changes Mar 11, 2020

View reviewed changes

YYStreet merged commit d87e930 into aws:master Mar 11, 2020

upgrade to latest sagemaker-experiments #286

upgrade to latest sagemaker-experiments #286

Uh oh!

Conversation

danabens commented Feb 12, 2020

Uh oh!

ghost commented Feb 17, 2020

Uh oh!

sagemaker-bot commented Feb 18, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 28, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 4, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 5, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 10, 2020

AWS CodeBuild CI Report

Uh oh!

danabens commented Mar 10, 2020

Uh oh!

sagemaker-bot commented Mar 10, 2020

AWS CodeBuild CI Report

Uh oh!

ghost commented Mar 10, 2020

Uh oh!

danabens commented Mar 10, 2020

Uh oh!

sagemaker-bot commented Mar 10, 2020

AWS CodeBuild CI Report

Uh oh!

ghost Mar 10, 2020

Choose a reason for hiding this comment

Uh oh!

danabens Mar 10, 2020

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 11, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 11, 2020

AWS CodeBuild CI Report

Uh oh!