Upload json tagging data, update upload/download functions to use singular data_access library and configuration. #35
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…m string subsitution. Upload handles visit&tag, visit¬ag, novisit Saving untagged to INCOMPLETE_TAG state works through local func testing cleaned up saving untagged images a bit Saving tags to DB also works Working upload flow with visit&tag, visit¬ag, novisit removing creds restructing vott_json_parser file. Deleting jsonpickle version Moving things around for importclean up purposes. Upload function still working, need to refactor download and check prior existing tests pytest working, had to rename db_access_v1 to fix conflicts Some more cleanup and restructring. Tested working through upload function and pytest
…ages from db then return Reverting some accidental renaming in other directories Removing vscode files download working, still need to 1. combine db config for all 3 functions, 2. write tests Download now works with new db_access_v2. Need to create shared_db config ditto prior
minor spacing
…sts. Should probably remove hardcoded resources
mtarng
force-pushed
the
upload-download-rebase
branch
from
3a1ad09 to
92959a8
Compare
andrebriggs
reviewed
Nov 12, 2018
andrebriggs
reviewed
Nov 13, 2018
| # TODO: images need more meaningful data than just download urls | ||
| image_urls = DB_Access.get_images_for_tagging(connection, imageCount) | ||
| # DB configuration | ||
| data_access = ImageTagDataAccess(get_postgres_provider()) |
There was a problem hiding this comment.
@mtarng this method raises an exception I believe. All the code should be in a try/catch
| # TODO: Should we add TagStateId = INCOMPLETE_TAG also for fetching images? | ||
| query = ("SELECT b.ImageId, b.ImageLocation, a.TagStateId FROM Image_Tagging_State a " | ||
| "JOIN Image_Info b ON a.ImageId = b.ImageId WHERE a.TagStateId = 1 order by " | ||
| "JOIN Image_Info b ON a.ImageId = b.ImageId WHERE a.TagStateId IN (1, 4) order by " |
There was a problem hiding this comment.
@mtarng I would suggest explicitly using the enums
ghost
reviewed
Nov 13, 2018
| # Create user id | ||
| user_id = data_access.create_user(DB_Access_V2.getpass.getuser()) | ||
| logging.info("The user id for '{0}' is {1}".format(DB_Access_V2.getpass.getuser(),user_id)) | ||
| user_id = data_access.create_user("testuser") # TODO: remove this hardcoding, should be passed in the request. |
There was a problem hiding this comment.
Will fix this during next wave of changes. Plan to make user ID passed in the request.
| from azure.storage.blob import BlockBlobService, ContentSettings | ||
|
|
||
| from ..shared.db_provider import get_postgres_provider | ||
| from ..shared.db_access import ImageTagDataAccess, ImageInfo |
There was a problem hiding this comment.
Aren't we exclusively using db_access_v2 at this point?
There was a problem hiding this comment.
yes, the code is in a file named db_access_v2.py, but it's been nested inside the db_access module. Check the /shared directory now. I'm going to delete the old db_access_v1 code in a follow up commit right now.
seushermsft
reviewed
Nov 13, 2018
| } | ||
|
|
||
| def __get_filename_from_fullpath(filename): | ||
| path_components = filename.split('/') |
There was a problem hiding this comment.
Nitpick: Why not use pathlib to avoid writing custom path handling and the various bugs that come with it? Pathlib can get the path, the filename, and the filename without extension.
bnookala
pushed a commit
to bnookala/active-learning-detect
that referenced
this pull request
Dec 19, 2018
…gular data_access library and configuration. (CatalystCode#35) * Upload flow reworked to use new db_accessv2. Fixing db update bug from string subsitution. Upload handles visit&tag, visit¬ag, novisit Saving untagged to INCOMPLETE_TAG state works through local func testing cleaned up saving untagged images a bit Saving tags to DB also works Working upload flow with visit&tag, visit¬ag, novisit removing creds restructing vott_json_parser file. Deleting jsonpickle version Moving things around for importclean up purposes. Upload function still working, need to refactor download and check prior existing tests pytest working, had to rename db_access_v1 to fix conflicts Some more cleanup and restructring. Tested working through upload function and pytest * Working on refactoring download function, saving work. Need to get images from db then return Reverting some accidental renaming in other directories Removing vscode files download working, still need to 1. combine db config for all 3 functions, 2. write tests Download now works with new db_access_v2. Need to create shared_db config ditto prior * single db configuration minor spacing * updating onboarding function to use new shared directory * temp commenting out onboarding/onboarding-client.py as it breaks pytests. Should probably remove hardcoded resources * feedback * minor TODO comment reshuffling * removing deprecated db_access_v1
abfleishman
pushed a commit
to abfleishman/active-learning-detect
that referenced
this pull request
Oct 10, 2019
Sync dev branch with master
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
leaving user_id for query param for now.