[Backfill corrections] hone AWS publish behavior #1796
Conversation
krivard
left a comment
krivard
left a comment
There was a problem hiding this comment.
👍 should work
I'm definitely squinting at bash for making this kind of operation awkward though, it seems like tests and pipes should be able to be friends
| AWS_MSG=`aws s3 cp $(USR_EXPORT_DIR) $(S3_BUCKET)/ --recursive --exclude "*" --include "*/prediction*.csv.gz"`; \ | ||
| echo "$${AWS_MSG}"; \ | ||
| [[ `echo "$${AWS_MSG}" | grep "upload:" | wc -l` -eq $$NUM_FILES ]] || ( echo "ERROR: Not all $${NUM_FILES} expected files were uploaded" && exit 74 ); \ |
There was a problem hiding this comment.
thought (nonblocking): it really feels like there should be a way to do this with tee or process substitution instead of variables and echo
I played with it for a while and the closest I got was teeing to stderr, which would probably be excluded from the log and that's not what we want... and your way is definitely more readable.
There was a problem hiding this comment.
Yeah, it's awkward to have the file upload running as a side effect.
There are some suggestions online to tee to /dev/tty or $(tty) but I can't actually get this to work on my own computer. Something to look into.
There was a problem hiding this comment.
that could work when running from a terminal, but normal pipeline operations typically run headless alas
|
This is ready to merge |
Description
--acloption from the upload command.Changelog