Consider using CodeArtifact to package up utility modules

[krivard]

Check out CodeArtifact to see if we can use it. Maybe try to get the delphi_utils code used in the indicators repo built as a Python package and stashed there so it can be imported into projects that live outside the indicators repo.

[vishakha1812]

Here is a brief workflow that looked feasible to me that we could adopt :

• For a single domain, create a new repository that would have packages in it (packages could be either custom built, ex. delphi_utils or a public python package - npm, pypi.)

• Authenticate the new CodeArtifact repo using the AWS Cli .

• Upload the package (in our case delphi-utils) using twine

Using this we can install our private packages using pip install in any indicator.

[krivard]

This is increasing in importance as we automate more indicators; doing N PRs every time we update the geo coding utils is a drag

[benjaminysmith]

Is it necessary to use CodeArtifact? Can we just upload stuff directly to PyPI?

[krivard]

These packages would be intended for use in other delphi software, not the general public. Is that something people put on PyPI?

[chinandrew]

You can also run your own pypi server with https://pypi.org/project/pypiserver/, but not sure it's any better than using codeartifact. Don't have experience with either so can't comment much more than that.

[benjaminysmith]

I guess I'm wondering -- is there a reason not to make them public? Even if no-one ends up using them (and they might) it sounds easier to do it with the vanilla PyPI flow rather than add more pieces of infra (with caveat being that I haven't actually tried either myself)

[krivard]

maybe? here are some possible reasons:

• if somebody searches for delphi on PyPI, we want them to find the epidata and covidcast clients, and not have to sift through our internal packages
• if we end up needing an internal utility that includes secrets, we'd have to spin up a private package repository anyhow

The first is a comms issue, but could be resolved using codenames if we really wanted to. The second is severely hypothetical, so unless @korlaxxalrok has something brewing we can probably disregard.

[korlaxxalrok]

Yeah, makes sense.

If we don't have an actual need for the private repo then it is probably something we can put aside for now. I think the main goal is to have some ability to turn our useful code into packages so we have less dependencies around keeping them in repos along with our projects, or needing to deploy several repos to a file system so we can call them as modules.

I guess if the package is useful to others (and maybe polished) and we want to make it public (and it is safe) then there isn't anything stopping us.