Migrate from AWS CodeArtifact

This page moves the packages you published to AWS CodeArtifact to CloudRepo and repoints your builds, with CodeArtifact left running until your builds pass against CloudRepo. It copies files and nothing else: your permissions and every other piece of data that is not a package file stay in AWS.

1. List what you have and decide where each repository goes.
2. Create the repositories in CloudRepo.
3. Export the files from CodeArtifact.
4. Import them into CloudRepo.
5. Repoint your builds.
6. Check, then cut over.

1. List what you have

You need a CloudRepo organization. If you have none, sign up.

List your repositories with the AWS CLI (ListRepositories), which reads the credentials you already use for aws:

Terminal

aws codeartifact list-repositories --output json | jq -r '.repositories[] | [.domainName, .name] | @tsv'

Then decide where each one goes. The left column uses AWS’s package formats and external connections:

In CodeArtifact	In CloudRepo
A repository’s packages in format maven, npm or pypi, published to it directly	A repository of that format. Its files are what you export and import.
Packages ingested from an external connection, such as Maven Central or npmjs.com	A proxy repository, if the upstream is one CloudRepo offers. Nothing to export: a proxy fills itself from the upstream.
Any other format (nuget, generic, ruby, swift, cargo)	No CloudRepo repository for it. CloudRepo hosts Maven, Python, npm and Docker.

The export in step 3 asks only for packages that were published directly (INTERNAL), so what a repository ingested from an external connection (EXTERNAL) stays out of your export. AWS defines both origin types on PackageVersionOrigin.

2. Create the repositories

For each CodeArtifact repository you are moving, create a repository of the same format in CloudRepo, and give it the name your builds already use if you can. See Creating a repository. Then add whoever needs access: add users, and create a repository token with Read + write for the import and Read only tokens for builds that only pull.

3. Export the files

For each repository and format, list its packages (ListPackages), versions (ListPackageVersions) and assets (ListPackageVersionAssets) with the AWS CLI, and download each asset (GetPackageVersionAsset) into a directory named for the repository. AWS documents a Maven package version’s assets as its JAR file, its POM file or any other asset in the version, so nothing is guessed.

Terminal

DOMAIN=my-domain
REPO=my-repo
FORMAT=maven
aws codeartifact list-packages --domain "$DOMAIN" --repository "$REPO" --format "$FORMAT" --output json \
  | jq -r '.packages[] | [.namespace // "", .package] | @tsv' \
  | while IFS=$'\t' read -r ns pkg; do
      key=(--domain "$DOMAIN" --repository "$REPO" --format "$FORMAT" --package "$pkg")
      [ -n "$ns" ] && key+=(--namespace "$ns")
      aws codeartifact list-package-versions "${key[@]}" --status Published --origin-type INTERNAL --output json \
        | jq -r '.versions[].version' \
        | while IFS= read -r ver; do
            aws codeartifact list-package-version-assets "${key[@]}" --package-version "$ver" --output json \
              | jq -r '.assets[].name' \
              | while IFS= read -r asset; do
                  dir="export/$REPO/$(printf '%s' "$ns" | tr . /)/$pkg/$ver"
                  mkdir -p "$dir"
                  aws codeartifact get-package-version-asset "${key[@]}" --package-version "$ver" \
                    --asset "$asset" "$dir/$asset" > /dev/null
                done
          done
    done

Run it once for each format: set FORMAT to maven, npm or pypi. Expected: for Maven, export/<repository>/<groupId as a path>/<artifactId>/<version>/ holds each version’s files, the layout CloudRepo takes as it is. AWS names a Maven package’s groupId as its namespace (GetPackageVersionAsset), which is the ns in the loop. The import loops look under ./export for npm tarballs (.tgz), Python wheels (.whl) and source distributions (.tar.gz), wherever they sit.

Only versions with the status Published are exported. List the others (Unlisted, Archived, and so on) with aws codeartifact list-package-versions and no --status, and decide whether you need them: ListPackageVersions names the statuses and returns all of them when you give none.

4. Import the files

Import artifacts into CloudRepo has one loop for each format. Run it for each repository, then check the bytes arrived. You can run a loop again after a failure, and what a second run skips and what it replaces depends on the format: see Running a loop again.

5. Repoint your builds

In each build, replace the CodeArtifact URL and credential with the CloudRepo repository’s URL and a repository token. The page for your build tool shows the file to change:

Maven

Pull Maven artifacts and Publish Maven artifacts: ~/.m2/settings.xml holds the credential, and pom.xml holds the repository URL.

Gradle

Pull Gradle dependencies and Publish Gradle artifacts: ~/.gradle/gradle.properties holds the credential, and your build script holds the repository URL.

npm

Pull npm packages and Publish npm packages: ~/.npmrc holds the registry and the token.

pip

Pull Python packages and Publish Python packages: pip.conf or PIP_INDEX_URL, and ~/.pypirc or TWINE_* for uploads.

The username for every client except npm is the email address of the account that created the token, and the password is the token. In CI, store the token in the CI system’s secret store and expose it as an environment variable, as the pages above show; do not write it into a script or a committed file.

6. Check, then cut over

Do this in order, and leave CodeArtifact running until the last step:

1. Build against CloudRepo, with CodeArtifact still up. Point one project at CloudRepo and run its full build, then the projects that depend on it. A 401, 403 or 404 has the same causes as when you publish by hand: see “When a Client Answers 401, 403 or 404” on Repository tokens.
2. Publish one new version to CloudRepo from CI and install it from a second machine.
3. Switch every build that still names CodeArtifact, and every CI job’s secret, to CloudRepo.
4. Stop publishing to CodeArtifact. From now on a new version exists in CloudRepo only.
5. Keep an export of CodeArtifact until you are sure nothing reads it. Step 3 of this page is that export.

If you are stuck, email support@cloudrepo.io with your organization name, the repositories involved, the command you ran and the error it printed.