Import artifacts into CloudRepo

You have the files on disk, exported from another repository manager. This page copies them into CloudRepo: one loop per format, each of which you can run again after a failure.

Before you start

• A CloudRepo repository for each repository whose files you are moving, of the same format. If you have none, create them. Import only those. For each other kind of repository in your old system, the page for that system says what to create in CloudRepo, if anything, such as a proxy repository.
• A repository token with Read + write that reaches those repositories. See Repository tokens. Your username is the email address of the account that created the token.
• The exported files in a directory, ./export, each repository in its own subdirectory. The page for the system you are leaving tells you how to get them: JFrog Artifactory, Sonatype Nexus, AWS CodeArtifact. For Azure Artifacts, see the Azure Artifacts guide.

Running a loop again

What a second run does depends on the format and on whether Overwrite Protection is on. It is on for every Maven, npm and Python repository unless someone turned it off for that repository. See Overwrite Protection.

• Maven. While Overwrite Protection is on, CloudRepo answers 409 Conflict for a release file, or a checksum or signature file that belongs to one, that is already in the repository, and leaves it as it is. maven-metadata.xml, its own checksum and signature files, and every file under a -SNAPSHOT version are rewritten and never refused.
• npm and Python. While Overwrite Protection is on, CloudRepo answers 409 Conflict for an npm version or a Python file that is already in the repository, and leaves it as it is.
• Docker. A Docker repository has no Overwrite Protection. Pushing a tag that already exists moves the tag to the new image.
• Overwrite Protection turned off. The repository accepts overwrites, so a second run replaces what it already holds.

So a second run skips the Maven releases, npm versions and Python files the first run finished and uploads the rest, only while Overwrite Protection is on, and it re-points every Docker tag it pushes again.

Import

Maven

A Maven repository is a directory tree, and CloudRepo’s Maven door takes a PUT at any path in that tree, which is what mvn deploy sends. So the whole tree goes up file by file, checksum files and maven-metadata.xml included, with the same paths.

1. Put the credential in ~/.netrc, so curl reads it from there and the token is on no command line. Keep the file readable by you alone (chmod 600 ~/.netrc).

~/.netrc

machine your-org.mycloudrepo.io
login you@example.com
password YOUR_REPOSITORY_TOKEN

2. Upload the tree. Run this from the repository’s directory in ./export. It prints one line per file: the status CloudRepo answered, then the path.

Terminal

BASE=https://your-org.mycloudrepo.io/repositories/your-repo
cd export/your-repo
find . -type f | sort | while IFS= read -r f; do
  code=$(curl --netrc --proto =https --silent --output /dev/null --write-out '%{http_code}' \
    --upload-file "$f" "$BASE/${f#./}")
  printf '%s %s\n' "$code" "$f"
done | tee ../import.log

2xx means the file is in. With Overwrite Protection on, 409 means it was already there: a release file, or a checksum file that belongs to one, from an earlier run. maven-metadata.xml and -SNAPSHOT versions are rewritten, never refused. Look at every other status in import.log: 401 is the credential, 403 is a token that cannot write here, 413 is a file over 50 GB.

npm

Each package version is a tarball (.tgz), and npm publish can publish a tarball directly. Put the registry and the token in ~/.npmrc, as on Publish npm packages:

~/.npmrc

registry=https://your-org.mycloudrepo.io/repositories/your-repo/
//your-org.mycloudrepo.io/repositories/your-repo/:_authToken=${CLOUDREPO_TOKEN}

Then publish every tarball, oldest version first:

Terminal

BASE=https://your-org.mycloudrepo.io/repositories/your-repo/
find ./export -name '*.tgz' | sort -V | while IFS= read -r t; do
  v=$(tar -xzOf "$t" package/package.json | jq -r .version)
  case "$v" in *-*) tag=next ;; *) tag=latest ;; esac
  npm publish "$t" --registry "$BASE" --tag "$tag" || echo "FAILED $t" >&2
done

Three things in that loop matter:

• --registry is not optional. A package’s own package.json can name a registry in publishConfig, and npm publishes there, whatever your .npmrc says. A package that came from your old registry may carry that line, and without --registry npm would send it straight back.
• --tag. npm refuses a prerelease version such as 1.0.0-beta.1 unless you name a tag, so the loop gives prereleases the tag next. Everything else gets latest.
• Oldest first. Each publish sets the tag it carries, so latest ends on the version you published last. Version-sorting the paths puts the highest version last. Check a package afterwards with npm view <name> dist-tags, and move latest with npm dist-tag add <name>@<version> latest.

While Overwrite Protection is on, an npm publish of a version already in the repository fails with 409, and the loop prints it and carries on.

pip

A Python repository holds wheels (.whl) and source distributions (.tar.gz), and twine uploads them. Give twine its credential through its environment, so no file is written and no token is on a command line, and upload one file at a time, so one refusal does not stop the rest:

Terminal

BASE=https://your-org.mycloudrepo.io/repositories/your-repo
export TWINE_USERNAME="you@example.com"
export TWINE_PASSWORD="$CLOUDREPO_TOKEN"
find ./export \( -name '*.whl' -o -name '*.tar.gz' \) | sort | while IFS= read -r f; do
  twine upload --non-interactive --repository-url "$BASE" "$f" || echo "FAILED $f" >&2
done

--repository-url is the repository’s URL without /simple/. While Overwrite Protection is on, a file already in the repository is refused with 409; twine prints only the status line, so rerun one file with --verbose to read the explanation. Python has no tags and no versions to order: the files can go up in any order.

Docker

A Docker repository holds images by name and tag. The Docker CLI copies one image at a time: pull it from the old registry, give it its new name, push it. Log in to both hosts with the token on standard input (docker login, as on Push Docker images), then list the images as name:tag lines in images.txt and run:

Terminal

OLD=registry.example.com/old-repo
NEW=your-org.mycloudrepo.io/repositories/your-repo
while IFS= read -r img; do
  docker pull "$OLD/$img" \
    && docker tag "$OLD/$img" "$NEW/$img" \
    && docker push "$NEW/$img" \
    || echo "FAILED $img" >&2
  docker rmi "$OLD/$img" "$NEW/$img" > /dev/null 2>&1
done < images.txt

The image name on CloudRepo carries repositories/ and your repository after the host; the login does not. docker rmi frees the disk after each image.

docker pull fetches the image for your machine’s platform, so an image published for several platforms arrives for one. Rebuild and push those from your CI instead, as on Push Docker images. Pushing a tag that already exists moves the tag to the new image, so a second run replaces what a tag points at.

Check the bytes arrived

A count is not proof. Fetch a few files back from CloudRepo and compare them with the originals:

Terminal

f=com/example/my-library/1.0.0/my-library-1.0.0.jar
curl --netrc --fail --silent "https://your-org.mycloudrepo.io/repositories/your-repo/$f" | sha256sum
sha256sum "export/your-repo/$f"

The two lines must match. For npm, Python and Docker, install or pull one package or image from CloudRepo in a clean directory or on a clean machine, as on Pull npm packages, Pull Python packages and Pull Docker images. The files are also listed in the repository in the admin portal.

When something fails

Each refusal has one cause, and it is the same as when you publish by hand. See the list at the end of Publish Maven artifacts, Publish npm packages, Publish Python packages or Push Docker images. If you are stuck, email support@cloudrepo.io with your organization name, the repository, the failing line from import.log and the command you ran.