Migrate from Artifactory to CloudRepo

View as Markdown

cloudrepo-migrate artifactory copies the local Maven and npm repositories of a JFrog Artifactory server into CloudRepo, then reads every file back from CloudRepo and checks it against the original. It moves every file of a Maven repository but a root archetype-catalog.xml, which the report names, and, for npm, every version of each package with its dist-tags. npm is tested against a stand-in for Artifactory’s npm API, not yet against a real Artifactory server. From version 0.7.0 it also moves every tag of each local Docker repository, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server. From version 0.7.0 it also moves every wheel and sdist of each local PyPI repository, tested against a stand-in for Artifactory’s PyPI API, not yet against a real Artifactory server. From version 0.8.0 it also moves a local Generic repository you name with --repo SOURCE=TARGET into TARGET, a CloudRepo Maven repository, every file at the same path and served as it is, tested against a real Artifactory OSS 7.146.39. See Generic repositories.

One command moves your Artifactory Maven and npm repositories into CloudRepo, every file checksum-verified.

cloudrepo-migrate artifactory \
  --artifactory-url https://artifactory.example.com/artifactory \
  --org your-org \
  --report-json report.json

Add --dry-run for a first run that uploads nothing. Artifactory is only read. It keeps running, unchanged, until you repoint your builds. Using an agent? Go to For agents. On Nexus? Use cloudrepo-migrate nexus3 or nexus2, the same tool with the same options.

What you need

A CloudRepo organization. If you have none, sign up.
A repository in CloudRepo for each Artifactory repository you move, of the same format. Give it the Artifactory repository’s key, or map the names with --repo. A CloudRepo repository name has lowercase letters, digits, _ and -, and at most 32 characters, so a key such as libs-release-local fits and a longer one needs --repo. Use a new, empty repository where you can: the tool puts each file at the path Artifactory has it, and a repository that allows overwrites replaces a file already there.
A repository token with Read + write on those repositories.
Python 3.8 or later with pipx (or pip), on a machine that can reach your Artifactory server and your CloudRepo organization over the network.
An Artifactory access token that can read the repositories. A legacy API key works too.

1. Install the tool

cloudrepo-migrate is one command for Python 3.8 or later, with no dependencies. Install it from PyPI:

pipx install cloudrepo-migrate

No pipx? pip install cloudrepo-migrate in a virtual environment works too. Then check that it runs:

cloudrepo-migrate --version

artifactory needs version 0.5.0 or later. If you installed the tool earlier for Nexus, run pipx upgrade cloudrepo-migrate. An older version answers invalid choice: 'artifactory'. Docker repositories move from version 0.7.0, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server. Earlier versions list them as skipped. PyPI repositories move from version 0.7.0, tested against a stand-in for Artifactory’s PyPI API, not yet against a real Artifactory server. Earlier versions list them as skipped. Generic repositories move from version 0.8.0, when --repo names them. Earlier versions list them as skipped.

2. Run it

Put the credentials in the environment of the shell that runs the tool: ARTIFACTORY_ACCESS_TOKEN for Artifactory, and CLOUDREPO_TOKEN for CloudRepo. Without an access token, the tool sends ARTIFACTORY_API_KEY instead (JFrog has deprecated API keys, so prefer a token). The tool reads them only from the environment. No option takes a credential, the tool never prints one or writes one to the report, and it sends the Artifactory credential only to the address you give it. It never follows a redirect, so the credential cannot be sent on to another host.

Run the command at the top of this page once with --dry-run added. A dry run lists what would move and uploads nothing. With CLOUDREPO_TOKEN set, it also checks the token and that each target repository exists. Then run it without --dry-run.

Option	Meaning
`--artifactory-url URL`	The base address of your Artifactory, ending in `/artifactory`, such as `https://artifactory.example.com/artifactory`. On JFrog Cloud, it is your server’s address followed by `/artifactory`.
`--org ORG` or `--cloudrepo-url URL`	Your CloudRepo organization, or its full `https` address.
`--repo SOURCE` or `SOURCE=TARGET`	Move only this Artifactory repository, to the CloudRepo repository `TARGET` (default: the same name). Repeat it for more. Default without it: every local Maven and npm repository, every local PyPI repository (from version 0.7.0, tested against a stand-in for Artifactory’s PyPI API, not yet against a real Artifactory server), and every local Docker repository (from version 0.7.0, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server). A local Generic repository moves only when named here, from version 0.8.0, into a CloudRepo Maven repository `TARGET`.
`--dry-run`	List what would move. Upload nothing.
`--latest-only`	For npm, move only the version each package’s `latest` dist-tag names. Default without it: every version, with its dist-tags.
`--report-json PATH`	Also write the report to `PATH` as JSON.
`--json`	Print the report as one JSON document on standard output, and the text on standard error.
`--jobs N`	Upload `N` Maven files at a time (default 4). npm packages and Docker tags move one at a time.
`--work-dir DIR`	Where a Maven file, or a Docker layer, is held while it moves (default: the system temporary directory).
`--artifactory-ca-file PEM`	A CA bundle, for an Artifactory with a private certificate.

What moves, and what does not

In Artifactory	What happens
Local repository, package type Maven	Every file moves, POMs and `maven-metadata.xml` included, except a root `archetype-catalog.xml` (below). Artifactory serves each file’s `.sha1` and `.md5` without storing them, so the tool writes both into CloudRepo from the file’s checked bytes, and Maven finds them as it did in Artifactory. The `maven-metadata.xml` files move last, so a build never reads a version whose files have not landed.
Local repository, package type npm	Every version of each package moves, oldest first, scoped packages included. Then each dist-tag Artifactory has (`latest`, `next` and any other) that CloudRepo lacks is added. A dist-tag CloudRepo already holds is never moved or removed: if it names another version than Artifactory does, or Artifactory has no such tag, the report lists it as `kept`. A deprecated version keeps its message. A version unpublished from Artifactory is not in its package, so it does not move. With `--latest-only`, only the version `latest` names moves. Tested against a stand-in, not yet a real Artifactory.
Local repository, package type PyPI	Every wheel (`.whl`) and sdist (`.tar.gz` or `.zip`) moves, every version, from version 0.7.0, tested against a stand-in for Artifactory’s PyPI API, not yet against a real Artifactory server. Each is uploaded under the project name its own metadata gives, so pip finds it by the name it always had. Other files beside them, such as signatures, do not move, and the report counts them.
Local repository, package type Docker	Every tag of every image moves, from version 0.7.0, with the manifest and every layer it names, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server. A tag that lists several platforms moves with each platform’s image. Each image keeps its name and tags, and each manifest its digest. The report counts each tag as a file. See Docker images.
Local repository, package type Generic	From version 0.8.0, moves only when `--repo` names it: every file it stores moves into `TARGET`, a CloudRepo Maven repository, at the same path and served as it is, a dot file and a root `archetype-catalog.xml` included, with no `.sha1` or `.md5` written beside it. Without `--repo` naming it, it is skipped with a reason that names the flag. See Generic repositories.
Remote repository	Skipped. Create a proxy repository in CloudRepo if its upstream is one CloudRepo offers.
Virtual repository	Skipped: it stores nothing of its own. Move the local repositories it includes. For Maven and Docker, you can then create a group repository in CloudRepo to serve them under one address.
Federated repository	Skipped. This version moves local repositories.
Any other package type	Skipped. From Artifactory, this version moves Maven, npm and Docker (from version 0.7.0, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server), and PyPI from version 0.7.0, tested against a stand-in for Artifactory’s PyPI API, not yet against a real Artifactory server. A Generic repository moves from version 0.8.0 when `--repo` names it. For the other formats CloudRepo hosts, export and import the files as Migrate from Sonatype Nexus Repository shows for any server.
Files Artifactory generates itself	Skipped: any file or folder whose name starts with a dot, such as the Maven index under `.index` and the npm metadata under `.npm`. In a Generic repository a dot file moves, as a file someone stored.
A root `archetype-catalog.xml`	Not moved from a Maven repository: this version moves no archetype catalog. The report says so on that repository’s line. Deploy the catalog to CloudRepo yourself. From a Generic repository it moves, as any other file.

Every skipped repository is in the report with its reason. The tool deletes nothing.

Docker images

From version 0.7.0, a local Docker repository moves image by image, tested against a stand-in for Artifactory’s Docker API, not yet against a real Artifactory server: every tag, its manifest and every layer the manifest names. A tag that lists several platforms moves each platform’s image first, then the list. The tool reads Artifactory through the Docker registry API Artifactory serves for each repository under /artifactory/api/docker/, with ARTIFACTORY_ACCESS_TOKEN (or ARTIFACTORY_API_KEY).

Every layer is checked against its digest. The bytes the tool reads from Artifactory must hash to the digest the manifest names, and the layer is uploaded by that digest. A layer CloudRepo already holds is not uploaded again, and a layer several images share is uploaded once.
Every manifest is read back. Each manifest is uploaded as Artifactory serves it, so it keeps its digest, then read back from CloudRepo and compared by digest.
A manifest that is not Docker schema 2 or OCI fails its tag. The tag is reported as failed, so the exit code is 1, with manifest REFERENCE is TYPE; CloudRepo takes Docker schema 2 and OCI manifests, where REFERENCE is the tag or digest it read and TYPE is the media type Artifactory served, or untyped.
A layer over 2 GiB fails its tag. The tag is reported as failed, so the exit code is 1, with blob DIGEST is SIZE bytes; CloudRepo takes a blob of at most 2147483648.
Remote, virtual and federated repositories are skipped, with the reason. A Docker remote repository is skipped as a proxy of a remote: create a proxy repository in CloudRepo instead. A Docker virtual repository is skipped as a virtual repository: move the local repositories it includes. A Docker federated repository is skipped as a federated repository.
CLOUDREPO_TOKEN must be a CloudRepo repository token. Create a repository token with Read + write on the target Docker repository. The tool signs in to CloudRepo’s Docker registry with it, so there is no username to set.

Create a Docker repository in CloudRepo for each Artifactory Docker repository you move. Then move one with:

cloudrepo-migrate artifactory \
  --artifactory-url https://artifactory.example.com/artifactory \
  --org your-org \
  --repo docker-local \
  --report-json report.json

Replace docker-local with the key of your Artifactory Docker repository, or write docker-local=TARGET to move it into a CloudRepo repository named TARGET. Add --dry-run first to count the tags and bytes without uploading anything. Run the same command again to resume: a tag CloudRepo already holds at the same digest is counted as already present, and nothing of it is uploaded. After the move, Docker clients pull each image from CloudRepo with repositories/<repository>/ in its name, as Pull Docker images shows.

Generic repositories

From version 0.8.0, a local Generic repository moves when you name it with --repo, into a CloudRepo Maven repository. CloudRepo has no Generic repository type: a Maven repository serves any file at any path, so installers, archives and other binaries keep the paths your scripts already fetch. Tested against a real Artifactory OSS 7.146.39.

Every file, at the same path. Every file the Generic repository stores moves into the CloudRepo Maven repository at the path Artifactory has it, and CloudRepo serves it as it is, so only the address before the path changes. A file or folder whose name starts with a dot, and a root archetype-catalog.xml, move too: in a Generic repository each is a file someone stored.
Every file is checked against the checksum Artifactory recorded. Before the upload, the bytes must match the SHA-256 and SHA-1 Artifactory lists for the file, and Artifactory must have listed one of them: a file with neither is not uploaded. After the upload, the tool reads the file back from CloudRepo and compares its SHA-1 and SHA-256.
No checksum file is added. Unlike for a Maven file, the tool writes no .sha1 or .md5 beside a Generic file: those are a Maven client’s, and only what Artifactory stores moves. A checksum file Artifactory stores moves as a file, and must name the digest of the file beside it.
A file CloudRepo cannot store fails, in a dry run too. A path with a character outside A-Z a-z 0-9 + _ - . ! * ' ( ) / @ ~, longer than 1024 characters, or with an empty, . or .. segment, and a file over 50,000,000,000 bytes, the most CloudRepo stores in one file, are each listed as a failure with the path, so the exit code is 1. A --dry-run lists the same failures, before anything moves.
Unnamed, it is skipped, with the flag to use. Without --repo naming it, a Generic repository is skipped as generic format is not supported by default: name it with --repo KEY=TARGET to move its files into a CloudRepo Maven repository TARGET, each at the same path.
--repo moves only what it names. Name your other repositories beside the Generic one, or move the Generic one in a run of its own.

Create a Maven repository in CloudRepo for each Artifactory Generic repository you move. Then move one with:

cloudrepo-migrate artifactory \
  --artifactory-url https://artifactory.example.com/artifactory \
  --org your-org \
  --repo generic-local=installers \
  --report-json report.json

Replace generic-local with the key of your Artifactory Generic repository, and installers with the CloudRepo Maven repository it moves into. Add --dry-run first to count the files and bytes, and to list each file CloudRepo cannot store, without uploading anything. Run the same command again to resume: a file CloudRepo already holds with the same bytes is counted as already present. After the move, fetch each file from CloudRepo at its same path, as Raw files over HTTP shows.

How every file is checked

Before the upload. A file’s bytes must match the SHA-256 and SHA-1 Artifactory lists for it. An npm tarball must also match its package’s dist.shasum and, when the package has one, its dist.integrity. A file that does not match is not uploaded.
After the upload. The tool reads the file back from CloudRepo and compares its SHA-1 and SHA-256 with the bytes it took from Artifactory. A wheel or sdist must also be listed in its project’s index, where pip looks for it.
Checksum files. The .sha1 and .md5 the tool writes for a Maven repository come from the bytes it downloaded and checked, never from Artifactory’s numbers alone, and are read back the same way. For a Generic repository it writes none.
Docker. Each layer must hash to the digest its manifest names before it is uploaded, and each manifest is read back from CloudRepo and compared by digest, as Docker images says.
Generic. A Generic file is checked as a Maven file is, and Artifactory must have listed its SHA-256 or SHA-1, or it is not uploaded, as Generic repositories says.

Any mismatch fails the run and names the file. The run then exits with 1, so a script or an agent cannot read a failed check as success.

Resume after a failure

Run the same command again. A file CloudRepo already holds is counted as already present, and not uploaded again, only after the tool has read it back and found the same bytes. A repository that refuses to overwrite a release answers 409 for a file it holds. The tool counts that file as moved only when the bytes CloudRepo holds match the source. A Docker tag CloudRepo already holds at the same digest is counted as already present on that one digest check, with nothing read back.

The end report

The last line of the output is the result:

Moved N repositories, F files, B bytes in M minutes; V of F verified, K failed.

N, F and B are the repositories, files and bytes that moved, M is the elapsed time, V is the number of Maven, npm and PyPI files read back and matching plus the number of Docker tags verified, and K is the number that failed. A Docker tag is verified when its manifest’s digest matches and its blobs were checked by digest before upload, or when CloudRepo already held it at the same digest; layers are not read back. Above it is one line for each repository: its Artifactory key, its CloudRepo name, its format, its counts and, for a skipped repository, why. A line REPO PACKAGE: kept: TAG -> VERSION (source: VERSION or none) names an npm dist-tag CloudRepo already holds that the tool left as it was. It is not a failure.

--report-json writes the same report as JSON, and --json prints it as one JSON document on standard output (the text then goes to standard error). A script reads these keys:

Key	Holds
`exit_code`	The exit code of the run, as in If something fails.
`error`	The message of a run that stopped before its report, such as a refused token. Such a document holds only `exit_code` and `error`.
`totals.files`, `.verified`, `.failed`	The counts in the last line.
`failures`	One entry for each failed file, with its `repository`, `path` and `reason`. Empty when nothing failed.
`kept`	One entry for each npm dist-tag CloudRepo already holds and the tool left as it was: its `repository`, `path` (the package), `tag`, `version` (CloudRepo’s) and `source` (the version Artifactory tags, or `null` when Artifactory has no such tag). Not a failure.
`repositories[].action`	`moved`, `would move` (a dry run), `skipped`, `ready` or `blocked`.
`repositories[].reason`	Why a repository was skipped or blocked, a root `archetype-catalog.xml` that did not move, the count of other PyPI files that did not move, how many Docker images a repository holds, in a dry run what moves from an npm repository, or, with `--latest-only`, the count of other npm versions that did not move.

If something fails

The exit code says which kind of failure it was:

Exit code	Meaning
`0`	Everything moved, every Maven, npm and PyPI file was read back and matched, and every Docker tag’s manifest matched its digest.
`1`	At least one file failed. The output lists each one with its path. Fix the cause and run the same command again.
`2`	A problem before anything moved, such as a missing target repository or a refused token. Nothing was uploaded.
`130`	Stopped with Ctrl-C. Run the same command again to resume.

The messages you are most likely to see:

The output says	What to do
`CloudRepo has no repository NAME: create it as ...`	Create the repository in CloudRepo, in the format named, then run again. Or map the Artifactory repository to another with `--repo`.
`CloudRepo has no Docker repository NAME: create one at ...`	Create a Docker repository with that name in CloudRepo, then run again. Or map the Artifactory repository to another with `--repo`.
`... is not a valid CloudRepo repository name`	The Artifactory key is not a valid CloudRepo name. Choose a name with `--repo SOURCE=TARGET`.
`CloudRepo refused the token`	Check `CLOUDREPO_TOKEN`, and that the token belongs to this organization.
`the token cannot reach CloudRepo repository NAME`, or `cannot publish`	Give the token Read + write on that repository.
`Artifactory refused the login`	Set `ARTIFACTORY_ACCESS_TOKEN` to a token that can read the repository.
`the bytes Artifactory served ... do not match the ... Artifactory recorded`	The copy in Artifactory does not match its own record, so the tool did not upload it. Check that file in Artifactory.
`Artifactory recorded no sha1 or sha256 for the file, so it cannot be checked before the upload`	A Generic file Artifactory lists no SHA-256 or SHA-1 for, so the tool did not upload it. Check that file in Artifactory.
`CloudRepo cannot store this path (...)`	The path has a character outside the ones the message lists, is longer than 1024 characters, or has an empty, `.` or `..` segment, so the file does not move. The other files do. Rename it in Artifactory to move it; until then the run keeps exiting 1.
`the file is SIZE bytes, over the 50000000000 bytes CloudRepo stores in one file`	The file is larger than CloudRepo stores in one file, so it does not move. The other files do, and the run keeps exiting 1 while the file is in the repository you move.
`answered 302, a redirect to ...`	Artifactory sent the download to another address, as some storage setups do. The tool never follows a redirect, so your credential stays with Artifactory. Email support@cloudrepo.io with the message.
`the package has no usable latest dist-tag`	Only with `--latest-only`: the package has no `latest` tag in Artifactory that names a version it lists, so the tool cannot choose a version. Tag one and run again, or run without `--latest-only`.
`dist-tags the source has, not added: ...`	A dist-tag Artifactory has, and CloudRepo lacks, names a version that did not move, whose own failure is listed too: fix that one and run the same command again. A tag that names a version Artifactory does not list stays unset until you retag it in Artifactory.
`CloudRepo already holds different bytes for this path and refuses overwrites`	The file in CloudRepo is not the one in Artifactory, and the repository will not replace it. Decide which copy is right, then email support@cloudrepo.io with the path.
`the bytes CloudRepo serves back differ from the source`	Run the same command again. If the same file fails twice, email support@cloudrepo.io with its path.

After the move

The files are in CloudRepo, and your builds still point at Artifactory. Repoint them one project at a time, as step 5 of Migrate from Sonatype Nexus Repository shows for each build tool, with your Artifactory address where that page has the Nexus one, and leave Artifactory running until your builds pass against CloudRepo.

For agents

Give your agent this instruction. It carries the address of this page as plain text, so the agent reads this page and nothing else:

Migrate my Artifactory to CloudRepo. Follow https://www.cloudrepo.io/docs/migrate/from-artifactory.md. Read my credentials from the environment variables ARTIFACTORY_ACCESS_TOKEN and CLOUDREPO_TOKEN, and ask me for the Artifactory URL and the CloudRepo organization if you do not have them.

The invocation, in two runs. Replace the address and the organization with the owner’s:

cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org your-org --dry-run --json
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org your-org --json

Each run prints one JSON document on standard output, its result, and the text report on standard error. The end report names the keys.

Steps:

Check that python3 --version is 3.8 or later. If cloudrepo-migrate --version does not answer, install the tool with the command in Install the tool, and check again. If it answers a version older than 0.5.0, run pipx upgrade cloudrepo-migrate, and check again. If cloudrepo-migrate artifactory --help does not list --json, run pipx upgrade cloudrepo-migrate. If it still does not, run each command with --report-json report.json in place of --json, and read report.json: the same document, without exit_code.
Check that CLOUDREPO_TOKEN is set without printing it, for example with test -n "$CLOUDREPO_TOKEN". Do the same for ARTIFACTORY_ACCESS_TOKEN, or for ARTIFACTORY_API_KEY if the owner uses an API key.
Run the first command, the dry run. It reads Artifactory and writes nothing to CloudRepo. Expect exit code 0, and in the document exit_code 0 and dry_run true. Each entry in repositories with the action would move is a repository that will move, with its format and files; each with the action skipped says why in its reason. On exit code 2, read error, or the reason of each repository with the action blocked, in If something fails. If a target repository is missing, ask the owner to create it in the portal, then repeat the dry run.
Run the second command.
Verify, all of these, in the document it printed:
- exit_code is 0, and so is the exit code of the command.
- failures is empty, totals.failed is 0 and totals.verified equals totals.files.
- No entry in repositories has the action blocked.
Tell the owner totals.repositories, totals.files and totals.verified, every repository with the action skipped and its reason, every entry in kept (a dist-tag CloudRepo already holds, left as it was), every PyPI repository whose reason counts other files that did not move and, with --latest-only, every npm repository whose reason counts versions that did not move.

If the exit code is 1, read each entry in failures, with its path and reason, remove its cause, and run the second command again, which resumes. If the same file fails twice, stop and report its path and reason to the owner.

Do not put a credential on a command line, in a file or in your answer. Do not change anything in Artifactory. Do not repoint builds or switch CI to CloudRepo until the owner tells you to.