Migrate from Nexus 2 to CloudRepo

View as Markdown

cloudrepo-migrate nexus2 copies the hosted Maven and npm repositories of a Nexus Repository 2 server into CloudRepo, then reads every file back from CloudRepo and checks it against the original. It moves every file of a Maven repository and, for npm, the version each package tags latest.

One command moves your Nexus 2 Maven and npm repositories into CloudRepo, every file checksum-verified.

cloudrepo-migrate nexus2 \
  --nexus-url https://nexus.example.com/nexus \
  --org your-org \
  --report-json report.json

Add --dry-run for a first run that uploads nothing. Nexus is only read. It keeps running, unchanged, until you repoint your builds. Using an agent? Go to For agents.

What you need

A CloudRepo organization. If you have none, sign up.
A repository in CloudRepo for each Nexus repository you move, of the same format. Give it the Nexus repository’s name, or map the names with --repo. A CloudRepo repository name has lowercase letters, digits, _ and -, and at most 32 characters. Use a new, empty repository where you can: the tool puts each file at the path Nexus has it, and a repository that allows overwrites replaces a file already there.
A repository token with Read + write on those repositories.
Python 3.8 or later with pipx (or pip), on a machine that can reach your Nexus server and your CloudRepo organization over the network.
A Nexus login that can read the repositories, unless your Nexus allows anonymous reads.

1. Install the tool

cloudrepo-migrate is one command for Python 3.8 or later, with no dependencies. Install it from PyPI:

pipx install cloudrepo-migrate

No pipx? pip install cloudrepo-migrate in a virtual environment works too. Then check that it runs:

cloudrepo-migrate --version

2. Run it

Put the credentials in the environment of the shell that runs the tool: NEXUS_USERNAME and NEXUS_PASSWORD for Nexus (leave both unset to read anonymously), and CLOUDREPO_TOKEN for CloudRepo. The tool reads them only from the environment, or asks for a missing one when it runs in a terminal. No option takes a credential, and the tool never prints one or writes one to the report.

Run the command at the top of this page once with --dry-run added. A dry run lists what would move and uploads nothing. With CLOUDREPO_TOKEN set, it also checks the token and that each target repository exists. Then run it without --dry-run.

Option	Meaning
`--nexus-url URL`	The base address of your Nexus 2 server, such as `https://nexus.example.com/nexus`.
`--org ORG` or `--cloudrepo-url URL`	Your CloudRepo organization, or its full `https` address.
`--repo SOURCE` or `SOURCE=TARGET`	Move only this Nexus repository, to the CloudRepo repository `TARGET` (default: the same name). Repeat it for more. Default without it: every hosted Maven and npm repository.
`--dry-run`	List what would move. Upload nothing.
`--report-json PATH`	Also write the report to `PATH` as JSON.
`--jobs N`	Upload `N` Maven files at a time (default 4). npm packages move one at a time.
`--work-dir DIR`	Where a Maven file is held while it moves (default: the system temporary directory).
`--nexus-ca-file PEM`	A CA bundle, for a Nexus with a private certificate.

What moves, and what does not

In Nexus 2	What happens
Hosted Maven repository	Every file moves, POMs and checksum files included. The `maven-metadata.xml` files move last, so a build never reads a version whose files have not landed.
Hosted npm repository	The version each package tags `latest` moves. Older versions do not move, and the report counts them.
Proxy repository	Skipped. Create a proxy repository in CloudRepo if its upstream is one CloudRepo offers.
Group repository	Skipped. Move the repositories it groups. For Maven, you can then create a group repository in CloudRepo to serve them under one address.
Virtual repository, other formats	Skipped. This version moves Maven and npm.
Files Nexus generates itself	Skipped: any file or folder whose name starts with a dot (such as `.index`, `.meta` and `.nexus`), and the root `archetype-catalog.xml`.

Every skipped repository is in the report with its reason. The tool deletes nothing.

How every file is checked

Before the upload. A Maven file’s bytes must match the SHA-1 Nexus recorded for it. An npm tarball must match its package’s dist.shasum and, when the package has one, its dist.integrity. A file that does not match is not uploaded.
After the upload. The tool reads the file back from CloudRepo and compares its SHA-1 and SHA-256 with the bytes it took from Nexus.
Checksum files. A checksum file Nexus holds (.sha1, .sha256, .sha512 or .md5) must name the digest of the file it sits beside.

Any mismatch fails the run and names the file. The run then exits with 1, so a script or an agent cannot read a failed check as success.

Resume after a failure

Run the same command again. A file CloudRepo already holds is counted as already present, and not uploaded again, only after the tool has read it back and found the same bytes. A repository that refuses to overwrite a release answers 409 for a file it holds. The tool counts that file as moved only when the bytes CloudRepo holds match the source.

The end report

The last line of the output is the result:

Moved N repositories, F files, B bytes in M minutes; V of F verified, K failed.

N, F and B are the repositories, files and bytes that moved, M is the elapsed time, V is the number of files read back and matching, and K is the number that failed. Above it is one line for each repository: its Nexus name, its CloudRepo name, its format, its counts and, for a skipped repository, why.

--report-json writes the same report as JSON. A script reads these keys:

Key	Holds
`totals.files`, `.verified`, `.failed`	The counts in the last line.
`failures`	One entry for each failed file, with its `repository`, `path` and `reason`. Empty when nothing failed.
`repositories[].action`	`moved`, `would move` (a dry run), `skipped`, `ready` or `blocked`.
`repositories[].reason`	Why a repository was skipped or blocked, or the count of older npm versions that did not move.

If something fails

The exit code says which kind of failure it was:

Exit code	Meaning
`0`	Everything moved, and every file was read back and matched.
`1`	At least one file failed. The output lists each one with its path. Fix the cause and run the same command again.
`2`	A problem before anything moved, such as a missing target repository or a refused token. Nothing was uploaded.

The messages you are most likely to see:

The output says	What to do
`CloudRepo has no repository NAME: create it as ...`	Create the repository in CloudRepo, in the format named, then run again. Or map the Nexus repository to another with `--repo`.
`... is not a valid CloudRepo repository name`	The Nexus name is not a valid CloudRepo name. Choose a name with `--repo SOURCE=TARGET`.
`CloudRepo refused the token`	Check `CLOUDREPO_TOKEN`, and that the token belongs to this organization.
`the token cannot reach CloudRepo repository NAME`, or `cannot publish`	Give the token Read + write on that repository.
`Nexus refused the login`	Set `NEXUS_USERNAME` and `NEXUS_PASSWORD` to a login that can read the repository.
`the bytes Nexus served ... do not match the sha1 Nexus recorded`	The copy in Nexus does not match its own record, so the tool did not upload it. Check that file in Nexus.
`the package has no usable latest dist-tag`	The package has no `latest` tag in Nexus, so the tool cannot choose a version. Tag one in Nexus and run again.
`CloudRepo already holds different bytes for this path and refuses overwrites`	The file in CloudRepo is not the one in Nexus, and the repository will not replace it. Decide which copy is right, then email support@cloudrepo.io with the path.
`its sha1 checksum file ... says ...`	The checksum file in Nexus disagrees with its file. The bytes moved unchanged. Only fixing the file in Nexus clears this, and the run keeps exiting 1 until you do.
`the bytes CloudRepo serves back differ from the source`	Run the same command again. If the same file fails twice, email support@cloudrepo.io with its path.

After the move

The files are in CloudRepo, except the older npm versions, which the report counts. A build that pins one cannot resolve it from CloudRepo until you export it from Nexus and import it with step 4 of Migrate from Sonatype Nexus Repository. Your builds still point at Nexus. Repoint them one project at a time, as step 5 of Migrate from Sonatype Nexus Repository shows for each build tool, and leave Nexus running until your builds pass against CloudRepo.

For agents

Give your agent this instruction. It carries the address of this page as plain text, so the agent reads this page and nothing else:

Migrate my Nexus 2 to CloudRepo. Follow https://www.cloudrepo.io/docs/migrate/from-nexus-2.md. Read my credentials from the environment variables NEXUS_USERNAME, NEXUS_PASSWORD and CLOUDREPO_TOKEN, and ask me for the Nexus URL and the CloudRepo organization if you do not have them.

The invocation, in two runs. Replace the address and the organization with the owner’s:

cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org your-org --dry-run
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org your-org --report-json report.json

Steps:

Check that python3 --version is 3.8 or later. If cloudrepo-migrate --version does not answer, install the tool with the command in Install the tool, and check again.
Check that CLOUDREPO_TOKEN is set without printing it, for example with test -n "$CLOUDREPO_TOKEN". Do the same for NEXUS_USERNAME and NEXUS_PASSWORD, unless the owner says Nexus allows anonymous reads.
Run the first command, the dry run. Expect exit code 0 and a last line that starts Would move. On exit code 2, read the message in If something fails. If a target repository is missing, ask the owner to create it in the portal, then repeat the dry run.
Run the second command.
Verify, all of these:
- The exit code is 0.
- The last line starts Moved and ends V of F verified, 0 failed., with V equal to F.
- In report.json, failures is empty, totals.failed is 0 and totals.verified equals totals.files.
- No entry in repositories has the action blocked.
Tell the owner the last line, every repository with the action skipped and its reason, and every npm repository whose reason counts older versions that did not move.

If the exit code is 1, read each failure, remove its cause, and run the second command again, which resumes. If the same file fails twice, stop and report its path and reason to the owner.

Do not put a credential on a command line, in a file or in your answer. Do not change anything in Nexus. Do not repoint builds or switch CI to CloudRepo until the owner tells you to.