REST calls in your scripts after Nexus or Artifactory

View as Markdown

A script often calls the REST API of Nexus or Artifactory where a build would call a package protocol: to ask for the latest version of an artifact, to upload a file, to search. Move your files and that script goes on calling Nexus or Artifactory, and fails the day you switch it off. This page takes each such call, shows it as the vendor documents it, and shows what to run against CloudRepo in its place. Where CloudRepo has nothing that matches, the page says so.

The Nexus calls are those of Nexus Repository 3. This page changes calls. It does not copy files or change the URLs in your builds: for those, see Migrate from Sonatype Nexus Repository, Migrate from JFrog Artifactory and Change every Nexus and Artifactory URL in your builds.

Where each call goes

Each row is a call as the vendor writes it. The last column links to the section that answers it.

Call in your script	Vendor	Where it goes
`GET /service/rest/v1/search/assets/download?sort=version`	Nexus	Latest version
`GET /artifactory/api/search/latestVersion`	Artifactory	Latest version
`[RELEASE]` in a download path	Artifactory	Latest version
`GET /artifactory/api/docker/<repoKey>/v2/<image>/tags/list`	Artifactory	Docker
`GET /artifactory/api/pypi/<repoKey>/pypi/<package>/json`	Artifactory	Python
`POST /service/rest/v1/components`	Nexus	Upload a file
`PUT /artifactory/<repoKey>/<filePath>`	Artifactory	Upload a file
`POST /artifactory/api/search/aql`	Artifactory	No counterpart
`GET /artifactory/api/search/prop`	Artifactory	No counterpart

Before you start

A CloudRepo repository that holds your files. If it does not yet, Migrate from Sonatype Nexus Repository and Migrate from JFrog Artifactory copy them.
A repository token that reaches it. A Read only token is enough to fetch; uploading takes Read + write. See Repository tokens. The username is the email address of the account that created the token, and the password is the token.
That login in ~/.netrc, which curl reads with --netrc, so the token is on no command line. Keep the file readable by you alone (chmod 600 ~/.netrc). Raw files over HTTP writes the same file.

machine your-org.mycloudrepo.io
login you@example.com
password YOUR_REPOSITORY_TOKEN

The Nexus and Artifactory examples on the vendors’ pages put the user and password on the command line. Every call below reads its login from ~/.netrc instead: add a machine entry for your Nexus or Artifactory host to the same file. The call itself is the vendor’s. On this page nexus.example and acme.jfrog.io stand for your Nexus and Artifactory hosts, your-org for your CloudRepo organization and your-repo for the repository that holds your files.

Fetch the latest version of an artifact

A script that deploys or reports the latest version of an artifact asks the repository for it. Nexus and Artifactory answer that on the server. CloudRepo does not, so this section shows the Nexus and Artifactory calls first, then what to read from CloudRepo for each format.

What Nexus and Artifactory do

Nexus Repository 3’s Search API downloads the latest version of an asset when you add sort=version: Sonatype says it downloads “the highest semantic version available for the Maven group ID and artifact ID specified”. The call answers with a redirect to the file, so curl needs --location.

curl --netrc --fail --location --output lib.jar \
  'https://nexus.example/service/rest/v1/search/assets/download?sort=version&repository=maven-releases&maven.groupId=com.example&maven.artifactId=lib&maven.extension=jar'

Artifactory’s Artifact Latest Version Search returns the latest version of a group ID and artifact ID as plain text, “based on a Maven-specific algorithm for sorting versions”. JFrog documents it as requiring Artifactory Pro.

curl --netrc --fail --silent \
  'https://acme.jfrog.io/artifactory/api/search/latestVersion?g=com.example&a=lib&repos=libs-release-local'

Artifactory’s Retrieve Artifact call takes the token [RELEASE] in place of the version in a download path: “Specify SNAPSHOT or [RELEASE] for the version in the requested path to get the latest Maven integration or release artifact.” curl reads square brackets as a range, so the command needs --globoff.

curl --netrc --fail --globoff --output lib.jar \
  'https://acme.jfrog.io/artifactory/libs-release-local/com/example/lib/[RELEASE]/lib-[RELEASE].jar'

What CloudRepo does

CloudRepo has no server-side latest resolver, so sort=version, api/search/latestVersion and [RELEASE] in a path have no counterpart on your repository host. Your script reads a list of versions instead, which each format keeps in its own place, and takes the highest itself. Decide which versions count (a snapshot, a release candidate, a branch tag) and write that rule into the script. The next four subsections give the list for Maven, Docker, npm and Python.

Maven

A Maven repository keeps an artifact’s version list in maven-metadata.xml, in the artifact’s folder: https://your-org.mycloudrepo.io/repositories/your-repo/com/example/lib/maven-metadata.xml for the group ID com.example and the artifact ID lib. mvn deploy and mvn deploy:deploy-file write that file as they publish, and cloudrepo-migrate (for Nexus, for Artifactory) copies the one your Nexus or Artifactory holds with the rest of your files. CloudRepo does not write or change maven-metadata.xml for a file you upload with curl, so a version uploaded that way is not in the file the repository holds.

The file holds <latest>, <release> and one <version> for each version. Maven’s reference defines <latest> as the last version added, snapshots included, and <release> as the last version added, releases only. Read <release> for the release deployed last:

curl --netrc --fail --silent \
  https://your-org.mycloudrepo.io/repositories/your-repo/com/example/lib/maven-metadata.xml \
  | sed -n 's:.*<release>\(.*\)</release>.*:\1:p'

<release> names the release version deployed last, so it is the highest version only when you deploy in version order. To take the highest, read every <version> and sort them as versions:

curl --netrc --fail --silent \
  https://your-org.mycloudrepo.io/repositories/your-repo/com/example/lib/maven-metadata.xml \
  | sed -n 's:.*<version>\(.*\)</version>.*:\1:p' \
  | grep -v -e '-SNAPSHOT' \
  | sort -V \
  | tail -n 1

sort -V is GNU sort’s version order and not Maven’s: it ranks 1.0.0-rc1 above 1.0.0, so a script that keeps release candidates in the same list needs its own rule. The grep drops snapshots, which sort -V would rank above their release.

Docker

Artifactory’s List Docker Tags lists “all tags of the specified Artifactory Docker repository”, and takes n and last to page through them.

curl --netrc --fail --silent \
  https://acme.jfrog.io/artifactory/api/docker/docker-local/v2/my-app/tags/list

CloudRepo answers the Docker registry’s own tag list on your organization’s host, at /v2/repositories/<repository>/<image>/tags/list. The repositories/<repository>/ part is the one an image name carries on CloudRepo: Docker image names after Nexus or Artifactory maps the Nexus and Artifactory forms. The answer is JSON with a name and a list of tags. The tags come in text order, so 1.10.0 comes before 1.9.0. A tag such as latest or a branch name is in the list too, and sorts after every number, so the command keeps the tags that are version numbers and then sorts them as versions:

curl --netrc --fail --silent \
  https://your-org.mycloudrepo.io/v2/repositories/your-repo/my-app/tags/list \
  | jq -r '.tags[]' \
  | grep -E '^[0-9]+(\.[0-9]+)*$' \
  | sort -V \
  | tail -n 1

CloudRepo takes n and last as Artifactory does. With n it answers that many tags, and when more remain it adds a Link header whose rel="next" address carries the last tag for the next page. Without n it answers every tag.

npm

npm keeps the latest version of a package as the version its latest dist-tag names. npm view reads “latest” when you name no version, and npm publish adds the latest tag unless you give it another with --tag. CloudRepo serves npm’s registry API, so the npm commands run against it as they run against Nexus or Artifactory. The token goes in the .npmrc that Install npm packages writes.

npm view @acme/my-package version --registry=https://your-org.mycloudrepo.io/repositories/your-repo/

npm dist-tag ls lists every tag of the package and the version each names:

npm dist-tag ls @acme/my-package --registry=https://your-org.mycloudrepo.io/repositories/your-repo/

Python

Artifactory’s PyPI JSON call retrieves “metadata in JSON format for the latest PyPI package version”.

curl --netrc --fail --silent \
  https://acme.jfrog.io/artifactory/api/pypi/pypi-local/pypi/my-package/json

CloudRepo serves no JSON API for a package. It serves the simple index that pip reads: the repository’s projects at the repository URL followed by /simple/, and for each project one HTML page at /simple/<package>/ that links every file of the package. A script can read that page:

curl --netrc --fail --silent \
  https://your-org.mycloudrepo.io/repositories/your-repo/simple/my-package/

Or it can let pip read the index. pip index versions lists the versions an index holds, highest first, and pip reads your login from ~/.netrc as Install Python packages shows:

python -m pip index versions my-package \
  --index-url https://your-org.mycloudrepo.io/repositories/your-repo/simple/

Upload a file

Nexus Repository 3 uploads a component with a POST, one form field for each coordinate and each file, and answers 204 No Content.

curl --netrc --fail --request POST \
  --form maven2.groupId=com.example \
  --form maven2.artifactId=lib \
  --form maven2.version=1.0.0 \
  --form maven2.asset1=@lib-1.0.0.jar \
  --form maven2.asset1.extension=jar \
  'https://nexus.example/service/rest/v1/components?repository=maven-releases'

Artifactory deploys an artifact with a PUT of the file to the path it will be read from.

curl --netrc --fail --upload-file lib-1.0.0.jar \
  https://acme.jfrog.io/artifactory/libs-release-local/com/example/lib/1.0.0/lib-1.0.0.jar

CloudRepo takes a file with a PUT at the path it will be read from, and has two ways in. For a Maven artifact, use mvn deploy:deploy-file against the repository URL. By default it also uploads a POM for the file, and it uploads maven-metadata.xml, so the version is listed for the commands under Maven above. -DrepositoryId names the <server> of ~/.m2/settings.xml that holds your login, as Publish with mvn deploy shows.

mvn deploy:deploy-file \
  -Dfile=lib-1.0.0.jar \
  -DgroupId=com.example \
  -DartifactId=lib \
  -Dversion=1.0.0 \
  -Dpackaging=jar \
  -DrepositoryId=cloudrepo \
  -Durl=https://your-org.mycloudrepo.io/repositories/your-repo

For any other file, use curl --upload-file, which sends a PUT of the file as it is. It writes no POM and no maven-metadata.xml:

curl --netrc --fail --upload-file lib-1.0.0.jar \
  https://your-org.mycloudrepo.io/repositories/your-repo/com/example/lib/1.0.0/lib-1.0.0.jar

Raw files over HTTP shows the upload, the download and the listing for a file that is not an artifact, and what each refusal answers. Connect Maven shows the Maven settings. Three differences from Nexus and Artifactory are worth checking in a script that uploads:

A checksum file is a file. CloudRepo serves lib-1.0.0.jar.sha1 if someone uploaded it, as mvn deploy does, and creates none for a file you upload with curl. A script that reads a .sha1 or .md5 beside such a file finds nothing until you upload that file too.
A second upload to a path is refused. A repository with Overwrite Protection on, as a new repository is, answers a second upload to a path that holds a file with 409 Conflict. A script that uploads the same path on every run needs a new path for each version, or the setting off.
A group repository stores nothing. Upload to a hosted repository, not to a group: a write to a group is refused with 405 Method Not Allowed.

Calls with no counterpart on CloudRepo

Artifactory’s Artifactory Query Language call, POST /artifactory/api/search/aql, and its Property Search call, GET /artifactory/api/search/prop, select files by a query or by the properties set on them. CloudRepo has no counterpart to either. Your repository host has no call that selects files by a query or by a property, and the API reference publishes no operation that searches files. A script that uses either keeps the list of files it needs, or lists a directory: Raw files over HTTP shows the listing, which is an HTML page.

Find the calls in your scripts

This command lists every line of the scripts below that Git tracks and that holds a call this page covers, or a Nexus or Artifactory path, so a call you did not think of shows up too. Run it in each repository you have, and change the list of file names to the scripts you keep:

git grep -n -i -E 'service/rest/v1/(search|components)|api/search/(latestVersion|aql|prop)|api/(docker|pypi)/|\[RELEASE\]|/artifactory/|/repository/|--upload-file|-X[[:space:]]*PUT' -- '*.sh' '*.bash' '*.zsh' '*.py' '*.js' '*.rb' '*.ps1' '*.groovy' '*.yml' '*.yaml' 'Jenkinsfile*' '*Makefile'

Each hit is a file and a line. An upload with curl -T is listed only when its line names a Nexus or Artifactory path, --upload-file or -X PUT, so look for -T in your scripts too. The same search for Docker image names and for build files is on Docker image names after Nexus or Artifactory. A script can also build a call from variables, or read it from a file the search does not name, so read the scripts your CI jobs run, not only the ones it lists.

Check a script before you cut over

Before the day you switch, run each rewritten call against CloudRepo and compare it with what Nexus or Artifactory gives:

For each artifact the script handles, compare the version it picks from CloudRepo with the version it picks from your Nexus or Artifactory. The two servers order versions by their own rules, and your script now holds the rule.
Make the call fail where it should. curl --fail exits 22 for a 401, 403, 404 or 409, and a script that ignores the exit code goes on with nothing. To see the status of a call, print it:

curl --netrc --silent --output /dev/null --write-out '%{http_code}\n' \
  https://your-org.mycloudrepo.io/repositories/your-repo/com/example/lib/maven-metadata.xml

Raw files over HTTP says what each status means, and Publish Maven artifacts says what to do about it. Then follow step 6 of Migrate from Sonatype Nexus Repository or Migrate from JFrog Artifactory, which sets the order of the switch.