Using BitBucket Pipelines With Maven Repositories
This tutorial will provide a professionally written, high quality, demonstration on how to use Pipelines and a Maven Repository in order to configure a Continuous Integration (CI) or Continuous Deployment workflow where build artifacts are stored, in a maven repository, for future use.
We accomplish this by providing a walkthrough of two examples:
First, we’ll use an Apache Maven based project in order to build a java library and publish it to an artifact repository. Once this library has been successfully published, we can use the library in other maven projects as a dependency.
If you want to dive right into the library example, you can view all of the source code in the Maven Library Example BitBucket Repository.
Once the library is available in a repository, we will provide brief instructions on how to modify the first example to pull down the library in subsequent builds.
We believe these examples represent a primary use case for using Maven, building and sharing Java Archive (JAR) files (dependencies).
This article will describe how to publish to CloudRepo, a fully managed, private maven repository server. However, in the interest of providing the greatest benefit, these instructions should apply to any Apache Maven compatible repository.
We assume the usage of the following technologies:
- Apache Maven
- BitBucket Git Repository
- BitBucket Pipelines
Apache Maven is a build tool used by Java software development teams to build software artifacts: libraries, binaries, documentation, etc.
Once an artifact has been successfully built with Maven, it can be deployed to a remote repository where other Maven based builds and processes can access it.
BitBucket Git Repository
Git is a version control system used by software development teams to manage their source code bases.. Source code is the input to build processes which produce software artifacts.
In the Java/Maven world, source code is typically compiled into Java Virtual Machine (JVM) Object Code and then distributed as JAR files.
BitBucket Cloud provides cloud based hosting for Git Repositories. A BitBucket hosted Git Repository is required in order to use Pipelines.
Pipelines provides a Continuous Integration and Continuous Delivery Platform as a Service (PaaS) used by teams of all sizes to build, test, and deliver software in a repeatable deployment pipeline.
Pipelines provides a CI server that can build source code, run automated tests, and automate deployment of your application to your production environment many times each day as part of your development practice and release process.
If you are building maven artifacts, you’ll want to publish them to a shared repository for use in other builds after tests (unit tests, integration tests, etc.) have run successfully.
This article assumes you’ll be using Pipelines (which provides a limited amount of build minutes for free each month) to build libraries and publish them to a central repository hosted by CloudRepo.
CloudRepo provides hosted private maven repositories so that you can publish your artifacts to persistent, highly available, accessible, and secure storage.
Once an artifact has been published to CloudRepo, it can be retrieved using a maven compatible client or via a HTTPS REST API. This allows you to reuse code by publishing it as a java library and then declaring it as a dependency in any other maven project.
For this example, we will store our artifacts using CloudRepo’s Private Maven Repositories.
Complete Build Process: Git, Apache Maven, BitBucket Pipelines, and CloudRepo
Tying these technologies together will give you a build pipeline that looks something like this:
There are two major use cases for integrating Pipelines and CloudRepo:
- Building a Library - Publishing Artifacts
- Building an Application - Reading Artifacts
This article will first setup a build pipeline which will publish artifacts to CloudRepo using Maven.
Once artifacts have been published we will show how to modify your pom.xml file to read these artifacts in a separate build.
Now that we’ve covered all the background information, let’s jump into the code.
Building and Publishing a Java Library to CloudRepo
The first step to building and publishing your java library is to get the source code into a BitBucket Git repository.
For this example, we’ve put together a library which performs basic mathematical functions that we’ll later use in an application. The complete source for this example can be found in our Example Git Repository
The functionality of this library is not important to this tutorial. It's just there to provide some code that we can publish as a library.
BitBucket Pipeline Configuration
This section will cover all the configuration you have to do in the BitBucket User Interface so that you can configure Maven in subsequent steps.
In order to use Pipelines, you must first enable them for your repository. Simply navigate to the settings of your repository and find the 'Pipelines' section. Toggle the 'Enable Pipelines' button in the Pipelines 'Settings' menu.
The following screenshot illustrates:
Add Secure Environment Variables for Your Pipeline
Private artifact repositories require credentials in order to access them. You don't want to store these credentials in your source code as that is a major no-no and security issue (Pro Tip: never store credentials in a git repository).
Pipelines allows you to set Secure Environment Variables that each pipeline will have access to. This is an ideal place to store your maven credentials and other variables related to our build process.
In your BitBucket Repository, navigate to the 'Settings' panel, then down to the 'Pipelines' section. Click 'Environment Variables' and add the following:
- This is the username for pushing and pulling artifacts from your maven repo. This will end up in your settings.xml file which we'll create later.
- Optional: To prevent logging of this username you can set the enviroment variable to secret.
- This is the passphrase for pushing and pulling artifacts from your maven repo. This will end up in your settings.xml file.
- RECOMMENDED: To prevent logging of this passphrase you can set the enviroment variable to secret.
- Used only when the 'mvn release:prepare' trigger is run. This is the user email used by Git when Pipelines commits back to this repository.
- Note: This does not have to correspond to an actual user's email, it will only show up in your Git commit logs as the author's email of any commit performed by Pipelines.
- Used only when the 'mvn release:prepare' trigger is run. This is the user name used by Git when Pipelines commits back to this repository.
- Note: This does not have to correspond to an actual user's name, it will only show up in your Git commit logs as the author's name of any commit performed by Pipelines.
The following screenshot illustrates:
Configuring SSH on Pipelines
When you run the Maven release plugin (ie. mvn release:prepare release:perform), maven will create git commits and attempt to push them back to your source code repository. A BitBucket Pipeline does not have the permissions to push to a repository so we have to enable this by adding an SSH key associated with a Pipeline.
Create SSH KeyPair (For Maven Releases Only)
We need a key pair to allow Pipelines to git push back to this repo when we are releasing an artifact via a mvn release:perform command.
This can be found in the 'Settings | Pipelines (SSH Keys) | Generate Keys' section of the BitBucket UI.
Generate a key to associate it with your pipeline as seen in the following screenshot:
Add Public Key to Your Accounts list of SSH Keys (For Maven Releases Only)
Once the Key Pair has been generated, copy the Public Key that was created and navigate to your BitBucket Account Settings (not the Pipelines Settings).
Add a new SSH to your account (Settings | Security | SSH Keys | Add Key).
Paste the public key from the BitBucket Pipeline here.
Now that you've configured the BitBucket UI, you're ready to proceed with configuring Maven in your pipeline.
Maven POM File Configuration
In order to use Maven to build a project you’ll need to have a valid working Project Object Model (POM) in your source code directory named pom.xml. We provide a pom.xml in our example for reference.
The <distributionManagement> tag is where we tell Maven where to deploy our Maven artifacts when we run commands like ‘mvn deploy’ (snapshots) or ‘mvn release:perform’ (releases).
A best practice for maven repository management is to have a separate repo for snapshots and one for releases.
Our distribution management section looks like this:
Important: Remember that the <distributionManagement> tag is used only for deploying artifacts TO maven repositories.
For reading artifacts from maven repositories, you'll use the <repositories> tag, which is discussed next:
The <repositories> tag is used when you want to retrieve dependencies from a Maven Repository.
For our example, we will only be deploying artifacts to CloudRepo. However, for completeness we provide the configuration for the <repositories> tag so that you can use it in projects that consume your libraries.
Here's what our <repositories> tag looks like:
Snapshots versus Releases?
As you can see, we follow best practices and use separate CloudRepo repositories for snapshots and repositories. We’ll quickly describe the differences between Snapshots and Releases here (skip ahead if this known to you).
Maven Snapshots are used when you’re developing code that isn’t quite ready to be released. Maven snapshots are indicted by the ‘-SNAPSHOT’ suffix appended to a version id. Using Snapshots allows you to point at a changing version of an artifact without having to update your POM file.
Using Maven snapshots should only be used during development. As the contents of a maven snapshot can change, you lose the benefit of repeatable builds by depending on snapshot versions of artifacts.
Maven Releases are considered to be static and should not change. This allows a project to depend on a release version of an artifact and achieve repeatability in a build. Depending on version 1.0.0 of an artifact should always return the same version of an artifact. When declaring dependencies from within your projects, you should attempt to use release versions whenever possible in order to better achieve repeatability.
Maven settings.xml file
Before you can successfully deploy your artifacts to CloudRepo from Pipelines, you’ll need to create a settings.xml file which contains the credentials for the repositories you specified in the <dependencyManagement> section of your POM file.
To simplify this process, we provide a script named `create-settings.sh` which will create a settings.xml file by using the settings.template.xml file as a base and replacing the %CLOUDREPO_USERNAME% and %CLOUDREPO_PASSPHRASE% tokens with values from the CLOUDREPO_USERNAME and CLOUDREPO_PASSPHRASE environment variables.
Testing settings.xml creation
To test the settings.xml creation on a local Linux based machine, execute the `create-settings.sh` script with your CloudRepo username and passphrase:
This command should run successfully and you should have a new file named settings.xml in your source directory. The contents should be the following:
`mvn deploy`- Deploying Snapshots to CloudRepo
Now that you’ve created the settings.xml file, you can test the deployment of snapshots locally by running the following command from your source directory:
This command will build and deploy your SNAPSHOT artifacts to the CloudRepo snapshot repository.
bitbucket-pipelines.yml - Pipelines Configuration File
Finally, we’ll need to add a bitbucket-pipelines.yml file to our source code repository. This file describes the steps and scripts that Pipelines should execute when building your library.
In our example, we create two different build steps.
The first step, "Build and Deploy Snapshot Artifact" will build and deploy a SNAPSHOT every time a commit is detected on the master branch.
As you develop, this will be the normal workflow that you use as your team collaborates on a development version of your library. This step is configured to run tests and to ensure that they all pass before a SNAPSHOT is published to maven.
The second step, which must be triggered manually, "Create Release Version" will prepare a Maven RELEASE version, and commit the changes to the master branch. It will then increment the project version and commit the next SNAPSHOT version to the repository.
This results in two commits to your master branch: one for the RELEASE version and one for the new SNAPSHOT version. Both of these commits will be detected by the build trigger in the first step and will be built accordingly.
After these builds complete, you will have the RELEASE version of your library in your repository as well as the new SNAPSHOT version.
Our bitbucket-pipelines.yml file looks like the following:
Now that you have everything setup properly, how do you know it's working? You can validate successful deployment in two places:
- BitBucket Pipeline Status
- CloudRepo Repository Storage
BitBucket Pipeline Status
You can refer to the 'Pipelines' menu option for your BitBucket Repository to see the status of your builds.
After successful execution of this example, you should see the following after your first successful build:
When you are ready to create a release version of your artifact, click the 'Run' button in the 'Create Release Version' step. This will kick of a 'mvn release:perform' which will create two new commits on your master branch.
When the 'Create Release Versions' step is done, the build results will look like the following:
When you go back to the Pipeline Summary you will see that two new builds have been completed, one for the new release version and another for the next version of the snapshot.
If your BitBucket Pipeline history looks like this then you've completed this example successfully.
CloudRepo Repository Storage
Since these pipelines deployed Snapshot artifacts to CloudRepo, log into your repositories user interface to verify the artifacts have been successfully deployed.
We'll show examples from the CloudRepo Admin Portal below:
Navigating to the 'maven-snapshots' repository in CloudRepo, we can validate that the next version of the snapshot has been release:
Navigating to the 'maven-releases' repository in CloudRepo, we can validate the 1.0.0 release version has been successfully deployed.
The source code for this example contains a README.md file with even more detailed instructions. Please look there if you need further information.
This is a living document and so we'd love to hear any feedback on how we can improve this tutorial for others. If you have any suggestions at all, please feel free to email us.
While we do our best to ensure a high quality example, we might have missed something. If you hit a snag, please open an issue so that we can fix it for others who are seeking similar knowledge.
Please feel free to share on Social Media, Medium, or other channels.