CircleCI CI/CD Integration

CircleCI provides modern, cloud-native CI/CD capabilities that integrate seamlessly with CloudRepo for publishing and consuming artifacts across all supported package types.

This guide covers complete integration patterns for Maven, Gradle, Python (pip, Poetry, UV, Pixi), and Docker projects using CircleCI 2.1 configuration with orbs, executors, and workflows, along with advanced patterns and security best practices.

Benefits & Overview

Why Use CloudRepo with CircleCI

  • Cost Effective: Save 90%+ compared to JFrog Artifactory with no egress fees

  • Native Integration: Works seamlessly with CircleCI orbs and executors

  • Secure: Leverage CircleCI Contexts for secure credential management

  • Scalable: Handle everything from small libraries to enterprise deployments

  • Support Included: All plans include professional support

  • Performance: Optimized caching strategies for faster builds

Repository URL Format

All CloudRepo repositories follow this URL pattern:

https://[organization-id].mycloudrepo.io/repositories/[repository-id]

Replace [organization-id] and [repository-id] with your actual values throughout this guide.

Prerequisites

CircleCI Account Setup

Before configuring CircleCI, ensure you have:

  1. A CircleCI account connected to your GitHub/GitLab/Bitbucket repository

  2. Appropriate CircleCI plan for your needs (Free tier works for most cases)

  3. CircleCI CLI installed locally (optional, for testing):

    # Install CircleCI CLI
curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/master/install.sh | bash

# Validate configuration locally
circleci config validate

CloudRepo Account Requirements

Ensure you have:

  1. A CloudRepo organization account

  2. At least one repository created (Maven, Python, or Raw)

  3. A user account with appropriate permissions:

    • Read permission for consuming artifacts

    • Write permission for publishing artifacts

    • Admin permission for repository management

  4. Your organization ID and repository ID from the CloudRepo dashboard

Context and Environment Variable Setup

CircleCI uses Contexts to securely store credentials across projects:

Creating a Context (Organization-wide):

  1. Navigate to CircleCI Organization Settings

  2. Click Contexts → Create Context

  3. Name it cloudrepo-credentials

  4. Add the following environment variables:

    • CLOUDREPO_USERNAME: Your CloudRepo username

    • CLOUDREPO_PASSWORD: Your CloudRepo password

    • CLOUDREPO_MAVEN_URL: Your Maven repository URL (if using Maven/Gradle)

    • CLOUDREPO_PYTHON_URL: Your Python repository URL (if using Python)

    • CLOUDREPO_RAW_URL: Your Raw repository URL (if using Raw/Docker)

Project-level Environment Variables:

For single projects, add variables in Project Settings → Environment Variables.

Authentication Configuration

Using CircleCI Contexts for Credentials

Contexts provide secure, centralized credential management:

.circleci/config.yml - Using Contexts
version: 2.1

workflows:
  build-and-deploy:
    jobs:
      - build:
          context:
            - cloudrepo-credentials

jobs:
  build:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Verify Credentials
          command: |
            if [ -z "$CLOUDREPO_USERNAME" ]; then
              echo "Error: CLOUDREPO_USERNAME not set"
              exit 1
            fi
            echo "Credentials loaded successfully"

Environment Variable Management

Best practices for managing environment variables:

.circleci/config.yml - Environment Variable Patterns
version: 2.1

# Define reusable environment variables
executors:
  cloudrepo-executor:
    docker:
      - image: cimg/base:stable
    environment:
      CLOUDREPO_ORG_ID: mycompany
      CLOUDREPO_MAVEN_REPO: maven-releases
      CLOUDREPO_PYTHON_REPO: python-releases

jobs:
  deploy:
    executor: cloudrepo-executor
    steps:
      - checkout
      - run:
          name: Configure Repository URLs
          command: |
            export CLOUDREPO_MAVEN_URL="https://${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${CLOUDREPO_MAVEN_REPO}"
            export CLOUDREPO_PYTHON_URL="https://${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${CLOUDREPO_PYTHON_REPO}"
            echo "Maven URL: $CLOUDREPO_MAVEN_URL"
            echo "Python URL: $CLOUDREPO_PYTHON_URL"

Security Best Practices

  1. Never commit credentials to your repository

  2. Use Contexts for organization-wide credentials

  3. Restrict Context access to specific projects/branches

  4. Rotate credentials regularly (every 90 days recommended)

  5. Use read-only credentials for dependency resolution

  6. Enable audit logging in CloudRepo for compliance

.circleci/config.yml - Security Configuration
version: 2.1

workflows:
  secure-deployment:
    jobs:
      - hold:
          type: approval
          filters:
            branches:
              only: main
      - deploy:
          requires:
            - hold
          context:
            - cloudrepo-production
          filters:
            branches:
              only: main

Maven/Gradle Projects

Orb Configuration for Maven

CircleCI provides official Maven orb for streamlined configuration:

.circleci/config.yml - Maven with Orb
version: 2.1

orbs:
  maven: circleci/maven@1.4.1

workflows:
  maven-workflow:
    jobs:
      - maven/test:
          context:
            - cloudrepo-credentials
      - maven/deploy:
          context:
            - cloudrepo-credentials
          requires:
            - maven/test
          filters:
            branches:
              only: main

jobs:
  maven/deploy:
    steps:
      - checkout
      - run:
          name: Configure Maven Settings
          command: |
            mkdir -p ~/.m2
            cat > ~/.m2/settings.xml <<EOF
            <settings>
              <servers>
                <server>
                  <id>cloudrepo</id>
                  <username>\${env.CLOUDREPO_USERNAME}</username>
                  <password>\${env.CLOUDREPO_PASSWORD}</password>
                </server>
              </servers>
            </settings>
            EOF
      - run:
          name: Deploy to CloudRepo
          command: mvn deploy -DaltDeploymentRepository=cloudrepo::default::${CLOUDREPO_MAVEN_URL}

Gradle Wrapper Integration

Configure Gradle projects with CloudRepo:

.circleci/config.yml - Gradle Configuration
version: 2.1

orbs:
  gradle: circleci/gradle@3.0.0

executors:
  gradle-executor:
    docker:
      - image: cimg/openjdk:17.0
    environment:
      GRADLE_OPTS: "-Xmx3200m -Dorg.gradle.daemon=false"

workflows:
  gradle-workflow:
    jobs:
      - test-and-build
      - deploy:
          requires:
            - test-and-build
          filters:
            branches:
              only: main
          context:
            - cloudrepo-credentials

jobs:
  test-and-build:
    executor: gradle-executor
    steps:
      - checkout
      - restore_cache:
          keys:
            - gradle-deps-{{ checksum "build.gradle" }}
            - gradle-deps-
      - run:
          name: Run Tests
          command: ./gradlew test
      - save_cache:
          key: gradle-deps-{{ checksum "build.gradle" }}
          paths:
            - ~/.gradle
      - persist_to_workspace:
          root: .
          paths:
            - build/libs

  deploy:
    executor: gradle-executor
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Configure Gradle Properties
          command: |
            cat > gradle.properties <<EOF
            cloudRepoUrl=${CLOUDREPO_MAVEN_URL}
            cloudRepoUsername=${CLOUDREPO_USERNAME}
            cloudRepoPassword=${CLOUDREPO_PASSWORD}
            EOF
      - run:
          name: Publish to CloudRepo
          command: ./gradlew publish
build.gradle - Publishing Configuration
publishing {
    repositories {
        maven {
            url = project.cloudRepoUrl
            credentials {
                username = project.cloudRepoUsername
                password = project.cloudRepoPassword
            }
        }
    }
    publications {
        maven(MavenPublication) {
            from components.java
            groupId = 'com.example'
            artifactId = 'my-library'
            version = '1.0.0'
        }
    }
}

Caching Dependencies

Optimize build times with intelligent caching:

.circleci/config.yml - Advanced Caching
version: 2.1

jobs:
  build-with-cache:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - restore_cache:
          keys:
            # Most specific cache first
            - v2-maven-{{ checksum "pom.xml" }}
            # Fallback to less specific cache
            - v2-maven-
      - run:
          name: Download Dependencies
          command: mvn dependency:go-offline
      - save_cache:
          key: v2-maven-{{ checksum "pom.xml" }}
          paths:
            - ~/.m2/repository
      - run:
          name: Build and Test
          command: mvn clean package

Publishing Artifacts

Complete example for publishing Maven artifacts:

.circleci/config.yml - Complete Maven Publishing
version: 2.1

orbs:
  maven: circleci/maven@1.4.1

executors:
  maven-executor:
    docker:
      - image: cimg/openjdk:17.0-node
    working_directory: ~/project

workflows:
  version: 2
  build-test-deploy:
    jobs:
      - test:
          context:
            - cloudrepo-credentials
      - hold:
          type: approval
          requires:
            - test
          filters:
            branches:
              only: main
      - deploy:
          requires:
            - hold
          context:
            - cloudrepo-credentials
          filters:
            branches:
              only: main

jobs:
  test:
    executor: maven-executor
    steps:
      - checkout
      - restore_cache:
          keys:
            - maven-deps-{{ checksum "pom.xml" }}
      - run:
          name: Configure Maven Settings
          command: |
            mkdir -p ~/.m2
            cat > ~/.m2/settings.xml <<EOF
            <settings>
              <servers>
                <server>
                  <id>cloudrepo</id>
                  <username>${CLOUDREPO_USERNAME}</username>
                  <password>${CLOUDREPO_PASSWORD}</password>
                </server>
              </servers>
              <profiles>
                <profile>
                  <id>cloudrepo</id>
                  <repositories>
                    <repository>
                      <id>cloudrepo</id>
                      <url>${CLOUDREPO_MAVEN_URL}</url>
                    </repository>
                  </repositories>
                </profile>
              </profiles>
              <activeProfiles>
                <activeProfile>cloudrepo</activeProfile>
              </activeProfiles>
            </settings>
            EOF
      - run:
          name: Run Tests
          command: mvn clean test
      - save_cache:
          key: maven-deps-{{ checksum "pom.xml" }}
          paths:
            - ~/.m2/repository
      - store_test_results:
          path: target/surefire-reports
      - store_artifacts:
          path: target/site

  deploy:
    executor: maven-executor
    steps:
      - checkout
      - restore_cache:
          keys:
            - maven-deps-{{ checksum "pom.xml" }}
      - run:
          name: Configure Maven Settings
          command: |
            mkdir -p ~/.m2
            cat > ~/.m2/settings.xml <<EOF
            <settings>
              <servers>
                <server>
                  <id>cloudrepo</id>
                  <username>${CLOUDREPO_USERNAME}</username>
                  <password>${CLOUDREPO_PASSWORD}</password>
                </server>
              </servers>
            </settings>
            EOF
      - run:
          name: Deploy to CloudRepo
          command: |
            mvn versions:set -DnewVersion=${CIRCLE_TAG:-${CIRCLE_SHA1:0:7}}
            mvn clean deploy -DskipTests \
              -DaltDeploymentRepository=cloudrepo::default::${CLOUDREPO_MAVEN_URL}

Python Projects

Python Orb Configuration

Use the CircleCI Python orb for standardized workflows:

.circleci/config.yml - Python Orb
version: 2.1

orbs:
  python: circleci/python@2.1.1

workflows:
  python-workflow:
    jobs:
      - python/test:
          pkg-manager: pip
          test-tool: pytest
      - publish-to-cloudrepo:
          requires:
            - python/test
          context:
            - cloudrepo-credentials
          filters:
            branches:
              only: main

jobs:
  publish-to-cloudrepo:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Build Distribution
          command: |
            pip install build twine
            python -m build
      - run:
          name: Upload to CloudRepo
          command: |
            twine upload \
              --repository-url ${CLOUDREPO_PYTHON_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*

Pip and Poetry Support

Configure pip and Poetry for CloudRepo:

.circleci/config.yml - Poetry Configuration
version: 2.1

orbs:
  python: circleci/python@2.1.1

executors:
  python-executor:
    docker:
      - image: cimg/python:3.11
    environment:
      POETRY_VERSION: 1.7.1

workflows:
  poetry-workflow:
    jobs:
      - test
      - publish:
          requires:
            - test
          context:
            - cloudrepo-credentials
          filters:
            tags:
              only: /^v.*/
            branches:
              ignore: /.*/

jobs:
  test:
    executor: python-executor
    steps:
      - checkout
      - run:
          name: Install Poetry
          command: |
            curl -sSL https://install.python-poetry.org | python -
            echo 'export PATH="$HOME/.local/bin:$PATH"' >> $BASH_ENV
      - restore_cache:
          keys:
            - poetry-deps-{{ checksum "poetry.lock" }}
      - run:
          name: Configure Poetry
          command: |
            poetry config virtualenvs.in-project true
            poetry config repositories.cloudrepo ${CLOUDREPO_PYTHON_URL}
            poetry config http-basic.cloudrepo ${CLOUDREPO_USERNAME} ${CLOUDREPO_PASSWORD}
      - run:
          name: Install Dependencies
          command: poetry install --no-interaction --no-ansi
      - save_cache:
          key: poetry-deps-{{ checksum "poetry.lock" }}
          paths:
            - .venv
      - run:
          name: Run Tests
          command: poetry run pytest --junitxml=test-results/junit.xml
      - store_test_results:
          path: test-results

  publish:
    executor: python-executor
    steps:
      - checkout
      - run:
          name: Install Poetry
          command: |
            curl -sSL https://install.python-poetry.org | python -
            echo 'export PATH="$HOME/.local/bin:$PATH"' >> $BASH_ENV
      - run:
          name: Configure Poetry for CloudRepo
          command: |
            poetry config repositories.cloudrepo ${CLOUDREPO_PYTHON_URL}
            poetry config http-basic.cloudrepo ${CLOUDREPO_USERNAME} ${CLOUDREPO_PASSWORD}
      - run:
          name: Build and Publish
          command: |
            poetry version ${CIRCLE_TAG#v}
            poetry build
            poetry publish -r cloudrepo

UV Ultra-fast Package Manager Integration

UV provides ultra-fast Python package management:

.circleci/config.yml - UV Configuration
version: 2.1

executors:
  uv-executor:
    docker:
      - image: cimg/python:3.11
    environment:
      UV_VERSION: 0.1.24

workflows:
  uv-workflow:
    jobs:
      - test-with-uv
      - publish-with-uv:
          requires:
            - test-with-uv
          context:
            - cloudrepo-credentials
          filters:
            branches:
              only: main

jobs:
  test-with-uv:
    executor: uv-executor
    steps:
      - checkout
      - run:
          name: Install UV
          command: |
            curl -LsSf https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-x86_64-unknown-linux-gnu.tar.gz | tar xz
            sudo mv uv /usr/local/bin/
      - restore_cache:
          keys:
            - uv-cache-{{ checksum "requirements.txt" }}
      - run:
          name: Configure UV for CloudRepo
          command: |
            export UV_EXTRA_INDEX_URL="https://${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}@${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${CLOUDREPO_PYTHON_REPO}/simple"
            echo "UV_EXTRA_INDEX_URL=$UV_EXTRA_INDEX_URL" >> $BASH_ENV
      - run:
          name: Install Dependencies with UV
          command: |
            uv venv
            source .venv/bin/activate
            uv pip install -r requirements.txt
            uv pip install pytest
      - save_cache:
          key: uv-cache-{{ checksum "requirements.txt" }}
          paths:
            - ~/.cache/uv
      - run:
          name: Run Tests
          command: |
            source .venv/bin/activate
            pytest tests/

  publish-with-uv:
    executor: uv-executor
    steps:
      - checkout
      - run:
          name: Install UV and Build Tools
          command: |
            curl -LsSf https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-x86_64-unknown-linux-gnu.tar.gz | tar xz
            sudo mv uv /usr/local/bin/
            uv venv
            source .venv/bin/activate
            uv pip install build twine
      - run:
          name: Build Package
          command: |
            source .venv/bin/activate
            python -m build
      - run:
          name: Upload to CloudRepo
          command: |
            source .venv/bin/activate
            twine upload \
              --repository-url ${CLOUDREPO_PYTHON_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*

Pixi Support

Configure Pixi for conda-forge compatible packages:

.circleci/config.yml - Pixi Configuration
version: 2.1

executors:
  pixi-executor:
    docker:
      - image: cimg/python:3.11
    environment:
      PIXI_VERSION: 0.15.2

workflows:
  pixi-workflow:
    jobs:
      - build-with-pixi
      - publish-conda-package:
          requires:
            - build-with-pixi
          context:
            - cloudrepo-credentials

jobs:
  build-with-pixi:
    executor: pixi-executor
    steps:
      - checkout
      - run:
          name: Install Pixi
          command: |
            curl -fsSL https://pixi.sh/install.sh | bash
            echo 'export PATH="$HOME/.pixi/bin:$PATH"' >> $BASH_ENV
      - run:
          name: Initialize Pixi Project
          command: |
            pixi init
            pixi add python=3.11 numpy pandas pytest
      - run:
          name: Configure Pixi for CloudRepo
          command: |
            cat >> pixi.toml <<EOF
            [project.pypi-options]
            extra-index-urls = ["https://${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}@${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${CLOUDREPO_PYTHON_REPO}/simple"]
            EOF
      - run:
          name: Run Tests
          command: pixi run pytest tests/

  publish-conda-package:
    executor: pixi-executor
    steps:
      - checkout
      - run:
          name: Install Conda Build Tools
          command: |
            curl -fsSL https://pixi.sh/install.sh | bash
            echo 'export PATH="$HOME/.pixi/bin:$PATH"' >> $BASH_ENV
            pixi global install conda-build anaconda-client
      - run:
          name: Build Conda Package
          command: |
            conda build recipe/
      - run:
          name: Convert and Upload to CloudRepo
          command: |
            # Convert conda package to wheel for CloudRepo Python repository
            pip install conda2wheel
            conda2wheel dist/*.tar.bz2
            twine upload \
              --repository-url ${CLOUDREPO_PYTHON_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*.whl

Publishing with Twine

Standard configuration for publishing Python packages:

.circleci/config.yml - Twine Publishing
version: 2.1

workflows:
  publish-workflow:
    jobs:
      - test
      - approve-release:
          type: approval
          requires:
            - test
          filters:
            branches:
              only: main
      - publish:
          requires:
            - approve-release
          context:
            - cloudrepo-credentials

jobs:
  test:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Install Dependencies
          command: |
            pip install -e .[dev]
            pip install pytest coverage
      - run:
          name: Run Tests with Coverage
          command: |
            coverage run -m pytest
            coverage report
            coverage html
      - store_artifacts:
          path: htmlcov

  publish:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Install Build Tools
          command: pip install build twine
      - run:
          name: Build Distribution
          command: python -m build
      - run:
          name: Check Distribution
          command: twine check dist/*
      - run:
          name: Create .pypirc
          command: |
            cat > ~/.pypirc <<EOF
            [distutils]
            index-servers = cloudrepo

            [cloudrepo]
            repository = ${CLOUDREPO_PYTHON_URL}
            username = ${CLOUDREPO_USERNAME}
            password = ${CLOUDREPO_PASSWORD}
            EOF
            chmod 600 ~/.pypirc
      - run:
          name: Upload to CloudRepo
          command: twine upload -r cloudrepo dist/*

Docker Integration

Building Docker Images with CloudRepo Dependencies

Build Docker images that consume artifacts from CloudRepo:

.circleci/config.yml - Docker Build
version: 2.1

orbs:
  docker: circleci/docker@2.2.0

executors:
  docker-executor:
    docker:
      - image: cimg/base:stable
    environment:
      DOCKER_BUILDKIT: 1

workflows:
  docker-workflow:
    jobs:
      - docker/hadolint:
          dockerfiles: ./Dockerfile
      - build-and-push:
          requires:
            - docker/hadolint
          context:
            - cloudrepo-credentials

jobs:
  build-and-push:
    executor: docker-executor
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
          version: 20.10.14
      - run:
          name: Create Docker Build Arguments
          command: |
            echo "export BUILDARGS='--build-arg CLOUDREPO_USERNAME=${CLOUDREPO_USERNAME} --build-arg CLOUDREPO_PASSWORD=${CLOUDREPO_PASSWORD} --build-arg CLOUDREPO_URL=${CLOUDREPO_MAVEN_URL}'" >> $BASH_ENV
      - run:
          name: Build Docker Image
          command: |
            docker build ${BUILDARGS} \
              --tag myapp:${CIRCLE_SHA1:0:7} \
              --tag myapp:latest \
              --cache-from myapp:latest \
              .
      - run:
          name: Save Docker Image
          command: |
            docker save -o image.tar myapp:${CIRCLE_SHA1:0:7}
      - persist_to_workspace:
          root: .
          paths:
            - image.tar

Multi-stage Builds

Optimize Docker builds with multi-stage patterns:

Dockerfile - Multi-stage Build with CloudRepo
# Build stage with CloudRepo dependencies
FROM maven:3.9-eclipse-temurin-17 AS builder

ARG CLOUDREPO_USERNAME
ARG CLOUDREPO_PASSWORD
ARG CLOUDREPO_URL

# Configure Maven settings for CloudRepo
RUN mkdir -p /root/.m2 && \
    echo "<settings><servers><server><id>cloudrepo</id><username>${CLOUDREPO_USERNAME}</username><password>${CLOUDREPO_PASSWORD}</password></server></servers><profiles><profile><id>cloudrepo</id><repositories><repository><id>cloudrepo</id><url>${CLOUDREPO_URL}</url></repository></repositories></profile></profiles><activeProfiles><activeProfile>cloudrepo</activeProfile></activeProfiles></settings>" > /root/.m2/settings.xml

WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline

COPY src ./src
RUN mvn clean package -DskipTests

# Runtime stage
FROM eclipse-temurin:17-jre-alpine
COPY --from=builder /app/target/*.jar app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
.circleci/config.yml - Multi-stage Docker Build
version: 2.1

workflows:
  docker-multistage:
    jobs:
      - build-multistage:
          context:
            - cloudrepo-credentials

jobs:
  build-multistage:
    machine:
      image: ubuntu-2204:2024.01.1
      docker_layer_caching: true
    resource_class: large
    steps:
      - checkout
      - run:
          name: Build with BuildKit
          command: |
            DOCKER_BUILDKIT=1 docker build \
              --build-arg CLOUDREPO_USERNAME=${CLOUDREPO_USERNAME} \
              --build-arg CLOUDREPO_PASSWORD=${CLOUDREPO_PASSWORD} \
              --build-arg CLOUDREPO_URL=${CLOUDREPO_MAVEN_URL} \
              --secret id=cloudrepo_user,env=CLOUDREPO_USERNAME \
              --secret id=cloudrepo_pass,env=CLOUDREPO_PASSWORD \
              --target production \
              --tag myapp:${CIRCLE_SHA1:0:7} \
              --cache-from type=registry,ref=myregistry/myapp:buildcache \
              --cache-to type=registry,ref=myregistry/myapp:buildcache,mode=max \
              .

Layer Caching Strategies

Optimize build times with intelligent Docker layer caching:

.circleci/config.yml - Advanced Docker Caching
version: 2.1

executors:
  docker-builder:
    machine:
      image: ubuntu-2204:2024.01.1
      docker_layer_caching: true
    resource_class: xlarge

workflows:
  cached-docker-build:
    jobs:
      - build-with-cache:
          context:
            - cloudrepo-credentials

jobs:
  build-with-cache:
    executor: docker-builder
    environment:
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Set up Docker Buildx
          command: |
            docker buildx create --name mybuilder --use
            docker buildx inspect --bootstrap
      - run:
          name: Build with Advanced Caching
          command: |
            # Use registry cache
            docker buildx build \
              --cache-from type=registry,ref=${DOCKER_REGISTRY}/myapp:buildcache \
              --cache-to type=registry,ref=${DOCKER_REGISTRY}/myapp:buildcache,mode=max \
              --build-arg CLOUDREPO_USERNAME=${CLOUDREPO_USERNAME} \
              --build-arg CLOUDREPO_PASSWORD=${CLOUDREPO_PASSWORD} \
              --build-arg CLOUDREPO_URL=${CLOUDREPO_MAVEN_URL} \
              --platform linux/amd64,linux/arm64 \
              --tag ${DOCKER_REGISTRY}/myapp:${CIRCLE_SHA1:0:7} \
              --tag ${DOCKER_REGISTRY}/myapp:latest \
              --push \
              .
      - run:
          name: Scan Image for Vulnerabilities
          command: |
            docker scout cves ${DOCKER_REGISTRY}/myapp:${CIRCLE_SHA1:0:7}

Advanced Workflows

Matrix Builds for Multiple Versions

Test against multiple language versions and dependencies:

.circleci/config.yml - Matrix Builds
version: 2.1

executors:
  python-multi:
    parameters:
      python-version:
        type: string
    docker:
      - image: cimg/python:<< parameters.python-version >>

workflows:
  matrix-test:
    jobs:
      - test-python:
          matrix:
            parameters:
              python-version: ["3.9", "3.10", "3.11", "3.12"]
          context:
            - cloudrepo-credentials
      - test-java:
          matrix:
            parameters:
              java-version: ["11", "17", "21"]
          context:
            - cloudrepo-credentials

jobs:
  test-python:
    parameters:
      python-version:
        type: string
    executor:
      name: python-multi
      python-version: << parameters.python-version >>
    steps:
      - checkout
      - run:
          name: Install Dependencies from CloudRepo
          command: |
            pip install --index-url https://${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}@${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${CLOUDREPO_PYTHON_REPO}/simple --extra-index-url https://pypi.org/simple -r requirements.txt
      - run:
          name: Run Tests
          command: pytest --junitxml=test-results/junit-<< parameters.python-version >>.xml
      - store_test_results:
          path: test-results

  test-java:
    parameters:
      java-version:
        type: string
    docker:
      - image: cimg/openjdk:<< parameters.java-version >>
    steps:
      - checkout
      - run:
          name: Configure Maven for CloudRepo
          command: |
            mkdir -p ~/.m2
            cat > ~/.m2/settings.xml <<EOF
            <settings>
              <servers>
                <server>
                  <id>cloudrepo</id>
                  <username>${CLOUDREPO_USERNAME}</username>
                  <password>${CLOUDREPO_PASSWORD}</password>
                </server>
              </servers>
              <profiles>
                <profile>
                  <id>cloudrepo</id>
                  <repositories>
                    <repository>
                      <id>cloudrepo</id>
                      <url>${CLOUDREPO_MAVEN_URL}</url>
                    </repository>
                  </repositories>
                </profile>
              </profiles>
              <activeProfiles>
                <activeProfile>cloudrepo</activeProfile>
              </activeProfiles>
            </settings>
            EOF
      - run:
          name: Run Tests
          command: mvn clean test
      - store_test_results:
          path: target/surefire-reports

Fan-out/Fan-in Patterns

Parallel execution with synchronized deployment:

.circleci/config.yml - Fan-out/Fan-in
version: 2.1

workflows:
  fan-out-in:
    jobs:
      - lint
      - unit-tests
      - integration-tests
      - security-scan
      - build:
          requires:
            - lint
            - unit-tests
      - deploy-staging:
          requires:
            - build
            - integration-tests
            - security-scan
          context:
            - cloudrepo-staging
      - smoke-tests:
          requires:
            - deploy-staging
      - deploy-production:
          requires:
            - smoke-tests
          context:
            - cloudrepo-production
          filters:
            branches:
              only: main

jobs:
  lint:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run: |
          pip install flake8 black mypy
          flake8 .
          black --check .
          mypy .

  unit-tests:
    docker:
      - image: cimg/python:3.11
    parallelism: 4
    steps:
      - checkout
      - run:
          name: Split and Run Tests
          command: |
            TEST_FILES=$(circleci tests glob "tests/unit/test_*.py" | circleci tests split --split-by=timings)
            pytest $TEST_FILES --junitxml=test-results/junit.xml
      - store_test_results:
          path: test-results

  integration-tests:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Start Test Services
          background: true
          command: docker-compose up -d
      - run:
          name: Wait for Services
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Run Integration Tests
          command: pytest tests/integration/ --junitxml=test-results/junit.xml
      - store_test_results:
          path: test-results

  security-scan:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Security Scanning
          command: |
            pip install safety bandit
            safety check
            bandit -r . -f json -o bandit-report.json
      - store_artifacts:
          path: bandit-report.json

  build:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Build Package
          command: |
            pip install build
            python -m build
      - persist_to_workspace:
          root: .
          paths:
            - dist/*

  deploy-staging:
    docker:
      - image: cimg/python:3.11
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Deploy to CloudRepo Staging
          command: |
            pip install twine
            twine upload \
              --repository-url ${CLOUDREPO_STAGING_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*

  smoke-tests:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Install from CloudRepo Staging
          command: |
            pip install --index-url https://${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}@${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/staging/simple mypackage
      - run:
          name: Run Smoke Tests
          command: python tests/smoke/test_deployment.py

  deploy-production:
    docker:
      - image: cimg/python:3.11
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Deploy to CloudRepo Production
          command: |
            pip install twine
            twine upload \
              --repository-url ${CLOUDREPO_PRODUCTION_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*

Conditional Deployment

Deploy based on branch, tag, or custom conditions:

.circleci/config.yml - Conditional Deployment
version: 2.1

parameters:
  deploy-to-production:
    type: boolean
    default: false

workflows:
  conditional-deploy:
    when:
      not: << pipeline.parameters.deploy-to-production >>
    jobs:
      - test
      - deploy-dev:
          requires:
            - test
          context:
            - cloudrepo-dev
          filters:
            branches:
              only: /feature-.*/

  production-deploy:
    when: << pipeline.parameters.deploy-to-production >>
    jobs:
      - test
      - hold:
          type: approval
      - deploy-production:
          requires:
            - hold
          context:
            - cloudrepo-production

  tag-based-deploy:
    jobs:
      - test:
          filters:
            tags:
              only: /^v.*/
            branches:
              ignore: /.*/
      - deploy-by-tag:
          requires:
            - test
          context:
            - cloudrepo-credentials
          filters:
            tags:
              only: /^v.*/
            branches:
              ignore: /.*/

jobs:
  deploy-by-tag:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Determine Deployment Target
          command: |
            if [[ "${CIRCLE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
              echo "Production release detected"
              export DEPLOY_URL=${CLOUDREPO_PRODUCTION_URL}
            elif [[ "${CIRCLE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+-rc ]]; then
              echo "Release candidate detected"
              export DEPLOY_URL=${CLOUDREPO_STAGING_URL}
            else
              echo "Pre-release detected"
              export DEPLOY_URL=${CLOUDREPO_DEV_URL}
            fi
            echo "export DEPLOY_URL=$DEPLOY_URL" >> $BASH_ENV
      - run:
          name: Build and Deploy
          command: |
            pip install build twine
            python -m build
            twine upload \
              --repository-url ${DEPLOY_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              dist/*

Scheduled Builds

Configure nightly builds and dependency updates:

.circleci/config.yml - Scheduled Builds
version: 2.1

workflows:
  nightly:
    triggers:
      - schedule:
          cron: "0 2 * * *"
          filters:
            branches:
              only:
                - main
    jobs:
      - dependency-update
      - security-audit
      - integration-test-full:
          context:
            - cloudrepo-credentials

  weekly-cleanup:
    triggers:
      - schedule:
          cron: "0 3 * * 0"
          filters:
            branches:
              only:
                - main
    jobs:
      - cleanup-old-artifacts:
          context:
            - cloudrepo-credentials

jobs:
  dependency-update:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Update Dependencies
          command: |
            pip install pip-upgrader
            pip-upgrade requirements.txt
      - run:
          name: Test with Updated Dependencies
          command: |
            pip install -r requirements.txt
            pytest tests/
      - run:
          name: Create PR if Changes
          command: |
            if [ -n "$(git status --porcelain)" ]; then
              git config user.email "ci@example.com"
              git config user.name "CircleCI"
              git checkout -b dependency-update-${CIRCLE_SHA1:0:7}
              git add requirements.txt
              git commit -m "chore: update dependencies"
              git push origin dependency-update-${CIRCLE_SHA1:0:7}
              # Create PR using GitHub CLI or API
            fi

  security-audit:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Security Audit
          command: |
            pip install safety pip-audit
            safety check --json > safety-report.json
            pip-audit --desc > audit-report.txt
      - store_artifacts:
          path: safety-report.json
      - store_artifacts:
          path: audit-report.txt
      - run:
          name: Notify if Vulnerabilities Found
          command: |
            if grep -q "vulnerability" audit-report.txt; then
              # Send notification via Slack, email, etc.
              echo "Vulnerabilities detected!"
            fi

  cleanup-old-artifacts:
    docker:
      - image: cimg/python:3.11
    steps:
      - checkout
      - run:
          name: Install CloudRepo CLI
          command: pip install cloudrepo-cli
      - run:
          name: Cleanup Old Artifacts
          command: |
            # Example: Remove artifacts older than 30 days from snapshot repository
            cloudrepo cleanup \
              --url ${CLOUDREPO_RAW_URL} \
              --username ${CLOUDREPO_USERNAME} \
              --password ${CLOUDREPO_PASSWORD} \
              --older-than 30d \
              --dry-run false

Approval Workflows

Implement manual approval gates for production deployments:

.circleci/config.yml - Approval Workflows
version: 2.1

workflows:
  deployment-with-approvals:
    jobs:
      - build-and-test:
          context:
            - cloudrepo-credentials
      - deploy-dev:
          requires:
            - build-and-test
          context:
            - cloudrepo-dev
      - request-staging-approval:
          type: approval
          requires:
            - deploy-dev
      - deploy-staging:
          requires:
            - request-staging-approval
          context:
            - cloudrepo-staging
      - staging-tests:
          requires:
            - deploy-staging
      - request-production-approval:
          type: approval
          requires:
            - staging-tests
      - deploy-production:
          requires:
            - request-production-approval
          context:
            - cloudrepo-production

jobs:
  build-and-test:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - run:
          name: Build and Test
          command: |
            ./gradlew clean build test
      - persist_to_workspace:
          root: .
          paths:
            - build/libs/*.jar

  deploy-dev:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Deploy to Dev
          command: |
            ./gradlew publish \
              -PrepoUrl=${CLOUDREPO_DEV_URL} \
              -PrepoUsername=${CLOUDREPO_USERNAME} \
              -PrepoPassword=${CLOUDREPO_PASSWORD}

  deploy-staging:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Deploy to Staging
          command: |
            ./gradlew publish \
              -PrepoUrl=${CLOUDREPO_STAGING_URL} \
              -PrepoUsername=${CLOUDREPO_USERNAME} \
              -PrepoPassword=${CLOUDREPO_PASSWORD}

  staging-tests:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - run:
          name: Run E2E Tests Against Staging
          command: |
            ./gradlew e2eTest \
              -PtargetEnv=staging \
              -PrepoUrl=${CLOUDREPO_STAGING_URL}

  deploy-production:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Deploy to Production
          command: |
            ./gradlew publish \
              -PrepoUrl=${CLOUDREPO_PRODUCTION_URL} \
              -PrepoUsername=${CLOUDREPO_USERNAME} \
              -PrepoPassword=${CLOUDREPO_PASSWORD}
      - run:
          name: Tag Release
          command: |
            git tag -a "release-${CIRCLE_SHA1:0:7}" -m "Production release"
            git push origin "release-${CIRCLE_SHA1:0:7}"

Performance Optimization

Dependency Caching

Optimize build times with intelligent caching strategies:

.circleci/config.yml - Advanced Caching
version: 2.1

jobs:
  optimized-build:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - restore_cache:
          keys:
            # Most specific cache key
            - v3-maven-{{ arch }}-{{ checksum "pom.xml" }}-{{ checksum "pom-parent.xml" }}
            # Fallback to less specific cache
            - v3-maven-{{ arch }}-{{ checksum "pom.xml" }}
            # Fallback to architecture specific cache
            - v3-maven-{{ arch }}-
            # Final fallback
            - v3-maven-
      - restore_cache:
          keys:
            - v2-gradle-{{ checksum "build.gradle" }}-{{ checksum "gradle.properties" }}
            - v2-gradle-{{ checksum "build.gradle" }}
            - v2-gradle-
      - run:
          name: Download Dependencies
          command: |
            # Maven dependencies
            mvn dependency:go-offline -DexcludeTransitive=false
            # Gradle dependencies
            ./gradlew dependencies
      - save_cache:
          key: v3-maven-{{ arch }}-{{ checksum "pom.xml" }}-{{ checksum "pom-parent.xml" }}
          paths:
            - ~/.m2/repository
      - save_cache:
          key: v2-gradle-{{ checksum "build.gradle" }}-{{ checksum "gradle.properties" }}
          paths:
            - ~/.gradle/caches
            - ~/.gradle/wrapper

Parallel Test Execution

Maximize test throughput with parallelism:

.circleci/config.yml - Parallel Testing
version: 2.1

jobs:
  parallel-tests:
    parallelism: 6
    docker:
      - image: cimg/openjdk:17.0
    resource_class: large
    steps:
      - checkout
      - restore_cache:
          keys:
            - v2-test-{{ checksum "pom.xml" }}
      - run:
          name: Configure Maven for CloudRepo
          command: |
            mkdir -p ~/.m2
            cat > ~/.m2/settings.xml <<EOF
            <settings>
              <servers>
                <server>
                  <id>cloudrepo</id>
                  <username>${CLOUDREPO_USERNAME}</username>
                  <password>${CLOUDREPO_PASSWORD}</password>
                </server>
              </servers>
              <profiles>
                <profile>
                  <id>cloudrepo</id>
                  <repositories>
                    <repository>
                      <id>cloudrepo</id>
                      <url>${CLOUDREPO_MAVEN_URL}</url>
                    </repository>
                  </repositories>
                </profile>
              </profiles>
              <activeProfiles>
                <activeProfile>cloudrepo</activeProfile>
              </activeProfiles>
            </settings>
            EOF
      - run:
          name: Split Tests
          command: |
            # Get list of test classes
            TEST_CLASSES=$(find src/test -name "*Test.java" | sed 's/src\/test\/java\///g' | sed 's/\.java//g' | sed 's/\//\./g')
            # Split tests across parallel containers
            echo "$TEST_CLASSES" | circleci tests split --split-by=timings > /tmp/tests-to-run
      - run:
          name: Run Tests
          command: |
            mvn test -Dtest=$(cat /tmp/tests-to-run | paste -sd "," -)
      - save_cache:
          key: v2-test-{{ checksum "pom.xml" }}
          paths:
            - ~/.m2/repository
      - store_test_results:
          path: target/surefire-reports
      - store_artifacts:
          path: target/site/jacoco

Docker Layer Caching

Optimize Docker builds with layer caching:

.circleci/config.yml - Docker Layer Cache
version: 2.1

jobs:
  docker-with-dlc:
    machine:
      image: ubuntu-2204:2024.01.1
      docker_layer_caching: true
    resource_class: xlarge
    steps:
      - checkout
      - run:
          name: Build with Layer Cache
          command: |
            # Enable BuildKit for better caching
            export DOCKER_BUILDKIT=1

            # Pull previous image for cache
            docker pull myregistry/myapp:latest || true

            # Build with cache from previous image
            docker build \
              --cache-from myregistry/myapp:latest \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              --build-arg CLOUDREPO_USERNAME=${CLOUDREPO_USERNAME} \
              --build-arg CLOUDREPO_PASSWORD=${CLOUDREPO_PASSWORD} \
              --build-arg CLOUDREPO_URL=${CLOUDREPO_MAVEN_URL} \
              --tag myregistry/myapp:${CIRCLE_SHA1:0:7} \
              --tag myregistry/myapp:latest \
              .

            # Push with cache metadata
            docker push myregistry/myapp:${CIRCLE_SHA1:0:7}
            docker push myregistry/myapp:latest

Resource Class Optimization

Choose appropriate resource classes for your workloads:

.circleci/config.yml - Resource Optimization
version: 2.1

executors:
  small-executor:
    docker:
      - image: cimg/base:stable
    resource_class: small

  medium-executor:
    docker:
      - image: cimg/openjdk:17.0
    resource_class: medium

  large-executor:
    docker:
      - image: cimg/openjdk:17.0
    resource_class: large

  xlarge-executor:
    docker:
      - image: cimg/openjdk:17.0
    resource_class: xlarge

  gpu-executor:
    machine:
      image: ubuntu-2204:2024.01.1
    resource_class: gpu.nvidia.medium

workflows:
  optimized-resources:
    jobs:
      - lint:
          executor: small-executor
      - unit-tests:
          executor: medium-executor
      - integration-tests:
          executor: large-executor
      - performance-tests:
          executor: xlarge-executor
      - ml-training:
          executor: gpu-executor

jobs:
  lint:
    executor: small-executor
    steps:
      - checkout
      - run: echo "Linting with minimal resources"

  unit-tests:
    executor: medium-executor
    steps:
      - checkout
      - run: mvn test

  integration-tests:
    executor: large-executor
    steps:
      - checkout
      - run: mvn verify

  performance-tests:
    executor: xlarge-executor
    steps:
      - checkout
      - run: mvn gatling:test

  ml-training:
    executor: gpu-executor
    steps:
      - checkout
      - run: python train_model.py

Security Best Practices

Secure Credential Storage

Implement defense-in-depth for credentials:

.circleci/config.yml - Secure Credentials
version: 2.1

workflows:
  secure-workflow:
    jobs:
      - security-check
      - deploy:
          requires:
            - security-check
          context:
            - cloudrepo-production
            - security-scanning
          filters:
            branches:
              only: main

jobs:
  security-check:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Check for Hardcoded Secrets
          command: |
            # Install secret scanning tools
            pip install detect-secrets truffleHog

            # Scan for secrets
            detect-secrets scan --baseline .secrets.baseline
            detect-secrets audit .secrets.baseline

            # Check git history
            truffleHog --regex --entropy=False .
      - run:
          name: Dependency Vulnerability Scan
          command: |
            # For Python
            pip install safety
            safety check

            # For Java
            mvn org.owasp:dependency-check-maven:check

  deploy:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Secure Deployment
          command: |
            # Never echo sensitive values
            if [ -z "$CLOUDREPO_PASSWORD" ]; then
              echo "Error: Credentials not configured"
              exit 1
            fi

            # Use credentials without exposing them
            echo "Deploying to CloudRepo..."
            # Deployment commands here

Restricted Contexts

Configure context restrictions for sensitive environments:

.circleci/config.yml - Context Restrictions
version: 2.1

# Production context restricted to main branch and specific users
workflows:
  restricted-deployment:
    jobs:
      - build:
          context:
            - cloudrepo-readonly
      - approval:
          type: approval
          requires:
            - build
          filters:
            branches:
              only: main
      - deploy-production:
          requires:
            - approval
          context:
            - cloudrepo-production
            - security-compliance
          filters:
            branches:
              only: main

jobs:
  build:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Build with Read-Only Access
          command: |
            # Use read-only credentials for dependencies
            export REPO_URL="https://${CLOUDREPO_RO_USER}:${CLOUDREPO_RO_PASS}@${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${REPO_ID}"
            # Build commands

  deploy-production:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Production Deployment
          command: |
            # Production deployment with write access
            # Only accessible after approval

Audit Logging

Implement comprehensive audit logging:

.circleci/config.yml - Audit Logging
version: 2.1

jobs:
  audited-deployment:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Initialize Audit Log
          command: |
            cat > audit.log <<EOF
            Deployment Audit Log
            ====================
            Build ID: ${CIRCLE_BUILD_NUM}
            Workflow: ${CIRCLE_WORKFLOW_ID}
            Branch: ${CIRCLE_BRANCH}
            User: ${CIRCLE_USERNAME}
            Timestamp: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
            SHA: ${CIRCLE_SHA1}
            EOF
      - run:
          name: Deploy with Audit Trail
          command: |
            echo "Starting deployment..." >> audit.log

            # Your deployment commands

            echo "Deployment completed successfully" >> audit.log
            echo "Artifacts published to CloudRepo" >> audit.log
            echo "Repository: ${CLOUDREPO_MAVEN_URL}" >> audit.log
      - run:
          name: Send Audit Log
          when: always
          command: |
            # Send to logging service
            curl -X POST https://logging.example.com/audit \
              -H "Content-Type: application/json" \
              -d @audit.log
      - store_artifacts:
          path: audit.log

Network Security

Implement network security best practices:

.circleci/config.yml - Network Security
version: 2.1

jobs:
  secure-network:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Configure Proxy Settings
          command: |
            # If using corporate proxy
            export HTTP_PROXY=${CORP_PROXY_URL}
            export HTTPS_PROXY=${CORP_PROXY_URL}
            export NO_PROXY=localhost,127.0.0.1,.mycloudrepo.io

            # Configure tools to use proxy
            npm config set proxy ${HTTP_PROXY}
            npm config set https-proxy ${HTTPS_PROXY}

            git config --global http.proxy ${HTTP_PROXY}
            git config --global https.proxy ${HTTPS_PROXY}
      - run:
          name: Verify SSL Certificates
          command: |
            # Test SSL connection to CloudRepo
            openssl s_client -connect ${CLOUDREPO_ORG_ID}.mycloudrepo.io:443 -showcerts

            # Verify certificate chain
            curl -v https://${CLOUDREPO_ORG_ID}.mycloudrepo.io 2>&1 | grep "SSL certificate verify"
      - run:
          name: IP Allowlist Check
          command: |
            # Get CircleCI IP for allowlisting
            CIRCLE_IP=$(curl -s https://checkip.amazonaws.com)
            echo "CircleCI IP: ${CIRCLE_IP}"

            # Verify IP is allowlisted in CloudRepo
            # (Configure in CloudRepo security settings)

Troubleshooting

Common Authentication Issues

Issue: 401 Unauthorized Error

Debugging Authentication
version: 2.1

jobs:
  debug-auth:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Debug Authentication
          command: |
            # Check if credentials are set
            if [ -z "$CLOUDREPO_USERNAME" ]; then
              echo "ERROR: CLOUDREPO_USERNAME not set"
              echo "Please check your context configuration"
              exit 1
            fi

            if [ -z "$CLOUDREPO_PASSWORD" ]; then
              echo "ERROR: CLOUDREPO_PASSWORD not set"
              exit 1
            fi

            # Test authentication
            curl -u "${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}" \
                 -I https://${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${REPO_ID}/ \
                 -w "HTTP Status: %{http_code}\n"

            # Check for special characters in password
            if echo "$CLOUDREPO_PASSWORD" | grep -q '[&$!@#%]'; then
              echo "Password contains special characters - ensure proper escaping"
            fi

Issue: Context Not Available

# Check context configuration
circleci context list
circleci context show cloudrepo-credentials

# Verify context restrictions
circleci context show-restrictions cloudrepo-credentials

Cache Invalidation

Force cache refresh when needed:

.circleci/config.yml - Cache Invalidation
version: 2.1

parameters:
  cache-version:
    type: string
    default: "v1"

jobs:
  build-with-cache-control:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - restore_cache:
          keys:
            - << pipeline.parameters.cache-version >>-deps-{{ checksum "pom.xml" }}
            - << pipeline.parameters.cache-version >>-deps-
      - run:
          name: Clear Cache if Needed
          command: |
            # Clear Maven cache
            if [ "${CLEAR_CACHE}" = "true" ]; then
              rm -rf ~/.m2/repository
              echo "Cache cleared"
            fi
      - run:
          name: Download Dependencies
          command: mvn dependency:go-offline
      - save_cache:
          key: << pipeline.parameters.cache-version >>-deps-{{ checksum "pom.xml" }}
          paths:
            - ~/.m2/repository

# Trigger with cache clear:
# circleci pipeline trigger --params '{"cache-version": "v2"}'

Network Timeouts

Handle and debug network timeout issues:

.circleci/config.yml - Timeout Handling
version: 2.1

jobs:
  handle-timeouts:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Configure Timeout Settings
          command: |
            # Maven timeout settings
            export MAVEN_OPTS="-Dhttp.socketTimeout=120000 -Dhttp.connectionTimeout=120000"

            # Gradle timeout settings
            echo "systemProp.http.socketTimeout=120000" >> gradle.properties
            echo "systemProp.http.connectionTimeout=120000" >> gradle.properties

            # NPM timeout settings
            npm config set fetch-timeout 120000
            npm config set fetch-retry-maxtimeout 120000

            # Git timeout settings
            git config --global http.postBuffer 524288000
            git config --global http.timeout 120
      - run:
          name: Retry on Timeout
          command: |
            # Retry function
            retry() {
              local n=1
              local max=5
              local delay=15
              while true; do
                "$@" && break || {
                  if [[ $n -lt $max ]]; then
                    ((n++))
                    echo "Command failed. Attempt $n/$max:"
                    sleep $delay
                  else
                    echo "The command has failed after $n attempts."
                    return 1
                  fi
                }
              done
            }

            # Use retry for network operations
            retry curl -f -u "${CLOUDREPO_USERNAME}:${CLOUDREPO_PASSWORD}" \
              https://${CLOUDREPO_ORG_ID}.mycloudrepo.io/repositories/${REPO_ID}/

Debugging Failed Builds

Comprehensive debugging strategies:

.circleci/config.yml - Debug Mode
version: 2.1

parameters:
  debug-mode:
    type: boolean
    default: false

jobs:
  debug-build:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - when:
          condition: << pipeline.parameters.debug-mode >>
          steps:
            - run:
                name: Enable Debug Output
                command: |
                  # Maven debug
                  export MAVEN_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"

                  # Gradle debug
                  export GRADLE_OPTS="-Dorg.gradle.debug=true"

                  # Shell debug
                  set -x
      - run:
          name: Environment Diagnostics
          command: |
            echo "=== System Information ==="
            uname -a
            echo ""
            echo "=== Java Version ==="
            java -version
            echo ""
            echo "=== Maven Version ==="
            mvn -version
            echo ""
            echo "=== Environment Variables ==="
            env | grep -E "CIRCLE|CLOUDREPO" | sed 's/PASSWORD=.*/PASSWORD=***/'
            echo ""
            echo "=== Network Diagnostics ==="
            # Test DNS resolution
            nslookup ${CLOUDREPO_ORG_ID}.mycloudrepo.io
            # Test connectivity
            ping -c 3 ${CLOUDREPO_ORG_ID}.mycloudrepo.io || true
            # Test HTTPS
            curl -I https://${CLOUDREPO_ORG_ID}.mycloudrepo.io
      - run:
          name: SSH Debug Session
          command: |
            # Enable SSH debugging for 30 minutes
            # Connect with: ssh -p 64535 34.211.9.15
            if [ "<< pipeline.parameters.debug-mode >>" = "true" ]; then
              echo "SSH debug session enabled for 30 minutes"
              sleep 1800
            fi

Note

For SSH debugging, you can rerun a job with SSH enabled through the CircleCI UI:

  1. Navigate to the failed job

  2. Click “Rerun job with SSH”

  3. Connect using the provided SSH command

  4. Debug interactively in the build environment

Warning

Common pitfalls to avoid:

  • Never commit credentials to your repository

  • Always use contexts for production credentials

  • Rotate credentials regularly

  • Monitor CloudRepo access logs for suspicious activity

  • Use read-only credentials where possible

  • Enable 2FA on CloudRepo accounts

See also