Poetry Repositories

This section provides comprehensive documentation for using CloudRepo with Poetry, the modern Python packaging and dependency management tool.

Introduction to Poetry

What is Poetry?

Poetry has emerged as the modern standard for Python dependency management and packaging, offering a more streamlined and deterministic approach compared to traditional tools like pip and setuptools. Poetry combines the best features of package managers from other ecosystems (like npm, cargo, and bundler) while addressing many of the pain points that Python developers have historically faced.

Key advantages of Poetry include:

Deterministic dependencies: Poetry uses a lock file (poetry.lock) to ensure everyone on your team gets the exact same dependencies
Simplified project management: One tool handles dependency resolution, virtual environment management, building, and publishing
Modern standards: Built on PEP 517/518, using pyproject.toml instead of multiple configuration files
Better dependency resolution: Sophisticated solver that handles complex dependency graphs more reliably
Intuitive workflow: Commands that feel natural and consistent across different operations

Why Use Poetry with CloudRepo?

CloudRepo provides a perfect complement to Poetry’s modern workflow by offering:

Private package hosting: Securely share internal Python packages within your organization
Cost-effective alternative: Save up to 90% compared to JFrog Artifactory or other enterprise solutions
No egress fees: Unlike cloud providers, CloudRepo doesn’t charge for package downloads
Simple authentication: Straightforward configuration that works seamlessly with Poetry
Enterprise features: Support included with all plans, no expensive service contracts

Prerequisites

Before configuring Poetry to work with CloudRepo, you’ll need:

CloudRepo Account and Repository
- An active CloudRepo organization account
- A Python repository created in the CloudRepo Admin Portal
- A repository user with appropriate permissions
Note

For security purposes, your admin user cannot access repositories directly. You must create a dedicated repository user in the CloudRepo Admin Portal.

If you haven’t completed these steps, please refer to:
- Creating a Repository
- Creating Users
Poetry Installation

Install Poetry using the official installer:
Installing Poetry (Recommended Method)
```
curl -sSL https://install.python-poetry.org | python3 -
```
Or via pip (though the installer is preferred):
Installing Poetry via pip
```
pip install poetry
```
Python Environment
- Python 3.8 or higher
- A project initialized with Poetry (poetry init or existing pyproject.toml)

Configuration

Basic Repository Configuration

Configure your pyproject.toml to use CloudRepo as a package source:

pyproject.toml - Basic CloudRepo Configuration

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "supplemental"  # or "primary" if you want CloudRepo to be the main source

The priority setting determines how Poetry uses your CloudRepo repository:

primary: Check CloudRepo first for all packages
supplemental: Only check CloudRepo for packages not found on PyPI
explicit: Only use CloudRepo when explicitly specified in dependencies

Authentication Configuration

Poetry provides multiple methods for authentication. Choose the one that best fits your workflow:

Method 1: Using Poetry Config (Recommended for Development)

Configure Authentication via Poetry CLI

# Set credentials for your CloudRepo repository
poetry config http-basic.cloudrepo [repository-user-email] [repository-user-password]

# Verify the configuration
poetry config --list | grep cloudrepo

Method 2: Environment Variables (Recommended for CI/CD)

Set Authentication via Environment Variables

export POETRY_HTTP_BASIC_CLOUDREPO_USERNAME="[repository-user-email]"
export POETRY_HTTP_BASIC_CLOUDREPO_PASSWORD="[repository-user-password]"

Method 3: Keyring Integration (Most Secure for Development)

Poetry can integrate with your system’s keyring for secure credential storage:

Using System Keyring

# Install keyring support
pip install keyring

# Store credentials in keyring
keyring set poetry-repository-cloudrepo [repository-user-email]
# You'll be prompted for the password

# Poetry will automatically use keyring when available

Multiple Repository Configuration

For projects that need packages from both PyPI and CloudRepo:

pyproject.toml - Multiple Sources Configuration

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "supplemental"

[[tool.poetry.source]]
name = "cloudrepo-ml"  # Example: Separate repo for ML packages
url = "https://[organization-id].mycloudrepo.io/repositories/[ml-repository-id]/simple"
priority = "explicit"  # Only use when explicitly referenced

Publishing Packages

Building Your Package

Poetry simplifies the build process with a single command:

Building Python Packages with Poetry

# Build both wheel and source distribution
poetry build

# Build only wheel
poetry build -f wheel

# Build only source distribution
poetry build -f sdist

This creates artifacts in the dist/ directory ready for publishing.

Publishing to CloudRepo

Configure CloudRepo as a publishing repository and upload your package:

Publishing to CloudRepo

# Configure the repository for publishing (one-time setup)
poetry config repositories.cloudrepo https://[organization-id].mycloudrepo.io/repositories/[repository-id]

# Configure authentication (if not already done)
poetry config http-basic.cloudrepo [repository-user-email] [repository-user-password]

# Publish your package
poetry publish --repository cloudrepo

# Build and publish in one command
poetry publish --build --repository cloudrepo

Version Management

Poetry makes version management straightforward:

Managing Package Versions

# Bump patch version (1.0.0 -> 1.0.1)
poetry version patch

# Bump minor version (1.0.1 -> 1.1.0)
poetry version minor

# Bump major version (1.1.0 -> 2.0.0)
poetry version major

# Set specific version
poetry version 2.3.4

# Use pre-release versions
poetry version prepatch  # 1.0.0 -> 1.0.1-alpha.0
poetry version prerelease  # 1.0.1-alpha.0 -> 1.0.1-alpha.1

Pre-release Versions

Publishing pre-release versions for testing:

pyproject.toml - Pre-release Version

[tool.poetry]
name = "my-package"
version = "1.0.0-alpha.1"  # Alpha release
# version = "1.0.0-beta.1"   # Beta release
# version = "1.0.0-rc.1"     # Release candidate

Publishing Pre-release Versions

# Automatic pre-release versioning
poetry version prerelease  # Increments pre-release number
poetry version prepatch    # Creates new patch pre-release
poetry version preminor    # Creates new minor pre-release
poetry version premajor    # Creates new major pre-release

# Publish the pre-release
poetry publish --repository cloudrepo

Platform-Specific Wheels

When building platform-specific wheels (e.g., with C extensions):

Building Platform-Specific Wheels

# Install build dependencies
poetry install --with dev

# Build wheels for current platform
poetry build -f wheel

# For multiple platforms, use cibuildwheel
pip install cibuildwheel
cibuildwheel --platform linux

# Upload all wheels to CloudRepo
poetry publish --repository cloudrepo

For complex multi-platform builds, configure cibuildwheel in your pyproject.toml:

pyproject.toml - Multi-platform Build Configuration

[tool.cibuildwheel]
build = ["cp38-*", "cp39-*", "cp310-*", "cp311-*", "cp312-*"]
skip = ["*-win32", "*-manylinux_i686"]

[tool.cibuildwheel.linux]
manylinux-x86_64-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"

Consuming Dependencies

Adding CloudRepo Packages

Install private packages from CloudRepo:

Installing Private Packages

# Add a private package from CloudRepo
poetry add my-private-package

# Add with version constraints
poetry add my-private-package@^2.0.0

# Add from specific source (when using explicit priority)
poetry add my-ml-package --source cloudrepo-ml

# Add as development dependency
poetry add --group dev my-test-utils

Dependency Groups and Extras

Poetry supports organizing dependencies into groups:

pyproject.toml - Dependency Groups

[tool.poetry.dependencies]
python = "^3.8"
my-core-lib = {version = "^1.0.0", source = "cloudrepo"}

[tool.poetry.group.dev.dependencies]
my-test-framework = {version = "^2.0.0", source = "cloudrepo"}
pytest = "^7.0.0"

[tool.poetry.group.docs]
optional = true

[tool.poetry.group.docs.dependencies]
my-doc-theme = {version = "^1.0.0", source = "cloudrepo"}
sphinx = "^5.0.0"

[tool.poetry.extras]
ml = ["tensorflow", "my-ml-utils"]
web = ["fastapi", "my-web-framework"]

Install specific groups or extras:

Installing Dependency Groups and Extras

# Install with all non-optional groups
poetry install

# Install including optional docs group
poetry install --with docs

# Install without dev dependencies
poetry install --without dev

# Install with specific extras
poetry install --extras "ml web"

# Install all extras
poetry install --all-extras

Lock File Management

Poetry’s lock file ensures reproducible installs:

Managing poetry.lock

# Generate/update lock file without installing
poetry lock

# Update specific packages in lock file
poetry update requests my-private-lib

# Update all dependencies to latest compatible versions
poetry update

# Install exactly what's in the lock file
poetry install --no-root

# Export lock file to requirements.txt format
poetry export -f requirements.txt --output requirements.txt
poetry export -f requirements.txt --with-credentials --output requirements-with-auth.txt

Mixing Public and Private Dependencies

Configure Poetry to efficiently handle both public and private packages:

pyproject.toml - Mixed Dependencies

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"  # Check PyPI first for public packages

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "supplemental"  # Fallback to CloudRepo for private packages

[tool.poetry.dependencies]
python = "^3.8"
# Public packages (from PyPI)
requests = "^2.28.0"
pandas = "^2.0.0"

# Private packages (from CloudRepo) - will be found via supplemental source
my-private-utils = "^1.2.0"
company-auth-lib = "^3.0.0"

# Explicitly specify source if needed
special-internal-tool = {version = "^2.0.0", source = "cloudrepo"}

Advanced Topics

Monorepo Support with Poetry

Poetry works well with monorepo structures. Here’s how to organize multiple packages:

Monorepo Directory Structure

my-monorepo/
├── pyproject.toml          # Root workspace configuration
├── packages/
│   ├── core-lib/
│   │   ├── pyproject.toml
│   │   └── src/
│   ├── web-api/
│   │   ├── pyproject.toml
│   │   └── src/
│   └── cli-tool/
│       ├── pyproject.toml
│       └── src/
└── poetry.lock

Root workspace configuration:

Root pyproject.toml - Workspace Configuration

[tool.poetry]
name = "my-monorepo"
version = "1.0.0"
description = "Monorepo workspace"
authors = ["Your Team <team@example.com>"]
packages = [
    {include = "packages/core-lib"},
    {include = "packages/web-api"},
    {include = "packages/cli-tool"}
]

[tool.poetry.dependencies]
python = "^3.8"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Individual package configuration:

packages/core-lib/pyproject.toml

[tool.poetry]
name = "company-core-lib"
version = "2.1.0"
description = "Shared core library"

[tool.poetry.dependencies]
python = "^3.8"

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "supplemental"

Publishing from monorepo:

Publishing Packages from Monorepo

# Navigate to specific package
cd packages/core-lib

# Build and publish
poetry build
poetry publish --repository cloudrepo

# Or use a script to publish all packages
for pkg in packages/*; do
    cd "$pkg"
    poetry build
    poetry publish --repository cloudrepo
    cd ../..
done

Plugin Development and Distribution

Create and distribute Poetry plugins through CloudRepo:

pyproject.toml - Poetry Plugin

[tool.poetry]
name = "poetry-cloudrepo-plugin"
version = "1.0.0"
description = "Custom Poetry plugin for CloudRepo integration"

[tool.poetry.dependencies]
python = "^3.8"
poetry = "^1.5.0"

[tool.poetry.plugins."poetry.application.plugin"]
cloudrepo = "poetry_cloudrepo_plugin:CloudRepoPlugin"

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"

Plugin Implementation Example

from poetry.plugins.application_plugin import ApplicationPlugin
from cleo.commands.command import Command

class CloudRepoCommand(Command):
    name = "cloudrepo:sync"
    description = "Sync packages with CloudRepo"

    def handle(self) -> int:
        self.line("Syncing with CloudRepo...")
        # Implementation here
        return 0

class CloudRepoPlugin(ApplicationPlugin):
    def activate(self, application):
        application.add(CloudRepoCommand())

Dependency Caching

Optimize Poetry’s performance with CloudRepo:

Configuring Poetry Caching

# Configure Poetry cache directory
poetry config cache-dir /path/to/cache

# Use CloudRepo as primary source to reduce PyPI lookups
poetry config repositories.cloudrepo https://[organization-id].mycloudrepo.io/repositories/[repository-id]
poetry config repositories.cloudrepo --priority primary

# Enable parallel downloads
poetry config installer.parallel true

# For CI/CD, pre-download dependencies
poetry export -f requirements.txt | pip download -r /dev/stdin -d ./cache

Using CloudRepo as Primary Index

Configure CloudRepo as your primary package index:

pyproject.toml - CloudRepo as Primary Index

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "primary"

[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"  # Fallback to PyPI only when needed

This configuration ensures:

All packages are first searched in CloudRepo
Only packages not found in CloudRepo are fetched from PyPI
Reduces external dependencies and improves security
Faster installs for internal packages

CI/CD Integration

GitHub Actions

.github/workflows/python-package.yml

name: Python Package CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  release:
    types: [published]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install Poetry
      run: |
        curl -sSL https://install.python-poetry.org | python3 -
        echo "$HOME/.local/bin" >> $GITHUB_PATH

    - name: Configure Poetry
      env:
        POETRY_HTTP_BASIC_CLOUDREPO_USERNAME: ${{ secrets.CLOUDREPO_USERNAME }}
        POETRY_HTTP_BASIC_CLOUDREPO_PASSWORD: ${{ secrets.CLOUDREPO_PASSWORD }}
      run: |
        poetry config virtualenvs.create true
        poetry config virtualenvs.in-project true

    - name: Cache Dependencies
      uses: actions/cache@v3
      with:
        path: .venv
        key: venv-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}

    - name: Install Dependencies
      run: poetry install --with dev

    - name: Run Tests
      run: poetry run pytest

  publish:
    needs: test
    runs-on: ubuntu-latest
    if: github.event_name == 'release'

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Poetry
      run: |
        curl -sSL https://install.python-poetry.org | python3 -
        echo "$HOME/.local/bin" >> $GITHUB_PATH

    - name: Configure CloudRepo
      run: |
        poetry config repositories.cloudrepo \
          https://${{ secrets.CLOUDREPO_ORG }}.mycloudrepo.io/repositories/${{ secrets.CLOUDREPO_REPO }}
        poetry config http-basic.cloudrepo \
          ${{ secrets.CLOUDREPO_USERNAME }} ${{ secrets.CLOUDREPO_PASSWORD }}

    - name: Build and Publish
      run: |
        poetry version $(echo "${{ github.ref }}" | sed 's/refs\/tags\/v//')
        poetry build
        poetry publish --repository cloudrepo

GitLab CI

.gitlab-ci.yml

stages:
  - test
  - build
  - publish

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  POETRY_HOME: "$CI_PROJECT_DIR/.poetry"
  POETRY_VIRTUALENVS_IN_PROJECT: "true"
  POETRY_HTTP_BASIC_CLOUDREPO_USERNAME: "${CLOUDREPO_USERNAME}"
  POETRY_HTTP_BASIC_CLOUDREPO_PASSWORD: "${CLOUDREPO_PASSWORD}"

before_script:
  - apt-get update -y
  - apt-get install -y curl
  - curl -sSL https://install.python-poetry.org | python3 -
  - export PATH="$POETRY_HOME/bin:$PATH"
  - poetry config repositories.cloudrepo ${CLOUDREPO_URL}

cache:
  paths:
    - .cache/pip
    - .venv

test:
  stage: test
  image: python:3.11
  script:
    - poetry install --with dev
    - poetry run pytest --cov=my_package tests/
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

build:
  stage: build
  image: python:3.11
  script:
    - poetry build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

publish:
  stage: publish
  image: python:3.11
  only:
    - tags
  dependencies:
    - build
  script:
    - poetry publish --repository cloudrepo

Troubleshooting

Common Issues and Solutions

Authentication Failures

Debugging Authentication Issues

# Check current configuration
poetry config --list | grep cloudrepo

# Clear cached credentials
poetry config --unset http-basic.cloudrepo

# Re-set credentials
poetry config http-basic.cloudrepo [email] [password]

# Test authentication
poetry search my-package --repository cloudrepo

SSL Certificate Issues

Resolving SSL Issues

# For development only - not recommended for production
poetry config certificates.cloudrepo.cert false

# Better solution - add corporate CA certificate
poetry config certificates.cloudrepo.cert /path/to/ca-bundle.crt

Dependency Resolution Conflicts

Resolving Dependency Conflicts

# Clear Poetry cache
poetry cache clear pypi --all
poetry cache clear cloudrepo --all

# Update lock file with verbose output
poetry lock -vvv

# Show dependency tree
poetry show --tree

# Check why a package is included
poetry show --tree my-package

Slow Package Downloads

pyproject.toml - Performance Optimization

[[tool.poetry.source]]
name = "cloudrepo"
url = "https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple"
priority = "primary"  # Check CloudRepo first

[tool.poetry]
# Enable parallel installer
installer = { parallel = true }

Package Not Found Errors

Debugging Package Discovery

# List all configured sources
poetry source show

# Search for package in specific repository
poetry search my-package --repository cloudrepo

# Check if package exists in CloudRepo
curl -u [email]:[password] \
  https://[organization-id].mycloudrepo.io/repositories/[repository-id]/simple/my-package/

# Force refresh of package list
poetry cache clear cloudrepo --all
poetry lock --no-update

Version Mismatch Issues

Handling Version Conflicts

# Show all available versions
poetry show my-package --all

# Update to specific version
poetry add my-package@2.0.0

# Allow pre-releases
poetry add my-package --allow-prereleases

# Use version range
poetry add "my-package>=1.0,<2.0"

Best Practices

Always commit poetry.lock: This ensures reproducible builds across your team
Use semantic versioning: Follow SemVer for your package versions
Separate dev dependencies: Use dependency groups to keep production dependencies minimal
Configure CI/CD credentials securely: Use environment variables or secrets management
Regular dependency updates: Run poetry update periodically and test thoroughly
Document custom sources: Include CloudRepo configuration instructions in your README
Use primary/supplemental wisely: Configure source priorities based on your package distribution

Getting Help

If you encounter issues not covered in this documentation:

Check the Poetry documentation
Review CloudRepo’s Knowledge Base
Contact CloudRepo support at support@cloudrepo.io

CloudRepo support is included with all plans - we typically respond within a few business hours and are here to help you succeed with your Python package management.