# AlmaLinux Build System (ALBS) - Complete Installation & Configuration Guide **Target**: Production-grade ALBS deployment for rpm-devel **OS**: AlmaLinux 9 **Duration**: 1-2 weeks to full production **Architecture Support**: x86_64, aarch64, ppc64le **Complexity**: Medium (Docker-based, simpler than Koji) --- ## Table of Contents 1. [Architecture & Components](#architecture--components) 2. [Prerequisites & Planning](#prerequisites--planning) 3. [Infrastructure Preparation](#infrastructure-preparation) 4. [ALBS Web Server Setup](#albs-web-server-setup) 5. [ALBS Build Node Setup](#albs-build-node-setup) 6. [Repository Configuration](#repository-configuration) 7. [GitHub Integration](#github-integration) 8. [First Build & Testing](#first-build--testing) 9. [Production Operations](#production-operations) 10. [Troubleshooting](#troubleshooting) 11. [Performance Tuning](#performance-tuning) --- ## Architecture & Components ### ALBS System Architecture ``` ┌────────────────────────────────────────────────────────┐ │ ALBS Web Server │ │ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ │ │ │ PostgreSQL │ │ Redis Cache │ │ FastAPI │ │ │ │ │ │ │ │ REST API │ │ │ └──────────────┘ └──────────────┘ └─────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ Pulp (Artifact Storage & Repo Management) │ │ │ │ - Package storage │ │ │ │ - Repository metadata │ │ │ │ - Distributions & releases │ │ │ └──────────────────────────────────────────────────┘ │ └────────────┬───────────────────────────────────────────┘ │ ┌────────┴────────┐ │ │ ┌───▼──────────┐ ┌───▼──────────┐ │ Build Node │ │ Build Node │ │ x86_64 │ │ aarch64 │ │ (Docker) │ │ (Docker) │ │ │ │ │ │ Mock chroots │ │ Mock chroots │ │ - EL8 │ │ - EL8 │ │ - EL9 │ │ - EL9 │ │ - Fedora39 │ │ - Fedora39 │ └──────────────┘ └──────────────┘ │ │ └─────────┬─────────┘ │ ┌──────────▼──────────┐ │ Pulp Artifacts │ │ (RPMs, logs) │ └─────────────────────┘ ``` ### Components **ALBS Web Server** (docker-compose) - FastAPI REST API for build management - PostgreSQL database (build metadata, tasks, platforms) - Redis cache (performance, session data) - Pulp integration (artifact coordination) - GitHub OAuth authentication - Task queue management **ALBS Build Nodes** (docker-compose, one per architecture) - Receives builds from Web Server queue - Executes builds in Mock chroots - Uploads artifacts to Pulp - Reports build status back to Web Server **Pulp** (artifact storage) - Stores built RPMs - Manages repository metadata - Handles multiple distributions/architectures - Serves repositories to clients **Supporting Services** - PostgreSQL: Persistent data storage - Redis: Session cache, queue management - Docker: Container runtime for all services - Gitea Listener: GitHub webhook receiver - Git Cacher: Source code caching --- ## Prerequisites & Planning ### Hardware Requirements **ALBS Web Server**: - CPU: 4 cores (Intel/AMD) - RAM: 8GB minimum (16GB recommended) - Storage: 100GB SSD - Network: 1Gbps **Build Node x86_64**: - CPU: 8 cores - RAM: 16GB - Storage: 300GB SSD - Network: 1Gbps **Build Node aarch64**: - CPU: 8 cores ARM (AWS Graviton, Ampere, Raspberry Pi 5) - RAM: 16GB - Storage: 300GB SSD - Network: 1Gbps **Pulp/Artifact Storage**: - Can be on Web Server or separate - Storage: 1TB minimum (grows 10-50GB/month) - SSD recommended for performance **Total**: ~1.5TB storage, 24-40 cores, 40-48GB RAM ### Software Requirements **All Servers**: - AlmaLinux 9 (minimal) - Docker & Docker Compose - Python 3.9+ - Git - SSH access between servers **Web Server Only**: - PostgreSQL 13+ - Redis - Nginx (reverse proxy, optional) ### Network Planning **Static IPs Required**: - albs-web.yourdomain.local - albs-builder-x86.yourdomain.local - albs-builder-arm.yourdomain.local **DNS Setup**: ``` albs-web.yourdomain.local A 10.x.x.10 albs-builder-x86.yourdomain.local A 10.x.x.20 albs-builder-arm.yourdomain.local A 10.x.x.30 ``` **Firewall Openings**: - Web Server: 443 (HTTPS), 8080 (Pulp), 5432 (PostgreSQL, internal only) - Build Nodes: SSH (22), Pulp sync (internal) - Between servers: All traffic (or specific ports: 5432, 6379, 8080) ### Git Repository Planning ALBS expects specs in git repositories: ``` https://github.com/rpm-devel/cas/ - cas.spec (root of repo or in SPECS/ directory) - Optional: source tarball or patch files https://github.com/rpm-devel/dockloom/ - dockloom.spec https://github.com/rpm-devel/cas-downloader/ - cas-downloader.spec (And 60+ other package repos from rpm-devel org) ``` --- ## Infrastructure Preparation ### Step 1: Provision AlmaLinux 9 Servers ```bash # On each server (Web, Builder x86, Builder ARM): # Update system dnf update -y # Install base packages dnf groupinstall -y "Development Tools" dnf install -y \ git \ curl \ wget \ vim \ net-tools \ htop \ tmux \ docker \ docker-compose \ python3.9 \ python3-pip \ postgresql-client # Add current user to docker group (to avoid sudo) usermod -aG docker $(whoami) # Start Docker systemctl start docker systemctl enable docker # Verify Docker docker run hello-world ``` ### Step 2: Network Configuration ```bash # Set hostnames on each server hostnamectl set-hostname albs-web.yourdomain.local # On Web Server hostnamectl set-hostname albs-builder-x86.yourdomain.local # On x86 builder hostnamectl set-hostname albs-builder-arm.yourdomain.local # On ARM builder # Edit /etc/hosts on all servers sudo vi /etc/hosts # Add all three servers to each /etc/hosts: 10.x.x.10 albs-web.yourdomain.local albs-web 10.x.x.20 albs-builder-x86.yourdomain.local albs-builder-x86 10.x.x.30 albs-builder-arm.yourdomain.local albs-builder-arm # Test connectivity ping albs-web.yourdomain.local ping albs-builder-x86.yourdomain.local ping albs-builder-arm.yourdomain.local ``` ### Step 3: Shared Storage Setup (Optional) For centralized artifact storage (recommended for production): ```bash # Option 1: NFS Export (on storage server) # Install NFS server dnf install -y nfs-utils systemctl start nfs-server systemctl enable nfs-server # Create export directory mkdir -p /exports/pulp-data chmod 777 /exports/pulp-data # Edit /etc/exports echo "/exports/pulp-data 10.0.0.0/8(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports exportfs -ra # Option 2: S3-Compatible Storage (MinIO, AWS S3) # More scalable; Pulp has native S3 support # Configure in Pulp settings (later) # Option 3: On-Disk (each node has local storage) # Simplest; less redundancy ``` ### Step 4: SSH Key Setup ```bash # On Web Server, create SSH key for automation ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N "" # Copy to build nodes (for build automation scripts) ssh-copy-id -i ~/.ssh/id_rsa.pub albs-builder-x86 ssh-copy-id -i ~/.ssh/id_rsa.pub albs-builder-arm # Verify passwordless SSH ssh albs-builder-x86 "echo 'Connected to x86 builder'" ``` --- ## ALBS Web Server Setup ### Step 1: Clone ALBS Repositories ```bash # Create directory structure mkdir -p /opt/albs cd /opt/albs # Clone ALBS Web Server git clone https://github.com/AlmaLinux/albs-web-server.git cd albs-web-server # Checkout stable release git checkout $(git describe --tags --abbrev=0) cd .. # Clone ALBS Node (for reference/setup) git clone https://github.com/AlmaLinux/albs-node.git # Clone Gitea Listener (GitHub webhook handler) git clone https://github.com/AlmaLinux/gitea_listener.git # Clone Pulp (will be containerized) # (Pulp image will be pulled via Docker) # Directory structure ls -la /opt/albs/ # albs-web-server/ # albs-node/ # gitea_listener/ ``` ### Step 2: PostgreSQL Setup ```bash # Install PostgreSQL server dnf install -y postgresql-server postgresql-contrib # Initialize database sudo -u postgres /usr/bin/postgresql-setup initdb # Edit /var/lib/pgsql/data/postgresql.conf sudo vi /var/lib/pgsql/data/postgresql.conf # Key settings: listen_addresses = '*' shared_buffers = 2GB # 25% of RAM effective_cache_size = 6GB # 75% of RAM work_mem = 32MB max_connections = 200 # Edit /var/lib/pgsql/data/pg_hba.conf sudo vi /var/lib/pgsql/data/pg_hba.conf # Add after local lines (for Docker containers): host all all 172.17.0.0/16 md5 host all all 127.0.0.1/32 md5 host all all 10.0.0.0/8 md5 # Restart PostgreSQL sudo systemctl restart postgresql sudo systemctl enable postgresql # Create ALBS database and users sudo -u postgres psql << EOF CREATE USER albs WITH PASSWORD 'albs-db-password-here'; CREATE DATABASE albs OWNER albs; CREATE USER pulp WITH PASSWORD 'pulp-db-password-here'; CREATE DATABASE pulp_app OWNER pulp; CREATE DATABASE pulp_content OWNER pulp; GRANT ALL ON DATABASE albs TO albs; GRANT ALL ON DATABASE pulp_app TO pulp; GRANT ALL ON DATABASE pulp_content TO pulp; EOF # Verify sudo -u postgres psql -l | grep albs ``` ### Step 3: Redis Setup ```bash # Install Redis dnf install -y redis # Configure Redis sudo vi /etc/redis/redis.conf # Key settings: bind 0.0.0.0 port 6379 requirepass redis-password-here appendonly yes # Start Redis sudo systemctl start redis sudo systemctl enable redis # Verify connection redis-cli ping # Should return: PONG ``` ### Step 4: Docker Compose Configuration ```bash # Create docker-compose.yml for Web Server in /opt/albs/ cd /opt/albs cat > docker-compose.yml << 'EOF' version: '3.8' services: albs-web: image: almalinux/albs-web-server:latest container_name: albs-web restart: always ports: - "8080:8080" - "443:443" environment: DATABASE_URL: postgresql://albs:albs-db-password-here@host.docker.internal:5432/albs REDIS_URL: redis://:redis-password-here@host.docker.internal:6379/0 PULP_URL: http://pulp:80 PULP_DOMAIN_NAME: albs-web.yourdomain.local SECRET_KEY: your-secret-key-here-change-this GITHUB_CLIENT_ID: your-github-client-id GITHUB_CLIENT_SECRET: your-github-client-secret GITHUB_CALLBACK_URL: https://albs-web.yourdomain.local/auth/github/callback volumes: - ./albs-web-server:/app - /etc/pki/albs:/etc/pki/albs:ro depends_on: - pulp networks: - albs-network extra_hosts: - "host.docker.internal:host-gateway" pulp: image: docker.io/pulp/pulp:latest container_name: albs-pulp restart: always ports: - "8000:80" environment: PULP_SECRET_KEY: pulp-secret-key-here-change-this PULP_CONTENT_ORIGIN: http://albs-web.yourdomain.local/pulp/content PULP_ALLOWED_IMPORT_PATHS: "['/var/lib/pulp']" PULP_ALLOWED_EXPORT_PATHS: "['/var/lib/pulp']" volumes: - pulp-data:/var/lib/pulp - pulp-assets:/var/lib/pulp/assets environment: POSTGRES_DB: pulp_app POSTGRES_USER: pulp POSTGRES_PASSWORD: pulp-db-password-here POSTGRES_HOST_ENVIRONMENT: host.docker.internal POSTGRES_PORT: 5432 networks: - albs-network extra_hosts: - "host.docker.internal:host-gateway" volumes: pulp-data: pulp-assets: networks: albs-network: driver: bridge EOF # Pull images docker-compose pull # Start services docker-compose up -d # Wait for services to start sleep 30 # Verify docker-compose ps docker-compose logs albs-web ``` ### Step 5: Nginx Reverse Proxy (Optional but Recommended) ```bash # Install Nginx dnf install -y nginx # Create SSL certificates sudo mkdir -p /etc/pki/albs cd /etc/pki/albs # Self-signed certificate (replace with real cert in production) sudo openssl req -x509 -nodes -days 365 \ -newkey rsa:2048 \ -keyout /etc/pki/albs/albs-key.key \ -out /etc/pki/albs/albs-cert.crt \ -subj "/CN=albs-web.yourdomain.local" # Configure Nginx sudo vi /etc/nginx/conf.d/albs.conf cat > /etc/nginx/conf.d/albs.conf << 'EOF' upstream albs_backend { server localhost:8080; } upstream pulp_backend { server localhost:8000; } server { listen 80; server_name albs-web.yourdomain.local; return 301 https://$server_name$request_uri; } server { listen 443 ssl http2; server_name albs-web.yourdomain.local; ssl_certificate /etc/pki/albs/albs-cert.crt; ssl_certificate_key /etc/pki/albs/albs-key.key; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; client_max_body_size 1G; # ALBS Web API location / { proxy_pass http://albs_backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # Pulp location /pulp/ { proxy_pass http://pulp_backend/pulp/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } EOF # Restart Nginx sudo systemctl restart nginx sudo systemctl enable nginx # Test curl -k https://albs-web.yourdomain.local/ ``` ### Step 6: GitHub OAuth Setup Log into GitHub and configure OAuth application: ``` GitHub Settings → Developer settings → OAuth Apps → New OAuth App Application name: ALBS rpm-devel Homepage URL: https://albs-web.yourdomain.local Authorization callback URL: https://albs-web.yourdomain.local/auth/github/callback Copy: Client ID and Client Secret Update in docker-compose.yml: GITHUB_CLIENT_ID: GITHUB_CLIENT_SECRET: Restart: docker-compose restart albs-web ``` ### Step 7: Initialize Platforms ```bash # Access ALBS Web API to create platforms # First, login with GitHub OAuth at: # https://albs-web.yourdomain.local # Then use API to create build platforms: curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "AlmaLinux-8", "type": "rpm", "architectures": ["x86_64", "aarch64"] }' curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "AlmaLinux-9", "type": "rpm", "architectures": ["x86_64", "aarch64"] }' curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "Fedora-39", "type": "rpm", "architectures": ["x86_64", "aarch64"] }' # Verify platforms created curl https://albs-web.yourdomain.local/api/v1/platforms \ -H "Authorization: Bearer " ``` --- ## ALBS Build Node Setup ### Step 1: Build Node Preparation ```bash # On each builder (x86_64 and aarch64) # Create working directories mkdir -p /opt/albs mkdir -p /var/lib/albs/builds mkdir -p /var/lib/albs/artifacts # Set permissions chmod 755 /opt/albs chmod 755 /var/lib/albs # Install dependencies dnf install -y \ python3.9 \ python3-pip \ mock \ rpm-build \ git \ podman ``` ### Step 2: Clone ALBS Node ```bash cd /opt/albs git clone https://github.com/AlmaLinux/albs-node.git cd albs-node git checkout $(git describe --tags --abbrev=0) # Install Python dependencies pip3 install -r requirements.txt ``` ### Step 3: Node Configuration ```bash # Create albs-node config mkdir -p /etc/albs cat > /etc/albs/albs-node.conf << 'EOF' [general] web_server_url = https://albs-web.yourdomain.local api_token = worker_count = 4 [pulp] url = http://albs-web.yourdomain.local/pulp username = pulp password = pulp-db-password-here [build] mock_config_path = /etc/mock enable_ccache = true max_parallel_builds = 4 [logging] level = INFO file = /var/log/albs-node.log EOF chmod 600 /etc/albs/albs-node.conf ``` ### Step 4: Mock Chroot Configuration ```bash # Create mock configs for all distros/archs # For EL8 x86_64 cat > /etc/mock/el8-x86_64.cfg << 'EOF' config_opts['chroot_name'] = 'el8-x86_64' config_opts['target_arch'] = 'x86_64' config_opts['releasever'] = '8' config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum debuglevel=2 reposdir=/etc/yum.repos.d [baseos] name=AlmaLinux 8 - BaseOS baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/ enabled=1 gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8 [appstream] name=AlmaLinux 8 - AppStream baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/ enabled=1 gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8 [extras] name=AlmaLinux 8 - Extras baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/ enabled=1 gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8 [epel] name=EPEL 8 baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/ enabled=1 gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8 """ config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir' EOF # For EL9 x86_64, EL9 aarch64, Fedora39 x86_64, etc. # Create similar configs for each distro/arch combination # Test mock chroots mock -r el8-x86_64 --init mock -r el9-x86_64 --init ``` ### Step 5: Systemd Service ```bash # Create systemd service for ALBS node sudo cat > /etc/systemd/system/albs-node.service << 'EOF' [Unit] Description=AlmaLinux Build System Build Node After=network.target docker.service [Service] Type=simple User=root WorkingDirectory=/opt/albs/albs-node ExecStart=/usr/bin/python3 -m albs_node Restart=always RestartSec=10 [Install] WantedBy=multi-user.target EOF # Enable and start sudo systemctl daemon-reload sudo systemctl enable albs-node sudo systemctl start albs-node # Check status sudo systemctl status albs-node sudo journalctl -u albs-node -f ``` ### Step 6: Register Node with Web Server ```bash # On Web Server, register the build node via API curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "builder-x86-1", "url": "http://albs-builder-x86.yourdomain.local:8000", "architectures": ["x86_64"], "platforms": ["AlmaLinux-8", "AlmaLinux-9", "Fedora-39"] }' # Repeat for aarch64 builder curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "builder-arm-1", "url": "http://albs-builder-arm.yourdomain.local:8000", "architectures": ["aarch64"], "platforms": ["AlmaLinux-8", "AlmaLinux-9", "Fedora-39"] }' # Verify nodes registered curl https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " ``` --- ## Repository Configuration ### Step 1: Create Repository Definitions ```bash # In ALBS Web Server, create distributions for each distro/arch # For each platform and architecture: curl -X POST https://albs-web.yourdomain.local/api/v1/distributions \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "AlmaLinux-8-x86_64", "platform": "AlmaLinux-8", "architecture": "x86_64", "build_group": "baseos", "repositories": [ { "name": "baseos", "enabled": true }, { "name": "appstream", "enabled": true }, { "name": "extras", "enabled": true }, { "name": "casjay-rpms", "enabled": true } ] }' # Create distributions for all combinations: # - AlmaLinux-8-x86_64, AlmaLinux-8-aarch64 # - AlmaLinux-9-x86_64, AlmaLinux-9-aarch64 # - Fedora-39-x86_64, Fedora-39-aarch64 ``` ### Step 2: Repository Mapping ```bash # Configure where repositories are synced from/to # Create publication endpoints curl -X POST https://albs-web.yourdomain.local/api/v1/repositories \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "casjay-el8-baseos", "description": "CasjaysDev EL8 BaseOS Repository", "distribution": "AlmaLinux-8-x86_64", "public_url": "http://repos.yourdomain.local/el8/x86_64/baseos/" }' ``` ### Step 3: Configure Pulp Sync ```bash # Sync official repos into Pulp (pull-through cache) # This allows builds to pull from cached official repos curl -X POST https://albs-web.yourdomain.local/api/v1/remotes \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "almalinux-8-baseos", "url": "https://mirrors.almalinux.org/almalinux/8/BaseOS/", "architecture": "x86_64", "distribution": "AlmaLinux-8-x86_64" }' ``` --- ## GitHub Integration ### Step 1: Gitea Listener Setup ```bash # On Web Server cd /opt/albs/gitea_listener # Create config cat > config.yaml << 'EOF' listen_address: 0.0.0.0 listen_port: 8888 web_server_url: https://albs-web.yourdomain.local web_server_token: log_level: INFO log_file: /var/log/gitea_listener.log github: webhook_secret: your-github-webhook-secret-here pulp: url: http://albs-web.yourdomain.local/pulp username: pulp password: pulp-db-password-here EOF chmod 600 config.yaml ``` ### Step 2: Docker Compose for Gitea Listener ```bash # Add to docker-compose.yml cat >> docker-compose.yml << 'EOF' gitea-listener: image: almalinux/gitea_listener:latest container_name: gitea-listener restart: always ports: - "8888:8888" environment: WEB_SERVER_URL: https://albs-web.yourdomain.local WEB_SERVER_TOKEN: GITHUB_WEBHOOK_SECRET: your-github-webhook-secret-here volumes: - ./gitea_listener/config.yaml:/etc/gitea_listener/config.yaml:ro networks: - albs-network EOF docker-compose up -d ``` ### Step 3: GitHub Webhook Configuration For each repository (github.com/rpm-devel/cas, etc.): ``` Repository Settings → Webhooks → Add webhook Payload URL: https://albs-web.yourdomain.local:8888/webhooks/github Content type: application/json Secret: your-github-webhook-secret-here Events: - Push - Release Active: Yes Save ``` ### Step 4: Test GitHub Hook ```bash # Push a change to your spec repo cd /tmp/cas-build git clone https://github.com/rpm-devel/cas cd cas echo "# Test" >> README.md git add README.md git commit -m "Test build trigger" git push # Check ALBS Web dashboard # https://albs-web.yourdomain.local/ # Should see build queued/running within minutes ``` --- ## First Build & Testing ### Step 1: Submit Test Build via API ```bash # Create a build from spec curl -X POST https://albs-web.yourdomain.local/api/v1/builds \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "cas", "version": "1.0.3", "release": "1", "platforms": ["AlmaLinux-8", "AlmaLinux-9"], "architectures": ["x86_64", "aarch64"], "git_ref": "https://github.com/rpm-devel/cas.git", "git_branch": "main" }' # Response will include build ID # Monitor via: https://albs-web.yourdomain.local/builds/ ``` ### Step 2: Monitor Build Progress ```bash # Via Web Dashboard https://albs-web.yourdomain.local/ # Via API curl https://albs-web.yourdomain.local/api/v1/builds/ \ -H "Authorization: Bearer " # Expected flow: # 1. Build queued # 2. Assigned to builder (x86_64) # 3. Build running # 4. Assigned to builder (aarch64) # 5. Build running # 6. Both complete # 7. Artifacts uploaded to Pulp ``` ### Step 3: Verify Build Artifacts ```bash # Check Pulp repository curl https://albs-web.yourdomain.local/pulp/api/v3/content/rpm/packages/ \ -H "Authorization: Bearer " # List built RPMs curl https://albs-web.yourdomain.local/pulp/content/el8/x86_64/baseos/ \ | grep -o 'cas.*\.rpm' # Should see: # cas-1.0.3-1.el8.x86_64.rpm # cas-1.0.3-1.el8.aarch64.rpm # cas-debuginfo-1.0.3-1.el8.x86_64.rpm (optional) # cas-debuginfo-1.0.3-1.el8.aarch64.rpm (optional) ``` ### Step 4: Publish Repository ```bash # Publish built RPMs to client-facing repository curl -X POST https://albs-web.yourdomain.local/api/v1/distributions//publish \ -H "Authorization: Bearer " # Repositories now available at: # http://repos.yourdomain.local/el8/x86_64/baseos/ # http://repos.yourdomain.local/el8/aarch64/baseos/ ``` ### Step 5: Test Client Installation ```bash # On test client (AlmaLinux 8 or 9) # Create repo config sudo tee /etc/yum.repos.d/casjay.repo > /dev/null << 'EOF' [casjay-baseos] name=CasjaysDev BaseOS baseurl=http://repos.yourdomain.local/el8/x86_64/baseos/ enabled=1 gpgcheck=0 [casjay-appstream] name=CasjaysDev AppStream baseurl=http://repos.yourdomain.local/el8/x86_64/appstream/ enabled=1 gpgcheck=0 EOF # Update repo cache sudo dnf makecache # Verify repo available sudo dnf repolist | grep casjay # Install package sudo dnf install cas # Verify installation cas --version ``` --- ## Production Operations ### Daily Monitoring ```bash # Check build node health curl https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " | jq '.[] | {name, status}' # Check active builds curl https://albs-web.yourdomain.local/api/v1/builds?status=in_progress \ -H "Authorization: Bearer " # Check failed builds curl https://albs-web.yourdomain.local/api/v1/builds?status=failed \ -H "Authorization: Bearer " # Check disk usage docker exec albs-pulp df -h /var/lib/pulp docker exec postgres du -sh /var/lib/postgresql/data ``` ### Automated Nightly Rebuilds ```bash # Script: /usr/local/bin/albs-rebuild-third-party.sh #!/bin/bash ALBS_URL="https://albs-web.yourdomain.local" ALBS_TOKEN="" # Remi packages to rebuild REMI_PACKAGES=( "php-8.2" "mariadb-10.6" "postgresql-15" "nodejs-20" ) for pkg in "${REMI_PACKAGES[@]}"; do echo "Rebuilding $pkg..." curl -X POST $ALBS_URL/api/v1/builds \ -H "Authorization: Bearer $ALBS_TOKEN" \ -H "Content-Type: application/json" \ -d "{ \"name\": \"$pkg\", \"platforms\": [\"AlmaLinux-8\", \"AlmaLinux-9\"], \"architectures\": [\"x86_64\", \"aarch64\"], \"git_ref\": \"https://rpms.remirepo.net/enterprise/SRPMS/$pkg.src.rpm\" }" done # Add to crontab # 0 2 * * * /usr/local/bin/albs-rebuild-third-party.sh ``` ### Backup Strategy ```bash # Backup PostgreSQL database 0 1 * * * docker exec postgres pg_dump -U albs albs | gzip > /backups/albs-$(date +\%Y\%m\%d).sql.gz # Backup Pulp data 0 2 * * * tar czf /backups/pulp-$(date +\%Y\%m\%d).tar.gz /var/lib/docker/volumes/albs_pulp-data/_data # Keep 30-day retention 0 3 * * * find /backups -name "albs-*.sql.gz" -mtime +30 -delete ``` ### Monitoring & Alerts ```bash # Health check script: /usr/local/bin/albs-health-check.sh #!/bin/bash ALERT_EMAIL="admin@rpm-devel.local" # Check web server if ! curl -s -k https://albs-web.yourdomain.local/health > /dev/null; then echo "ALERT: ALBS Web Server unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL fi # Check build nodes OFFLINE=$(curl -s https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " | jq '.[] | select(.status != "online") | .name' | wc -l) if [[ $OFFLINE -gt 0 ]]; then echo "ALERT: $OFFLINE build nodes offline" | mail -s "ALBS Alert" $ALERT_EMAIL fi # Check disk space PULP_USAGE=$(docker exec albs-pulp df -h /var/lib/pulp | tail -1 | awk '{print $5}' | tr -d '%') if [[ $PULP_USAGE -gt 80 ]]; then echo "ALERT: Pulp storage at ${PULP_USAGE}% capacity" | mail -s "ALBS Alert" $ALERT_EMAIL fi # Add to crontab (run every 30 minutes) # */30 * * * * /usr/local/bin/albs-health-check.sh ``` --- ## Troubleshooting ### Issue: Build Nodes Not Connecting ```bash # Check node logs docker-compose logs -f albs-node # Verify network connectivity docker exec albs-node ping albs-web.yourdomain.local # Check API token validity curl -i https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " # Verify node config cat /etc/albs/albs-node.conf # Restart node systemctl restart albs-node ``` ### Issue: Build Fails with Mock Error ```bash # Check mock chroot mock -r el8-x86_64 --shell # Verify repo access in chroot mock -r el8-x86_64 --shell dnf repolist # Check mock config for typos grep -A20 "\[baseos\]" /etc/mock/el8-x86_64.cfg # Rebuild chroot mock -r el8-x86_64 --scrub=all mock -r el8-x86_64 --init ``` ### Issue: Artifacts Not Uploaded to Pulp ```bash # Check Pulp status docker-compose logs pulp | tail -50 # Verify Pulp API curl -u admin:admin http://localhost:8000/pulp/api/v3/status/ # Check artifact upload path ls -la /var/lib/docker/volumes/albs_pulp-data/_data/ # Verify ALBS node upload settings grep -A5 "\[pulp\]" /etc/albs/albs-node.conf ``` ### Issue: GitHub Webhooks Not Triggering Builds ```bash # Check Gitea listener logs docker-compose logs gitea-listener | tail -50 # Test webhook manually curl -X POST https://albs-web.yourdomain.local:8888/webhooks/github \ -H "Content-Type: application/json" \ -H "X-Hub-Signature: sha256=test" \ -d '{"action":"opened","pull_request":{}}' # Verify webhook URL on GitHub # Repository Settings → Webhooks → Check delivery history # Check web server logs for auth errors docker-compose logs albs-web | grep -i webhook ``` ### Issue: PostgreSQL Connection Refused ```bash # Verify PostgreSQL is running sudo systemctl status postgresql # Check PostgreSQL logs sudo tail -50 /var/log/postgresql/*.log # Verify listening on all interfaces sudo grep "listen_addresses" /var/lib/pgsql/data/postgresql.conf # Test connection psql -h 127.0.0.1 -U albs -d albs -c "SELECT 1" # Check pg_hba.conf for correct auth entries sudo grep "host.*albs" /var/lib/pgsql/data/pg_hba.conf ``` --- ## Performance Tuning ### PostgreSQL Optimization ```bash # Edit /var/lib/pgsql/data/postgresql.conf # Memory tuning (for 16GB RAM) shared_buffers = 4GB # 25% of RAM effective_cache_size = 12GB # 75% of RAM work_mem = 64MB maintenance_work_mem = 2GB # Connection tuning max_connections = 300 max_parallel_workers = 8 # Query planning random_page_cost = 1.1 # For SSD storage effective_io_concurrency = 200 # WAL (Write-Ahead Logging) wal_level = replica max_wal_senders = 5 # Logging log_statement = 'mod' log_min_duration_statement = 1000 # Log queries > 1 second # Restart PostgreSQL sudo systemctl restart postgresql ``` ### Mock Cache Optimization ```bash # In /etc/mock/*.cfg files config_opts['keep_mounted'] = True config_opts['use_host_resolv'] = True config_opts['basedir'] = '/mnt/mock-cache' # Fast SSD config_opts['cache_topdir'] = '/var/cache/mock' config_opts['internal_dev_setup'] = True ``` ### Docker Optimization ```bash # Edit /etc/docker/daemon.json { "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" }, "max-concurrent-downloads": 10, "max-concurrent-uploads": 10 } # Restart Docker sudo systemctl restart docker ``` ### Pulp Performance ```bash # Tune in docker-compose.yml environment: PULP_EXPORT_PER_PAGE: 1000 PULP_IMPORT_PER_PAGE: 1000 PULP_CONTENT_PATH_PREFIX: /pulp/content/ PULP_WORKERS: 4 ``` --- ## Scaling ### Add More Build Nodes ```bash # Repeat ALBS Build Node Setup (Step 1-6) on new servers # For additional x86_64 builders: builder-x86-2, builder-x86-3, etc. # For additional aarch64 builders: builder-arm-2, builder-arm-3, etc. # Register new nodes via API curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "builder-x86-2", "url": "http://albs-builder-x86-2.yourdomain.local:8000", "architectures": ["x86_64"], "platforms": ["AlmaLinux-8", "AlmaLinux-9"] }' # ALBS automatically load-balances builds across all nodes ``` ### Distributed Pulp Storage ```bash # Use S3-compatible storage (MinIO, AWS S3) # Configure in docker-compose.yml environment: PULP_STORAGE_CLASS: storages.backends.s3boto3.S3Boto3Storage AWS_ACCESS_KEY_ID: your-access-key AWS_SECRET_ACCESS_KEY: your-secret-key AWS_STORAGE_BUCKET_NAME: albs-artifacts AWS_S3_ENDPOINT_URL: https://s3.yourdomain.local ``` --- ## Complete Docker Compose Reference ```yaml version: '3.8' services: postgres: image: postgres:13 container_name: albs-postgres restart: always environment: POSTGRES_PASSWORD: postgres-password volumes: - postgres-data:/var/lib/postgresql/data ports: - "5432:5432" networks: - albs-network redis: image: redis:7-alpine container_name: albs-redis restart: always command: redis-server --requirepass redis-password ports: - "6379:6379" networks: - albs-network albs-web: image: almalinux/albs-web-server:latest container_name: albs-web restart: always ports: - "8080:8080" environment: DATABASE_URL: postgresql://albs:albs-password@postgres:5432/albs REDIS_URL: redis://:redis-password@redis:6379/0 PULP_URL: http://pulp:80 SECRET_KEY: your-secret-key GITHUB_CLIENT_ID: your-github-id GITHUB_CLIENT_SECRET: your-github-secret depends_on: - postgres - redis networks: - albs-network pulp: image: docker.io/pulp/pulp:latest container_name: albs-pulp restart: always ports: - "8000:80" environment: PULP_SECRET_KEY: pulp-secret PULP_CONTENT_ORIGIN: http://albs-web.yourdomain.local/pulp/content volumes: - pulp-data:/var/lib/pulp networks: - albs-network volumes: postgres-data: pulp-data: networks: albs-network: driver: bridge ``` --- ## Success Criteria You've successfully deployed ALBS when: - ✅ Web Server accessible at `https://albs-web.yourdomain.local` - ✅ GitHub OAuth login works - ✅ Both build nodes show as "online" - ✅ Test package builds successfully on both x86_64 and aarch64 - ✅ Built RPMs appear in Pulp repository - ✅ RPMs downloadable from public repo URL - ✅ Client can install packages via `dnf install` - ✅ GitHub webhook triggers automatic builds - ✅ Health checks pass without errors --- ## Key Commands Reference ### Build Management ```bash # Submit build curl -X POST https://albs-web.yourdomain.local/api/v1/builds \ -H "Authorization: Bearer " -d '{...}' # List builds curl https://albs-web.yourdomain.local/api/v1/builds \ -H "Authorization: Bearer " # Get build status curl https://albs-web.yourdomain.local/api/v1/builds/ \ -H "Authorization: Bearer " ``` ### Docker Management ```bash # View logs docker-compose logs -f albs-web docker-compose logs -f albs-node # Restart services docker-compose restart albs-web docker-compose restart albs-node # Stop all docker-compose down ``` ### System Health ```bash # Check nodes curl https://albs-web.yourdomain.local/api/v1/build_nodes \ -H "Authorization: Bearer " # Check platforms curl https://albs-web.yourdomain.local/api/v1/platforms \ -H "Authorization: Bearer " # Check Pulp status curl -u admin:admin http://localhost:8000/pulp/api/v3/status/ ``` --- End of ALBS Complete Guide