jason / Setup RPM Devel

Last active 3 days ago

Like 0

Revision f065fc11160ce5015c10ed7e4f55033f0c5c3864

ALBS.md · 27 KiB · Markdown Raw

# ALBS (AlmaLinux Build System) - Single Server Complete Guide

**Target**: Production ALBS on single 128GB/32-core/8TB server  
**OS**: AlmaLinux 9 (minimal)  
**Deployment**: 1-2 weeks  
**Architecture Support**: x86_64, aarch64 (parallel builds on same server)  
**Complexity**: Medium (Docker-based, simplified single-server)

---

## Table of Contents

1. [Architecture & Overview](#architecture--overview)
2. [Prerequisites & Planning](#prerequisites--planning)
3. [System Preparation](#system-preparation)
4. [PostgreSQL Setup](#postgresql-setup)
5. [Redis Setup](#redis-setup)
6. [ALBS Services Installation](#albs-services-installation)
7. [Mock Chroot Configuration](#mock-chroot-configuration)
8. [Repository Structure](#repository-structure)
9. [GitHub Integration](#github-integration)
10. [First Build & Testing](#first-build--testing)
11. [Production Operations](#production-operations)
12. [Performance & Scaling](#performance--scaling)
13. [Troubleshooting](#troubleshooting)

---

## Architecture & Overview

### Single Server ALBS Setup

```
┌────────────────────────────────────────────────────┐
│           Single Server (128GB/32c/8TB)            │
│                                                    │
│  ┌──────────────┐  ┌──────────────┐               │
│  │ PostgreSQL   │  │ Redis Cache  │               │
│  │ (16GB alloc) │  │ (4GB alloc)  │               │
│  └──────────────┘  └──────────────┘               │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  ALBS Web Server (Docker)                    │ │
│  │  - FastAPI REST API                          │ │
│  │  - Nginx reverse proxy                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Pulp (Docker)                               │ │
│  │  - Artifact storage (2TB allocated)          │ │
│  │  - Repository metadata                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Build Execution (Local)                     │ │
│  │  ┌─────────────┐  ┌─────────────┐            │ │
│  │  │ Mock x86_64 │  │ Mock aarch64 │           │ │
│  │  │ (max 4      │  │ (max 4      │            │ │
│  │  │ parallel)   │  │ parallel)   │            │ │
│  │  └─────────────┘  └─────────────┘            │ │
│  │  - EL8, EL9, Fedora chroots                  │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Repository Storage                          │ │
│  │  /var/lib/repos/ (5TB allocated)              │ │
│  │  - el8/x86_64, el8/aarch64                   │ │
│  │  - el9/x86_64, el9/aarch64                   │ │
│  │  - fedora39/x86_64, fedora39/aarch64         │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
└────────────────────────────────────────────────────┘
```

### Resource Allocation

**Available**: 128GB RAM, 32 cores, 8TB disk

**Allocation**:
- PostgreSQL: 16GB RAM
- Redis: 4GB RAM
- Docker containers (ALBS + Pulp): 8GB RAM
- Mock chroots (in-memory): 16GB RAM (2x 8GB simultaneous builds)
- OS + kernel cache: 20GB
- Reserved: 64GB (headroom)

**Storage**:
- OS/system: 200GB
- PostgreSQL data: 100GB
- Mock cache: 500GB
- Pulp artifacts: 2TB
- Repositories: 5TB
- Logs: 100GB
- Remaining: 300GB (headroom)

---

## Prerequisites & Planning

### Hardware Confirmation
- ✅ 128GB RAM (more than enough)
- ✅ 32 cores (can run 4-8 parallel builds)
- ✅ 8TB disk (5TB for repos + artifacts + OS)
- ✅ Network: 1Gbps (for GitHub webhooks, client downloads)

### Software Required
- AlmaLinux 9 (minimal install)
- Docker & Docker Compose
- PostgreSQL 13+
- Redis 6+
- Python 3.9+
- Mock
- createrepo_c

### Ports Needed
- 80: HTTP (redirect to HTTPS)
- 443: HTTPS (ALBS Web UI + Pulp)
- 5432: PostgreSQL (localhost only)
- 6379: Redis (localhost only)
- 22: SSH

### DNS Setup

```
albs.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>  (or same IP)
```

---

## System Preparation

### Step 1: Update & Install Base Packages

```bash
# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install required packages
sudo dnf install -y \
  git curl wget vim net-tools htop tmux \
  python3.9 python3-pip python3-devel \
  libffi-devel openssl-devel \
  postgresql-server postgresql-contrib postgresql-devel \
  redis \
  docker docker-compose \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Start Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add current user to docker group
sudo usermod -aG docker $(whoami)

# Verify
docker run hello-world
```

### Step 2: Hostname & Network

```bash
# Set hostname
sudo hostnamectl set-hostname albs.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost
<your-ip>   albs.yourdomain.local albs
<your-ip>   repos.yourdomain.local repos

# Test
ping albs.yourdomain.local
```

### Step 3: Create Directory Structure

```bash
# Create all working directories
sudo mkdir -p /var/lib/albs/{artifacts,builds,cache}
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo mkdir -p /var/log/albs

# Set permissions
sudo chown -R $(whoami):$(whoami) /var/lib/albs /var/lib/repos /var/lib/mock /var/log/albs
sudo chmod -R 755 /var/lib/albs /var/lib/repos

# Verify
du -sh /var/lib/albs /var/lib/repos /var/lib/mock
```

### Step 4: Disk Mounting (if using separate partitions)

```bash
# If you have separate disks for storage:
sudo mkfs.ext4 /dev/sdX1
sudo mkdir -p /mnt/repos
sudo mount /dev/sdX1 /mnt/repos

# Make permanent in /etc/fstab
echo "/dev/sdX1 /mnt/repos ext4 defaults 0 2" | sudo tee -a /etc/fstab
sudo mount -a

# For this guide, we'll use /var/lib/repos (single disk)
```

---

## PostgreSQL Setup

### Step 1: Initialize & Configure

```bash
# Initialize PostgreSQL
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78% of 128GB
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1             # For SSD
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL settings
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9
wal_buffers = 16MB

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql
```

### Step 2: Create ALBS Database

```bash
# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER albs WITH PASSWORD 'albs-secure-password-here';
CREATE DATABASE albs OWNER albs;
CREATE USER pulp WITH PASSWORD 'pulp-secure-password-here';
CREATE DATABASE pulp_app OWNER pulp;
CREATE DATABASE pulp_content OWNER pulp;

GRANT ALL ON DATABASE albs TO albs;
GRANT ALL ON DATABASE pulp_app TO pulp;
GRANT ALL ON DATABASE pulp_content TO pulp;

-- Allow connections
ALTER ROLE albs CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep -E "albs|pulp"
```

---

## Redis Setup

### Step 1: Configure

```bash
# Edit Redis config
sudo vi /etc/redis/redis.conf

# Key settings:
bind 127.0.0.1
port 6379
requirepass your-redis-password-here
appendonly yes
appendfsync everysec
maxmemory 4gb
maxmemory-policy allkeys-lru

# Start
sudo systemctl start redis
sudo systemctl enable redis

# Verify
redis-cli ping
# Should return: PONG
```

---

## ALBS Services Installation

### Step 1: Clone ALBS Repository

```bash
# Create working directory
mkdir -p ~/albs-build
cd ~/albs-build

# Clone ALBS
git clone https://github.com/AlmaLinux/albs-web-server.git
cd albs-web-server
git checkout $(git describe --tags --abbrev=0)

# Back to build directory
cd ~/albs-build
```

### Step 2: Create SSL Certificates

```bash
# Create certificate directory
mkdir -p certs
cd certs

# Self-signed certificates (replace with real certs in production)
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout albs-key.key \
  -out albs-cert.crt \
  -subj "/CN=albs.yourdomain.local/O=CasjaysDev/C=US"

# Copy to Docker accessible location
sudo cp albs-cert.crt /etc/pki/albs/ 2>/dev/null || sudo mkdir -p /etc/pki/albs && sudo cp albs-cert.crt /etc/pki/albs/
sudo cp albs-key.key /etc/pki/albs/
sudo chmod 644 /etc/pki/albs/albs-cert.crt
sudo chmod 600 /etc/pki/albs/albs-key.key
```

### Step 3: Docker Compose Setup

```bash
# In ~/albs-build, create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  albs-web:
    image: almalinux/albs-web-server:latest
    container_name: albs-web
    restart: always
    ports:
      - "8080:8080"
    environment:
      DATABASE_URL: postgresql://albs:albs-secure-password-here@localhost:5432/albs
      REDIS_URL: redis://:your-redis-password-here@localhost:6379/0
      PULP_URL: http://localhost:8000
      PULP_DOMAIN_NAME: albs.yourdomain.local
      SECRET_KEY: your-secret-key-here-change-this
      GITHUB_CLIENT_ID: your-github-client-id
      GITHUB_CLIENT_SECRET: your-github-client-secret
      GITHUB_CALLBACK_URL: https://albs.yourdomain.local/auth/github/callback
    volumes:
      - ./albs-web-server:/app
      - /etc/pki/albs:/etc/pki/albs:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

pulp:
    image: docker.io/pulp/pulp:latest
    container_name: albs-pulp
    restart: always
    ports:
      - "8000:80"
    environment:
      PULP_SECRET_KEY: pulp-secret-key-here-change-this
      PULP_CONTENT_ORIGIN: http://albs.yourdomain.local/pulp/content
      PULP_ALLOWED_IMPORT_PATHS: "['/var/lib/pulp']"
      PULP_ALLOWED_EXPORT_PATHS: "['/var/lib/pulp']"
    volumes:
      - pulp-data:/var/lib/pulp
      - pulp-assets:/var/lib/pulp/assets
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

volumes:
  pulp-data:
  pulp-assets:

networks:
  albs-network:
    driver: bridge
EOF

# Pull images
docker-compose pull

# Start services
docker-compose up -d

# Wait for startup
sleep 30

# Verify
docker-compose ps
```

### Step 4: GitHub OAuth Configuration

Get GitHub OAuth credentials:

```
1. Go to GitHub Settings → Developer settings → OAuth Apps
2. Create New OAuth App:
   - Application name: ALBS rpm-devel
   - Homepage URL: https://albs.yourdomain.local
   - Authorization callback URL: https://albs.yourdomain.local/auth/github/callback

3. Copy Client ID and Client Secret

4. Update in docker-compose.yml:
   GITHUB_CLIENT_ID: <your-id>
   GITHUB_CLIENT_SECRET: <your-secret>

5. Restart:
   docker-compose restart albs-web
```

### Step 5: Nginx Reverse Proxy

```bash
# Configure Nginx
sudo vi /etc/nginx/conf.d/albs.conf

cat > /etc/nginx/conf.d/albs.conf << 'EOF'
upstream albs_backend {
    server localhost:8080;
}

upstream pulp_backend {
    server localhost:8000;
}

server {
    listen 80;
    server_name albs.yourdomain.local;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name albs.yourdomain.local;

ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

client_max_body_size 1G;

location / {
        proxy_pass http://albs_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

location /pulp/ {
        proxy_pass http://pulp_backend/pulp/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

# Start Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx

# Test
curl -k https://albs.yourdomain.local/
```

---

## Mock Chroot Configuration

### Step 1: Create Mock Configs for All Distro/Arch Combos

```bash
# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64
sudo tee /etc/mock/el8-aarch64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-aarch64'
config_opts['target_arch'] = 'aarch64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-aarch64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init
```

---

## Repository Structure

### Step 1: Initialize Repositories

```bash
# Create repo directories
mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}/{baseos,appstream,extras,casjay-rpms,casjay-extras}
mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}

# Initialize empty repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# Verify
ls -la /var/lib/repos/el9/x86_64/
```

### Step 2: Configure Nginx Repo Server

```bash
# Create Nginx config for repos
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;

root /var/lib/repos;

location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx
```

---

## GitHub Integration

### Step 1: Gitea Listener Setup

```bash
# Clone Gitea Listener
cd ~/albs-build
git clone https://github.com/AlmaLinux/gitea_listener.git

# Create config
mkdir -p gitea_listener/config
cat > gitea_listener/config.yaml << 'EOF'
listen_address: 0.0.0.0
listen_port: 8888

web_server_url: https://albs.yourdomain.local
web_server_token: <generated-from-web-ui>

log_level: INFO
log_file: /var/log/gitea_listener.log

github:
  webhook_secret: your-github-webhook-secret-here
EOF

chmod 600 gitea_listener/config.yaml
```

### Step 2: Add to Docker Compose

```bash
# Add to docker-compose.yml:
cat >> docker-compose.yml << 'EOF'

gitea-listener:
    image: almalinux/gitea_listener:latest
    container_name: gitea-listener
    restart: always
    ports:
      - "8888:8888"
    environment:
      WEB_SERVER_URL: https://albs.yourdomain.local
      WEB_SERVER_TOKEN: <your-token>
      GITHUB_WEBHOOK_SECRET: your-github-webhook-secret-here
    volumes:
      - ./gitea_listener/config.yaml:/etc/gitea_listener/config.yaml:ro
    networks:
      - albs-network
EOF

docker-compose up -d gitea-listener
```

### Step 3: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

```
Settings → Webhooks → Add webhook

Payload URL: https://albs.yourdomain.local:8888/webhooks/github
Content type: application/json
Secret: your-github-webhook-secret-here
Events: Push
Active: Yes
```

---

## First Build & Testing

### Step 1: Access ALBS Web UI

```
https://albs.yourdomain.local/

1. Click "Sign in with GitHub"
2. Authorize the app
3. You should see dashboard
```

### Step 2: Create Platforms

```bash
# Via API (get token from web UI first):
curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-8",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'

curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-9",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'
```

### Step 3: Submit Test Build

```bash
# Create test SRPM
mkdir -p /tmp/cas-build
cd /tmp/cas-build
git clone https://github.com/rpm-devel/cas
cd cas
rpmbuild -bs cas.spec

# Submit to ALBS
curl -X POST https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cas",
    "version": "1.0.3",
    "release": "1",
    "platforms": ["AlmaLinux-9"],
    "architectures": ["x86_64", "aarch64"],
    "git_ref": "https://github.com/rpm-devel/cas.git",
    "git_branch": "main"
  }'
```

### Step 4: Monitor Build

```
Dashboard: https://albs.yourdomain.local/builds/

Watch progress:
1. Build queued
2. x86_64 build running
3. x86_64 complete
4. aarch64 build running
5. aarch64 complete
6. Artifacts uploaded to Pulp
```

### Step 5: Verify Artifacts

```bash
# Check Pulp repository
curl https://albs.yourdomain.local/pulp/api/v3/content/rpm/packages/ \
  -H "Authorization: Bearer <pulp-token>"

# List RPMs
curl https://albs.yourdomain.local/pulp/content/el9/x86_64/baseos/ | grep "\.rpm"
```

### Step 6: Test Client Installation

```bash
# On test machine (AlmaLinux 9)
cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-baseos]
name=CasjaysDev BaseOS
baseurl=https://repos.yourdomain.local/el9/x86_64/baseos/
enabled=1
gpgcheck=0
EOF

dnf install cas
cas --version
```

---

## Production Operations

### Daily Monitoring

```bash
# Check ALBS status
curl https://albs.yourdomain.local/api/v1/health

# List builds
curl https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <token>"

# Check disk usage
df -h /var/lib/repos /var/lib/albs

# Check database
sudo -u postgres psql -d albs -c "SELECT count(*) FROM builds;"
```

### Nightly Rebuilds Script

```bash
# Create /usr/local/bin/albs-rebuild-third-party.sh
#!/bin/bash

ALBS_URL="https://albs.yourdomain.local"
ALBS_TOKEN="<your-token>"

# Packages to rebuild
PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  curl -X POST $ALBS_URL/api/v1/builds \
    -H "Authorization: Bearer $ALBS_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"name\": \"$pkg\",
      \"platforms\": [\"AlmaLinux-9\"],
      \"architectures\": [\"x86_64\", \"aarch64\"],
      \"git_ref\": \"https://github.com/rpm-devel/$pkg.git\"
    }"
done

# Cron job: 0 2 * * * /usr/local/bin/albs-rebuild-third-party.sh
```

### Automated Repository Sync

```bash
# Copy built RPMs to repos
#!/bin/bash

BUILD_DIR="/var/lib/docker/volumes/pulp-data/_data"
REPO_DIR="/var/lib/repos"

# Sync el9 x86_64
cp $BUILD_DIR/artifacts/el9/x86_64/*.rpm $REPO_DIR/el9/x86_64/casjay-rpms/
createrepo_c $REPO_DIR/el9/x86_64/casjay-rpms/

# Sync el9 aarch64
cp $BUILD_DIR/artifacts/el9/aarch64/*.rpm $REPO_DIR/el9/aarch64/casjay-rpms/
createrepo_c $REPO_DIR/el9/aarch64/casjay-rpms/

# Run via cron: 0 */6 * * * /usr/local/bin/albs-sync-repos.sh
```

### Database Backup

```bash
# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump albs | gzip > /var/backups/albs-$(date +\%Y\%m\%d).sql.gz

# Backup Pulp data
0 2 * * * tar czf /var/backups/pulp-$(date +\%Y\%m\%d).tar.gz /var/lib/docker/volumes/pulp-data/_data

# Retention
find /var/backups -name "albs-*.sql.gz" -mtime +30 -delete
```

### Health Checks

```bash
# Script: /usr/local/bin/albs-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check web server
if ! curl -s -k https://albs.yourdomain.local/health > /dev/null; then
  echo "ALERT: ALBS unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: Repos at ${USAGE}% capacity" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql -d albs -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: Database unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/albs-health-check.sh
```

---

## Performance & Scaling

### PostgreSQL Tuning (128GB)

```bash
# Already configured in postgresql.conf with:
shared_buffers = 16GB
effective_cache_size = 100GB
work_mem = 256MB
```

### Mock Cache Optimization

```bash
# In /etc/mock/*.cfg
config_opts['keep_mounted'] = True
config_opts['use_host_resolv'] = True
config_opts['basedir'] = '/var/lib/mock'
```

### Parallel Build Execution

```bash
# ALBS will automatically parallelize builds across cores
# With 32 cores, can run:
# - 4 simultaneous x86_64 builds (8 cores each)
# - 4 simultaneous aarch64 builds (8 cores each)

# Monitor via:
watch -n 5 'ps aux | grep mock'
```

### Docker Memory Limits

Edit docker-compose.yml to set memory limits:

```yaml
albs-web:
  mem_limit: 4g

pulp:
  mem_limit: 4g
```

---

## Troubleshooting

### ALBS Web Not Accessible

```bash
# Check Docker containers
docker-compose ps

# Check logs
docker-compose logs albs-web | tail -50
docker-compose logs pulp | tail -50

# Check Nginx
sudo systemctl status nginx
sudo nginx -t

# Check SSL cert
openssl x509 -in /etc/pki/albs/albs-cert.crt -text -noout
```

### Builds Failing

```bash
# Check Mock chroot
sudo mock -r el9-x86_64 --shell

# Test repo access
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init

# Check Docker logs
docker-compose logs albs-web | grep -i error
```

### Database Connection Errors

```bash
# Check PostgreSQL running
sudo systemctl status postgresql

# Test connection
psql -h localhost -U albs -d albs -c "SELECT 1;"

# Check PostgreSQL logs
sudo tail -50 /var/log/postgresql/*.log
```

### RPMs Not in Pulp

```bash
# Verify Pulp running
docker-compose logs pulp | tail -20

# Check Pulp API
curl -u admin:admin http://localhost:8000/pulp/api/v3/status/

# Check artifact upload directory
ls -la /var/lib/docker/volumes/pulp-data/_data/
```

### GitHub Webhook Not Triggering

```bash
# Check Gitea Listener logs
docker-compose logs gitea-listener | tail -50

# Verify webhook configuration in GitHub
# Settings → Webhooks → Check delivery history

# Test webhook manually
curl -X POST https://albs.yourdomain.local:8888/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature: sha256=test" \
  -d '{}'
```

---

## Success Criteria

✅ Web dashboard accessible at https://albs.yourdomain.local  
✅ GitHub OAuth login works  
✅ Test build submitted and completed  
✅ Both x86_64 and aarch64 builds successful  
✅ Built RPMs in Pulp repository  
✅ Client can install RPMs via dnf  
✅ GitHub webhook triggers automatic builds  
✅ Daily backups running  
✅ Health checks passing

---

## Quick Command Reference

```bash
# Docker operations
docker-compose ps
docker-compose logs -f albs-web
docker-compose restart albs-web
docker-compose down && docker-compose up -d

# Database
sudo -u postgres psql -d albs
sudo -u postgres pg_dump albs > /backup/albs.sql

# Repository management
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/

# Mock testing
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --clean

# System monitoring
docker stats
df -h
du -sh /var/lib/repos/* /var/lib/albs/*

# API calls
curl -H "Authorization: Bearer <token>" \
  https://albs.yourdomain.local/api/v1/builds
```

---

End of ALBS Single-Server Guide

ALBS (AlmaLinux Build System) - Single Server Complete Guide

Target: Production ALBS on single 128GB/32-core/8TB server
OS: AlmaLinux 9 (minimal)
Deployment: 1-2 weeks
Architecture Support: x86_64, aarch64 (parallel builds on same server)
Complexity: Medium (Docker-based, simplified single-server)

Architecture & Overview
Prerequisites & Planning
System Preparation
PostgreSQL Setup
Redis Setup
ALBS Services Installation
Mock Chroot Configuration
Repository Structure
GitHub Integration
First Build & Testing
Production Operations
Performance & Scaling
Troubleshooting

Architecture & Overview

Single Server ALBS Setup

┌────────────────────────────────────────────────────┐
│           Single Server (128GB/32c/8TB)            │
│                                                    │
│  ┌──────────────┐  ┌──────────────┐               │
│  │ PostgreSQL   │  │ Redis Cache  │               │
│  │ (16GB alloc) │  │ (4GB alloc)  │               │
│  └──────────────┘  └──────────────┘               │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  ALBS Web Server (Docker)                    │ │
│  │  - FastAPI REST API                          │ │
│  │  - Nginx reverse proxy                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Pulp (Docker)                               │ │
│  │  - Artifact storage (2TB allocated)          │ │
│  │  - Repository metadata                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Build Execution (Local)                     │ │
│  │  ┌─────────────┐  ┌─────────────┐            │ │
│  │  │ Mock x86_64 │  │ Mock aarch64 │           │ │
│  │  │ (max 4      │  │ (max 4      │            │ │
│  │  │ parallel)   │  │ parallel)   │            │ │
│  │  └─────────────┘  └─────────────┘            │ │
│  │  - EL8, EL9, Fedora chroots                  │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Repository Storage                          │ │
│  │  /var/lib/repos/ (5TB allocated)              │ │
│  │  - el8/x86_64, el8/aarch64                   │ │
│  │  - el9/x86_64, el9/aarch64                   │ │
│  │  - fedora39/x86_64, fedora39/aarch64         │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
└────────────────────────────────────────────────────┘

Resource Allocation

Available: 128GB RAM, 32 cores, 8TB disk

Allocation:

PostgreSQL: 16GB RAM
Redis: 4GB RAM
Docker containers (ALBS + Pulp): 8GB RAM
Mock chroots (in-memory): 16GB RAM (2x 8GB simultaneous builds)
OS + kernel cache: 20GB
Reserved: 64GB (headroom)

Storage:

OS/system: 200GB
PostgreSQL data: 100GB
Mock cache: 500GB
Pulp artifacts: 2TB
Repositories: 5TB
Logs: 100GB
Remaining: 300GB (headroom)

Prerequisites & Planning

Hardware Confirmation

✅ 128GB RAM (more than enough)
✅ 32 cores (can run 4-8 parallel builds)
✅ 8TB disk (5TB for repos + artifacts + OS)
✅ Network: 1Gbps (for GitHub webhooks, client downloads)

Software Required

AlmaLinux 9 (minimal install)
Docker & Docker Compose
PostgreSQL 13+
Redis 6+
Python 3.9+
Mock
createrepo_c

Ports Needed

80: HTTP (redirect to HTTPS)
443: HTTPS (ALBS Web UI + Pulp)
5432: PostgreSQL (localhost only)
6379: Redis (localhost only)
22: SSH

DNS Setup

albs.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>  (or same IP)

System Preparation

Step 1: Update & Install Base Packages

# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install required packages
sudo dnf install -y \
  git curl wget vim net-tools htop tmux \
  python3.9 python3-pip python3-devel \
  libffi-devel openssl-devel \
  postgresql-server postgresql-contrib postgresql-devel \
  redis \
  docker docker-compose \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Start Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add current user to docker group
sudo usermod -aG docker $(whoami)

# Verify
docker run hello-world

Step 2: Hostname & Network

# Set hostname
sudo hostnamectl set-hostname albs.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost
<your-ip>   albs.yourdomain.local albs
<your-ip>   repos.yourdomain.local repos

# Test
ping albs.yourdomain.local

Step 3: Create Directory Structure

# Create all working directories
sudo mkdir -p /var/lib/albs/{artifacts,builds,cache}
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo mkdir -p /var/log/albs

# Set permissions
sudo chown -R $(whoami):$(whoami) /var/lib/albs /var/lib/repos /var/lib/mock /var/log/albs
sudo chmod -R 755 /var/lib/albs /var/lib/repos

# Verify
du -sh /var/lib/albs /var/lib/repos /var/lib/mock

Step 4: Disk Mounting (if using separate partitions)

# If you have separate disks for storage:
sudo mkfs.ext4 /dev/sdX1
sudo mkdir -p /mnt/repos
sudo mount /dev/sdX1 /mnt/repos

# Make permanent in /etc/fstab
echo "/dev/sdX1 /mnt/repos ext4 defaults 0 2" | sudo tee -a /etc/fstab
sudo mount -a

# For this guide, we'll use /var/lib/repos (single disk)

PostgreSQL Setup

Step 1: Initialize & Configure

# Initialize PostgreSQL
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78% of 128GB
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1             # For SSD
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL settings
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9
wal_buffers = 16MB

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql

Step 2: Create ALBS Database

# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER albs WITH PASSWORD 'albs-secure-password-here';
CREATE DATABASE albs OWNER albs;
CREATE USER pulp WITH PASSWORD 'pulp-secure-password-here';
CREATE DATABASE pulp_app OWNER pulp;
CREATE DATABASE pulp_content OWNER pulp;

GRANT ALL ON DATABASE albs TO albs;
GRANT ALL ON DATABASE pulp_app TO pulp;
GRANT ALL ON DATABASE pulp_content TO pulp;

-- Allow connections
ALTER ROLE albs CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep -E "albs|pulp"

Redis Setup

Step 1: Configure

# Edit Redis config
sudo vi /etc/redis/redis.conf

# Key settings:
bind 127.0.0.1
port 6379
requirepass your-redis-password-here
appendonly yes
appendfsync everysec
maxmemory 4gb
maxmemory-policy allkeys-lru

# Start
sudo systemctl start redis
sudo systemctl enable redis

# Verify
redis-cli ping
# Should return: PONG

ALBS Services Installation

Step 1: Clone ALBS Repository

# Create working directory
mkdir -p ~/albs-build
cd ~/albs-build

# Clone ALBS
git clone https://github.com/AlmaLinux/albs-web-server.git
cd albs-web-server
git checkout $(git describe --tags --abbrev=0)

# Back to build directory
cd ~/albs-build

Step 2: Create SSL Certificates

# Create certificate directory
mkdir -p certs
cd certs

# Self-signed certificates (replace with real certs in production)
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout albs-key.key \
  -out albs-cert.crt \
  -subj "/CN=albs.yourdomain.local/O=CasjaysDev/C=US"

# Copy to Docker accessible location
sudo cp albs-cert.crt /etc/pki/albs/ 2>/dev/null || sudo mkdir -p /etc/pki/albs && sudo cp albs-cert.crt /etc/pki/albs/
sudo cp albs-key.key /etc/pki/albs/
sudo chmod 644 /etc/pki/albs/albs-cert.crt
sudo chmod 600 /etc/pki/albs/albs-key.key

Step 3: Docker Compose Setup

# In ~/albs-build, create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  albs-web:
    image: almalinux/albs-web-server:latest
    container_name: albs-web
    restart: always
    ports:
      - "8080:8080"
    environment:
      DATABASE_URL: postgresql://albs:albs-secure-password-here@localhost:5432/albs
      REDIS_URL: redis://:your-redis-password-here@localhost:6379/0
      PULP_URL: http://localhost:8000
      PULP_DOMAIN_NAME: albs.yourdomain.local
      SECRET_KEY: your-secret-key-here-change-this
      GITHUB_CLIENT_ID: your-github-client-id
      GITHUB_CLIENT_SECRET: your-github-client-secret
      GITHUB_CALLBACK_URL: https://albs.yourdomain.local/auth/github/callback
    volumes:
      - ./albs-web-server:/app
      - /etc/pki/albs:/etc/pki/albs:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

  pulp:
    image: docker.io/pulp/pulp:latest
    container_name: albs-pulp
    restart: always
    ports:
      - "8000:80"
    environment:
      PULP_SECRET_KEY: pulp-secret-key-here-change-this
      PULP_CONTENT_ORIGIN: http://albs.yourdomain.local/pulp/content
      PULP_ALLOWED_IMPORT_PATHS: "['/var/lib/pulp']"
      PULP_ALLOWED_EXPORT_PATHS: "['/var/lib/pulp']"
    volumes:
      - pulp-data:/var/lib/pulp
      - pulp-assets:/var/lib/pulp/assets
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

volumes:
  pulp-data:
  pulp-assets:

networks:
  albs-network:
    driver: bridge
EOF

# Pull images
docker-compose pull

# Start services
docker-compose up -d

# Wait for startup
sleep 30

# Verify
docker-compose ps

Step 4: GitHub OAuth Configuration

Get GitHub OAuth credentials:

1. Go to GitHub Settings → Developer settings → OAuth Apps
2. Create New OAuth App:
   - Application name: ALBS rpm-devel
   - Homepage URL: https://albs.yourdomain.local
   - Authorization callback URL: https://albs.yourdomain.local/auth/github/callback

3. Copy Client ID and Client Secret

4. Update in docker-compose.yml:
   GITHUB_CLIENT_ID: <your-id>
   GITHUB_CLIENT_SECRET: <your-secret>

5. Restart:
   docker-compose restart albs-web

Step 5: Nginx Reverse Proxy

# Configure Nginx
sudo vi /etc/nginx/conf.d/albs.conf

cat > /etc/nginx/conf.d/albs.conf << 'EOF'
upstream albs_backend {
    server localhost:8080;
}

upstream pulp_backend {
    server localhost:8000;
}

server {
    listen 80;
    server_name albs.yourdomain.local;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name albs.yourdomain.local;

    ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    client_max_body_size 1G;

    location / {
        proxy_pass http://albs_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /pulp/ {
        proxy_pass http://pulp_backend/pulp/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

# Start Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx

# Test
curl -k https://albs.yourdomain.local/

Mock Chroot Configuration

Step 1: Create Mock Configs for All Distro/Arch Combos

# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64
sudo tee /etc/mock/el8-aarch64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-aarch64'
config_opts['target_arch'] = 'aarch64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-aarch64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init

Repository Structure

Step 1: Initialize Repositories

# Create repo directories
mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}/{baseos,appstream,extras,casjay-rpms,casjay-extras}
mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}

# Initialize empty repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# Verify
ls -la /var/lib/repos/el9/x86_64/

Step 2: Configure Nginx Repo Server

# Create Nginx config for repos
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

    ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;

    root /var/lib/repos;

    location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx

GitHub Integration

Step 1: Gitea Listener Setup

# Clone Gitea Listener
cd ~/albs-build
git clone https://github.com/AlmaLinux/gitea_listener.git

# Create config
mkdir -p gitea_listener/config
cat > gitea_listener/config.yaml << 'EOF'
listen_address: 0.0.0.0
listen_port: 8888

web_server_url: https://albs.yourdomain.local
web_server_token: <generated-from-web-ui>

log_level: INFO
log_file: /var/log/gitea_listener.log

github:
  webhook_secret: your-github-webhook-secret-here
EOF

chmod 600 gitea_listener/config.yaml

Step 2: Add to Docker Compose

# Add to docker-compose.yml:
cat >> docker-compose.yml << 'EOF'

  gitea-listener:
    image: almalinux/gitea_listener:latest
    container_name: gitea-listener
    restart: always
    ports:
      - "8888:8888"
    environment:
      WEB_SERVER_URL: https://albs.yourdomain.local
      WEB_SERVER_TOKEN: <your-token>
      GITHUB_WEBHOOK_SECRET: your-github-webhook-secret-here
    volumes:
      - ./gitea_listener/config.yaml:/etc/gitea_listener/config.yaml:ro
    networks:
      - albs-network
EOF

docker-compose up -d gitea-listener

Step 3: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

Settings → Webhooks → Add webhook

Payload URL: https://albs.yourdomain.local:8888/webhooks/github
Content type: application/json
Secret: your-github-webhook-secret-here
Events: Push
Active: Yes

First Build & Testing

Step 1: Access ALBS Web UI

https://albs.yourdomain.local/

1. Click "Sign in with GitHub"
2. Authorize the app
3. You should see dashboard

Step 2: Create Platforms

# Via API (get token from web UI first):
curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-8",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'

curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-9",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'

Step 3: Submit Test Build

# Create test SRPM
mkdir -p /tmp/cas-build
cd /tmp/cas-build
git clone https://github.com/rpm-devel/cas
cd cas
rpmbuild -bs cas.spec

# Submit to ALBS
curl -X POST https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cas",
    "version": "1.0.3",
    "release": "1",
    "platforms": ["AlmaLinux-9"],
    "architectures": ["x86_64", "aarch64"],
    "git_ref": "https://github.com/rpm-devel/cas.git",
    "git_branch": "main"
  }'

Step 4: Monitor Build

Dashboard: https://albs.yourdomain.local/builds/

Watch progress:
1. Build queued
2. x86_64 build running
3. x86_64 complete
4. aarch64 build running
5. aarch64 complete
6. Artifacts uploaded to Pulp

Step 5: Verify Artifacts

# Check Pulp repository
curl https://albs.yourdomain.local/pulp/api/v3/content/rpm/packages/ \
  -H "Authorization: Bearer <pulp-token>"

# List RPMs
curl https://albs.yourdomain.local/pulp/content/el9/x86_64/baseos/ | grep "\.rpm"

Step 6: Test Client Installation

# On test machine (AlmaLinux 9)
cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-baseos]
name=CasjaysDev BaseOS
baseurl=https://repos.yourdomain.local/el9/x86_64/baseos/
enabled=1
gpgcheck=0
EOF

dnf install cas
cas --version

Production Operations

Daily Monitoring

# Check ALBS status
curl https://albs.yourdomain.local/api/v1/health

# List builds
curl https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <token>"

# Check disk usage
df -h /var/lib/repos /var/lib/albs

# Check database
sudo -u postgres psql -d albs -c "SELECT count(*) FROM builds;"

Nightly Rebuilds Script

# Create /usr/local/bin/albs-rebuild-third-party.sh
#!/bin/bash

ALBS_URL="https://albs.yourdomain.local"
ALBS_TOKEN="<your-token>"

# Packages to rebuild
PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  curl -X POST $ALBS_URL/api/v1/builds \
    -H "Authorization: Bearer $ALBS_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"name\": \"$pkg\",
      \"platforms\": [\"AlmaLinux-9\"],
      \"architectures\": [\"x86_64\", \"aarch64\"],
      \"git_ref\": \"https://github.com/rpm-devel/$pkg.git\"
    }"
done

# Cron job: 0 2 * * * /usr/local/bin/albs-rebuild-third-party.sh

Automated Repository Sync

# Copy built RPMs to repos
#!/bin/bash

BUILD_DIR="/var/lib/docker/volumes/pulp-data/_data"
REPO_DIR="/var/lib/repos"

# Sync el9 x86_64
cp $BUILD_DIR/artifacts/el9/x86_64/*.rpm $REPO_DIR/el9/x86_64/casjay-rpms/
createrepo_c $REPO_DIR/el9/x86_64/casjay-rpms/

# Sync el9 aarch64
cp $BUILD_DIR/artifacts/el9/aarch64/*.rpm $REPO_DIR/el9/aarch64/casjay-rpms/
createrepo_c $REPO_DIR/el9/aarch64/casjay-rpms/

# Run via cron: 0 */6 * * * /usr/local/bin/albs-sync-repos.sh

Database Backup

# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump albs | gzip > /var/backups/albs-$(date +\%Y\%m\%d).sql.gz

# Backup Pulp data
0 2 * * * tar czf /var/backups/pulp-$(date +\%Y\%m\%d).tar.gz /var/lib/docker/volumes/pulp-data/_data

# Retention
find /var/backups -name "albs-*.sql.gz" -mtime +30 -delete

Health Checks

# Script: /usr/local/bin/albs-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check web server
if ! curl -s -k https://albs.yourdomain.local/health > /dev/null; then
  echo "ALERT: ALBS unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: Repos at ${USAGE}% capacity" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql -d albs -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: Database unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/albs-health-check.sh

Performance & Scaling

PostgreSQL Tuning (128GB)

# Already configured in postgresql.conf with:
shared_buffers = 16GB
effective_cache_size = 100GB
work_mem = 256MB

Mock Cache Optimization

# In /etc/mock/*.cfg
config_opts['keep_mounted'] = True
config_opts['use_host_resolv'] = True
config_opts['basedir'] = '/var/lib/mock'

Parallel Build Execution

# ALBS will automatically parallelize builds across cores
# With 32 cores, can run:
# - 4 simultaneous x86_64 builds (8 cores each)
# - 4 simultaneous aarch64 builds (8 cores each)

# Monitor via:
watch -n 5 'ps aux | grep mock'

Docker Memory Limits

Edit docker-compose.yml to set memory limits:

albs-web:
  mem_limit: 4g

pulp:
  mem_limit: 4g

Troubleshooting

ALBS Web Not Accessible

# Check Docker containers
docker-compose ps

# Check logs
docker-compose logs albs-web | tail -50
docker-compose logs pulp | tail -50

# Check Nginx
sudo systemctl status nginx
sudo nginx -t

# Check SSL cert
openssl x509 -in /etc/pki/albs/albs-cert.crt -text -noout

Builds Failing

# Check Mock chroot
sudo mock -r el9-x86_64 --shell

# Test repo access
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init

# Check Docker logs
docker-compose logs albs-web | grep -i error

Database Connection Errors

# Check PostgreSQL running
sudo systemctl status postgresql

# Test connection
psql -h localhost -U albs -d albs -c "SELECT 1;"

# Check PostgreSQL logs
sudo tail -50 /var/log/postgresql/*.log

RPMs Not in Pulp

# Verify Pulp running
docker-compose logs pulp | tail -20

# Check Pulp API
curl -u admin:admin http://localhost:8000/pulp/api/v3/status/

# Check artifact upload directory
ls -la /var/lib/docker/volumes/pulp-data/_data/

GitHub Webhook Not Triggering

# Check Gitea Listener logs
docker-compose logs gitea-listener | tail -50

# Verify webhook configuration in GitHub
# Settings → Webhooks → Check delivery history

# Test webhook manually
curl -X POST https://albs.yourdomain.local:8888/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature: sha256=test" \
  -d '{}'

Success Criteria

✅ Web dashboard accessible at https://albs.yourdomain.local
✅ GitHub OAuth login works
✅ Test build submitted and completed
✅ Both x86_64 and aarch64 builds successful
✅ Built RPMs in Pulp repository
✅ Client can install RPMs via dnf
✅ GitHub webhook triggers automatic builds
✅ Daily backups running
✅ Health checks passing

Quick Command Reference

# Docker operations
docker-compose ps
docker-compose logs -f albs-web
docker-compose restart albs-web
docker-compose down && docker-compose up -d

# Database
sudo -u postgres psql -d albs
sudo -u postgres pg_dump albs > /backup/albs.sql

# Repository management
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/

# Mock testing
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --clean

# System monitoring
docker stats
df -h
du -sh /var/lib/repos/* /var/lib/albs/*

# API calls
curl -H "Authorization: Bearer <token>" \
  https://albs.yourdomain.local/api/v1/builds

End of ALBS Single-Server Guide

KOJI.md · 30 KiB · Markdown Raw

# Koji Build System - Single Server Complete Guide

**Target**: Production Koji on single 128GB/32-core/8TB server  
**OS**: AlmaLinux 9 (minimal)  
**Deployment**: 3-4 weeks  
**Architecture Support**: x86_64, aarch64 (parallel builds on same server)  
**Complexity**: High (but single-server simplified)

---

## Table of Contents

1. [Architecture & Overview](#architecture--overview)
2. [Prerequisites & Planning](#prerequisites--planning)
3. [System Preparation](#system-preparation)
4. [PostgreSQL Setup](#postgresql-setup)
5. [RabbitMQ Setup](#rabbitmq-setup)
6. [SSL Certificate Generation](#ssl-certificate-generation)
7. [Koji Hub Installation](#koji-hub-installation)
8. [Koji Web Installation](#koji-web-installation)
9. [Koji Builders Configuration](#koji-builders-configuration)
10. [Mock Chroot Setup](#mock-chroot-setup)
11. [Koji Tags & Targets](#koji-tags--targets)
12. [Repository Structure](#repository-structure)
13. [GitHub Integration](#github-integration)
14. [First Build & Testing](#first-build--testing)
15. [Production Operations](#production-operations)
16. [Troubleshooting](#troubleshooting)

---

## Architecture & Overview

### Single Server Koji Setup

```
┌──────────────────────────────────────────────────────────┐
│          Single Server (128GB/32c/8TB)                   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  PostgreSQL (16GB)                              │   │
│  │  - Build metadata, packages, tasks              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  RabbitMQ (4GB)                                 │   │
│  │  - Task queue, async processing                 │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Hub (XMLRPC API)                          │   │
│  │  - Apache + mod_wsgi                            │   │
│  │  - Task orchestration                           │   │
│  │  - Package tracking                             │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Web (Dashboard)                           │   │
│  │  - Package browser                              │   │
│  │  - Build history                                │   │
│  │  - Task monitoring                              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Local Build Execution                          │   │
│  │  ┌──────────────┐  ┌──────────────┐             │   │
│  │  │ Mock x86_64  │  │ Mock aarch64  │            │   │
│  │  │ (8 parallel) │  │ (8 parallel)  │            │   │
│  │  └──────────────┘  └──────────────┘             │   │
│  │  - EL8, EL9, Fedora chroots                     │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Package Storage                                │   │
│  │  /var/lib/koji/packages/ (3TB allocated)        │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Repository Storage                             │   │
│  │  /var/lib/repos/ (5TB allocated)                │   │
│  │  - el8/9/fedora × x86_64/aarch64                │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
└──────────────────────────────────────────────────────────┘
```

### Resource Allocation

**Available**: 128GB RAM, 32 cores, 8TB disk

**Allocation**:
- PostgreSQL: 16GB RAM
- RabbitMQ: 4GB RAM
- Apache/Koji Hub: 4GB RAM
- Mock builds (simultaneous): 16GB RAM (2x 8GB)
- Kernel cache/OS: 20GB
- Reserved headroom: 52GB

**Storage**:
- OS/system: 200GB
- PostgreSQL: 100GB
- Mock cache: 500GB
- Koji packages: 3TB
- Repositories: 5TB
- Logs: 100GB
- Remaining: 100GB (headroom)

---

## Prerequisites & Planning

### Hardware Confirmation
- ✅ 128GB RAM (excellent)
- ✅ 32 cores (run 16 parallel builds)
- ✅ 8TB disk (plenty for everything)
- ✅ Network: 1Gbps

### Ports Required
- 80: HTTP (redirect to HTTPS)
- 443: HTTPS (Koji Web + Hub)
- 5432: PostgreSQL (localhost only)
- 5672: RabbitMQ (localhost only)
- 22: SSH

### DNS Setup

```
koji.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>
```

### Directory Structure

```
/var/lib/koji/
  packages/          (3TB for built packages)
  work/              (temporary build space)
  scratch/           (scratch builds)
  
/var/lib/repos/
  el8/x86_64/        (repositories)
  el8/aarch64/
  el9/x86_64/
  el9/aarch64/
  fedora39/x86_64/
  fedora39/aarch64/
  srpms/

/etc/koji-hub/       (Koji Hub config)
/etc/koji-web/       (Koji Web config)
/etc/mock/           (Mock chroot configs)
```

---

## System Preparation

### Step 1: Update & Install Packages

```bash
# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install Koji packages
sudo dnf install -y \
  koji-hub koji-web koji-utils koji-builder \
  koji-cli koji-client-lib \
  postgresql-server postgresql-contrib postgresql-devel \
  rabbitmq-server \
  httpd mod_wsgi mod_ssl \
  python3.9 python3-pip python3-devel \
  git curl wget vim net-tools htop tmux \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Install EPEL packages that may be needed
sudo dnf install -y --enablerepo=epel koji-{builder,hub,web}

# Start essential services
sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server
```

### Step 2: Set Hostname & Network

```bash
# Set hostname
sudo hostnamectl set-hostname koji.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost koji
<your-ip>   koji.yourdomain.local koji
<your-ip>   repos.yourdomain.local repos

# Test
ping koji.yourdomain.local
```

### Step 3: Create Directory Structure

```bash
# Koji directories
sudo mkdir -p /var/lib/koji/{packages,work,scratch,mock-root}
sudo chown -R apache:apache /var/lib/koji
sudo chmod -R 755 /var/lib/koji

# Repository directories
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}
sudo chown -R apache:apache /var/lib/repos
sudo chmod -R 755 /var/lib/repos

# Mock cache
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo chmod -R 777 /var/lib/mock

# Logs
sudo mkdir -p /var/log/koji-{hub,builder}
sudo chown apache:apache /var/log/koji-*

# Verify
df -h /var/lib
```

---

## PostgreSQL Setup

### Step 1: Initialize & Configure

```bash
# Initialize database cluster
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78%
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql
```

### Step 2: Create Koji Database

```bash
# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER koji WITH PASSWORD 'koji-secure-password-here';
CREATE DATABASE koji OWNER koji;
ALTER ROLE koji CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep koji
```

### Step 3: Load Koji Schema

```bash
# Find schema location
rpm -ql koji-hub | grep schema

# Load schema (adjust path based on Koji version)
sudo -u postgres psql koji < /usr/share/doc/koji/docs/schema.sql

# Verify tables created
sudo -u postgres psql koji -c "\dt" | head -20
```

---

## RabbitMQ Setup

### Step 1: Configure & Initialize

```bash
# Start RabbitMQ
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server

# Wait for startup
sleep 5

# Create Koji user and vhost
sudo rabbitmqctl add_user koji koji-mq-password-here
sudo rabbitmqctl add_vhost /koji
sudo rabbitmqctl set_permissions -p /koji koji ".*" ".*" ".*"

# Verify
sudo rabbitmqctl list_users
sudo rabbitmqctl list_vhosts
```

---

## SSL Certificate Generation

### Step 1: Create Certificates

```bash
# Create certificate directory
sudo mkdir -p /etc/pki/koji
cd /tmp/cert-build

# Create CA certificate
sudo openssl genrsa -out koji-ca-key.key 2048
sudo openssl req -new -x509 -days 3650 -key koji-ca-key.key \
  -out koji-ca-cert.crt \
  -subj "/CN=koji-ca/O=CasjaysDev/C=US"

# Create Koji Hub certificate
sudo openssl genrsa -out koji-hub-key.key 2048
sudo openssl req -new -key koji-hub-key.key \
  -out koji-hub.csr \
  -subj "/CN=koji.yourdomain.local/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-hub.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-hub-cert.crt

# Create client certificate (for koji CLI)
sudo openssl genrsa -out koji-client-key.key 2048
sudo openssl req -new -key koji-client-key.key \
  -out koji-client.csr \
  -subj "/CN=koji-admin/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-client.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-client-cert.crt

# Copy to Koji directory
sudo cp koji-ca-cert.crt koji-hub-cert.crt koji-hub-key.key /etc/pki/koji/
sudo cp koji-ca-cert.crt koji-client-cert.crt koji-client-key.key /etc/pki/koji/

# Permissions
sudo chmod 600 /etc/pki/koji/*-key.key
sudo chmod 644 /etc/pki/koji/*-cert.crt
sudo chown -R apache:apache /etc/pki/koji

# Verify
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt
```

---

## Koji Hub Installation

### Step 1: Configure Koji Hub

```bash
# Edit hub configuration
sudo vi /etc/koji-hub/hub.conf

# Key settings:
[hub]
DBName = koji
DBUser = koji
DBPassword = koji-secure-password-here
DBHost = localhost
DBPort = 5432

KojiDir = /var/lib/koji

MQHost = localhost
MQPort = 5672
MQUser = koji
MQPassword = koji-mq-password-here
MQVHost = /koji

KojiHubCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCertFile = /etc/pki/koji/koji-hub-cert.crt
KojiHubKeyFile = /etc/pki/koji/koji-hub-key.key

AuthMethod = ssl
ProxyPrincipals = koji-admin

# Task management
AutoRebuilds = true
BuildForce = false
```

### Step 2: Apache/HTTPD Configuration

```bash
# Enable modules
sudo a2enmod wsgi
sudo a2enmod ssl

# Create Koji VirtualHost
sudo tee /etc/httpd/conf.d/koji.conf > /dev/null << 'EOF'
LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName koji.yourdomain.local
    Redirect permanent / https://koji.yourdomain.local/
</VirtualHost>

<VirtualHost *:443>
    ServerName koji.yourdomain.local
    SSLEngine on
    SSLCertificateFile /etc/pki/koji/koji-hub-cert.crt
    SSLCertificateKeyFile /etc/pki/koji/koji-hub-key.key
    SSLCACertificateFile /etc/pki/koji/koji-ca-cert.crt
    SSLVerifyClient optional
    SSLVerifyDepth 10

WSGIScriptAlias / /usr/share/koji-web/kojiweb.wsgi
    WSGICallableObject application

<Directory "/usr/share/koji-web/">
        AllowOverride All
        Options All
        Require all granted
    </Directory>

ErrorLog /var/log/httpd/koji-error_log
    CustomLog /var/log/httpd/koji-access_log combined
</VirtualHost>

Listen 443
EOF

# Start Apache
sudo systemctl restart httpd
sudo systemctl enable httpd
```

### Step 3: Start Koji Hub

```bash
# Start Koji daemon
sudo systemctl start kojid
sudo systemctl enable kojid

# Verify
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Should see: "Connected to database..."
```

---

## Koji Web Installation

### Step 1: Configure Koji Web

```bash
# Edit web configuration
sudo vi /etc/koji-web/web.conf

[web]
SiteName = CasjaysDev Koji
KojiHubURL = https://koji.yourdomain.local/koji
WebCertFile = /etc/pki/koji/koji-web-cert.crt
WebKeyFile = /etc/pki/koji/koji-web-key.key
WebCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCA = /etc/pki/koji/koji-ca-cert.crt

# Make self-signed certs if needed
sudo cp /etc/pki/koji/koji-hub-cert.crt /etc/pki/koji/koji-web-cert.crt
sudo cp /etc/pki/koji/koji-hub-key.key /etc/pki/koji/koji-web-key.key
```

### Step 2: Verify Web Access

```bash
# Test HTTPS
curl -k https://koji.yourdomain.local/koji/

# Should see web interface (HTML output)
```

---

## Koji Builders Configuration

### Step 1: Koji Builder Configuration

```bash
# Since this is single-server, we'll run kojid locally
# It acts as both hub and builder

# Configuration is already in /etc/koji-hub/hub.conf

# Verify kojid is running
sudo systemctl status kojid

# Check logs for builder registration
tail -f /var/log/koji-hub/kojid.log
```

### Step 2: Register Builder Hosts

```bash
# Install koji-cli to add hosts
sudo dnf install -y koji-cli

# Copy client certs
mkdir -p ~/.koji
cp /etc/pki/koji/koji-ca-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-key.key ~/.koji/

# Create koji config
cat > ~/.koji/config << 'EOF'
[koji]
server = https://koji.yourdomain.local/koji
cert = ~/.koji/koji-client-cert.crt
ca = ~/.koji/koji-ca-cert.crt
key = ~/.koji/koji-client-key.key
EOF

chmod 600 ~/.koji/config

# Add builder hosts (for documentation; single-server uses localhost)
koji add-host localhost x86_64
koji add-host localhost aarch64

# Create channels
koji add-channel x86_64-chan
koji add-channel aarch64-chan

# Assign to channels
koji edit-host localhost --channel-arches x86_64
koji edit-host localhost --channel-arches aarch64

# Verify
koji list-hosts --ready
```

---

## Mock Chroot Setup

### Step 1: Create Mock Configurations

```bash
# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64 (copy and modify)
sudo cp /etc/mock/el8-x86_64.cfg /etc/mock/el8-aarch64.cfg
sudo sed -i "s/el8-x86_64/el8-aarch64/g" /etc/mock/el8-aarch64.cfg
sudo sed -i "s/target_arch = 'x86_64'/target_arch = 'aarch64'/g" /etc/mock/el8-aarch64.cfg

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init
```

---

## Koji Tags & Targets

### Step 1: Create Tag Hierarchy

```bash
# Set up koji CLI first
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key
export KOJI_SERVER=https://koji.yourdomain.local/koji

# Create tags for AlmaLinux 8
koji add-tag el8-build
koji add-repo el8-build koji-build-repo

koji add-tag el8-release --parent el8-build

# Create targets
koji add-target el8-candidate el8-build el8-release

# Create tags for AlmaLinux 9
koji add-tag el9-build
koji add-repo el9-build koji-build-repo

koji add-tag el9-release --parent el9-build
koji add-target el9-candidate el9-build el9-release

# Create tags for Fedora 39
koji add-tag fedora39-build
koji add-repo fedora39-build koji-build-repo

koji add-tag fedora39-release --parent fedora39-build
koji add-target fedora39-candidate fedora39-build fedora39-release

# For third-party rebuilds
koji add-tag el9-rebuild-build --parent el9-build
koji add-target el9-rebuild-candidate el9-rebuild-build el9-release

# Verify
koji list-tags
koji list-targets
```

### Step 2: Add Packages to Tags

```bash
# Import package sets for dependency resolution
koji import-comps -t el8-build /etc/comps-el8.xml
koji import-comps -t el9-build /etc/comps-el9.xml

# Add key packages
koji add-pkg el8-build kernel vim git
koji add-pkg el9-build kernel vim git
```

---

## Repository Structure

### Step 1: Initialize Repositories

```bash
# Create all repo directories with subdirs
for dist in el8 el9 fedora39; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      mkdir -p /var/lib/repos/${dist}/${arch}/${repo}
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# SRPM repos
for dist in el8 el9 fedora39; do
  mkdir -p /var/lib/repos/srpms/${dist}
  createrepo_c /var/lib/repos/srpms/${dist}/
done

# Verify
ls -la /var/lib/repos/el9/x86_64/
```

### Step 2: Configure Nginx Repository Server

```bash
# Configure Nginx for repo serving
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

ssl_certificate /etc/pki/koji/koji-hub-cert.crt;
    ssl_certificate_key /etc/pki/koji/koji-hub-key.key;

root /var/lib/repos;

location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx
sudo systemctl enable nginx
```

---

## GitHub Integration

### Step 1: Create Webhook Handler Script

```bash
# Create /usr/local/bin/koji-github-builder.sh
sudo tee /usr/local/bin/koji-github-builder.sh > /dev/null << 'SCRIPT'
#!/bin/bash

SPEC_URL="git+https://github.com/rpm-devel/${REPO}.git#${COMMIT}"

# Default targets (can override per-repo)
TARGETS="${KOJI_TARGETS:-el9-candidate}"

for TARGET in $TARGETS; do
  echo "[$(date)] Building: ${REPO} @ ${COMMIT:0:7} in ${TARGET}"
  
  # Use koji from CLI
  koji build ${TARGET} ${SPEC_URL} 2>&1 | tee -a /var/log/koji-builds.log
done
SCRIPT

sudo chmod +x /usr/local/bin/koji-github-builder.sh
```

### Step 2: Flask Webhook Receiver

```bash
# Create /usr/local/bin/koji-webhook.py
sudo tee /usr/local/bin/koji-webhook.py > /dev/null << 'PYTHON'
#!/usr/bin/env python3

from flask import Flask, request
import json
import subprocess
import logging

app = Flask(__name__)
logging.basicConfig(
    filename='/var/log/koji-webhook.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

@app.route('/webhook/github', methods=['POST'])
def github_webhook():
    payload = request.get_json()
    
    if not payload:
        return 'No payload', 400
    
    repo_name = payload['repository']['name']
    commit = payload['head_commit']['id'][:7]
    
    logging.info(f"Webhook: {repo_name} @ {commit}")
    
    try:
        result = subprocess.run(
            ['/usr/local/bin/koji-github-builder.sh', json.dumps(payload)],
            capture_output=True, text=True, timeout=60
        )
        logging.info(f"Build result: {result.stdout}")
        return 'Build submitted', 202
    except Exception as e:
        logging.error(f"Build failed: {e}")
        return 'Build failed', 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8888, debug=False)
PYTHON

sudo chmod +x /usr/local/bin/koji-webhook.py

# Install Flask
pip3 install flask
```

### Step 3: Systemd Service

```bash
# Create systemd service for webhook
sudo tee /etc/systemd/system/koji-webhook.service > /dev/null << 'EOF'
[Unit]
Description=Koji GitHub Webhook Handler
After=network.target

[Service]
Type=simple
User=koji
ExecStart=/usr/local/bin/koji-webhook.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Create koji user if needed
sudo useradd -r koji 2>/dev/null || true

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable koji-webhook
sudo systemctl start koji-webhook
```

### Step 4: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

```
Settings → Webhooks → Add webhook

Payload URL: https://koji.yourdomain.local:8888/webhook/github
Content type: application/json
Secret: your-secret-here
Events: Push events
Active: Yes
```

---

## First Build & Testing

### Step 1: Create Test SRPM

```bash
# Clone test package
git clone https://github.com/rpm-devel/cas /tmp/cas-build
cd /tmp/cas-build

# Build SRPM
rpmbuild -bs cas.spec

# Result in ~/rpmbuild/SRPMS/cas-*.src.rpm
```

### Step 2: Submit Build via koji CLI

```bash
# Set environment
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

# Submit build
koji build el9-candidate ~/rpmbuild/SRPMS/cas-*.src.rpm

# Watch build
koji watch-task <task-id>

# Expected flow:
# 1. Build submitted
# 2. x86_64 build starting
# 3. x86_64 build complete
# 4. aarch64 build starting
# 5. aarch64 build complete
# 6. All complete
```

### Step 3: Verify Build Success

```bash
# List builds
koji list-builds cas --latest=1

# Get build info
koji buildinfo <build-id>

# Verify both architectures
koji buildinfo <build-id> | grep arch

# List built packages
ls -la /var/lib/koji/packages/cas/*/1/
```

### Step 4: Copy to Repository

```bash
# Copy RPMs to repo
sudo cp /var/lib/koji/packages/cas/*/1/x86_64/*.rpm \
  /var/lib/repos/el9/x86_64/casjay-rpms/
sudo cp /var/lib/koji/packages/cas/*/1/aarch64/*.rpm \
  /var/lib/repos/el9/aarch64/casjay-rpms/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
sudo createrepo_c /var/lib/repos/el9/aarch64/casjay-rpms/
```

### Step 5: Test Client Installation

```bash
# On test machine (AlmaLinux 9)
sudo cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-rpms]
name=CasjaysDev RPMs
baseurl=https://repos.yourdomain.local/el9/x86_64/casjay-rpms/
enabled=1
gpgcheck=0
EOF

# Install
sudo dnf install cas

# Verify
cas --version
```

---

## Production Operations

### Daily Monitoring

```bash
# Check Koji hub status
koji list-hosts --ready

# Active tasks
koji list-tasks --state=active

# Failed builds
koji list-builds --state=failed --limit=10

# Database size
sudo -u postgres psql koji -c "SELECT pg_size_pretty(pg_database_size('koji'));"

# Disk usage
du -sh /var/lib/koji/packages /var/lib/repos
```

### Automated Nightly Rebuilds

```bash
# Script: /usr/local/bin/koji-nightly-rebuilds.sh
#!/bin/bash

export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
  "nginx"
  "apache"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  koji build el9-rebuild-candidate \
    "git+https://github.com/rpm-devel/$pkg.git"
done

# Cron: 0 2 * * * /usr/local/bin/koji-nightly-rebuilds.sh
```

### Build Promotion Workflow

```bash
# After testing, promote to release
koji move-build el9-candidate el9-release cas

# Or tag directly
koji tag-pkg el9-release cas-1.0.3-1

# Verify promotion
koji list-builds --tag=el9-release --latest=5
```

### Repository Sync

```bash
# Script: /usr/local/bin/koji-sync-repos.sh
#!/bin/bash

# Sync all built packages to public repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    # Get recently built packages
    src=/var/lib/koji/packages
    dst=/var/lib/repos/${dist}/${arch}/casjay-rpms
    
    # Copy and regenerate
    find $src -name "*.rpm" -mtime -1 | xargs -I {} cp {} $dst/
    createrepo_c $dst
  done
done

# Cron: 0 */6 * * * /usr/local/bin/koji-sync-repos.sh
```

### Database Backup

```bash
# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump koji | gzip > /var/backups/koji-$(date +\%Y\%m\%d).sql.gz

# Backup built packages (important!)
0 2 * * * tar czf /var/backups/koji-pkgs-$(date +\%Y\%m\%d).tar.gz /var/lib/koji/packages/

# Retention (30 days)
find /var/backups -name "koji-*.sql.gz" -mtime +30 -delete
find /var/backups -name "koji-pkgs-*.tar.gz" -mtime +30 -delete
```

### Health Checks

```bash
# Script: /usr/local/bin/koji-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check Koji hub
if ! koji list-hosts > /dev/null 2>&1; then
  echo "ALERT: Koji hub unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: /var/lib/repos at ${USAGE}% capacity" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql koji -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: PostgreSQL unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check RabbitMQ
if ! sudo rabbitmqctl status > /dev/null 2>&1; then
  echo "ALERT: RabbitMQ unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/koji-health-check.sh
```

---

## Troubleshooting

### Koji Hub Not Accessible

```bash
# Check Apache
sudo systemctl status httpd
sudo apache2ctl configtest

# Check Koji daemon
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Verify SSL certs
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt
```

### Build Failures

```bash
# Check specific build
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Test Mock chroot directly
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init
```

### Database Connection Errors

```bash
# Check PostgreSQL
sudo systemctl status postgresql
sudo -u postgres psql -d koji -c "SELECT 1;"

# Check logs
sudo tail -50 /var/log/postgresql/*.log

# Verify configuration
sudo grep "listen_addresses" /var/lib/pgsql/data/postgresql.conf
```

### RabbitMQ Issues

```bash
# Check RabbitMQ status
sudo systemctl status rabbitmq-server
sudo rabbitmqctl status

# Check koji user
sudo rabbitmqctl list_users | grep koji

# Verify queue
sudo rabbitmqctl list_queues -p /koji
```

### Packages Not in Repository

```bash
# Verify build completed
koji list-builds | grep <package>

# Check package location
find /var/lib/koji/packages -name "*.rpm" | grep <package>

# Manual copy
sudo cp /var/lib/koji/packages/<pkg>/*/* /var/lib/repos/el9/x86_64/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
```

---

## Success Criteria

✅ Koji web accessible at https://koji.yourdomain.local/koji/  
✅ koji list-hosts --ready shows builder online  
✅ Test build submitted and completed successfully  
✅ Both x86_64 and aarch64 builds successful  
✅ Built packages in /var/lib/koji/packages/  
✅ RPMs copied to /var/lib/repos/  
✅ Repository metadata generated (repomd.xml exists)  
✅ Client can install packages via dnf  
✅ GitHub webhook triggers automatic builds  
✅ Daily backups completing  
✅ Health checks passing

---

## Quick Reference

```bash
# Koji commands
koji build <target> <SRPM-or-git-url>
koji watch-task <task-id>
koji list-tasks --state=active
koji list-builds --latest=20
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Tag management
koji add-tag <tag-name>
koji add-target <target> <build-tag> <dest-tag>
koji list-tags
koji list-targets
koji tag-pkg <tag> <package>
koji move-build <src-tag> <dest-tag> <package>

# System commands
sudo systemctl status {httpd,kojid,postgresql,rabbitmq-server}
tail -f /var/log/koji-hub/kojid.log
tail -f /var/log/httpd/koji-access_log

# Repository
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
koji regen-repo <tag>
```

---

End of Koji Single-Server Guide

Koji Build System - Single Server Complete Guide

Target: Production Koji on single 128GB/32-core/8TB server
OS: AlmaLinux 9 (minimal)
Deployment: 3-4 weeks
Architecture Support: x86_64, aarch64 (parallel builds on same server)
Complexity: High (but single-server simplified)

Architecture & Overview
Prerequisites & Planning
System Preparation
PostgreSQL Setup
RabbitMQ Setup
SSL Certificate Generation
Koji Hub Installation
Koji Web Installation
Koji Builders Configuration
Mock Chroot Setup
Koji Tags & Targets
Repository Structure
GitHub Integration
First Build & Testing
Production Operations
Troubleshooting

Architecture & Overview

Single Server Koji Setup

┌──────────────────────────────────────────────────────────┐
│          Single Server (128GB/32c/8TB)                   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  PostgreSQL (16GB)                              │   │
│  │  - Build metadata, packages, tasks              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  RabbitMQ (4GB)                                 │   │
│  │  - Task queue, async processing                 │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Hub (XMLRPC API)                          │   │
│  │  - Apache + mod_wsgi                            │   │
│  │  - Task orchestration                           │   │
│  │  - Package tracking                             │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Web (Dashboard)                           │   │
│  │  - Package browser                              │   │
│  │  - Build history                                │   │
│  │  - Task monitoring                              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Local Build Execution                          │   │
│  │  ┌──────────────┐  ┌──────────────┐             │   │
│  │  │ Mock x86_64  │  │ Mock aarch64  │            │   │
│  │  │ (8 parallel) │  │ (8 parallel)  │            │   │
│  │  └──────────────┘  └──────────────┘             │   │
│  │  - EL8, EL9, Fedora chroots                     │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Package Storage                                │   │
│  │  /var/lib/koji/packages/ (3TB allocated)        │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Repository Storage                             │   │
│  │  /var/lib/repos/ (5TB allocated)                │   │
│  │  - el8/9/fedora × x86_64/aarch64                │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
└──────────────────────────────────────────────────────────┘

Resource Allocation

Available: 128GB RAM, 32 cores, 8TB disk

Allocation:

PostgreSQL: 16GB RAM
RabbitMQ: 4GB RAM
Apache/Koji Hub: 4GB RAM
Mock builds (simultaneous): 16GB RAM (2x 8GB)
Kernel cache/OS: 20GB
Reserved headroom: 52GB

Storage:

OS/system: 200GB
PostgreSQL: 100GB
Mock cache: 500GB
Koji packages: 3TB
Repositories: 5TB
Logs: 100GB
Remaining: 100GB (headroom)

Prerequisites & Planning

Hardware Confirmation

✅ 128GB RAM (excellent)
✅ 32 cores (run 16 parallel builds)
✅ 8TB disk (plenty for everything)
✅ Network: 1Gbps

Ports Required

80: HTTP (redirect to HTTPS)
443: HTTPS (Koji Web + Hub)
5432: PostgreSQL (localhost only)
5672: RabbitMQ (localhost only)
22: SSH

DNS Setup

koji.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>

Directory Structure

/var/lib/koji/
  packages/          (3TB for built packages)
  work/              (temporary build space)
  scratch/           (scratch builds)
  
/var/lib/repos/
  el8/x86_64/        (repositories)
  el8/aarch64/
  el9/x86_64/
  el9/aarch64/
  fedora39/x86_64/
  fedora39/aarch64/
  srpms/

/etc/koji-hub/       (Koji Hub config)
/etc/koji-web/       (Koji Web config)
/etc/mock/           (Mock chroot configs)

System Preparation

Step 1: Update & Install Packages

# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install Koji packages
sudo dnf install -y \
  koji-hub koji-web koji-utils koji-builder \
  koji-cli koji-client-lib \
  postgresql-server postgresql-contrib postgresql-devel \
  rabbitmq-server \
  httpd mod_wsgi mod_ssl \
  python3.9 python3-pip python3-devel \
  git curl wget vim net-tools htop tmux \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Install EPEL packages that may be needed
sudo dnf install -y --enablerepo=epel koji-{builder,hub,web}

# Start essential services
sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server

Step 2: Set Hostname & Network

# Set hostname
sudo hostnamectl set-hostname koji.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost koji
<your-ip>   koji.yourdomain.local koji
<your-ip>   repos.yourdomain.local repos

# Test
ping koji.yourdomain.local

Step 3: Create Directory Structure

# Koji directories
sudo mkdir -p /var/lib/koji/{packages,work,scratch,mock-root}
sudo chown -R apache:apache /var/lib/koji
sudo chmod -R 755 /var/lib/koji

# Repository directories
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}
sudo chown -R apache:apache /var/lib/repos
sudo chmod -R 755 /var/lib/repos

# Mock cache
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo chmod -R 777 /var/lib/mock

# Logs
sudo mkdir -p /var/log/koji-{hub,builder}
sudo chown apache:apache /var/log/koji-*

# Verify
df -h /var/lib

PostgreSQL Setup

Step 1: Initialize & Configure

# Initialize database cluster
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78%
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql

Step 2: Create Koji Database

# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER koji WITH PASSWORD 'koji-secure-password-here';
CREATE DATABASE koji OWNER koji;
ALTER ROLE koji CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep koji

Step 3: Load Koji Schema

# Find schema location
rpm -ql koji-hub | grep schema

# Load schema (adjust path based on Koji version)
sudo -u postgres psql koji < /usr/share/doc/koji/docs/schema.sql

# Verify tables created
sudo -u postgres psql koji -c "\dt" | head -20

RabbitMQ Setup

Step 1: Configure & Initialize

# Start RabbitMQ
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server

# Wait for startup
sleep 5

# Create Koji user and vhost
sudo rabbitmqctl add_user koji koji-mq-password-here
sudo rabbitmqctl add_vhost /koji
sudo rabbitmqctl set_permissions -p /koji koji ".*" ".*" ".*"

# Verify
sudo rabbitmqctl list_users
sudo rabbitmqctl list_vhosts

SSL Certificate Generation

Step 1: Create Certificates

# Create certificate directory
sudo mkdir -p /etc/pki/koji
cd /tmp/cert-build

# Create CA certificate
sudo openssl genrsa -out koji-ca-key.key 2048
sudo openssl req -new -x509 -days 3650 -key koji-ca-key.key \
  -out koji-ca-cert.crt \
  -subj "/CN=koji-ca/O=CasjaysDev/C=US"

# Create Koji Hub certificate
sudo openssl genrsa -out koji-hub-key.key 2048
sudo openssl req -new -key koji-hub-key.key \
  -out koji-hub.csr \
  -subj "/CN=koji.yourdomain.local/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-hub.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-hub-cert.crt

# Create client certificate (for koji CLI)
sudo openssl genrsa -out koji-client-key.key 2048
sudo openssl req -new -key koji-client-key.key \
  -out koji-client.csr \
  -subj "/CN=koji-admin/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-client.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-client-cert.crt

# Copy to Koji directory
sudo cp koji-ca-cert.crt koji-hub-cert.crt koji-hub-key.key /etc/pki/koji/
sudo cp koji-ca-cert.crt koji-client-cert.crt koji-client-key.key /etc/pki/koji/

# Permissions
sudo chmod 600 /etc/pki/koji/*-key.key
sudo chmod 644 /etc/pki/koji/*-cert.crt
sudo chown -R apache:apache /etc/pki/koji

# Verify
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt

Koji Hub Installation

Step 1: Configure Koji Hub

# Edit hub configuration
sudo vi /etc/koji-hub/hub.conf

# Key settings:
[hub]
DBName = koji
DBUser = koji
DBPassword = koji-secure-password-here
DBHost = localhost
DBPort = 5432

KojiDir = /var/lib/koji

MQHost = localhost
MQPort = 5672
MQUser = koji
MQPassword = koji-mq-password-here
MQVHost = /koji

KojiHubCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCertFile = /etc/pki/koji/koji-hub-cert.crt
KojiHubKeyFile = /etc/pki/koji/koji-hub-key.key

AuthMethod = ssl
ProxyPrincipals = koji-admin

# Task management
AutoRebuilds = true
BuildForce = false

Step 2: Apache/HTTPD Configuration

# Enable modules
sudo a2enmod wsgi
sudo a2enmod ssl

# Create Koji VirtualHost
sudo tee /etc/httpd/conf.d/koji.conf > /dev/null << 'EOF'
LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName koji.yourdomain.local
    Redirect permanent / https://koji.yourdomain.local/
</VirtualHost>

<VirtualHost *:443>
    ServerName koji.yourdomain.local
    SSLEngine on
    SSLCertificateFile /etc/pki/koji/koji-hub-cert.crt
    SSLCertificateKeyFile /etc/pki/koji/koji-hub-key.key
    SSLCACertificateFile /etc/pki/koji/koji-ca-cert.crt
    SSLVerifyClient optional
    SSLVerifyDepth 10

    WSGIScriptAlias / /usr/share/koji-web/kojiweb.wsgi
    WSGICallableObject application

    <Directory "/usr/share/koji-web/">
        AllowOverride All
        Options All
        Require all granted
    </Directory>

    ErrorLog /var/log/httpd/koji-error_log
    CustomLog /var/log/httpd/koji-access_log combined
</VirtualHost>

Listen 443
EOF

# Start Apache
sudo systemctl restart httpd
sudo systemctl enable httpd

Step 3: Start Koji Hub

# Start Koji daemon
sudo systemctl start kojid
sudo systemctl enable kojid

# Verify
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Should see: "Connected to database..."

Koji Web Installation

Step 1: Configure Koji Web

# Edit web configuration
sudo vi /etc/koji-web/web.conf

[web]
SiteName = CasjaysDev Koji
KojiHubURL = https://koji.yourdomain.local/koji
WebCertFile = /etc/pki/koji/koji-web-cert.crt
WebKeyFile = /etc/pki/koji/koji-web-key.key
WebCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCA = /etc/pki/koji/koji-ca-cert.crt

# Make self-signed certs if needed
sudo cp /etc/pki/koji/koji-hub-cert.crt /etc/pki/koji/koji-web-cert.crt
sudo cp /etc/pki/koji/koji-hub-key.key /etc/pki/koji/koji-web-key.key

Step 2: Verify Web Access

# Test HTTPS
curl -k https://koji.yourdomain.local/koji/

# Should see web interface (HTML output)

Koji Builders Configuration

Step 1: Koji Builder Configuration

# Since this is single-server, we'll run kojid locally
# It acts as both hub and builder

# Configuration is already in /etc/koji-hub/hub.conf

# Verify kojid is running
sudo systemctl status kojid

# Check logs for builder registration
tail -f /var/log/koji-hub/kojid.log

Step 2: Register Builder Hosts

# Install koji-cli to add hosts
sudo dnf install -y koji-cli

# Copy client certs
mkdir -p ~/.koji
cp /etc/pki/koji/koji-ca-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-key.key ~/.koji/

# Create koji config
cat > ~/.koji/config << 'EOF'
[koji]
server = https://koji.yourdomain.local/koji
cert = ~/.koji/koji-client-cert.crt
ca = ~/.koji/koji-ca-cert.crt
key = ~/.koji/koji-client-key.key
EOF

chmod 600 ~/.koji/config

# Add builder hosts (for documentation; single-server uses localhost)
koji add-host localhost x86_64
koji add-host localhost aarch64

# Create channels
koji add-channel x86_64-chan
koji add-channel aarch64-chan

# Assign to channels
koji edit-host localhost --channel-arches x86_64
koji edit-host localhost --channel-arches aarch64

# Verify
koji list-hosts --ready

Mock Chroot Setup

Step 1: Create Mock Configurations

# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64 (copy and modify)
sudo cp /etc/mock/el8-x86_64.cfg /etc/mock/el8-aarch64.cfg
sudo sed -i "s/el8-x86_64/el8-aarch64/g" /etc/mock/el8-aarch64.cfg
sudo sed -i "s/target_arch = 'x86_64'/target_arch = 'aarch64'/g" /etc/mock/el8-aarch64.cfg

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init

Koji Tags & Targets

Step 1: Create Tag Hierarchy

# Set up koji CLI first
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key
export KOJI_SERVER=https://koji.yourdomain.local/koji

# Create tags for AlmaLinux 8
koji add-tag el8-build
koji add-repo el8-build koji-build-repo

koji add-tag el8-release --parent el8-build

# Create targets
koji add-target el8-candidate el8-build el8-release

# Create tags for AlmaLinux 9
koji add-tag el9-build
koji add-repo el9-build koji-build-repo

koji add-tag el9-release --parent el9-build
koji add-target el9-candidate el9-build el9-release

# Create tags for Fedora 39
koji add-tag fedora39-build
koji add-repo fedora39-build koji-build-repo

koji add-tag fedora39-release --parent fedora39-build
koji add-target fedora39-candidate fedora39-build fedora39-release

# For third-party rebuilds
koji add-tag el9-rebuild-build --parent el9-build
koji add-target el9-rebuild-candidate el9-rebuild-build el9-release

# Verify
koji list-tags
koji list-targets

Step 2: Add Packages to Tags

# Import package sets for dependency resolution
koji import-comps -t el8-build /etc/comps-el8.xml
koji import-comps -t el9-build /etc/comps-el9.xml

# Add key packages
koji add-pkg el8-build kernel vim git
koji add-pkg el9-build kernel vim git

Repository Structure

Step 1: Initialize Repositories

# Create all repo directories with subdirs
for dist in el8 el9 fedora39; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      mkdir -p /var/lib/repos/${dist}/${arch}/${repo}
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# SRPM repos
for dist in el8 el9 fedora39; do
  mkdir -p /var/lib/repos/srpms/${dist}
  createrepo_c /var/lib/repos/srpms/${dist}/
done

# Verify
ls -la /var/lib/repos/el9/x86_64/

Step 2: Configure Nginx Repository Server

# Configure Nginx for repo serving
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

    ssl_certificate /etc/pki/koji/koji-hub-cert.crt;
    ssl_certificate_key /etc/pki/koji/koji-hub-key.key;

    root /var/lib/repos;

    location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx
sudo systemctl enable nginx

GitHub Integration

Step 1: Create Webhook Handler Script

# Create /usr/local/bin/koji-github-builder.sh
sudo tee /usr/local/bin/koji-github-builder.sh > /dev/null << 'SCRIPT'
#!/bin/bash

PAYLOAD=$1
REPO=$(echo $PAYLOAD | jq -r '.repository.name')
BRANCH=$(echo $PAYLOAD | jq -r '.ref' | cut -d'/' -f3)
COMMIT=$(echo $PAYLOAD | jq -r '.head_commit.id' | cut -c1-7)

SPEC_URL="git+https://github.com/rpm-devel/${REPO}.git#${COMMIT}"

# Default targets (can override per-repo)
TARGETS="${KOJI_TARGETS:-el9-candidate}"

for TARGET in $TARGETS; do
  echo "[$(date)] Building: ${REPO} @ ${COMMIT:0:7} in ${TARGET}"
  
  # Use koji from CLI
  koji build ${TARGET} ${SPEC_URL} 2>&1 | tee -a /var/log/koji-builds.log
done
SCRIPT

sudo chmod +x /usr/local/bin/koji-github-builder.sh

Step 2: Flask Webhook Receiver

# Create /usr/local/bin/koji-webhook.py
sudo tee /usr/local/bin/koji-webhook.py > /dev/null << 'PYTHON'
#!/usr/bin/env python3

from flask import Flask, request
import json
import subprocess
import logging

app = Flask(__name__)
logging.basicConfig(
    filename='/var/log/koji-webhook.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

@app.route('/webhook/github', methods=['POST'])
def github_webhook():
    payload = request.get_json()
    
    if not payload:
        return 'No payload', 400
    
    repo_name = payload['repository']['name']
    commit = payload['head_commit']['id'][:7]
    
    logging.info(f"Webhook: {repo_name} @ {commit}")
    
    try:
        result = subprocess.run(
            ['/usr/local/bin/koji-github-builder.sh', json.dumps(payload)],
            capture_output=True, text=True, timeout=60
        )
        logging.info(f"Build result: {result.stdout}")
        return 'Build submitted', 202
    except Exception as e:
        logging.error(f"Build failed: {e}")
        return 'Build failed', 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8888, debug=False)
PYTHON

sudo chmod +x /usr/local/bin/koji-webhook.py

# Install Flask
pip3 install flask

Step 3: Systemd Service

# Create systemd service for webhook
sudo tee /etc/systemd/system/koji-webhook.service > /dev/null << 'EOF'
[Unit]
Description=Koji GitHub Webhook Handler
After=network.target

[Service]
Type=simple
User=koji
ExecStart=/usr/local/bin/koji-webhook.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Create koji user if needed
sudo useradd -r koji 2>/dev/null || true

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable koji-webhook
sudo systemctl start koji-webhook

Step 4: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

Settings → Webhooks → Add webhook

Payload URL: https://koji.yourdomain.local:8888/webhook/github
Content type: application/json
Secret: your-secret-here
Events: Push events
Active: Yes

First Build & Testing

Step 1: Create Test SRPM

# Clone test package
git clone https://github.com/rpm-devel/cas /tmp/cas-build
cd /tmp/cas-build

# Build SRPM
rpmbuild -bs cas.spec

# Result in ~/rpmbuild/SRPMS/cas-*.src.rpm

Step 2: Submit Build via koji CLI

# Set environment
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

# Submit build
koji build el9-candidate ~/rpmbuild/SRPMS/cas-*.src.rpm

# Watch build
koji watch-task <task-id>

# Expected flow:
# 1. Build submitted
# 2. x86_64 build starting
# 3. x86_64 build complete
# 4. aarch64 build starting
# 5. aarch64 build complete
# 6. All complete

Step 3: Verify Build Success

# List builds
koji list-builds cas --latest=1

# Get build info
koji buildinfo <build-id>

# Verify both architectures
koji buildinfo <build-id> | grep arch

# List built packages
ls -la /var/lib/koji/packages/cas/*/1/

Step 4: Copy to Repository

# Copy RPMs to repo
sudo cp /var/lib/koji/packages/cas/*/1/x86_64/*.rpm \
  /var/lib/repos/el9/x86_64/casjay-rpms/
sudo cp /var/lib/koji/packages/cas/*/1/aarch64/*.rpm \
  /var/lib/repos/el9/aarch64/casjay-rpms/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
sudo createrepo_c /var/lib/repos/el9/aarch64/casjay-rpms/

Step 5: Test Client Installation

# On test machine (AlmaLinux 9)
sudo cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-rpms]
name=CasjaysDev RPMs
baseurl=https://repos.yourdomain.local/el9/x86_64/casjay-rpms/
enabled=1
gpgcheck=0
EOF

# Install
sudo dnf install cas

# Verify
cas --version

Production Operations

Daily Monitoring

# Check Koji hub status
koji list-hosts --ready

# Active tasks
koji list-tasks --state=active

# Failed builds
koji list-builds --state=failed --limit=10

# Database size
sudo -u postgres psql koji -c "SELECT pg_size_pretty(pg_database_size('koji'));"

# Disk usage
du -sh /var/lib/koji/packages /var/lib/repos

Automated Nightly Rebuilds

# Script: /usr/local/bin/koji-nightly-rebuilds.sh
#!/bin/bash

export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
  "nginx"
  "apache"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  koji build el9-rebuild-candidate \
    "git+https://github.com/rpm-devel/$pkg.git"
done

# Cron: 0 2 * * * /usr/local/bin/koji-nightly-rebuilds.sh

Build Promotion Workflow

# After testing, promote to release
koji move-build el9-candidate el9-release cas

# Or tag directly
koji tag-pkg el9-release cas-1.0.3-1

# Verify promotion
koji list-builds --tag=el9-release --latest=5

Repository Sync

# Script: /usr/local/bin/koji-sync-repos.sh
#!/bin/bash

# Sync all built packages to public repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    # Get recently built packages
    src=/var/lib/koji/packages
    dst=/var/lib/repos/${dist}/${arch}/casjay-rpms
    
    # Copy and regenerate
    find $src -name "*.rpm" -mtime -1 | xargs -I {} cp {} $dst/
    createrepo_c $dst
  done
done

# Cron: 0 */6 * * * /usr/local/bin/koji-sync-repos.sh

Database Backup

# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump koji | gzip > /var/backups/koji-$(date +\%Y\%m\%d).sql.gz

# Backup built packages (important!)
0 2 * * * tar czf /var/backups/koji-pkgs-$(date +\%Y\%m\%d).tar.gz /var/lib/koji/packages/

# Retention (30 days)
find /var/backups -name "koji-*.sql.gz" -mtime +30 -delete
find /var/backups -name "koji-pkgs-*.tar.gz" -mtime +30 -delete

Health Checks

# Script: /usr/local/bin/koji-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check Koji hub
if ! koji list-hosts > /dev/null 2>&1; then
  echo "ALERT: Koji hub unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: /var/lib/repos at ${USAGE}% capacity" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql koji -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: PostgreSQL unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check RabbitMQ
if ! sudo rabbitmqctl status > /dev/null 2>&1; then
  echo "ALERT: RabbitMQ unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/koji-health-check.sh

Troubleshooting

Koji Hub Not Accessible

# Check Apache
sudo systemctl status httpd
sudo apache2ctl configtest

# Check Koji daemon
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Verify SSL certs
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt

Build Failures

# Check specific build
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Test Mock chroot directly
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init

Database Connection Errors

# Check PostgreSQL
sudo systemctl status postgresql
sudo -u postgres psql -d koji -c "SELECT 1;"

# Check logs
sudo tail -50 /var/log/postgresql/*.log

# Verify configuration
sudo grep "listen_addresses" /var/lib/pgsql/data/postgresql.conf

RabbitMQ Issues

# Check RabbitMQ status
sudo systemctl status rabbitmq-server
sudo rabbitmqctl status

# Check koji user
sudo rabbitmqctl list_users | grep koji

# Verify queue
sudo rabbitmqctl list_queues -p /koji

Packages Not in Repository

# Verify build completed
koji list-builds | grep <package>

# Check package location
find /var/lib/koji/packages -name "*.rpm" | grep <package>

# Manual copy
sudo cp /var/lib/koji/packages/<pkg>/*/* /var/lib/repos/el9/x86_64/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/

Success Criteria

✅ Koji web accessible at https://koji.yourdomain.local/koji/
✅ koji list-hosts --ready shows builder online
✅ Test build submitted and completed successfully
✅ Both x86_64 and aarch64 builds successful
✅ Built packages in /var/lib/koji/packages/
✅ RPMs copied to /var/lib/repos/
✅ Repository metadata generated (repomd.xml exists)
✅ Client can install packages via dnf
✅ GitHub webhook triggers automatic builds
✅ Daily backups completing
✅ Health checks passing

Quick Reference

# Koji commands
koji build <target> <SRPM-or-git-url>
koji watch-task <task-id>
koji list-tasks --state=active
koji list-builds --latest=20
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Tag management
koji add-tag <tag-name>
koji add-target <target> <build-tag> <dest-tag>
koji list-tags
koji list-targets
koji tag-pkg <tag> <package>
koji move-build <src-tag> <dest-tag> <package>

# System commands
sudo systemctl status {httpd,kojid,postgresql,rabbitmq-server}
tail -f /var/log/koji-hub/kojid.log
tail -f /var/log/httpd/koji-access_log

# Repository
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
koji regen-repo <tag>

End of Koji Single-Server Guide

README.md · 38 B · Markdown Raw

ALBS

[#ALBS.md]

KOJI

[#KOJI.md]