# Koji Build System - Single Server Complete Guide

**Target**: Production Koji on single 128GB/32-core/8TB server  
**OS**: AlmaLinux 9 (minimal)  
**Deployment**: 3-4 weeks  
**Architecture Support**: x86_64, aarch64 (parallel builds on same server)  
**Complexity**: High (but single-server simplified)

---

## Table of Contents

1. [Architecture & Overview](#architecture--overview)
2. [Prerequisites & Planning](#prerequisites--planning)
3. [System Preparation](#system-preparation)
4. [PostgreSQL Setup](#postgresql-setup)
5. [RabbitMQ Setup](#rabbitmq-setup)
6. [SSL Certificate Generation](#ssl-certificate-generation)
7. [Koji Hub Installation](#koji-hub-installation)
8. [Koji Web Installation](#koji-web-installation)
9. [Koji Builders Configuration](#koji-builders-configuration)
10. [Mock Chroot Setup](#mock-chroot-setup)
11. [Koji Tags & Targets](#koji-tags--targets)
12. [Repository Structure](#repository-structure)
13. [GitHub Integration](#github-integration)
14. [First Build & Testing](#first-build--testing)
15. [Production Operations](#production-operations)
16. [Troubleshooting](#troubleshooting)

---

## Architecture & Overview

### Single Server Koji Setup

```
┌──────────────────────────────────────────────────────────┐
│          Single Server (128GB/32c/8TB)                   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  PostgreSQL (16GB)                              │   │
│  │  - Build metadata, packages, tasks              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  RabbitMQ (4GB)                                 │   │
│  │  - Task queue, async processing                 │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Hub (XMLRPC API)                          │   │
│  │  - Apache + mod_wsgi                            │   │
│  │  - Task orchestration                           │   │
│  │  - Package tracking                             │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Koji Web (Dashboard)                           │   │
│  │  - Package browser                              │   │
│  │  - Build history                                │   │
│  │  - Task monitoring                              │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Local Build Execution                          │   │
│  │  ┌──────────────┐  ┌──────────────┐             │   │
│  │  │ Mock x86_64  │  │ Mock aarch64  │            │   │
│  │  │ (8 parallel) │  │ (8 parallel)  │            │   │
│  │  └──────────────┘  └──────────────┘             │   │
│  │  - EL8, EL9, Fedora chroots                     │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Package Storage                                │   │
│  │  /var/lib/koji/packages/ (3TB allocated)        │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Repository Storage                             │   │
│  │  /var/lib/repos/ (5TB allocated)                │   │
│  │  - el8/9/fedora × x86_64/aarch64                │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
└──────────────────────────────────────────────────────────┘
```

### Resource Allocation

**Available**: 128GB RAM, 32 cores, 8TB disk

**Allocation**:
- PostgreSQL: 16GB RAM
- RabbitMQ: 4GB RAM
- Apache/Koji Hub: 4GB RAM
- Mock builds (simultaneous): 16GB RAM (2x 8GB)
- Kernel cache/OS: 20GB
- Reserved headroom: 52GB

**Storage**:
- OS/system: 200GB
- PostgreSQL: 100GB
- Mock cache: 500GB
- Koji packages: 3TB
- Repositories: 5TB
- Logs: 100GB
- Remaining: 100GB (headroom)

---

## Prerequisites & Planning

### Hardware Confirmation
- ✅ 128GB RAM (excellent)
- ✅ 32 cores (run 16 parallel builds)
- ✅ 8TB disk (plenty for everything)
- ✅ Network: 1Gbps

### Ports Required
- 80: HTTP (redirect to HTTPS)
- 443: HTTPS (Koji Web + Hub)
- 5432: PostgreSQL (localhost only)
- 5672: RabbitMQ (localhost only)
- 22: SSH

### DNS Setup

```
koji.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>
```

### Directory Structure

```
/var/lib/koji/
  packages/          (3TB for built packages)
  work/              (temporary build space)
  scratch/           (scratch builds)
  
/var/lib/repos/
  el8/x86_64/        (repositories)
  el8/aarch64/
  el9/x86_64/
  el9/aarch64/
  fedora39/x86_64/
  fedora39/aarch64/
  srpms/

/etc/koji-hub/       (Koji Hub config)
/etc/koji-web/       (Koji Web config)
/etc/mock/           (Mock chroot configs)
```

---

## System Preparation

### Step 1: Update & Install Packages

```bash
# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install Koji packages
sudo dnf install -y \
  koji-hub koji-web koji-utils koji-builder \
  koji-cli koji-client-lib \
  postgresql-server postgresql-contrib postgresql-devel \
  rabbitmq-server \
  httpd mod_wsgi mod_ssl \
  python3.9 python3-pip python3-devel \
  git curl wget vim net-tools htop tmux \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Install EPEL packages that may be needed
sudo dnf install -y --enablerepo=epel koji-{builder,hub,web}

# Start essential services
sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server
```

### Step 2: Set Hostname & Network

```bash
# Set hostname
sudo hostnamectl set-hostname koji.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost koji
<your-ip>   koji.yourdomain.local koji
<your-ip>   repos.yourdomain.local repos

# Test
ping koji.yourdomain.local
```

### Step 3: Create Directory Structure

```bash
# Koji directories
sudo mkdir -p /var/lib/koji/{packages,work,scratch,mock-root}
sudo chown -R apache:apache /var/lib/koji
sudo chmod -R 755 /var/lib/koji

# Repository directories
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}
sudo chown -R apache:apache /var/lib/repos
sudo chmod -R 755 /var/lib/repos

# Mock cache
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo chmod -R 777 /var/lib/mock

# Logs
sudo mkdir -p /var/log/koji-{hub,builder}
sudo chown apache:apache /var/log/koji-*

# Verify
df -h /var/lib
```

---

## PostgreSQL Setup

### Step 1: Initialize & Configure

```bash
# Initialize database cluster
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78%
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql
```

### Step 2: Create Koji Database

```bash
# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER koji WITH PASSWORD 'koji-secure-password-here';
CREATE DATABASE koji OWNER koji;
ALTER ROLE koji CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep koji
```

### Step 3: Load Koji Schema

```bash
# Find schema location
rpm -ql koji-hub | grep schema

# Load schema (adjust path based on Koji version)
sudo -u postgres psql koji < /usr/share/doc/koji/docs/schema.sql

# Verify tables created
sudo -u postgres psql koji -c "\dt" | head -20
```

---

## RabbitMQ Setup

### Step 1: Configure & Initialize

```bash
# Start RabbitMQ
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server

# Wait for startup
sleep 5

# Create Koji user and vhost
sudo rabbitmqctl add_user koji koji-mq-password-here
sudo rabbitmqctl add_vhost /koji
sudo rabbitmqctl set_permissions -p /koji koji ".*" ".*" ".*"

# Verify
sudo rabbitmqctl list_users
sudo rabbitmqctl list_vhosts
```

---

## SSL Certificate Generation

### Step 1: Create Certificates

```bash
# Create certificate directory
sudo mkdir -p /etc/pki/koji
cd /tmp/cert-build

# Create CA certificate
sudo openssl genrsa -out koji-ca-key.key 2048
sudo openssl req -new -x509 -days 3650 -key koji-ca-key.key \
  -out koji-ca-cert.crt \
  -subj "/CN=koji-ca/O=CasjaysDev/C=US"

# Create Koji Hub certificate
sudo openssl genrsa -out koji-hub-key.key 2048
sudo openssl req -new -key koji-hub-key.key \
  -out koji-hub.csr \
  -subj "/CN=koji.yourdomain.local/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-hub.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-hub-cert.crt

# Create client certificate (for koji CLI)
sudo openssl genrsa -out koji-client-key.key 2048
sudo openssl req -new -key koji-client-key.key \
  -out koji-client.csr \
  -subj "/CN=koji-admin/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
  -in koji-client.csr \
  -CA koji-ca-cert.crt -CAkey koji-ca-key.key \
  -CAcreateserial -out koji-client-cert.crt

# Copy to Koji directory
sudo cp koji-ca-cert.crt koji-hub-cert.crt koji-hub-key.key /etc/pki/koji/
sudo cp koji-ca-cert.crt koji-client-cert.crt koji-client-key.key /etc/pki/koji/

# Permissions
sudo chmod 600 /etc/pki/koji/*-key.key
sudo chmod 644 /etc/pki/koji/*-cert.crt
sudo chown -R apache:apache /etc/pki/koji

# Verify
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt
```

---

## Koji Hub Installation

### Step 1: Configure Koji Hub

```bash
# Edit hub configuration
sudo vi /etc/koji-hub/hub.conf

# Key settings:
[hub]
DBName = koji
DBUser = koji
DBPassword = koji-secure-password-here
DBHost = localhost
DBPort = 5432

KojiDir = /var/lib/koji

MQHost = localhost
MQPort = 5672
MQUser = koji
MQPassword = koji-mq-password-here
MQVHost = /koji

KojiHubCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCertFile = /etc/pki/koji/koji-hub-cert.crt
KojiHubKeyFile = /etc/pki/koji/koji-hub-key.key

AuthMethod = ssl
ProxyPrincipals = koji-admin

# Task management
AutoRebuilds = true
BuildForce = false
```

### Step 2: Apache/HTTPD Configuration

```bash
# Enable modules
sudo a2enmod wsgi
sudo a2enmod ssl

# Create Koji VirtualHost
sudo tee /etc/httpd/conf.d/koji.conf > /dev/null << 'EOF'
LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName koji.yourdomain.local
    Redirect permanent / https://koji.yourdomain.local/
</VirtualHost>

<VirtualHost *:443>
    ServerName koji.yourdomain.local
    SSLEngine on
    SSLCertificateFile /etc/pki/koji/koji-hub-cert.crt
    SSLCertificateKeyFile /etc/pki/koji/koji-hub-key.key
    SSLCACertificateFile /etc/pki/koji/koji-ca-cert.crt
    SSLVerifyClient optional
    SSLVerifyDepth 10

    WSGIScriptAlias / /usr/share/koji-web/kojiweb.wsgi
    WSGICallableObject application

    <Directory "/usr/share/koji-web/">
        AllowOverride All
        Options All
        Require all granted
    </Directory>

    ErrorLog /var/log/httpd/koji-error_log
    CustomLog /var/log/httpd/koji-access_log combined
</VirtualHost>

Listen 443
EOF

# Start Apache
sudo systemctl restart httpd
sudo systemctl enable httpd
```

### Step 3: Start Koji Hub

```bash
# Start Koji daemon
sudo systemctl start kojid
sudo systemctl enable kojid

# Verify
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Should see: "Connected to database..."
```

---

## Koji Web Installation

### Step 1: Configure Koji Web

```bash
# Edit web configuration
sudo vi /etc/koji-web/web.conf

[web]
SiteName = CasjaysDev Koji
KojiHubURL = https://koji.yourdomain.local/koji
WebCertFile = /etc/pki/koji/koji-web-cert.crt
WebKeyFile = /etc/pki/koji/koji-web-key.key
WebCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCA = /etc/pki/koji/koji-ca-cert.crt

# Make self-signed certs if needed
sudo cp /etc/pki/koji/koji-hub-cert.crt /etc/pki/koji/koji-web-cert.crt
sudo cp /etc/pki/koji/koji-hub-key.key /etc/pki/koji/koji-web-key.key
```

### Step 2: Verify Web Access

```bash
# Test HTTPS
curl -k https://koji.yourdomain.local/koji/

# Should see web interface (HTML output)
```

---

## Koji Builders Configuration

### Step 1: Koji Builder Configuration

```bash
# Since this is single-server, we'll run kojid locally
# It acts as both hub and builder

# Configuration is already in /etc/koji-hub/hub.conf

# Verify kojid is running
sudo systemctl status kojid

# Check logs for builder registration
tail -f /var/log/koji-hub/kojid.log
```

### Step 2: Register Builder Hosts

```bash
# Install koji-cli to add hosts
sudo dnf install -y koji-cli

# Copy client certs
mkdir -p ~/.koji
cp /etc/pki/koji/koji-ca-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-cert.crt ~/.koji/
cp /etc/pki/koji/koji-client-key.key ~/.koji/

# Create koji config
cat > ~/.koji/config << 'EOF'
[koji]
server = https://koji.yourdomain.local/koji
cert = ~/.koji/koji-client-cert.crt
ca = ~/.koji/koji-ca-cert.crt
key = ~/.koji/koji-client-key.key
EOF

chmod 600 ~/.koji/config

# Add builder hosts (for documentation; single-server uses localhost)
koji add-host localhost x86_64
koji add-host localhost aarch64

# Create channels
koji add-channel x86_64-chan
koji add-channel aarch64-chan

# Assign to channels
koji edit-host localhost --channel-arches x86_64
koji edit-host localhost --channel-arches aarch64

# Verify
koji list-hosts --ready
```

---

## Mock Chroot Setup

### Step 1: Create Mock Configurations

```bash
# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64 (copy and modify)
sudo cp /etc/mock/el8-x86_64.cfg /etc/mock/el8-aarch64.cfg
sudo sed -i "s/el8-x86_64/el8-aarch64/g" /etc/mock/el8-aarch64.cfg
sudo sed -i "s/target_arch = 'x86_64'/target_arch = 'aarch64'/g" /etc/mock/el8-aarch64.cfg

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init
```

---

## Koji Tags & Targets

### Step 1: Create Tag Hierarchy

```bash
# Set up koji CLI first
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key
export KOJI_SERVER=https://koji.yourdomain.local/koji

# Create tags for AlmaLinux 8
koji add-tag el8-build
koji add-repo el8-build koji-build-repo

koji add-tag el8-release --parent el8-build

# Create targets
koji add-target el8-candidate el8-build el8-release

# Create tags for AlmaLinux 9
koji add-tag el9-build
koji add-repo el9-build koji-build-repo

koji add-tag el9-release --parent el9-build
koji add-target el9-candidate el9-build el9-release

# Create tags for Fedora 39
koji add-tag fedora39-build
koji add-repo fedora39-build koji-build-repo

koji add-tag fedora39-release --parent fedora39-build
koji add-target fedora39-candidate fedora39-build fedora39-release

# For third-party rebuilds
koji add-tag el9-rebuild-build --parent el9-build
koji add-target el9-rebuild-candidate el9-rebuild-build el9-release

# Verify
koji list-tags
koji list-targets
```

### Step 2: Add Packages to Tags

```bash
# Import package sets for dependency resolution
koji import-comps -t el8-build /etc/comps-el8.xml
koji import-comps -t el9-build /etc/comps-el9.xml

# Add key packages
koji add-pkg el8-build kernel vim git
koji add-pkg el9-build kernel vim git
```

---

## Repository Structure

### Step 1: Initialize Repositories

```bash
# Create all repo directories with subdirs
for dist in el8 el9 fedora39; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      mkdir -p /var/lib/repos/${dist}/${arch}/${repo}
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# SRPM repos
for dist in el8 el9 fedora39; do
  mkdir -p /var/lib/repos/srpms/${dist}
  createrepo_c /var/lib/repos/srpms/${dist}/
done

# Verify
ls -la /var/lib/repos/el9/x86_64/
```

### Step 2: Configure Nginx Repository Server

```bash
# Configure Nginx for repo serving
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

    ssl_certificate /etc/pki/koji/koji-hub-cert.crt;
    ssl_certificate_key /etc/pki/koji/koji-hub-key.key;

    root /var/lib/repos;

    location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx
sudo systemctl enable nginx
```

---

## GitHub Integration

### Step 1: Create Webhook Handler Script

```bash
# Create /usr/local/bin/koji-github-builder.sh
sudo tee /usr/local/bin/koji-github-builder.sh > /dev/null << 'SCRIPT'
#!/bin/bash

PAYLOAD=$1
REPO=$(echo $PAYLOAD | jq -r '.repository.name')
BRANCH=$(echo $PAYLOAD | jq -r '.ref' | cut -d'/' -f3)
COMMIT=$(echo $PAYLOAD | jq -r '.head_commit.id' | cut -c1-7)

SPEC_URL="git+https://github.com/rpm-devel/${REPO}.git#${COMMIT}"

# Default targets (can override per-repo)
TARGETS="${KOJI_TARGETS:-el9-candidate}"

for TARGET in $TARGETS; do
  echo "[$(date)] Building: ${REPO} @ ${COMMIT:0:7} in ${TARGET}"
  
  # Use koji from CLI
  koji build ${TARGET} ${SPEC_URL} 2>&1 | tee -a /var/log/koji-builds.log
done
SCRIPT

sudo chmod +x /usr/local/bin/koji-github-builder.sh
```

### Step 2: Flask Webhook Receiver

```bash
# Create /usr/local/bin/koji-webhook.py
sudo tee /usr/local/bin/koji-webhook.py > /dev/null << 'PYTHON'
#!/usr/bin/env python3

from flask import Flask, request
import json
import subprocess
import logging

app = Flask(__name__)
logging.basicConfig(
    filename='/var/log/koji-webhook.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

@app.route('/webhook/github', methods=['POST'])
def github_webhook():
    payload = request.get_json()
    
    if not payload:
        return 'No payload', 400
    
    repo_name = payload['repository']['name']
    commit = payload['head_commit']['id'][:7]
    
    logging.info(f"Webhook: {repo_name} @ {commit}")
    
    try:
        result = subprocess.run(
            ['/usr/local/bin/koji-github-builder.sh', json.dumps(payload)],
            capture_output=True, text=True, timeout=60
        )
        logging.info(f"Build result: {result.stdout}")
        return 'Build submitted', 202
    except Exception as e:
        logging.error(f"Build failed: {e}")
        return 'Build failed', 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8888, debug=False)
PYTHON

sudo chmod +x /usr/local/bin/koji-webhook.py

# Install Flask
pip3 install flask
```

### Step 3: Systemd Service

```bash
# Create systemd service for webhook
sudo tee /etc/systemd/system/koji-webhook.service > /dev/null << 'EOF'
[Unit]
Description=Koji GitHub Webhook Handler
After=network.target

[Service]
Type=simple
User=koji
ExecStart=/usr/local/bin/koji-webhook.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Create koji user if needed
sudo useradd -r koji 2>/dev/null || true

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable koji-webhook
sudo systemctl start koji-webhook
```

### Step 4: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

```
Settings → Webhooks → Add webhook

Payload URL: https://koji.yourdomain.local:8888/webhook/github
Content type: application/json
Secret: your-secret-here
Events: Push events
Active: Yes
```

---

## First Build & Testing

### Step 1: Create Test SRPM

```bash
# Clone test package
git clone https://github.com/rpm-devel/cas /tmp/cas-build
cd /tmp/cas-build

# Build SRPM
rpmbuild -bs cas.spec

# Result in ~/rpmbuild/SRPMS/cas-*.src.rpm
```

### Step 2: Submit Build via koji CLI

```bash
# Set environment
export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

# Submit build
koji build el9-candidate ~/rpmbuild/SRPMS/cas-*.src.rpm

# Watch build
koji watch-task <task-id>

# Expected flow:
# 1. Build submitted
# 2. x86_64 build starting
# 3. x86_64 build complete
# 4. aarch64 build starting
# 5. aarch64 build complete
# 6. All complete
```

### Step 3: Verify Build Success

```bash
# List builds
koji list-builds cas --latest=1

# Get build info
koji buildinfo <build-id>

# Verify both architectures
koji buildinfo <build-id> | grep arch

# List built packages
ls -la /var/lib/koji/packages/cas/*/1/
```

### Step 4: Copy to Repository

```bash
# Copy RPMs to repo
sudo cp /var/lib/koji/packages/cas/*/1/x86_64/*.rpm \
  /var/lib/repos/el9/x86_64/casjay-rpms/
sudo cp /var/lib/koji/packages/cas/*/1/aarch64/*.rpm \
  /var/lib/repos/el9/aarch64/casjay-rpms/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
sudo createrepo_c /var/lib/repos/el9/aarch64/casjay-rpms/
```

### Step 5: Test Client Installation

```bash
# On test machine (AlmaLinux 9)
sudo cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-rpms]
name=CasjaysDev RPMs
baseurl=https://repos.yourdomain.local/el9/x86_64/casjay-rpms/
enabled=1
gpgcheck=0
EOF

# Install
sudo dnf install cas

# Verify
cas --version
```

---

## Production Operations

### Daily Monitoring

```bash
# Check Koji hub status
koji list-hosts --ready

# Active tasks
koji list-tasks --state=active

# Failed builds
koji list-builds --state=failed --limit=10

# Database size
sudo -u postgres psql koji -c "SELECT pg_size_pretty(pg_database_size('koji'));"

# Disk usage
du -sh /var/lib/koji/packages /var/lib/repos
```

### Automated Nightly Rebuilds

```bash
# Script: /usr/local/bin/koji-nightly-rebuilds.sh
#!/bin/bash

export KOJI_CERT_FILE=~/.koji/koji-client-cert.crt
export KOJI_CA_CERT=~/.koji/koji-ca-cert.crt
export KOJI_KEY_FILE=~/.koji/koji-client-key.key

PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
  "nginx"
  "apache"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  koji build el9-rebuild-candidate \
    "git+https://github.com/rpm-devel/$pkg.git"
done

# Cron: 0 2 * * * /usr/local/bin/koji-nightly-rebuilds.sh
```

### Build Promotion Workflow

```bash
# After testing, promote to release
koji move-build el9-candidate el9-release cas

# Or tag directly
koji tag-pkg el9-release cas-1.0.3-1

# Verify promotion
koji list-builds --tag=el9-release --latest=5
```

### Repository Sync

```bash
# Script: /usr/local/bin/koji-sync-repos.sh
#!/bin/bash

# Sync all built packages to public repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    # Get recently built packages
    src=/var/lib/koji/packages
    dst=/var/lib/repos/${dist}/${arch}/casjay-rpms
    
    # Copy and regenerate
    find $src -name "*.rpm" -mtime -1 | xargs -I {} cp {} $dst/
    createrepo_c $dst
  done
done

# Cron: 0 */6 * * * /usr/local/bin/koji-sync-repos.sh
```

### Database Backup

```bash
# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump koji | gzip > /var/backups/koji-$(date +\%Y\%m\%d).sql.gz

# Backup built packages (important!)
0 2 * * * tar czf /var/backups/koji-pkgs-$(date +\%Y\%m\%d).tar.gz /var/lib/koji/packages/

# Retention (30 days)
find /var/backups -name "koji-*.sql.gz" -mtime +30 -delete
find /var/backups -name "koji-pkgs-*.tar.gz" -mtime +30 -delete
```

### Health Checks

```bash
# Script: /usr/local/bin/koji-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check Koji hub
if ! koji list-hosts > /dev/null 2>&1; then
  echo "ALERT: Koji hub unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: /var/lib/repos at ${USAGE}% capacity" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql koji -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: PostgreSQL unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Check RabbitMQ
if ! sudo rabbitmqctl status > /dev/null 2>&1; then
  echo "ALERT: RabbitMQ unreachable" | mail -s "Koji Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/koji-health-check.sh
```

---

## Troubleshooting

### Koji Hub Not Accessible

```bash
# Check Apache
sudo systemctl status httpd
sudo apache2ctl configtest

# Check Koji daemon
sudo systemctl status kojid
tail -f /var/log/koji-hub/kojid.log

# Verify SSL certs
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
  /etc/pki/koji/koji-hub-cert.crt
```

### Build Failures

```bash
# Check specific build
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Test Mock chroot directly
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init
```

### Database Connection Errors

```bash
# Check PostgreSQL
sudo systemctl status postgresql
sudo -u postgres psql -d koji -c "SELECT 1;"

# Check logs
sudo tail -50 /var/log/postgresql/*.log

# Verify configuration
sudo grep "listen_addresses" /var/lib/pgsql/data/postgresql.conf
```

### RabbitMQ Issues

```bash
# Check RabbitMQ status
sudo systemctl status rabbitmq-server
sudo rabbitmqctl status

# Check koji user
sudo rabbitmqctl list_users | grep koji

# Verify queue
sudo rabbitmqctl list_queues -p /koji
```

### Packages Not in Repository

```bash
# Verify build completed
koji list-builds | grep <package>

# Check package location
find /var/lib/koji/packages -name "*.rpm" | grep <package>

# Manual copy
sudo cp /var/lib/koji/packages/<pkg>/*/* /var/lib/repos/el9/x86_64/

# Regenerate metadata
sudo createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
```

---

## Success Criteria

✅ Koji web accessible at https://koji.yourdomain.local/koji/  
✅ koji list-hosts --ready shows builder online  
✅ Test build submitted and completed successfully  
✅ Both x86_64 and aarch64 builds successful  
✅ Built packages in /var/lib/koji/packages/  
✅ RPMs copied to /var/lib/repos/  
✅ Repository metadata generated (repomd.xml exists)  
✅ Client can install packages via dnf  
✅ GitHub webhook triggers automatic builds  
✅ Daily backups completing  
✅ Health checks passing  

---

## Quick Reference

```bash
# Koji commands
koji build <target> <SRPM-or-git-url>
koji watch-task <task-id>
koji list-tasks --state=active
koji list-builds --latest=20
koji buildinfo <build-id>
koji taskinfo <task-id>
koji getlogs <task-id>

# Tag management
koji add-tag <tag-name>
koji add-target <target> <build-tag> <dest-tag>
koji list-tags
koji list-targets
koji tag-pkg <tag> <package>
koji move-build <src-tag> <dest-tag> <package>

# System commands
sudo systemctl status {httpd,kojid,postgresql,rabbitmq-server}
tail -f /var/log/koji-hub/kojid.log
tail -f /var/log/httpd/koji-access_log

# Repository
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/
koji regen-repo <tag>
```

---

End of Koji Single-Server Guide