# Koji Build System - Complete Installation & Configuration Guide
**Target**: Production-grade Koji deployment for rpm-devel multi-distro builds
**OS**: AlmaLinux 9
**Duration**: 3-4 weeks to full production
**Architecture Support**: x86_64, aarch64, s390x, ppc64le
**Complexity**: High (distributed, powerful, flexible)
---
## Table of Contents
1. [Architecture & Components](#architecture--components)
2. [Prerequisites & Planning](#prerequisites--planning)
3. [Infrastructure Preparation](#infrastructure-preparation)
4. [Koji Hub Installation](#koji-hub-installation)
5. [Koji Builders Setup](#koji-builders-setup)
6. [Repository Structure](#repository-structure)
7. [Koji Tags & Targets](#koji-tags--targets)
8. [Client Configuration](#client-configuration)
9. [GitHub Integration](#github-integration)
10. [First Build & Testing](#first-build--testing)
11. [Production Operations](#production-operations)
12. [Performance Tuning & Scaling](#performance-tuning--scaling)
13. [Troubleshooting](#troubleshooting)
---
## Architecture & Components
### Koji System Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Koji Hub (Central) │
│ ┌────────────────┐ ┌─────────────┐ ┌──────────────┐ │
│ │ PostgreSQL │ │RabbitMQ/MQ │ │ Koji Hub │ │
│ │ (metadata) │ │ (task queue)│ │ (manager) │ │
│ └────────────────┘ └─────────────┘ └──────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Koji Web (Dashboard/REST API) │ │
│ │ Nginx/Apache with SSL │ │
│ └──────────────────────────────────────────────────┘ │
└──────────────┬──────────────────────────────────────────┘
│
┌──────────┴──────────┐
│ │
┌───▼─────────────┐ ┌───▼──────────────┐
│ Koji Builder │ │ Koji Builder │
│ x86_64 (8c/16G)│ │ aarch64 (8c/16G)│
│ │ │ │
│ Mock chroots: │ │ Mock chroots: │
│ - el8-x86_64 │ │ - el8-aarch64 │
│ - el9-x86_64 │ │ - el9-aarch64 │
│ - f39-x86_64 │ │ - f39-aarch64 │
│ │ │ │
│ Build repos: │ │ Build repos: │
│ - Official OS │ │ - Official OS │
│ - casjay-* │ │ - casjay-* │
│ - Third-party │ │ - Third-party │
└─────────────────┘ └──────────────────┘
│ │
└──────────┬──────────┘
│
┌──────────▼──────────┐
│ createrepo_c │
│ (Metadata Regen) │
└──────────┬──────────┘
│
┌──────────▼──────────────────────┐
│ Repository Storage │
│ (/var/www/repos/) │
│ - el8/x86_64, el8/aarch64 │
│ - el9/x86_64, el9/aarch64 │
│ - fedora39/x86_64, ... │
└──────────┬──────────────────────┘
│
┌──────────▼──────────┐
│ Nginx Web Server │
│ (Public Access) │
└─────────────────────┘
```
### Components
**Koji Hub** (Central orchestration)
- XML-RPC API for build submission
- Task scheduling & delegation
- Package tracking & versioning
- Build history & management
- Tag/repo management
**PostgreSQL** (Metadata database)
- Build metadata, tasks, packages
- User management & permissions
- Version history, build logs
**RabbitMQ** (Message broker)
- Asynchronous task queue
- Hub → Builder communication
- Build notifications
**Koji Builders** (2+, one per arch)
- Mock-based build environment
- Automatic architecture detection
- Parallel build execution
- Artifact upload to central storage
**Mock** (Build isolation)
- Chroot-based build environment
- All distro/arch combinations
- Dependency resolution
- Clean builds
**Koji Web** (Dashboard)
- Web UI for monitoring
- REST API for integrations
- Build history browser
- Package search
**Nginx** (Repository server)
- Serves built RPMs to clients
- Repository metadata hosting
- Multi-distro/arch support
---
## Prerequisites & Planning
### Hardware Requirements
**Koji Hub**:
- CPU: 8 cores (Intel/AMD)
- RAM: 16GB minimum (32GB recommended)
- Storage: 500GB SSD
- Network: 1Gbps
**PostgreSQL** (can be on Hub):
- CPU: 4 cores
- RAM: 8GB
- Storage: 200GB SSD
**Koji Builder x86_64**:
- CPU: 8 cores
- RAM: 16GB
- Storage: 300GB SSD
- Network: 1Gbps
**Koji Builder aarch64**:
- CPU: 8 cores ARM (Graviton, Ampere, RPI5)
- RAM: 16GB
- Storage: 300GB SSD
- Network: 1Gbps
**Repository Storage**:
- Capacity: 1TB minimum (shared NFS or local)
- SSD recommended for performance
- Growth: 10-50GB/month
**Total**: ~1.5TB storage, 32-40 cores, 40-48GB RAM
### Network Planning
**Static IPs Required**:
- koji-hub.yourdomain.local (10.x.x.10)
- koji-builder-x86.yourdomain.local (10.x.x.20)
- koji-builder-arm.yourdomain.local (10.x.x.30)
**DNS Setup**:
```
koji-hub.yourdomain.local A 10.x.x.10
koji-builder-x86.yourdomain.local A 10.x.x.20
koji-builder-arm.yourdomain.local A 10.x.x.30
repos.yourdomain.local A 10.x.x.10 (or separate)
```
**Firewall Ports**:
- Hub: 443 (HTTPS), 80 (HTTP redirect)
- PostgreSQL: 5432 (internal only)
- RabbitMQ: 5672 (internal only)
- Builders: 22 (SSH), outbound to Hub
- Repos: 80/443 (public)
### Git Repository Structure
All specs in dedicated repos:
```
github.com/rpm-devel/cas/
cas.spec
Makefile or build script
github.com/rpm-devel/dockloom/
dockloom.spec
(65+ other package repos)
```
Specs should contain remote URLs for sources, not committed tarballs.
---
## Infrastructure Preparation
### Step 1: Provision AlmaLinux 9 Servers
```bash
# On each server (hub, builder-x86, builder-arm):
# Update system
dnf update -y
# Install base packages
dnf groupinstall -y "Development Tools"
dnf install -y \
git curl wget vim net-tools htop tmux \
python3.9 python3-pip python3-devel \
libffi-devel openssl-devel \
postgresql-client nfs-utils
# On Hub only, add:
dnf install -y \
postgresql-server postgresql-contrib postgresql-devel \
rabbitmq-server \
httpd mod_wsgi mod_ssl \
koji-hub koji-web koji-utils \
createrepo_c nginx \
epel-release # For some Koji packages
```
### Step 2: Network Configuration
```bash
# Set static hostnames
hostnamectl set-hostname koji-hub.yourdomain.local # Hub
hostnamectl set-hostname koji-builder-x86.yourdomain.local # x86
hostnamectl set-hostname koji-builder-arm.yourdomain.local # ARM
# Edit /etc/hosts on all servers
sudo vi /etc/hosts
# Add to all /etc/hosts:
10.x.x.10 koji-hub.yourdomain.local koji-hub
10.x.x.20 koji-builder-x86.yourdomain.local koji-builder-x86
10.x.x.30 koji-builder-arm.yourdomain.local koji-builder-arm
# Test connectivity
ping koji-hub.yourdomain.local
ping koji-builder-x86.yourdomain.local
ping koji-builder-arm.yourdomain.local
```
### Step 3: SSH Key Setup
```bash
# Create koji user on all servers
sudo useradd -r -m koji
# On hub, create SSH key
sudo -u koji ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
# Copy to builders
sudo -u koji ssh-copy-id -i ~/.ssh/id_rsa.pub koji@koji-builder-x86
sudo -u koji ssh-copy-id -i ~/.ssh/id_rsa.pub koji@koji-builder-arm
# Test passwordless SSH
sudo -u koji ssh koji@koji-builder-x86 "echo 'Connected'"
```
### Step 4: Firewall Configuration
```bash
# On all servers
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-service=http
# On hub, restrict internal services to builders only
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.x.x.0/24" port protocol="tcp" port="5432" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.x.x.0/24" port protocol="tcp" port="5672" accept'
sudo firewall-cmd --reload
```
---
## Koji Hub Installation
### Step 1: PostgreSQL Installation & Configuration
```bash
# Install & initialize
sudo dnf install -y postgresql-server postgresql-contrib postgresql-devel
sudo /usr/bin/postgresql-setup initdb
# Configure for Koji
sudo vi /var/lib/pgsql/data/postgresql.conf
# Key settings:
listen_addresses = '*'
shared_buffers = 4GB # 25% of 16GB RAM
effective_cache_size = 12GB # 75%
work_mem = 32MB
maintenance_work_mem = 2GB
max_connections = 500
random_page_cost = 1.1 # For SSD
effective_io_concurrency = 200
# Authentication
sudo vi /var/lib/pgsql/data/pg_hba.conf
# Add (after local lines):
host all all 10.0.0.0/8 md5
host all all 127.0.0.1/32 md5
# Start PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql
# Verify
sudo -u postgres psql -c "\l"
```
### Step 2: Create Koji Database
```bash
# Create database and users
sudo -u postgres psql << 'EOF'
CREATE USER koji WITH PASSWORD 'koji-db-password-change-this';
CREATE DATABASE koji OWNER koji;
GRANT ALL ON DATABASE koji TO koji;
-- Allow koji user to create tables
ALTER ROLE koji CREATEDB;
EOF
# Verify
sudo -u postgres psql -l | grep koji
```
### Step 3: RabbitMQ Installation & Configuration
```bash
# Install
sudo dnf install -y rabbitmq-server
# Configure for Koji
sudo systemctl start rabbitmq-server
sudo systemctl enable rabbitmq-server
# Add Koji user
sudo rabbitmqctl add_user koji koji-mq-password-change-this
sudo rabbitmqctl add_vhost koji
sudo rabbitmqctl set_permissions -p koji koji ".*" ".*" ".*"
# Verify
sudo rabbitmqctl list_users
sudo rabbitmqctl list_vhosts
```
### Step 4: SSL Certificate Generation
```bash
# Create certificate directory
sudo mkdir -p /etc/pki/koji
cd /etc/pki/koji
# Create CA certificate
sudo openssl genrsa -out koji-ca-key.key 2048
sudo openssl req -new -x509 -days 3650 -key koji-ca-key.key \
-out koji-ca-cert.crt \
-subj "/CN=koji-ca/O=CasjaysDev/C=US"
# Create Koji Hub certificate
sudo openssl genrsa -out koji-hub-key.key 2048
sudo openssl req -new -key koji-hub-key.key \
-out koji-hub.csr \
-subj "/CN=koji-hub.yourdomain.local/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
-in koji-hub.csr \
-CA koji-ca-cert.crt -CAkey koji-ca-key.key \
-CAcreateserial -out koji-hub-cert.crt
# Create builder certificates
for builder in x86 arm; do
sudo openssl genrsa -out koji-builder-${builder}-key.key 2048
sudo openssl req -new -key koji-builder-${builder}-key.key \
-out koji-builder-${builder}.csr \
-subj "/CN=koji-builder-${builder}.yourdomain.local/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
-in koji-builder-${builder}.csr \
-CA koji-ca-cert.crt -CAkey koji-ca-key.key \
-CAcreateserial -out koji-builder-${builder}-cert.crt
done
# Create client certificate
sudo openssl genrsa -out koji-client-key.key 2048
sudo openssl req -new -key koji-client-key.key \
-out koji-client.csr \
-subj "/CN=koji-admin/O=CasjaysDev/C=US"
sudo openssl x509 -req -days 365 \
-in koji-client.csr \
-CA koji-ca-cert.crt -CAkey koji-ca-key.key \
-CAcreateserial -out koji-client-cert.crt
# Permissions
sudo chmod 600 *-key.key
sudo chown -R apache:apache *-cert.crt koji-web*
```
### Step 5: Koji Hub Configuration
```bash
# Install Koji Hub
sudo dnf install -y koji-hub koji-web koji-utils createrepo_c
# Configure hub
sudo vi /etc/koji-hub/hub.conf
[hub]
DBName = koji
DBUser = koji
DBPassword = koji-db-password-change-this
DBHost = localhost
DBPort = 5432
KojiDir = /var/lib/koji
MQHost = localhost
MQPort = 5672
MQUser = koji
MQPassword = koji-mq-password-change-this
MQVHost = /koji
KojiHubCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCertFile = /etc/pki/koji/koji-hub-cert.crt
KojiHubKeyFile = /etc/pki/koji/koji-hub-key.key
AuthMethod = ssl
ProxyPrincipals = koji-admin
[policy]
build = action :: allow
default = deny
```
### Step 6: Initialize Koji Database
```bash
# Load schema
sudo -u postgres psql koji < /usr/share/doc/koji/docs/schema.sql
# Verify tables created
sudo -u postgres psql koji -c "\dt"
```
### Step 7: Koji Web Configuration
```bash
# Configure web
sudo vi /etc/koji-web/web.conf
[web]
SiteName = CasjaysDev Koji Build System
KojiHubURL = https://koji-hub.yourdomain.local/koji
WebCertFile = /etc/pki/koji/koji-web-cert.crt
WebKeyFile = /etc/pki/koji/koji-web-key.key
WebCA = /etc/pki/koji/koji-ca-cert.crt
KojiHubCA = /etc/pki/koji/koji-ca-cert.crt
```
### Step 8: Apache/HTTPD Configuration
```bash
# Enable modules
sudo a2enmod wsgi
sudo a2enmod ssl
# Create Koji VirtualHost
sudo vi /etc/httpd/conf.d/koji.conf
LoadModule wsgi_module modules/mod_wsgi.so
ServerName koji-hub.yourdomain.local
SSLEngine on
SSLCertificateFile /etc/pki/koji/koji-hub-cert.crt
SSLCertificateKeyFile /etc/pki/koji/koji-hub-key.key
SSLCACertificateFile /etc/pki/koji/koji-ca-cert.crt
SSLVerifyClient optional
SSLVerifyDepth 10
WSGIScriptAlias / /usr/share/koji-web/kojiweb.wsgi
WSGICallableObject application
AllowOverride All
Options All
ErrorLog /var/log/httpd/koji-error_log
CustomLog /var/log/httpd/koji-access_log combined
ServerName koji-hub.yourdomain.local
Redirect permanent / https://koji-hub.yourdomain.local/
```
### Step 9: Start Koji Hub Services
```bash
# Create directories
sudo mkdir -p /var/lib/koji/{packages,work,scratch}
sudo chown -R apache:apache /var/lib/koji
# Start services
sudo systemctl start kojid
sudo systemctl start httpd
sudo systemctl enable kojid
sudo systemctl enable httpd
# Verify
sudo systemctl status kojid
sudo systemctl status httpd
# Check logs
tail -f /var/log/koji-hub/kojid.log
```
---
## Koji Builders Setup
### Step 1: Builder Installation
```bash
# On each builder (x86 and ARM):
# Install koji-builder
sudo dnf install -y koji-builder koji-utils mock rpm-build
# Create koji user
sudo useradd -r koji # (may already exist)
# Create working directories
sudo mkdir -p /mnt/koji/{packages,work,scratch,mock}
sudo chown -R koji:koji /mnt/koji
```
### Step 2: Copy SSL Certificates
```bash
# On hub, copy certs to builders
for builder in koji-builder-x86 koji-builder-arm; do
scp /etc/pki/koji/koji-ca-cert.crt koji@${builder}:/etc/pki/koji/
scp /etc/pki/koji/koji-builder-${ARCH}-cert.crt koji@${builder}:/etc/pki/koji/
scp /etc/pki/koji/koji-builder-${ARCH}-key.key koji@${builder}:/etc/pki/koji/
done
# On each builder, set permissions
sudo chown koji:koji /etc/pki/koji/*.crt /etc/pki/koji/*.key
sudo chmod 600 /etc/pki/koji/*.key
```
### Step 3: Builder Configuration
```bash
# On each builder
sudo vi /etc/kojid/kojid.conf
[kojid]
server = https://koji-hub.yourdomain.local/koji
user = koji-builder-x86.yourdomain.local # Change for ARM
password = # Empty, using cert auth
ca = /etc/pki/koji/koji-ca-cert.crt
cert = /etc/pki/koji/koji-builder-x86-cert.crt
privkey = /etc/pki/koji/koji-builder-x86-key.key
topdir = /mnt/koji
workdir = /mnt/koji/work
mockdir = /mnt/koji/mock
maxjobs = 4
```
### Step 4: Mock Chroot Setup
```bash
# Create mock configs for all distro/arch combos
# EL8 x86_64
sudo vi /etc/mock/el8-x86_64.cfg
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'
config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d
[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""
# Repeat for el8-aarch64, el9-x86_64, el9-aarch64, etc.
# Test mock chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
```
### Step 5: Start Builder Daemon
```bash
# On each builder
sudo systemctl start kojid
sudo systemctl enable kojid
# Verify connection to hub
sudo systemctl status kojid
sudo tail -f /var/log/kojid.log
# Should see: "Connected to hub..."
```
### Step 6: Register Builders with Koji Hub
```bash
# On hub, install koji CLI
sudo dnf install -y koji
# Add builder hosts
koji add-host koji-builder-x86.yourdomain.local x86_64
koji add-host koji-builder-arm.yourdomain.local aarch64
# Create channels
koji add-channel x86_64-chan
koji add-channel aarch64-chan
# Assign builders to channels
koji edit-host koji-builder-x86.yourdomain.local --channel-arches x86_64
koji edit-host koji-builder-arm.yourdomain.local --channel-arches aarch64
# Verify builders are ready
koji list-hosts --ready
```
---
## Repository Structure
### Step 1: Create Directory Structure
```bash
# On hub (or NFS share)
sudo mkdir -p /var/www/repos/{el8,el9,el10,fedora39,fedora40}/x86_64
sudo mkdir -p /var/www/repos/{el8,el9,el10,fedora39,fedora40}/aarch64
sudo mkdir -p /var/www/repos/srpms/{el8,el9,fedora39}
# Create subdirs for each repo (baseos, appstream, etc.)
for dist in el8 el9; do
for arch in x86_64 aarch64; do
for repo in baseos appstream extras crb casjay-rpms casjay-extras casjay-addons debug sources; do
sudo mkdir -p /var/www/repos/${dist}/${arch}/${repo}
done
done
done
# Permissions
sudo chown -R apache:apache /var/www/repos
sudo chmod -R 755 /var/www/repos
```
### Step 2: Koji Tag Creation
```bash
# Create build tags
koji add-tag el8-build
koji add-tag el9-build
koji add-tag fedora39-build
# Create destination tags
koji add-tag el8-release --parent el8-build
koji add-tag el9-release --parent el9-build
# Add packages to build tag (repos it pulls from during build)
# Add official repos, casjay repos, third-party repos
# Create build targets
koji add-target el8-candidate el8-build el8-release
koji add-target el9-candidate el9-build el9-release
# Verify
koji list-tags
koji list-targets
```
### Step 3: Nginx Repository Server
```bash
# Install Nginx
sudo dnf install -y nginx
# Configure
sudo vi /etc/nginx/conf.d/repos.conf
server {
listen 80;
server_name repos.yourdomain.local;
location / {
root /var/www/repos;
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
}
location ~ \.repo$ {
alias /var/www/repos$request_uri;
}
}
# Start Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx
# Test
curl http://repos.yourdomain.local/
```
---
## Koji Tags & Targets
### Step 1: Tag Hierarchy Setup
```bash
# Tag structure for multi-distro builds
# AlmaLinux 8
koji add-tag el8-build
koji add-repo el8-build koji-build-repo
koji add-tag el8-release --parent el8-build
koji add-target el8-candidate el8-build el8-release
# AlmaLinux 9
koji add-tag el9-build
koji add-repo el9-build koji-build-repo
koji add-tag el9-release --parent el9-build
koji add-target el9-candidate el9-build el9-release
# Fedora 39
koji add-tag fedora39-build
koji add-repo fedora39-build koji-build-repo
koji add-tag fedora39-release --parent fedora39-build
koji add-target fedora39-candidate fedora39-build fedora39-release
# Third-party rebuild tags
koji add-tag el9-rebuild-build
koji add-target el9-rebuild-candidate el9-rebuild-build el9-release
```
### Step 2: Add Official Repos to Tags
```bash
# For each tag, add package set (what packages/repos build against)
koji add-pkg el8-build kernel vim git
koji add-pkg el9-build kernel vim git
# Import official packages into tags
koji import-comps -t el8-build /etc/comps-el8.xml
koji import-comps -t el9-build /etc/comps-el9.xml
```
### Step 3: User Permissions
```bash
# Add users
koji add-user your-username
koji grant-permission build your-username
koji grant-permission repo your-username
koji grant-permission admin koji-admin
# Verify
koji list-users
koji list-permissions
```
---
## Client Configuration
### Step 1: Create casjay-release Package
```bash
# Create spec file
mkdir -p /tmp/casjay-release/{SPECS,SOURCES}
cat > /tmp/casjay-release/SPECS/casjay-release.spec << 'EOF'
Name: casjay-release
Version: 1.0
Release: 1%{?dist}
Summary: CasjaysDev Repository Configuration
License: WTFPL
Source0: almalinux.9.repo
Source1: rockylinux.9.repo
Source2: RPM-GPG-KEY-casjay
%description
Repository configuration for CasjaysDev packages
%install
install -D -m 644 %{SOURCE0} %{buildroot}/etc/yum.repos.d/almalinux.9.repo
install -D -m 644 %{SOURCE1} %{buildroot}/etc/yum.repos.d/rockylinux.9.repo
install -D -m 644 %{SOURCE2} %{buildroot}/etc/pki/rpm-gpg/RPM-GPG-KEY-casjay
%files
/etc/yum.repos.d/almalinux.9.repo
/etc/yum.repos.d/rockylinux.9.repo
/etc/pki/rpm-gpg/RPM-GPG-KEY-casjay
%changelog
* $(date +'%a %b %d %Y') Jason - 1.0-1
- Initial release
EOF
# Build SRPM
cd /tmp/casjay-release
rpmbuild -bs SPECS/casjay-release.spec
# Result: ~/rpmbuild/SRPMS/casjay-release-1.0-1.src.rpm
```
### Step 2: Configure Repository URLs
```bash
# Edit repo files (from github.com/rpm-devel/casjay-release)
# Update baseurl/mirrorlist to point to your Koji/Nginx:
[casjay-rpms]
name=CasjaysDev RPMs
baseurl=http://repos.yourdomain.local/el9/x86_64/casjay-rpms/
enabled=1
gpgcheck=1
gpgkey=http://repos.yourdomain.local/RPM-GPG-KEY-casjay
[casjay-extras]
name=CasjaysDev Extras
baseurl=http://repos.yourdomain.local/el9/x86_64/casjay-extras/
enabled=1
gpgcheck=1
gpgkey=http://repos.yourdomain.local/RPM-GPG-KEY-casjay
```
### Step 3: Build and Publish casjay-release
```bash
# Build casjay-release package
koji build el9-candidate ~/rpmbuild/SRPMS/casjay-release-1.0-1.src.rpm
# Monitor
koji watch-task
# Verify both archs built
koji list-builds casjay-release --latest=1
# Promote to release
koji move-build el9-candidate el9-release casjay-release
# Copy to repo
cp /var/lib/koji/packages/casjay-release/1.0/1/noarch/*.rpm \
/var/www/repos/el9/x86_64/casjay-rpms/
# Regenerate repo metadata
createrepo_c /var/www/repos/el9/x86_64/casjay-rpms/
```
### Step 4: Test Client Installation
```bash
# On test client
dnf install casjay-release-1.0-1.noarch.rpm
# Verify repos available
dnf repolist | grep casjay
# Install from repo
dnf install
```
---
## GitHub Integration
### Step 1: Webhook Receiver Script
```bash
# Create webhook handler
cat > /usr/local/bin/koji-github-builder.sh << 'EOF'
#!/bin/bash
PAYLOAD=$1
REPO=$(echo $PAYLOAD | jq -r '.repository.name')
BRANCH=$(echo $PAYLOAD | jq -r '.ref' | cut -d'/' -f3)
COMMIT=$(echo $PAYLOAD | jq -r '.head_commit.id')
SPEC_URL="git+https://github.com/rpm-devel/${REPO}.git#${COMMIT}"
# Default targets (can be overridden per-repo)
TARGETS="${KOJI_TARGETS:-el9-candidate}"
for TARGET in $TARGETS; do
echo "Building: ${REPO} @ ${COMMIT} in ${TARGET}"
koji build ${TARGET} ${SPEC_URL}
done
EOF
chmod +x /usr/local/bin/koji-github-builder.sh
```
### Step 2: Flask Webhook Handler
```bash
# Install Flask
pip3 install flask pyyaml
# Create handler
cat > /usr/local/bin/koji-webhook-handler.py << 'EOF'
#!/usr/bin/env python3
from flask import Flask, request
import json
import subprocess
import logging
app = Flask(__name__)
logging.basicConfig(filename='/var/log/koji-webhook.log', level=logging.INFO)
@app.route('/webhook/github', methods=['POST'])
def github_webhook():
payload = request.get_json()
if not payload:
return 'No payload', 400
repo_name = payload['repository']['name']
commit = payload['head_commit']['id']
logging.info(f"Webhook: {repo_name} @ {commit}")
try:
result = subprocess.run(
['/usr/local/bin/koji-github-builder.sh', json.dumps(payload)],
capture_output=True, text=True, timeout=60
)
logging.info(f"Build result: {result.stdout}")
return 'Build submitted', 202
except Exception as e:
logging.error(f"Build failed: {e}")
return 'Build failed', 500
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000, debug=False)
EOF
chmod +x /usr/local/bin/koji-webhook-handler.py
```
### Step 3: Systemd Service
```bash
# Create service
sudo cat > /etc/systemd/system/koji-webhook.service << 'EOF'
[Unit]
Description=Koji GitHub Webhook Handler
After=network.target
[Service]
Type=simple
User=koji
ExecStart=/usr/local/bin/koji-webhook-handler.py
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable koji-webhook
sudo systemctl start koji-webhook
```
### Step 4: GitHub Webhook Setup
For each repo (github.com/rpm-devel/cas, etc.):
```
Settings → Webhooks → Add webhook
Payload URL: https://koji-hub.yourdomain.local:5000/webhook/github
Content type: application/json
Events: Push events
Active: Yes
Save
```
---
## First Build & Testing
### Step 1: Submit Test Build
```bash
# Prepare test SRPM
cd /tmp/cas
git clone https://github.com/rpm-devel/cas
cd cas
rpmbuild -bs cas.spec
# Submit to Koji
koji build el9-candidate ~/rpmbuild/SRPMS/cas-*.src.rpm
# Monitor build
koji watch-task
# Expected output:
# Task started
# Build --
# [x86_64] started
# [aarch64] started
# (both complete)
```
### Step 2: Verify Build Success
```bash
# Check build status
koji list-builds cas --latest=1
# List built packages
koji buildinfo
# Verify both architectures
ls -la /var/lib/koji/packages/cas/*/*/
# Should show x86_64/ and aarch64/ subdirectories
```
### Step 3: Publish to Repository
```bash
# Copy RPMs to repo
cp /var/lib/koji/packages/cas/*/*/x86_64/*.rpm /var/www/repos/el9/x86_64/casjay-rpms/
cp /var/lib/koji/packages/cas/*/*/aarch64/*.rpm /var/www/repos/el9/aarch64/casjay-rpms/
# Regenerate metadata
createrepo_c /var/www/repos/el9/x86_64/casjay-rpms/
createrepo_c /var/www/repos/el9/aarch64/casjay-rpms/
# Sign metadata (if using GPG)
gpg --detach-sign --armor /var/www/repos/el9/x86_64/casjay-rpms/repodata/repomd.xml
```
### Step 4: Test Client Installation
```bash
# On test machine (AlmaLinux 9)
cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-rpms]
name=CasjaysDev RPMs
baseurl=http://repos.yourdomain.local/el9/x86_64/casjay-rpms/
enabled=1
gpgcheck=0
EOF
# Install
dnf install cas
# Verify
cas --version
```
---
## Production Operations
### Daily Build Submission
```bash
# Via CLI
koji build el9-candidate git+https://github.com/rpm-devel/package.git
# Via GitHub webhook (automatic)
# Push to spec repo → webhook fires → build auto-submits
# Monitor via dashboard
https://koji-hub.yourdomain.local/koji/
```
### Third-Party Package Rebuilds
```bash
# Script: /usr/local/bin/koji-rebuild-third-party.sh
#!/bin/bash
SRPMS_TO_BUILD=(
"https://rpms.remirepo.net/enterprise/9/remi/SRPMS/php-8.2.src.rpm"
"https://mirrors.elrepo.org/linux/elrepo/el9/SRPMS/kernel-ml.src.rpm"
"https://download1.rpmfusion.org/free/el/updates/9/SRPMS/ffmpeg.src.rpm"
)
for srpm in "${SRPMS_TO_BUILD[@]}"; do
koji build el9-rebuild-candidate "$srpm"
done
# Add to crontab
# 0 2 * * 0 /usr/local/bin/koji-rebuild-third-party.sh
```
### Build Promotion
```bash
# Move from candidate to release after testing
koji move-build el9-candidate el9-release
# Or tag directly
koji tag-pkg el9-release cas-1.0.3-1
# Verify promotion
koji list-builds --tag=el9-release --latest=5
```
### Repository Regeneration
```bash
# Script to regen all repos
for dist in el8 el9; do
for arch in x86_64 aarch64; do
for repo in baseos appstream extras casjay-*; do
repo_path="/var/www/repos/${dist}/${arch}/${repo}"
if [[ -d "$repo_path" ]]; then
createrepo_c "$repo_path"
fi
done
done
done
```
### Database Backup
```bash
# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump koji | gzip > /backups/koji-$(date +\%Y\%m\%d).sql.gz
# Keep 30-day retention
find /backups -name "koji-*.sql.gz" -mtime +30 -delete
```
### Health Monitoring
```bash
# Check builder status
koji list-hosts --ready
# Check active tasks
koji list-tasks --state=active
# Check failed builds
koji list-builds --state=failed --all
# Check repo sizes
du -sh /var/www/repos/el9/*/
# PostgreSQL stats
sudo -u postgres psql koji -c "SELECT count(*) FROM builds;"
```
---
## Performance Tuning & Scaling
### PostgreSQL Optimization
```bash
# For production (32GB RAM system)
sudo vi /var/lib/pgsql/data/postgresql.conf
shared_buffers = 8GB # 25% of RAM
effective_cache_size = 24GB # 75%
work_mem = 128MB
maintenance_work_mem = 4GB
max_connections = 500
max_parallel_workers = 8
random_page_cost = 1.1
effective_io_concurrency = 200
# Restart PostgreSQL
sudo systemctl restart postgresql
```
### Mock Cache Optimization
```bash
# Edit /etc/mock/el9-x86_64.cfg
config_opts['keep_mounted'] = True
config_opts['use_host_resolv'] = True
config_opts['basedir'] = '/mnt/koji/mock' # Fast SSD
config_opts['cache_topdir'] = '/var/cache/mock'
```
### Adding More Builders
```bash
# Register additional builder
koji add-host koji-builder-x86-2.yourdomain.local x86_64
# Copy certificates and config (same as initial setup)
# Start kojid on new builder
# Koji automatically load-balances across all builders
koji list-hosts --ready # Should show all builders
```
---
## Troubleshooting
### Builders Not Connecting
```bash
# Check SSL certificates
openssl verify -CAfile /etc/pki/koji/koji-ca-cert.crt \
/etc/pki/koji/koji-builder-x86-cert.crt
# Check builder daemon logs
tail -f /var/log/kojid.log
# Verify hostname resolution
koji list-hosts
# Test network connectivity
ssh koji@koji-builder-x86 "echo 'Connected'"
```
### Build Dependency Errors
```bash
# Verify Mock config repos
grep -A5 "\[baseos\]" /etc/mock/el9-x86_64.cfg
# Test repo access
mock -r el9-x86_64 --shell dnf repolist
# Rebuild mock chroot
mock -r el9-x86_64 --scrub=all
mock -r el9-x86_64 --init
```
### Packages Not in Repository
```bash
# Verify build completed
koji list-builds | grep
# Find RPM location
find /var/lib/koji/packages -name "*.rpm" | grep
# Copy to repo if missing
cp /var/lib/koji/packages/*/*/*/x86_64/*.rpm /var/www/repos/el9/x86_64/
# Regenerate metadata
createrepo_c /var/www/repos/el9/x86_64/
```
### Client Installation Failures
```bash
# Verify repo exists
curl http://repos.yourdomain.local/el9/x86_64/casjay-rpms/repodata/repomd.xml
# Check repo config on client
cat /etc/yum.repos.d/casjay.repo
# Update cache
dnf clean all
dnf makecache
# Test with verbose output
dnf install -v
```
---
## Success Criteria
You've successfully deployed Koji when:
- ✅ Koji Hub accessible at `https://koji-hub.yourdomain.local/koji/`
- ✅ `koji list-hosts --ready` shows both builders
- ✅ Test package builds successfully
- ✅ Both x86_64 and aarch64 builds complete
- ✅ Built RPMs in `/var/www/repos/el9/x86_64/` and `/aarch64/`
- ✅ Repository metadata (repomd.xml) generated
- ✅ Client can `dnf install casjay-release`
- ✅ All 40+ repos show in `dnf repolist`
- ✅ GitHub webhook triggers automatic builds
- ✅ Monitoring alerts working
- ✅ Database backups completing
---
## Quick Command Reference
### Build Management
```bash
koji build
koji watch-task
koji list-tasks --state=active
koji list-builds --latest=20
koji list-builds --all
koji buildinfo
```
### Tag Management
```bash
koji add-tag
koji add-target
koji list-tags
koji list-targets
koji tag-pkg
koji move-build
```
### Builder Management
```bash
koji list-hosts --ready
koji add-host
koji edit-host --channel-arches x86_64
koji disable-host
```
### Repository Management
```bash
koji regen-repo
koji list-packages
koji import-comps -t comps.xml
```
### User Management
```bash
koji add-user
koji grant-permission
koji list-users
koji list-permissions
```
---
End of Koji Complete Guide