AlmaLinux Build System (ALBS) - Complete Installation & Configuration Guide
Target: Production-grade ALBS deployment for rpm-devel
OS: AlmaLinux 9
Duration: 1-2 weeks to full production
Architecture Support: x86_64, aarch64, ppc64le
Complexity: Medium (Docker-based, simpler than Koji)
Table of Contents
- Architecture & Components
- Prerequisites & Planning
- Infrastructure Preparation
- ALBS Web Server Setup
- ALBS Build Node Setup
- Repository Configuration
- GitHub Integration
- First Build & Testing
- Production Operations
- Troubleshooting
- Performance Tuning
Architecture & Components
ALBS System Architecture
┌────────────────────────────────────────────────────────┐
│ ALBS Web Server │
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ │
│ │ PostgreSQL │ │ Redis Cache │ │ FastAPI │ │
│ │ │ │ │ │ REST API │ │
│ └──────────────┘ └──────────────┘ └─────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Pulp (Artifact Storage & Repo Management) │ │
│ │ - Package storage │ │
│ │ - Repository metadata │ │
│ │ - Distributions & releases │ │
│ └──────────────────────────────────────────────────┘ │
└────────────┬───────────────────────────────────────────┘
│
┌────────┴────────┐
│ │
┌───▼──────────┐ ┌───▼──────────┐
│ Build Node │ │ Build Node │
│ x86_64 │ │ aarch64 │
│ (Docker) │ │ (Docker) │
│ │ │ │
│ Mock chroots │ │ Mock chroots │
│ - EL8 │ │ - EL8 │
│ - EL9 │ │ - EL9 │
│ - Fedora39 │ │ - Fedora39 │
└──────────────┘ └──────────────┘
│ │
└─────────┬─────────┘
│
┌──────────▼──────────┐
│ Pulp Artifacts │
│ (RPMs, logs) │
└─────────────────────┘
Components
ALBS Web Server (docker-compose)
- FastAPI REST API for build management
- PostgreSQL database (build metadata, tasks, platforms)
- Redis cache (performance, session data)
- Pulp integration (artifact coordination)
- GitHub OAuth authentication
- Task queue management
ALBS Build Nodes (docker-compose, one per architecture)
- Receives builds from Web Server queue
- Executes builds in Mock chroots
- Uploads artifacts to Pulp
- Reports build status back to Web Server
Pulp (artifact storage)
- Stores built RPMs
- Manages repository metadata
- Handles multiple distributions/architectures
- Serves repositories to clients
Supporting Services
- PostgreSQL: Persistent data storage
- Redis: Session cache, queue management
- Docker: Container runtime for all services
- Gitea Listener: GitHub webhook receiver
- Git Cacher: Source code caching
Prerequisites & Planning
Hardware Requirements
ALBS Web Server:
- CPU: 4 cores (Intel/AMD)
- RAM: 8GB minimum (16GB recommended)
- Storage: 100GB SSD
- Network: 1Gbps
Build Node x86_64:
- CPU: 8 cores
- RAM: 16GB
- Storage: 300GB SSD
- Network: 1Gbps
Build Node aarch64:
- CPU: 8 cores ARM (AWS Graviton, Ampere, Raspberry Pi 5)
- RAM: 16GB
- Storage: 300GB SSD
- Network: 1Gbps
Pulp/Artifact Storage:
- Can be on Web Server or separate
- Storage: 1TB minimum (grows 10-50GB/month)
- SSD recommended for performance
Total: ~1.5TB storage, 24-40 cores, 40-48GB RAM
Software Requirements
All Servers:
- AlmaLinux 9 (minimal)
- Docker & Docker Compose
- Python 3.9+
- Git
- SSH access between servers
Web Server Only:
- PostgreSQL 13+
- Redis
- Nginx (reverse proxy, optional)
Network Planning
Static IPs Required:
- albs-web.yourdomain.local
- albs-builder-x86.yourdomain.local
- albs-builder-arm.yourdomain.local
DNS Setup:
albs-web.yourdomain.local A 10.x.x.10
albs-builder-x86.yourdomain.local A 10.x.x.20
albs-builder-arm.yourdomain.local A 10.x.x.30
Firewall Openings:
- Web Server: 443 (HTTPS), 8080 (Pulp), 5432 (PostgreSQL, internal only)
- Build Nodes: SSH (22), Pulp sync (internal)
- Between servers: All traffic (or specific ports: 5432, 6379, 8080)
Git Repository Planning
ALBS expects specs in git repositories:
https://github.com/rpm-devel/cas/
- cas.spec (root of repo or in SPECS/ directory)
- Optional: source tarball or patch files
https://github.com/rpm-devel/dockloom/
- dockloom.spec
https://github.com/rpm-devel/cas-downloader/
- cas-downloader.spec
(And 60+ other package repos from rpm-devel org)
Infrastructure Preparation
Step 1: Provision AlmaLinux 9 Servers
# On each server (Web, Builder x86, Builder ARM):
# Update system
dnf update -y
# Install base packages
dnf groupinstall -y "Development Tools"
dnf install -y \
git \
curl \
wget \
vim \
net-tools \
htop \
tmux \
docker \
docker-compose \
python3.9 \
python3-pip \
postgresql-client
# Add current user to docker group (to avoid sudo)
usermod -aG docker $(whoami)
# Start Docker
systemctl start docker
systemctl enable docker
# Verify Docker
docker run hello-world
Step 2: Network Configuration
# Set hostnames on each server
hostnamectl set-hostname albs-web.yourdomain.local # On Web Server
hostnamectl set-hostname albs-builder-x86.yourdomain.local # On x86 builder
hostnamectl set-hostname albs-builder-arm.yourdomain.local # On ARM builder
# Edit /etc/hosts on all servers
sudo vi /etc/hosts
# Add all three servers to each /etc/hosts:
10.x.x.10 albs-web.yourdomain.local albs-web
10.x.x.20 albs-builder-x86.yourdomain.local albs-builder-x86
10.x.x.30 albs-builder-arm.yourdomain.local albs-builder-arm
# Test connectivity
ping albs-web.yourdomain.local
ping albs-builder-x86.yourdomain.local
ping albs-builder-arm.yourdomain.local
Step 3: Shared Storage Setup (Optional)
For centralized artifact storage (recommended for production):
# Option 1: NFS Export (on storage server)
# Install NFS server
dnf install -y nfs-utils
systemctl start nfs-server
systemctl enable nfs-server
# Create export directory
mkdir -p /exports/pulp-data
chmod 777 /exports/pulp-data
# Edit /etc/exports
echo "/exports/pulp-data 10.0.0.0/8(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
exportfs -ra
# Option 2: S3-Compatible Storage (MinIO, AWS S3)
# More scalable; Pulp has native S3 support
# Configure in Pulp settings (later)
# Option 3: On-Disk (each node has local storage)
# Simplest; less redundancy
Step 4: SSH Key Setup
# On Web Server, create SSH key for automation
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
# Copy to build nodes (for build automation scripts)
ssh-copy-id -i ~/.ssh/id_rsa.pub albs-builder-x86
ssh-copy-id -i ~/.ssh/id_rsa.pub albs-builder-arm
# Verify passwordless SSH
ssh albs-builder-x86 "echo 'Connected to x86 builder'"
ALBS Web Server Setup
Step 1: Clone ALBS Repositories
# Create directory structure
mkdir -p /opt/albs
cd /opt/albs
# Clone ALBS Web Server
git clone https://github.com/AlmaLinux/albs-web-server.git
cd albs-web-server
# Checkout stable release
git checkout $(git describe --tags --abbrev=0)
cd ..
# Clone ALBS Node (for reference/setup)
git clone https://github.com/AlmaLinux/albs-node.git
# Clone Gitea Listener (GitHub webhook handler)
git clone https://github.com/AlmaLinux/gitea_listener.git
# Clone Pulp (will be containerized)
# (Pulp image will be pulled via Docker)
# Directory structure
ls -la /opt/albs/
# albs-web-server/
# albs-node/
# gitea_listener/
Step 2: PostgreSQL Setup
# Install PostgreSQL server
dnf install -y postgresql-server postgresql-contrib
# Initialize database
sudo -u postgres /usr/bin/postgresql-setup initdb
# Edit /var/lib/pgsql/data/postgresql.conf
sudo vi /var/lib/pgsql/data/postgresql.conf
# Key settings:
listen_addresses = '*'
shared_buffers = 2GB # 25% of RAM
effective_cache_size = 6GB # 75% of RAM
work_mem = 32MB
max_connections = 200
# Edit /var/lib/pgsql/data/pg_hba.conf
sudo vi /var/lib/pgsql/data/pg_hba.conf
# Add after local lines (for Docker containers):
host all all 172.17.0.0/16 md5
host all all 127.0.0.1/32 md5
host all all 10.0.0.0/8 md5
# Restart PostgreSQL
sudo systemctl restart postgresql
sudo systemctl enable postgresql
# Create ALBS database and users
sudo -u postgres psql << EOF
CREATE USER albs WITH PASSWORD 'albs-db-password-here';
CREATE DATABASE albs OWNER albs;
CREATE USER pulp WITH PASSWORD 'pulp-db-password-here';
CREATE DATABASE pulp_app OWNER pulp;
CREATE DATABASE pulp_content OWNER pulp;
GRANT ALL ON DATABASE albs TO albs;
GRANT ALL ON DATABASE pulp_app TO pulp;
GRANT ALL ON DATABASE pulp_content TO pulp;
EOF
# Verify
sudo -u postgres psql -l | grep albs
Step 3: Redis Setup
# Install Redis
dnf install -y redis
# Configure Redis
sudo vi /etc/redis/redis.conf
# Key settings:
bind 0.0.0.0
port 6379
requirepass redis-password-here
appendonly yes
# Start Redis
sudo systemctl start redis
sudo systemctl enable redis
# Verify connection
redis-cli ping
# Should return: PONG
Step 4: Docker Compose Configuration
# Create docker-compose.yml for Web Server in /opt/albs/
cd /opt/albs
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
albs-web:
image: almalinux/albs-web-server:latest
container_name: albs-web
restart: always
ports:
- "8080:8080"
- "443:443"
environment:
DATABASE_URL: postgresql://albs:albs-db-password-here@host.docker.internal:5432/albs
REDIS_URL: redis://:redis-password-here@host.docker.internal:6379/0
PULP_URL: http://pulp:80
PULP_DOMAIN_NAME: albs-web.yourdomain.local
SECRET_KEY: your-secret-key-here-change-this
GITHUB_CLIENT_ID: your-github-client-id
GITHUB_CLIENT_SECRET: your-github-client-secret
GITHUB_CALLBACK_URL: https://albs-web.yourdomain.local/auth/github/callback
volumes:
- ./albs-web-server:/app
- /etc/pki/albs:/etc/pki/albs:ro
depends_on:
- pulp
networks:
- albs-network
extra_hosts:
- "host.docker.internal:host-gateway"
pulp:
image: docker.io/pulp/pulp:latest
container_name: albs-pulp
restart: always
ports:
- "8000:80"
environment:
PULP_SECRET_KEY: pulp-secret-key-here-change-this
PULP_CONTENT_ORIGIN: http://albs-web.yourdomain.local/pulp/content
PULP_ALLOWED_IMPORT_PATHS: "['/var/lib/pulp']"
PULP_ALLOWED_EXPORT_PATHS: "['/var/lib/pulp']"
volumes:
- pulp-data:/var/lib/pulp
- pulp-assets:/var/lib/pulp/assets
environment:
POSTGRES_DB: pulp_app
POSTGRES_USER: pulp
POSTGRES_PASSWORD: pulp-db-password-here
POSTGRES_HOST_ENVIRONMENT: host.docker.internal
POSTGRES_PORT: 5432
networks:
- albs-network
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
pulp-data:
pulp-assets:
networks:
albs-network:
driver: bridge
EOF
# Pull images
docker-compose pull
# Start services
docker-compose up -d
# Wait for services to start
sleep 30
# Verify
docker-compose ps
docker-compose logs albs-web
Step 5: Nginx Reverse Proxy (Optional but Recommended)
# Install Nginx
dnf install -y nginx
# Create SSL certificates
sudo mkdir -p /etc/pki/albs
cd /etc/pki/albs
# Self-signed certificate (replace with real cert in production)
sudo openssl req -x509 -nodes -days 365 \
-newkey rsa:2048 \
-keyout /etc/pki/albs/albs-key.key \
-out /etc/pki/albs/albs-cert.crt \
-subj "/CN=albs-web.yourdomain.local"
# Configure Nginx
sudo vi /etc/nginx/conf.d/albs.conf
cat > /etc/nginx/conf.d/albs.conf << 'EOF'
upstream albs_backend {
server localhost:8080;
}
upstream pulp_backend {
server localhost:8000;
}
server {
listen 80;
server_name albs-web.yourdomain.local;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name albs-web.yourdomain.local;
ssl_certificate /etc/pki/albs/albs-cert.crt;
ssl_certificate_key /etc/pki/albs/albs-key.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
client_max_body_size 1G;
# ALBS Web API
location / {
proxy_pass http://albs_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Pulp
location /pulp/ {
proxy_pass http://pulp_backend/pulp/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
EOF
# Restart Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx
# Test
curl -k https://albs-web.yourdomain.local/
Step 6: GitHub OAuth Setup
Log into GitHub and configure OAuth application:
GitHub Settings → Developer settings → OAuth Apps → New OAuth App
Application name: ALBS rpm-devel
Homepage URL: https://albs-web.yourdomain.local
Authorization callback URL: https://albs-web.yourdomain.local/auth/github/callback
Copy: Client ID and Client Secret
Update in docker-compose.yml:
GITHUB_CLIENT_ID: <your-client-id>
GITHUB_CLIENT_SECRET: <your-client-secret>
Restart:
docker-compose restart albs-web
Step 7: Initialize Platforms
# Access ALBS Web API to create platforms
# First, login with GitHub OAuth at:
# https://albs-web.yourdomain.local
# Then use API to create build platforms:
curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "AlmaLinux-8",
"type": "rpm",
"architectures": ["x86_64", "aarch64"]
}'
curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "AlmaLinux-9",
"type": "rpm",
"architectures": ["x86_64", "aarch64"]
}'
curl -X POST https://albs-web.yourdomain.local/api/v1/platforms \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "Fedora-39",
"type": "rpm",
"architectures": ["x86_64", "aarch64"]
}'
# Verify platforms created
curl https://albs-web.yourdomain.local/api/v1/platforms \
-H "Authorization: Bearer <your-token>"
ALBS Build Node Setup
Step 1: Build Node Preparation
# On each builder (x86_64 and aarch64)
# Create working directories
mkdir -p /opt/albs
mkdir -p /var/lib/albs/builds
mkdir -p /var/lib/albs/artifacts
# Set permissions
chmod 755 /opt/albs
chmod 755 /var/lib/albs
# Install dependencies
dnf install -y \
python3.9 \
python3-pip \
mock \
rpm-build \
git \
podman
Step 2: Clone ALBS Node
cd /opt/albs
git clone https://github.com/AlmaLinux/albs-node.git
cd albs-node
git checkout $(git describe --tags --abbrev=0)
# Install Python dependencies
pip3 install -r requirements.txt
Step 3: Node Configuration
# Create albs-node config
mkdir -p /etc/albs
cat > /etc/albs/albs-node.conf << 'EOF'
[general]
web_server_url = https://albs-web.yourdomain.local
api_token = <generated-at-web-server>
worker_count = 4
[pulp]
url = http://albs-web.yourdomain.local/pulp
username = pulp
password = pulp-db-password-here
[build]
mock_config_path = /etc/mock
enable_ccache = true
max_parallel_builds = 4
[logging]
level = INFO
file = /var/log/albs-node.log
EOF
chmod 600 /etc/albs/albs-node.conf
Step 4: Mock Chroot Configuration
# Create mock configs for all distros/archs
# For EL8 x86_64
cat > /etc/mock/el8-x86_64.cfg << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'
config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d
[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8
[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""
config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
EOF
# For EL9 x86_64, EL9 aarch64, Fedora39 x86_64, etc.
# Create similar configs for each distro/arch combination
# Test mock chroots
mock -r el8-x86_64 --init
mock -r el9-x86_64 --init
Step 5: Systemd Service
# Create systemd service for ALBS node
sudo cat > /etc/systemd/system/albs-node.service << 'EOF'
[Unit]
Description=AlmaLinux Build System Build Node
After=network.target docker.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/albs/albs-node
ExecStart=/usr/bin/python3 -m albs_node
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable albs-node
sudo systemctl start albs-node
# Check status
sudo systemctl status albs-node
sudo journalctl -u albs-node -f
Step 6: Register Node with Web Server
# On Web Server, register the build node via API
curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "builder-x86-1",
"url": "http://albs-builder-x86.yourdomain.local:8000",
"architectures": ["x86_64"],
"platforms": ["AlmaLinux-8", "AlmaLinux-9", "Fedora-39"]
}'
# Repeat for aarch64 builder
curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "builder-arm-1",
"url": "http://albs-builder-arm.yourdomain.local:8000",
"architectures": ["aarch64"],
"platforms": ["AlmaLinux-8", "AlmaLinux-9", "Fedora-39"]
}'
# Verify nodes registered
curl https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <your-token>"
Repository Configuration
Step 1: Create Repository Definitions
# In ALBS Web Server, create distributions for each distro/arch
# For each platform and architecture:
curl -X POST https://albs-web.yourdomain.local/api/v1/distributions \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "AlmaLinux-8-x86_64",
"platform": "AlmaLinux-8",
"architecture": "x86_64",
"build_group": "baseos",
"repositories": [
{
"name": "baseos",
"enabled": true
},
{
"name": "appstream",
"enabled": true
},
{
"name": "extras",
"enabled": true
},
{
"name": "casjay-rpms",
"enabled": true
}
]
}'
# Create distributions for all combinations:
# - AlmaLinux-8-x86_64, AlmaLinux-8-aarch64
# - AlmaLinux-9-x86_64, AlmaLinux-9-aarch64
# - Fedora-39-x86_64, Fedora-39-aarch64
Step 2: Repository Mapping
# Configure where repositories are synced from/to
# Create publication endpoints
curl -X POST https://albs-web.yourdomain.local/api/v1/repositories \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "casjay-el8-baseos",
"description": "CasjaysDev EL8 BaseOS Repository",
"distribution": "AlmaLinux-8-x86_64",
"public_url": "http://repos.yourdomain.local/el8/x86_64/baseos/"
}'
Step 3: Configure Pulp Sync
# Sync official repos into Pulp (pull-through cache)
# This allows builds to pull from cached official repos
curl -X POST https://albs-web.yourdomain.local/api/v1/remotes \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "almalinux-8-baseos",
"url": "https://mirrors.almalinux.org/almalinux/8/BaseOS/",
"architecture": "x86_64",
"distribution": "AlmaLinux-8-x86_64"
}'
GitHub Integration
Step 1: Gitea Listener Setup
# On Web Server
cd /opt/albs/gitea_listener
# Create config
cat > config.yaml << 'EOF'
listen_address: 0.0.0.0
listen_port: 8888
web_server_url: https://albs-web.yourdomain.local
web_server_token: <generated-at-web-server>
log_level: INFO
log_file: /var/log/gitea_listener.log
github:
webhook_secret: your-github-webhook-secret-here
pulp:
url: http://albs-web.yourdomain.local/pulp
username: pulp
password: pulp-db-password-here
EOF
chmod 600 config.yaml
Step 2: Docker Compose for Gitea Listener
# Add to docker-compose.yml
cat >> docker-compose.yml << 'EOF'
gitea-listener:
image: almalinux/gitea_listener:latest
container_name: gitea-listener
restart: always
ports:
- "8888:8888"
environment:
WEB_SERVER_URL: https://albs-web.yourdomain.local
WEB_SERVER_TOKEN: <your-token>
GITHUB_WEBHOOK_SECRET: your-github-webhook-secret-here
volumes:
- ./gitea_listener/config.yaml:/etc/gitea_listener/config.yaml:ro
networks:
- albs-network
EOF
docker-compose up -d
Step 3: GitHub Webhook Configuration
For each repository (github.com/rpm-devel/cas, etc.):
Repository Settings → Webhooks → Add webhook
Payload URL: https://albs-web.yourdomain.local:8888/webhooks/github
Content type: application/json
Secret: your-github-webhook-secret-here
Events:
- Push
- Release
Active: Yes
Save
Step 4: Test GitHub Hook
# Push a change to your spec repo
cd /tmp/cas-build
git clone https://github.com/rpm-devel/cas
cd cas
echo "# Test" >> README.md
git add README.md
git commit -m "Test build trigger"
git push
# Check ALBS Web dashboard
# https://albs-web.yourdomain.local/
# Should see build queued/running within minutes
First Build & Testing
Step 1: Submit Test Build via API
# Create a build from spec
curl -X POST https://albs-web.yourdomain.local/api/v1/builds \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "cas",
"version": "1.0.3",
"release": "1",
"platforms": ["AlmaLinux-8", "AlmaLinux-9"],
"architectures": ["x86_64", "aarch64"],
"git_ref": "https://github.com/rpm-devel/cas.git",
"git_branch": "main"
}'
# Response will include build ID
# Monitor via: https://albs-web.yourdomain.local/builds/<build-id>
Step 2: Monitor Build Progress
# Via Web Dashboard
https://albs-web.yourdomain.local/
# Via API
curl https://albs-web.yourdomain.local/api/v1/builds/<build-id> \
-H "Authorization: Bearer <your-token>"
# Expected flow:
# 1. Build queued
# 2. Assigned to builder (x86_64)
# 3. Build running
# 4. Assigned to builder (aarch64)
# 5. Build running
# 6. Both complete
# 7. Artifacts uploaded to Pulp
Step 3: Verify Build Artifacts
# Check Pulp repository
curl https://albs-web.yourdomain.local/pulp/api/v3/content/rpm/packages/ \
-H "Authorization: Bearer <pulp-token>"
# List built RPMs
curl https://albs-web.yourdomain.local/pulp/content/el8/x86_64/baseos/ \
| grep -o 'cas.*\.rpm'
# Should see:
# cas-1.0.3-1.el8.x86_64.rpm
# cas-1.0.3-1.el8.aarch64.rpm
# cas-debuginfo-1.0.3-1.el8.x86_64.rpm (optional)
# cas-debuginfo-1.0.3-1.el8.aarch64.rpm (optional)
Step 4: Publish Repository
# Publish built RPMs to client-facing repository
curl -X POST https://albs-web.yourdomain.local/api/v1/distributions/<dist-id>/publish \
-H "Authorization: Bearer <your-token>"
# Repositories now available at:
# http://repos.yourdomain.local/el8/x86_64/baseos/
# http://repos.yourdomain.local/el8/aarch64/baseos/
Step 5: Test Client Installation
# On test client (AlmaLinux 8 or 9)
# Create repo config
sudo tee /etc/yum.repos.d/casjay.repo > /dev/null << 'EOF'
[casjay-baseos]
name=CasjaysDev BaseOS
baseurl=http://repos.yourdomain.local/el8/x86_64/baseos/
enabled=1
gpgcheck=0
[casjay-appstream]
name=CasjaysDev AppStream
baseurl=http://repos.yourdomain.local/el8/x86_64/appstream/
enabled=1
gpgcheck=0
EOF
# Update repo cache
sudo dnf makecache
# Verify repo available
sudo dnf repolist | grep casjay
# Install package
sudo dnf install cas
# Verify installation
cas --version
Production Operations
Daily Monitoring
# Check build node health
curl https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <your-token>" | jq '.[] | {name, status}'
# Check active builds
curl https://albs-web.yourdomain.local/api/v1/builds?status=in_progress \
-H "Authorization: Bearer <your-token>"
# Check failed builds
curl https://albs-web.yourdomain.local/api/v1/builds?status=failed \
-H "Authorization: Bearer <your-token>"
# Check disk usage
docker exec albs-pulp df -h /var/lib/pulp
docker exec postgres du -sh /var/lib/postgresql/data
Automated Nightly Rebuilds
# Script: /usr/local/bin/albs-rebuild-third-party.sh
#!/bin/bash
ALBS_URL="https://albs-web.yourdomain.local"
ALBS_TOKEN="<your-token>"
# Remi packages to rebuild
REMI_PACKAGES=(
"php-8.2"
"mariadb-10.6"
"postgresql-15"
"nodejs-20"
)
for pkg in "${REMI_PACKAGES[@]}"; do
echo "Rebuilding $pkg..."
curl -X POST $ALBS_URL/api/v1/builds \
-H "Authorization: Bearer $ALBS_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"name\": \"$pkg\",
\"platforms\": [\"AlmaLinux-8\", \"AlmaLinux-9\"],
\"architectures\": [\"x86_64\", \"aarch64\"],
\"git_ref\": \"https://rpms.remirepo.net/enterprise/SRPMS/$pkg.src.rpm\"
}"
done
# Add to crontab
# 0 2 * * * /usr/local/bin/albs-rebuild-third-party.sh
Backup Strategy
# Backup PostgreSQL database
0 1 * * * docker exec postgres pg_dump -U albs albs | gzip > /backups/albs-$(date +\%Y\%m\%d).sql.gz
# Backup Pulp data
0 2 * * * tar czf /backups/pulp-$(date +\%Y\%m\%d).tar.gz /var/lib/docker/volumes/albs_pulp-data/_data
# Keep 30-day retention
0 3 * * * find /backups -name "albs-*.sql.gz" -mtime +30 -delete
Monitoring & Alerts
# Health check script: /usr/local/bin/albs-health-check.sh
#!/bin/bash
ALERT_EMAIL="admin@rpm-devel.local"
# Check web server
if ! curl -s -k https://albs-web.yourdomain.local/health > /dev/null; then
echo "ALERT: ALBS Web Server unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi
# Check build nodes
OFFLINE=$(curl -s https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <token>" | jq '.[] | select(.status != "online") | .name' | wc -l)
if [[ $OFFLINE -gt 0 ]]; then
echo "ALERT: $OFFLINE build nodes offline" | mail -s "ALBS Alert" $ALERT_EMAIL
fi
# Check disk space
PULP_USAGE=$(docker exec albs-pulp df -h /var/lib/pulp | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $PULP_USAGE -gt 80 ]]; then
echo "ALERT: Pulp storage at ${PULP_USAGE}% capacity" | mail -s "ALBS Alert" $ALERT_EMAIL
fi
# Add to crontab (run every 30 minutes)
# */30 * * * * /usr/local/bin/albs-health-check.sh
Troubleshooting
Issue: Build Nodes Not Connecting
# Check node logs
docker-compose logs -f albs-node
# Verify network connectivity
docker exec albs-node ping albs-web.yourdomain.local
# Check API token validity
curl -i https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <token>"
# Verify node config
cat /etc/albs/albs-node.conf
# Restart node
systemctl restart albs-node
Issue: Build Fails with Mock Error
# Check mock chroot
mock -r el8-x86_64 --shell
# Verify repo access in chroot
mock -r el8-x86_64 --shell dnf repolist
# Check mock config for typos
grep -A20 "\[baseos\]" /etc/mock/el8-x86_64.cfg
# Rebuild chroot
mock -r el8-x86_64 --scrub=all
mock -r el8-x86_64 --init
Issue: Artifacts Not Uploaded to Pulp
# Check Pulp status
docker-compose logs pulp | tail -50
# Verify Pulp API
curl -u admin:admin http://localhost:8000/pulp/api/v3/status/
# Check artifact upload path
ls -la /var/lib/docker/volumes/albs_pulp-data/_data/
# Verify ALBS node upload settings
grep -A5 "\[pulp\]" /etc/albs/albs-node.conf
Issue: GitHub Webhooks Not Triggering Builds
# Check Gitea listener logs
docker-compose logs gitea-listener | tail -50
# Test webhook manually
curl -X POST https://albs-web.yourdomain.local:8888/webhooks/github \
-H "Content-Type: application/json" \
-H "X-Hub-Signature: sha256=test" \
-d '{"action":"opened","pull_request":{}}'
# Verify webhook URL on GitHub
# Repository Settings → Webhooks → Check delivery history
# Check web server logs for auth errors
docker-compose logs albs-web | grep -i webhook
Issue: PostgreSQL Connection Refused
# Verify PostgreSQL is running
sudo systemctl status postgresql
# Check PostgreSQL logs
sudo tail -50 /var/log/postgresql/*.log
# Verify listening on all interfaces
sudo grep "listen_addresses" /var/lib/pgsql/data/postgresql.conf
# Test connection
psql -h 127.0.0.1 -U albs -d albs -c "SELECT 1"
# Check pg_hba.conf for correct auth entries
sudo grep "host.*albs" /var/lib/pgsql/data/pg_hba.conf
Performance Tuning
PostgreSQL Optimization
# Edit /var/lib/pgsql/data/postgresql.conf
# Memory tuning (for 16GB RAM)
shared_buffers = 4GB # 25% of RAM
effective_cache_size = 12GB # 75% of RAM
work_mem = 64MB
maintenance_work_mem = 2GB
# Connection tuning
max_connections = 300
max_parallel_workers = 8
# Query planning
random_page_cost = 1.1 # For SSD storage
effective_io_concurrency = 200
# WAL (Write-Ahead Logging)
wal_level = replica
max_wal_senders = 5
# Logging
log_statement = 'mod'
log_min_duration_statement = 1000 # Log queries > 1 second
# Restart PostgreSQL
sudo systemctl restart postgresql
Mock Cache Optimization
# In /etc/mock/*.cfg files
config_opts['keep_mounted'] = True
config_opts['use_host_resolv'] = True
config_opts['basedir'] = '/mnt/mock-cache' # Fast SSD
config_opts['cache_topdir'] = '/var/cache/mock'
config_opts['internal_dev_setup'] = True
Docker Optimization
# Edit /etc/docker/daemon.json
{
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 10
}
# Restart Docker
sudo systemctl restart docker
Pulp Performance
# Tune in docker-compose.yml
environment:
PULP_EXPORT_PER_PAGE: 1000
PULP_IMPORT_PER_PAGE: 1000
PULP_CONTENT_PATH_PREFIX: /pulp/content/
PULP_WORKERS: 4
Scaling
Add More Build Nodes
# Repeat ALBS Build Node Setup (Step 1-6) on new servers
# For additional x86_64 builders: builder-x86-2, builder-x86-3, etc.
# For additional aarch64 builders: builder-arm-2, builder-arm-3, etc.
# Register new nodes via API
curl -X POST https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <your-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "builder-x86-2",
"url": "http://albs-builder-x86-2.yourdomain.local:8000",
"architectures": ["x86_64"],
"platforms": ["AlmaLinux-8", "AlmaLinux-9"]
}'
# ALBS automatically load-balances builds across all nodes
Distributed Pulp Storage
# Use S3-compatible storage (MinIO, AWS S3)
# Configure in docker-compose.yml
environment:
PULP_STORAGE_CLASS: storages.backends.s3boto3.S3Boto3Storage
AWS_ACCESS_KEY_ID: your-access-key
AWS_SECRET_ACCESS_KEY: your-secret-key
AWS_STORAGE_BUCKET_NAME: albs-artifacts
AWS_S3_ENDPOINT_URL: https://s3.yourdomain.local
Complete Docker Compose Reference
version: '3.8'
services:
postgres:
image: postgres:13
container_name: albs-postgres
restart: always
environment:
POSTGRES_PASSWORD: postgres-password
volumes:
- postgres-data:/var/lib/postgresql/data
ports:
- "5432:5432"
networks:
- albs-network
redis:
image: redis:7-alpine
container_name: albs-redis
restart: always
command: redis-server --requirepass redis-password
ports:
- "6379:6379"
networks:
- albs-network
albs-web:
image: almalinux/albs-web-server:latest
container_name: albs-web
restart: always
ports:
- "8080:8080"
environment:
DATABASE_URL: postgresql://albs:albs-password@postgres:5432/albs
REDIS_URL: redis://:redis-password@redis:6379/0
PULP_URL: http://pulp:80
SECRET_KEY: your-secret-key
GITHUB_CLIENT_ID: your-github-id
GITHUB_CLIENT_SECRET: your-github-secret
depends_on:
- postgres
- redis
networks:
- albs-network
pulp:
image: docker.io/pulp/pulp:latest
container_name: albs-pulp
restart: always
ports:
- "8000:80"
environment:
PULP_SECRET_KEY: pulp-secret
PULP_CONTENT_ORIGIN: http://albs-web.yourdomain.local/pulp/content
volumes:
- pulp-data:/var/lib/pulp
networks:
- albs-network
volumes:
postgres-data:
pulp-data:
networks:
albs-network:
driver: bridge
Success Criteria
You've successfully deployed ALBS when:
- ✅ Web Server accessible at
https://albs-web.yourdomain.local - ✅ GitHub OAuth login works
- ✅ Both build nodes show as "online"
- ✅ Test package builds successfully on both x86_64 and aarch64
- ✅ Built RPMs appear in Pulp repository
- ✅ RPMs downloadable from public repo URL
- ✅ Client can install packages via
dnf install - ✅ GitHub webhook triggers automatic builds
- ✅ Health checks pass without errors
Key Commands Reference
Build Management
# Submit build
curl -X POST https://albs-web.yourdomain.local/api/v1/builds \
-H "Authorization: Bearer <token>" -d '{...}'
# List builds
curl https://albs-web.yourdomain.local/api/v1/builds \
-H "Authorization: Bearer <token>"
# Get build status
curl https://albs-web.yourdomain.local/api/v1/builds/<build-id> \
-H "Authorization: Bearer <token>"
Docker Management
# View logs
docker-compose logs -f albs-web
docker-compose logs -f albs-node
# Restart services
docker-compose restart albs-web
docker-compose restart albs-node
# Stop all
docker-compose down
System Health
# Check nodes
curl https://albs-web.yourdomain.local/api/v1/build_nodes \
-H "Authorization: Bearer <token>"
# Check platforms
curl https://albs-web.yourdomain.local/api/v1/platforms \
-H "Authorization: Bearer <token>"
# Check Pulp status
curl -u admin:admin http://localhost:8000/pulp/api/v3/status/
End of ALBS Complete Guide