# ALBS (AlmaLinux Build System) - Single Server Complete Guide

**Target**: Production ALBS on single 128GB/32-core/8TB server  
**OS**: AlmaLinux 9 (minimal)  
**Deployment**: 1-2 weeks  
**Architecture Support**: x86_64, aarch64 (parallel builds on same server)  
**Complexity**: Medium (Docker-based, simplified single-server)

---

## Table of Contents

1. [Architecture & Overview](#architecture--overview)
2. [Prerequisites & Planning](#prerequisites--planning)
3. [System Preparation](#system-preparation)
4. [PostgreSQL Setup](#postgresql-setup)
5. [Redis Setup](#redis-setup)
6. [ALBS Services Installation](#albs-services-installation)
7. [Mock Chroot Configuration](#mock-chroot-configuration)
8. [Repository Structure](#repository-structure)
9. [GitHub Integration](#github-integration)
10. [First Build & Testing](#first-build--testing)
11. [Production Operations](#production-operations)
12. [Performance & Scaling](#performance--scaling)
13. [Troubleshooting](#troubleshooting)

---

## Architecture & Overview

### Single Server ALBS Setup

```
┌────────────────────────────────────────────────────┐
│           Single Server (128GB/32c/8TB)            │
│                                                    │
│  ┌──────────────┐  ┌──────────────┐               │
│  │ PostgreSQL   │  │ Redis Cache  │               │
│  │ (16GB alloc) │  │ (4GB alloc)  │               │
│  └──────────────┘  └──────────────┘               │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  ALBS Web Server (Docker)                    │ │
│  │  - FastAPI REST API                          │ │
│  │  - Nginx reverse proxy                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Pulp (Docker)                               │ │
│  │  - Artifact storage (2TB allocated)          │ │
│  │  - Repository metadata                       │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Build Execution (Local)                     │ │
│  │  ┌─────────────┐  ┌─────────────┐            │ │
│  │  │ Mock x86_64 │  │ Mock aarch64 │           │ │
│  │  │ (max 4      │  │ (max 4      │            │ │
│  │  │ parallel)   │  │ parallel)   │            │ │
│  │  └─────────────┘  └─────────────┘            │ │
│  │  - EL8, EL9, Fedora chroots                  │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  Repository Storage                          │ │
│  │  /var/lib/repos/ (5TB allocated)              │ │
│  │  - el8/x86_64, el8/aarch64                   │ │
│  │  - el9/x86_64, el9/aarch64                   │ │
│  │  - fedora39/x86_64, fedora39/aarch64         │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
└────────────────────────────────────────────────────┘
```

### Resource Allocation

**Available**: 128GB RAM, 32 cores, 8TB disk

**Allocation**:
- PostgreSQL: 16GB RAM
- Redis: 4GB RAM
- Docker containers (ALBS + Pulp): 8GB RAM
- Mock chroots (in-memory): 16GB RAM (2x 8GB simultaneous builds)
- OS + kernel cache: 20GB
- Reserved: 64GB (headroom)

**Storage**:
- OS/system: 200GB
- PostgreSQL data: 100GB
- Mock cache: 500GB
- Pulp artifacts: 2TB
- Repositories: 5TB
- Logs: 100GB
- Remaining: 300GB (headroom)

---

## Prerequisites & Planning

### Hardware Confirmation
- ✅ 128GB RAM (more than enough)
- ✅ 32 cores (can run 4-8 parallel builds)
- ✅ 8TB disk (5TB for repos + artifacts + OS)
- ✅ Network: 1Gbps (for GitHub webhooks, client downloads)

### Software Required
- AlmaLinux 9 (minimal install)
- Docker & Docker Compose
- PostgreSQL 13+
- Redis 6+
- Python 3.9+
- Mock
- createrepo_c

### Ports Needed
- 80: HTTP (redirect to HTTPS)
- 443: HTTPS (ALBS Web UI + Pulp)
- 5432: PostgreSQL (localhost only)
- 6379: Redis (localhost only)
- 22: SSH

### DNS Setup

```
albs.yourdomain.local    A  <your-server-ip>
repos.yourdomain.local   A  <your-server-ip>  (or same IP)
```

---

## System Preparation

### Step 1: Update & Install Base Packages

```bash
# Update system
sudo dnf update -y

# Install development tools
sudo dnf groupinstall -y "Development Tools"

# Install required packages
sudo dnf install -y \
  git curl wget vim net-tools htop tmux \
  python3.9 python3-pip python3-devel \
  libffi-devel openssl-devel \
  postgresql-server postgresql-contrib postgresql-devel \
  redis \
  docker docker-compose \
  mock rpm-build \
  createrepo_c \
  nginx \
  epel-release

# Start Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add current user to docker group
sudo usermod -aG docker $(whoami)

# Verify
docker run hello-world
```

### Step 2: Hostname & Network

```bash
# Set hostname
sudo hostnamectl set-hostname albs.yourdomain.local

# Edit /etc/hosts
sudo vi /etc/hosts

# Add:
127.0.0.1   localhost
<your-ip>   albs.yourdomain.local albs
<your-ip>   repos.yourdomain.local repos

# Test
ping albs.yourdomain.local
```

### Step 3: Create Directory Structure

```bash
# Create all working directories
sudo mkdir -p /var/lib/albs/{artifacts,builds,cache}
sudo mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}
sudo mkdir -p /var/lib/mock/{cache,tmp}
sudo mkdir -p /var/log/albs

# Set permissions
sudo chown -R $(whoami):$(whoami) /var/lib/albs /var/lib/repos /var/lib/mock /var/log/albs
sudo chmod -R 755 /var/lib/albs /var/lib/repos

# Verify
du -sh /var/lib/albs /var/lib/repos /var/lib/mock
```

### Step 4: Disk Mounting (if using separate partitions)

```bash
# If you have separate disks for storage:
sudo mkfs.ext4 /dev/sdX1
sudo mkdir -p /mnt/repos
sudo mount /dev/sdX1 /mnt/repos

# Make permanent in /etc/fstab
echo "/dev/sdX1 /mnt/repos ext4 defaults 0 2" | sudo tee -a /etc/fstab
sudo mount -a

# For this guide, we'll use /var/lib/repos (single disk)
```

---

## PostgreSQL Setup

### Step 1: Initialize & Configure

```bash
# Initialize PostgreSQL
sudo /usr/bin/postgresql-setup initdb

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Configure for single-server (128GB available)
sudo vi /var/lib/pgsql/data/postgresql.conf

# Key settings:
listen_addresses = 'localhost'
shared_buffers = 16GB              # 12.5% of 128GB
effective_cache_size = 100GB       # 78% of 128GB
work_mem = 256MB
maintenance_work_mem = 4GB
max_connections = 500
random_page_cost = 1.1             # For SSD
effective_io_concurrency = 200
max_parallel_workers = 16

# WAL settings
wal_level = replica
max_wal_senders = 3
checkpoint_completion_target = 0.9
wal_buffers = 16MB

# Logging
log_statement = 'mod'
log_min_duration_statement = 1000

# Restart
sudo systemctl restart postgresql
```

### Step 2: Create ALBS Database

```bash
# Create database and user
sudo -u postgres psql << 'EOF'
CREATE USER albs WITH PASSWORD 'albs-secure-password-here';
CREATE DATABASE albs OWNER albs;
CREATE USER pulp WITH PASSWORD 'pulp-secure-password-here';
CREATE DATABASE pulp_app OWNER pulp;
CREATE DATABASE pulp_content OWNER pulp;

GRANT ALL ON DATABASE albs TO albs;
GRANT ALL ON DATABASE pulp_app TO pulp;
GRANT ALL ON DATABASE pulp_content TO pulp;

-- Allow connections
ALTER ROLE albs CREATEDB;
EOF

# Verify
sudo -u postgres psql -l | grep -E "albs|pulp"
```

---

## Redis Setup

### Step 1: Configure

```bash
# Edit Redis config
sudo vi /etc/redis/redis.conf

# Key settings:
bind 127.0.0.1
port 6379
requirepass your-redis-password-here
appendonly yes
appendfsync everysec
maxmemory 4gb
maxmemory-policy allkeys-lru

# Start
sudo systemctl start redis
sudo systemctl enable redis

# Verify
redis-cli ping
# Should return: PONG
```

---

## ALBS Services Installation

### Step 1: Clone ALBS Repository

```bash
# Create working directory
mkdir -p ~/albs-build
cd ~/albs-build

# Clone ALBS
git clone https://github.com/AlmaLinux/albs-web-server.git
cd albs-web-server
git checkout $(git describe --tags --abbrev=0)

# Back to build directory
cd ~/albs-build
```

### Step 2: Create SSL Certificates

```bash
# Create certificate directory
mkdir -p certs
cd certs

# Self-signed certificates (replace with real certs in production)
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout albs-key.key \
  -out albs-cert.crt \
  -subj "/CN=albs.yourdomain.local/O=CasjaysDev/C=US"

# Copy to Docker accessible location
sudo cp albs-cert.crt /etc/pki/albs/ 2>/dev/null || sudo mkdir -p /etc/pki/albs && sudo cp albs-cert.crt /etc/pki/albs/
sudo cp albs-key.key /etc/pki/albs/
sudo chmod 644 /etc/pki/albs/albs-cert.crt
sudo chmod 600 /etc/pki/albs/albs-key.key
```

### Step 3: Docker Compose Setup

```bash
# In ~/albs-build, create docker-compose.yml
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  albs-web:
    image: almalinux/albs-web-server:latest
    container_name: albs-web
    restart: always
    ports:
      - "8080:8080"
    environment:
      DATABASE_URL: postgresql://albs:albs-secure-password-here@localhost:5432/albs
      REDIS_URL: redis://:your-redis-password-here@localhost:6379/0
      PULP_URL: http://localhost:8000
      PULP_DOMAIN_NAME: albs.yourdomain.local
      SECRET_KEY: your-secret-key-here-change-this
      GITHUB_CLIENT_ID: your-github-client-id
      GITHUB_CLIENT_SECRET: your-github-client-secret
      GITHUB_CALLBACK_URL: https://albs.yourdomain.local/auth/github/callback
    volumes:
      - ./albs-web-server:/app
      - /etc/pki/albs:/etc/pki/albs:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

  pulp:
    image: docker.io/pulp/pulp:latest
    container_name: albs-pulp
    restart: always
    ports:
      - "8000:80"
    environment:
      PULP_SECRET_KEY: pulp-secret-key-here-change-this
      PULP_CONTENT_ORIGIN: http://albs.yourdomain.local/pulp/content
      PULP_ALLOWED_IMPORT_PATHS: "['/var/lib/pulp']"
      PULP_ALLOWED_EXPORT_PATHS: "['/var/lib/pulp']"
    volumes:
      - pulp-data:/var/lib/pulp
      - pulp-assets:/var/lib/pulp/assets
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - albs-network

volumes:
  pulp-data:
  pulp-assets:

networks:
  albs-network:
    driver: bridge
EOF

# Pull images
docker-compose pull

# Start services
docker-compose up -d

# Wait for startup
sleep 30

# Verify
docker-compose ps
```

### Step 4: GitHub OAuth Configuration

Get GitHub OAuth credentials:

```
1. Go to GitHub Settings → Developer settings → OAuth Apps
2. Create New OAuth App:
   - Application name: ALBS rpm-devel
   - Homepage URL: https://albs.yourdomain.local
   - Authorization callback URL: https://albs.yourdomain.local/auth/github/callback

3. Copy Client ID and Client Secret

4. Update in docker-compose.yml:
   GITHUB_CLIENT_ID: <your-id>
   GITHUB_CLIENT_SECRET: <your-secret>

5. Restart:
   docker-compose restart albs-web
```

### Step 5: Nginx Reverse Proxy

```bash
# Configure Nginx
sudo vi /etc/nginx/conf.d/albs.conf

cat > /etc/nginx/conf.d/albs.conf << 'EOF'
upstream albs_backend {
    server localhost:8080;
}

upstream pulp_backend {
    server localhost:8000;
}

server {
    listen 80;
    server_name albs.yourdomain.local;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name albs.yourdomain.local;

    ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    client_max_body_size 1G;

    location / {
        proxy_pass http://albs_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /pulp/ {
        proxy_pass http://pulp_backend/pulp/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

# Start Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx

# Test
curl -k https://albs.yourdomain.local/
```

---

## Mock Chroot Configuration

### Step 1: Create Mock Configs for All Distro/Arch Combos

```bash
# EL8 x86_64
sudo tee /etc/mock/el8-x86_64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-x86_64'
config_opts['target_arch'] = 'x86_64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-x86_64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# EL8 aarch64
sudo tee /etc/mock/el8-aarch64.cfg > /dev/null << 'EOF'
config_opts['chroot_name'] = 'el8-aarch64'
config_opts['target_arch'] = 'aarch64'
config_opts['releasever'] = '8'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=2
reposdir=/etc/yum.repos.d

[baseos]
name=AlmaLinux 8 - BaseOS
baseurl=https://mirrors.almalinux.org/almalinux/8/BaseOS/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[appstream]
name=AlmaLinux 8 - AppStream
baseurl=https://mirrors.almalinux.org/almalinux/8/AppStream/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[extras]
name=AlmaLinux 8 - Extras
baseurl=https://mirrors.almalinux.org/almalinux/8/extras/$basearch/os/
enabled=1
gpgkey=https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux-8

[epel]
name=EPEL 8
baseurl=https://download.fedoraproject.org/pub/epel/8/Everything/$basearch/
enabled=1
gpgkey=https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8
"""

config_opts['macros']['%_topdir'] = '/var/lib/mock/el8-aarch64/root/builddir'
config_opts['use_host_resolv'] = True
config_opts['keep_mounted'] = True
EOF

# Repeat for el9-x86_64, el9-aarch64, fedora39-x86_64, fedora39-aarch64

# Test chroots
sudo mock -r el8-x86_64 --init
sudo mock -r el8-aarch64 --init
sudo mock -r el9-x86_64 --init
sudo mock -r el9-aarch64 --init
```

---

## Repository Structure

### Step 1: Initialize Repositories

```bash
# Create repo directories
mkdir -p /var/lib/repos/{el8,el9,fedora39}/{x86_64,aarch64}/{baseos,appstream,extras,casjay-rpms,casjay-extras}
mkdir -p /var/lib/repos/srpms/{el8,el9,fedora39}

# Initialize empty repos
for dist in el8 el9; do
  for arch in x86_64 aarch64; do
    for repo in baseos appstream extras casjay-rpms casjay-extras; do
      createrepo_c /var/lib/repos/${dist}/${arch}/${repo}/
    done
  done
done

# Verify
ls -la /var/lib/repos/el9/x86_64/
```

### Step 2: Configure Nginx Repo Server

```bash
# Create Nginx config for repos
sudo tee /etc/nginx/conf.d/repos.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen 443 ssl http2;
    server_name repos.yourdomain.local;

    ssl_certificate /etc/pki/albs/albs-cert.crt;
    ssl_certificate_key /etc/pki/albs/albs-key.key;

    root /var/lib/repos;

    location / {
        autoindex on;
        autoindex_exact_size off;
        autoindex_localtime on;
    }
}
EOF

sudo systemctl restart nginx
```

---

## GitHub Integration

### Step 1: Gitea Listener Setup

```bash
# Clone Gitea Listener
cd ~/albs-build
git clone https://github.com/AlmaLinux/gitea_listener.git

# Create config
mkdir -p gitea_listener/config
cat > gitea_listener/config.yaml << 'EOF'
listen_address: 0.0.0.0
listen_port: 8888

web_server_url: https://albs.yourdomain.local
web_server_token: <generated-from-web-ui>

log_level: INFO
log_file: /var/log/gitea_listener.log

github:
  webhook_secret: your-github-webhook-secret-here
EOF

chmod 600 gitea_listener/config.yaml
```

### Step 2: Add to Docker Compose

```bash
# Add to docker-compose.yml:
cat >> docker-compose.yml << 'EOF'

  gitea-listener:
    image: almalinux/gitea_listener:latest
    container_name: gitea-listener
    restart: always
    ports:
      - "8888:8888"
    environment:
      WEB_SERVER_URL: https://albs.yourdomain.local
      WEB_SERVER_TOKEN: <your-token>
      GITHUB_WEBHOOK_SECRET: your-github-webhook-secret-here
    volumes:
      - ./gitea_listener/config.yaml:/etc/gitea_listener/config.yaml:ro
    networks:
      - albs-network
EOF

docker-compose up -d gitea-listener
```

### Step 3: GitHub Webhook Setup

For each package repo (github.com/rpm-devel/cas, etc.):

```
Settings → Webhooks → Add webhook

Payload URL: https://albs.yourdomain.local:8888/webhooks/github
Content type: application/json
Secret: your-github-webhook-secret-here
Events: Push
Active: Yes
```

---

## First Build & Testing

### Step 1: Access ALBS Web UI

```
https://albs.yourdomain.local/

1. Click "Sign in with GitHub"
2. Authorize the app
3. You should see dashboard
```

### Step 2: Create Platforms

```bash
# Via API (get token from web UI first):
curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-8",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'

curl -X POST https://albs.yourdomain.local/api/v1/platforms \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AlmaLinux-9",
    "type": "rpm",
    "architectures": ["x86_64", "aarch64"]
  }'
```

### Step 3: Submit Test Build

```bash
# Create test SRPM
mkdir -p /tmp/cas-build
cd /tmp/cas-build
git clone https://github.com/rpm-devel/cas
cd cas
rpmbuild -bs cas.spec

# Submit to ALBS
curl -X POST https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cas",
    "version": "1.0.3",
    "release": "1",
    "platforms": ["AlmaLinux-9"],
    "architectures": ["x86_64", "aarch64"],
    "git_ref": "https://github.com/rpm-devel/cas.git",
    "git_branch": "main"
  }'
```

### Step 4: Monitor Build

```
Dashboard: https://albs.yourdomain.local/builds/

Watch progress:
1. Build queued
2. x86_64 build running
3. x86_64 complete
4. aarch64 build running
5. aarch64 complete
6. Artifacts uploaded to Pulp
```

### Step 5: Verify Artifacts

```bash
# Check Pulp repository
curl https://albs.yourdomain.local/pulp/api/v3/content/rpm/packages/ \
  -H "Authorization: Bearer <pulp-token>"

# List RPMs
curl https://albs.yourdomain.local/pulp/content/el9/x86_64/baseos/ | grep "\.rpm"
```

### Step 6: Test Client Installation

```bash
# On test machine (AlmaLinux 9)
cat > /etc/yum.repos.d/casjay.repo << 'EOF'
[casjay-baseos]
name=CasjaysDev BaseOS
baseurl=https://repos.yourdomain.local/el9/x86_64/baseos/
enabled=1
gpgcheck=0
EOF

dnf install cas
cas --version
```

---

## Production Operations

### Daily Monitoring

```bash
# Check ALBS status
curl https://albs.yourdomain.local/api/v1/health

# List builds
curl https://albs.yourdomain.local/api/v1/builds \
  -H "Authorization: Bearer <token>"

# Check disk usage
df -h /var/lib/repos /var/lib/albs

# Check database
sudo -u postgres psql -d albs -c "SELECT count(*) FROM builds;"
```

### Nightly Rebuilds Script

```bash
# Create /usr/local/bin/albs-rebuild-third-party.sh
#!/bin/bash

ALBS_URL="https://albs.yourdomain.local"
ALBS_TOKEN="<your-token>"

# Packages to rebuild
PACKAGES=(
  "php-8.2"
  "mariadb-10.6"
  "postgresql-15"
  "nodejs-20"
)

for pkg in "${PACKAGES[@]}"; do
  echo "Building: $pkg"
  curl -X POST $ALBS_URL/api/v1/builds \
    -H "Authorization: Bearer $ALBS_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"name\": \"$pkg\",
      \"platforms\": [\"AlmaLinux-9\"],
      \"architectures\": [\"x86_64\", \"aarch64\"],
      \"git_ref\": \"https://github.com/rpm-devel/$pkg.git\"
    }"
done

# Cron job: 0 2 * * * /usr/local/bin/albs-rebuild-third-party.sh
```

### Automated Repository Sync

```bash
# Copy built RPMs to repos
#!/bin/bash

BUILD_DIR="/var/lib/docker/volumes/pulp-data/_data"
REPO_DIR="/var/lib/repos"

# Sync el9 x86_64
cp $BUILD_DIR/artifacts/el9/x86_64/*.rpm $REPO_DIR/el9/x86_64/casjay-rpms/
createrepo_c $REPO_DIR/el9/x86_64/casjay-rpms/

# Sync el9 aarch64
cp $BUILD_DIR/artifacts/el9/aarch64/*.rpm $REPO_DIR/el9/aarch64/casjay-rpms/
createrepo_c $REPO_DIR/el9/aarch64/casjay-rpms/

# Run via cron: 0 */6 * * * /usr/local/bin/albs-sync-repos.sh
```

### Database Backup

```bash
# Backup PostgreSQL
0 1 * * * sudo -u postgres pg_dump albs | gzip > /var/backups/albs-$(date +\%Y\%m\%d).sql.gz

# Backup Pulp data
0 2 * * * tar czf /var/backups/pulp-$(date +\%Y\%m\%d).tar.gz /var/lib/docker/volumes/pulp-data/_data

# Retention
find /var/backups -name "albs-*.sql.gz" -mtime +30 -delete
```

### Health Checks

```bash
# Script: /usr/local/bin/albs-health-check.sh
#!/bin/bash

ALERT_EMAIL="admin@yourdomain.local"

# Check web server
if ! curl -s -k https://albs.yourdomain.local/health > /dev/null; then
  echo "ALERT: ALBS unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check disk space
USAGE=$(df /var/lib/repos | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $USAGE -gt 80 ]]; then
  echo "ALERT: Repos at ${USAGE}% capacity" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Check PostgreSQL
if ! sudo -u postgres psql -d albs -c "SELECT 1" > /dev/null 2>&1; then
  echo "ALERT: Database unreachable" | mail -s "ALBS Alert" $ALERT_EMAIL
fi

# Cron: */30 * * * * /usr/local/bin/albs-health-check.sh
```

---

## Performance & Scaling

### PostgreSQL Tuning (128GB)

```bash
# Already configured in postgresql.conf with:
shared_buffers = 16GB
effective_cache_size = 100GB
work_mem = 256MB
```

### Mock Cache Optimization

```bash
# In /etc/mock/*.cfg
config_opts['keep_mounted'] = True
config_opts['use_host_resolv'] = True
config_opts['basedir'] = '/var/lib/mock'
```

### Parallel Build Execution

```bash
# ALBS will automatically parallelize builds across cores
# With 32 cores, can run:
# - 4 simultaneous x86_64 builds (8 cores each)
# - 4 simultaneous aarch64 builds (8 cores each)

# Monitor via:
watch -n 5 'ps aux | grep mock'
```

### Docker Memory Limits

Edit docker-compose.yml to set memory limits:

```yaml
albs-web:
  mem_limit: 4g

pulp:
  mem_limit: 4g
```

---

## Troubleshooting

### ALBS Web Not Accessible

```bash
# Check Docker containers
docker-compose ps

# Check logs
docker-compose logs albs-web | tail -50
docker-compose logs pulp | tail -50

# Check Nginx
sudo systemctl status nginx
sudo nginx -t

# Check SSL cert
openssl x509 -in /etc/pki/albs/albs-cert.crt -text -noout
```

### Builds Failing

```bash
# Check Mock chroot
sudo mock -r el9-x86_64 --shell

# Test repo access
sudo mock -r el9-x86_64 --shell dnf repolist

# Rebuild chroot
sudo mock -r el9-x86_64 --scrub=all
sudo mock -r el9-x86_64 --init

# Check Docker logs
docker-compose logs albs-web | grep -i error
```

### Database Connection Errors

```bash
# Check PostgreSQL running
sudo systemctl status postgresql

# Test connection
psql -h localhost -U albs -d albs -c "SELECT 1;"

# Check PostgreSQL logs
sudo tail -50 /var/log/postgresql/*.log
```

### RPMs Not in Pulp

```bash
# Verify Pulp running
docker-compose logs pulp | tail -20

# Check Pulp API
curl -u admin:admin http://localhost:8000/pulp/api/v3/status/

# Check artifact upload directory
ls -la /var/lib/docker/volumes/pulp-data/_data/
```

### GitHub Webhook Not Triggering

```bash
# Check Gitea Listener logs
docker-compose logs gitea-listener | tail -50

# Verify webhook configuration in GitHub
# Settings → Webhooks → Check delivery history

# Test webhook manually
curl -X POST https://albs.yourdomain.local:8888/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature: sha256=test" \
  -d '{}'
```

---

## Success Criteria

✅ Web dashboard accessible at https://albs.yourdomain.local  
✅ GitHub OAuth login works  
✅ Test build submitted and completed  
✅ Both x86_64 and aarch64 builds successful  
✅ Built RPMs in Pulp repository  
✅ Client can install RPMs via dnf  
✅ GitHub webhook triggers automatic builds  
✅ Daily backups running  
✅ Health checks passing  

---

## Quick Command Reference

```bash
# Docker operations
docker-compose ps
docker-compose logs -f albs-web
docker-compose restart albs-web
docker-compose down && docker-compose up -d

# Database
sudo -u postgres psql -d albs
sudo -u postgres pg_dump albs > /backup/albs.sql

# Repository management
createrepo_c /var/lib/repos/el9/x86_64/casjay-rpms/

# Mock testing
sudo mock -r el9-x86_64 --shell
sudo mock -r el9-x86_64 --clean

# System monitoring
docker stats
df -h
du -sh /var/lib/repos/* /var/lib/albs/*

# API calls
curl -H "Authorization: Bearer <token>" \
  https://albs.yourdomain.local/api/v1/builds
```

---

End of ALBS Single-Server Guide