How to build *BSD CVSup mirror

Motoyuki Konno <motoyuki@bsdclub.org>

This document describes how to build *BSD (FreeBSD, NetBSD and OpenBSD) CVSup mirror site. Comments and suggestions are always welcome.


Contents

  1. Hardware and Software Requirements
  2. Setup
    1. Common Setup
    2. Getting FreeBSD CVS repository using CVSup
    3. Getting NetBSD CVS repository using CVSup
    4. Getting OpenBSD CVS repository using CVSup
    5. Setup CVSup daemon
  3. Getting server stats
  4. Tuning CVSup server
  5. Security

1. Hardware and Software Requirements


2. Setup

2.1. Common Setup

2.1.1. cvsup and cvsupd user

Example of /etc/master.passwd entries:

cvsup:*:10001:10001::0:0:CVSup Owner:/d/cvsup:/sbin/nologin
cvsupd:*:10002:10002::0:0:CVSupd Owner:/d/cvsupd:/sbin/nologin

Example of /etc/group entries:

cvsup:*:10001:
cvsupd:*:10002:

2.1.2. create directories

# mkdir /d/cvsup                        # cvsup home dir
# mkdir /d/cvsupd                       # cvsupd home dir
# chown root:wheel /d/cvsup /d/cvsupd
# chmod 755 /d/cvsup /d/cvsupd
# cd /d/cvsupd
# mkdir prefixes                        # cvsupd collection (data) dir
# mkdir scan                            # scan file dir
# chown root:wheel prefixes scan
# chmod 755 prefixes scan
# cd /d/cvsup
# ln -s ../cvsupd/scan sup
# mkdir log                             # cvsup log dir
# mkdir lock                            # cvsupd lock file dir
# chown cvsup:cvsup /d/cvsup/log /d/cvsup/lock
# chmod 755 /d/cvsup/log /d/cvsup/lock

2.2. Getting FreeBSD CVS repository using CVSup

2.2.1. create directory

# cd /d/cvsupd
# mkdir prefixes/FreeBSD.cvs                # CVS repository
# mkdir prefixes/FreeBSD-gnats.current      # GNATS DB
# mkdir prefixes/FreeBSD-mail.current       # mailing list archive
# mkdir prefixes/FreeBSD-www.current        # www.FreeBSD.org data
# mkdir prefixes/distrib.self               # CVSup config file
# mkdir prefixes/FreeBSD-jp.cvs             # JP CVS repository (for JP ONLY)
# mkdir prefixes/FreeBSD-jp-distrib.self    # CVSup config file (for JP ONLY)
# chown cvsup:cvsup /d/cvsupd/prefixes/*
# chmod 755 /d/cvsupd/prefixes/*
# mkdir scan/cvs-all
# mkdir scan/gnats
# mkdir scan/www
# mkdir scan/mail-archive
# mkdir scan/distrib
# mkdir scan/jp-all                         # for JP ONLY
# mkdir scan/jp-distrib                     # for JP ONLY
# chown cvsup:cvsup /d/cvsupd/scan/*
# chmod 755 /d/cvsupd/scan/*
# ln -s prefixes/distrib.self/sup sup
# ln -s prefixes/jp-distrib.self/sup-jp sup-jp  # for JP ONLY

2.2.2. create cvsup refuse file

Create cvsup refuse files (/d/cvsupd/scan/distrib/refuse.self and /d/cvsupd/scan/jp-distrib/refuse.self) containing the following contents. Please make sure that refuse files are owned by root and their file modes are 0444.

*.sh
cvsupd.access
cvsupd.passwd
prefixes
sup.client
supfile*

2.2.3. create supfile-freebsd

Create /d/cvsup/supfile-freebsd. The following is an example. Please specify the nearest CVSup mirror site instead of cvsup.jp.FreeBSD.org.

If you want to become an official FreeBSD CVSup mirror, you can access CVSup master server. Please contact hub@FreeBSD.org (cvsupadm@jp.FreeBSD.org if your site is in Japan).

# Supfile for FreeBSD CVSup mirror
*default host=cvsup.jp.FreeBSD.org
*default base=/d/cvsup
*default delete use-rel-suffix umask=002

cvs-all      release=cvs     prefix=/d/cvsupd/prefixes/FreeBSD.cvs
gnats        release=current prefix=/d/cvsupd/prefixes/FreeBSD-gnats.current
www          release=current prefix=/d/cvsupd/prefixes/FreeBSD-www.current
mail-archive release=current prefix=/d/cvsupd/prefixes/FreeBSD-mail.current
distrib      release=self    prefix=/d/cvsupd/prefixes/distrib.self

# following 2 lines are for JP mirror sites ONLY.
jp-all       release=cvs     prefix=/d/cvsupd/prefixes/FreeBSD-jp.cvs
jp-distrib   release=self    prefix=/d/cvsupd/prefixes/FreeBSD-jp-distrib.self

2.2.4. create update script

Create a shell script update-freebsd.sh. Please specify S_TIME so that cvsup runs without -s flag once a day.

#!/bin/sh
export PATH=/bin:/usr/bin:/usr/local/bin
export LANG=C

SUPFILE=/d/cvsup/supfile-freebsd
LOCK=/d/cvsup/lock/freebsd.lock
LOG=/d/cvsup/log/freebsd.log
ADMIN="your@address"
S_TIME=0430

if [ x`date "+%H%M"` = x$S_TIME ]; then
        SFLAG=
else
        SFLAG=-s
fi
(
        echo
        echo "========== CVSup $SFLAG started at `date`"
        if ! cvsup $SFLAG -g -L 2 -r 1 -l $LOCK $SUPFILE; then
                tail $LOG |\
                mail -s "CVSup FreeBSD failed" $ADMIN
        fi
        echo "========== CVSup $SFLAG ended at `date`"
) >> $LOG

2.2.5. Execute update script

Become cvsup user (using sudo or su) and execute the update script. If everything goes well, add an entry of the update script to crontab. Most of FreeBSD cvsup mirror sites in Japan update their mirror at least once an hour.


2.3. Getting NetBSD CVS repository using CVSup

2.3.1. create directory

# cd /d/cvsupd
# mkdir prefixes/NetBSD.cvs
# mkdir prefixes/NetBSD-jp.cvs              # for JP ONLY
# mkdir prefixes/NetBSD-distrib.self
# chown cvsup:cvsup /d/cvsupd/prefixes/*
# chmod 755 /d/cvsupd/prefixes/*
# mkdir scan/netbsd
# mkdir scan/netbsd-distrib
# mkdir scan/netbsd-jp-all                  # for JP ONLY
# mkdir scan/netbsd-jp-distrib              # for JP ONLY
# chown cvsup:cvsup /d/cvsupd/scan/*
# chmod 755 /d/cvsupd/scan/*
# ln -s prefixes/NetBSD-distrib.self/sup-netbsd sup-netbsd

2.3.2. create cvsup refuse file

Create cvsup refuse files (/d/cvsupd/scan/netbsd-distrib/refuse.self and /d/cvsupd/scan/netbsd-jp-distrib/refuse.self) containing the following contents. Please make sure that refuse files are owned by root and their file modes are 0444.

*.sh
cvsupd.access
cvsupd.passwd
prefixes
sup.client
supfile*

2.3.3. create supfile-netbsd

Create /d/cvsup/supfile-netbsd.

# Supfile for NetBSD CVSup mirror
*default host=cvsup.jp.NetBSD.org
*default base=/d/cvsup
*default delete use-rel-suffix umask=002

netbsd            release=cvs  prefix=/d/cvsupd/prefixes/NetBSD.cvs
netbsd-distrib    release=self prefix=/d/cvsupd/prefixes/NetBSD-distrib.self

# the following 2 lines are for JP mirror sites ONLY.
netbsd-jp-all     release=cvs  prefix=/d/cvsupd/prefixes/NetBSD-jp.cvs
netbsd-jp-distrib release=self prefix=/d/cvsupd/prefixes/NetBSD-distrib.self

2.3.4. create update script

Create a shell script update-netbsd.sh. Please specify S_TIME so that cvsup runs without -s flag once a day.

#!/bin/sh
export PATH=/bin:/usr/bin:/usr/local/bin
export LANG=C

SUPFILE=/d/cvsup/supfile-netbsd
LOCK=/d/cvsup/lock/netbsd.lock
LOG=/d/cvsup/log/netbsd.log
ADMIN="your@address"
S_TIME=0430

if [ x`date "+%H%M"` = x$S_TIME ]; then
        SFLAG=
else
        SFLAG=-s
fi
(
        echo
        echo "========== CVSup $SFLAG started at `date`"
        if ! cvsup $SFLAG -g -L 2 -r 1 -l $LOCK $SUPFILE; then
                tail $LOG |\
                mail -s "CVSup NetBSD failed" $ADMIN
        fi
        echo "========== CVSup $SFLAG ended at `date`"
) >> $LOG

2.3.5. Execute update script

Become cvsup user (using sudo or su) and execute the update script. If everything goes well, add an entry for the update script to crontab. cvsup.jp.NetBSD.org updates its contents every 8 hours.


2.4. Getting OpenBSD CVS repository using CVSup

2.4.1. create directory

# cd /d/cvsupd
# mkdir prefixes/OpenBSD.cvs
# mkdir prefixes/OpenBSD-distrib.self
# chown cvsup:cvsup /d/cvsupd/prefixes/*
# chmod 755 /d/cvsupd/prefixes/*
# mkdir scan/OpenBSD-all
# mkdir scan/OpenBSD-distrib
# chown cvsup:cvsup /d/cvsupd/scan/*
# chmod 755 /d/cvsupd/scan/*
# ln -s prefixes/OpenBSD-distrib.self/sup-openbsd sup-openbsd

2.4.2. create cvsup refuse file

Create cvsup refuse files (/d/cvsupd/scan/OpenBSD-distrib/refuse.self) containing the following contents. Please make sure that refuse files are owned by root and their file modes are 0444.

*.sh
cvsupd.access
cvsupd.passwd
prefixes
sup.client
supfile*

2.4.3. create supfile-openbsd

Create /d/cvsup/supfile-openbsd.

If you want to become an official OpenBSD CVSup mirror, please contact brian@openbsd.org or beck@openbsd.org. OpenBSD does not have the central CVSup server, so you have to use sup to update the repository from an official root server. If you want to use cvsup.jp.OpenBSD.org as a master server instead, please contact cvsupadm@openbsd.bsdlab.org.

# Supfile for OpenBSD CVSup mirror
*default host=cvsup.jp.OpenBSD.org
*default base=/d/cvsup
*default delete use-rel-suffix umask=002

OpenBSD-all      release=cvs  prefix=/d/cvsupd/prefixes/OpenBSD.cvs

# NOTE: -distrib collection is available from cvsup.jp.OpenBSD.org only
OpenBSD-distrib  release=self prefix=/d/cvsupd/prefixes/OpenBSD-distrib.self

2.4.4. create update script

Create shell script update-openbsd.sh. Please specify S_TIME so that cvsup runs without -s flag once a day.

#!/bin/sh
export PATH=/bin:/usr/bin:/usr/local/bin
export LANG=C

SUPFILE=/d/cvsup/supfile-openbsd
LOCK=/d/cvsup/lock/openbsd.lock
LOG=/d/cvsup/log/openbsd.log
ADMIN="your@address"
S_TIME=0430

if [ x`date "+%H%M"` = x$S_TIME ]; then
        SFLAG=
else
        SFLAG=-s
fi
(
        echo
        echo "========== CVSup $SFLAG started at `date`"
        if ! cvsup $SFLAG -g -L 2 -r 1 -l $LOCK $SUPFILE; then
                tail $LOG |\
                mail -s "CVSup OpenBSD failed" $ADMIN
        fi
        echo "========== CVSup $SFLAG ended at `date`"
) >> $LOG

2.4.5. Execute update script

Become cvsup user (using sudo or su) and execute the update script. If everything goes well, add an entry for the update script to crontab. cvsup.jp.OpenBSD.org updates its CVS repository every 2 hours. For more information, please refer to http://cvsup.jp.OpenBSD.org/.


2.5. Setup CVSup daemon

2.5.1 Create cvsupd.access

Create a CVSup daemon access control file /d/cvsupd/cvsupd.access. A sample is as follows:

# ============================================================
# Limit 1 connection at a time for each host
# ============================================================
-0.0.0.0/0/32 1

# ============================================================
# Always allow access from localhost (or ssh port forwarding)
# ============================================================
+localhost

# ============================================================
# Limit 10 connections total at any time
# ============================================================
-0.0.0.0/0 10

# ============================================================
# Allow anonymous access
# ============================================================
+0.0.0.0/0

2.5.2 logging

When "-l" flag is specified in the form of "-l @local0," the logging is done via syslog to the local0 facility. This is a recommended way since it makes the maintenance easier.

Probably you may want to add the following lines into /etc/syslog.conf in order to separate log messages generated by cvsupd processes from others.

!cvsupd
*.*                                     /var/log/cvsupd.log

Also, the default syslog.conf files in *BSD include "*" that matches all facilities, for example:

*.notice;kern.debug;lpr.info;mail.crit;news.err /var/log/messages

When such lines are included, "local0" messages will also be logged in /var/log/messages. To prevent that behavior, you can add "local0.none" into the lines where the "*" is used:

*.notice;kern.debug;lpr.info;mail.crit;news.err;local0.none /var/log/messages

For more information, see syslog.conf(5).

2.5.3 create cvsupd startup script

An example of a startup script (/d/cvsupd/cvsupd.sh) is as follows:

#!/bin/sh
export PATH=/usr/bin:/usr/local/bin
export LANG=C

exec nice -10 cvsupd -C 128 -l @local0 -b /proj/cvsupd -s scan \
        -c sup:sup-jp:sup-netbsd:sup-openbsd

Add the following lines into /etc/rc.local:

[ -x /proj/cvsupd/cvsupd.sh ] && \
    su -m cvsupd -c /proj/cvsupd/cvsupd.sh && echo -n ' cvsupd'

If your server is run by FreeBSD, you may use another startup script that follows the framework of FreeBSD's local startup scripts. Here is a sample of /usr/local/etc/rc.d/cvsupd.sh:

#!/bin/sh

case "$1" in
start)
    [ -x /d/cvsupd/cvsupd.sh ] || exit 1
    su -m cvsupd -c /d/cvsupd/cvsupd.sh && echo -n ' cvsupd'
    ;;
stop)
    killall cvsupd && echo -n ' cvsupd'
    ;;
esac

3. Getting server stats

Not yet.


4. Tuning CVSup server

Not yet.


5. Security

5.1. cvsup

CVSup client (cvsup) needs write permission of the following directories. Make sure that there is no file or directory (except the following directories) that cvsup user can write.

5.2. cvsupd

CVSup daemon (cvsupd) does NOT create and/or write any file. Make sure that there is no file or directory that cvsupd user can write.