<= back to main index


syncopt - A Flexible and Simple Approach to Package Install

Last modified: Jun 4 11:03

Overview

The syncopt script and its associated work practices are yet another approach to the standard sysadmin problem of keeping multiple machines' software installations up to date. It is independent of the vendor's packaging scheme and thus you can use either your vendor's system or syncopt, or both!

Also see the manual entry: syncopt.1.html.

The core notion is that the install is done once on a central machine and then syncopt takes care of attaching that to every client, often via cron.

Like most such solutions, syncopt has to achieve a few goals:

As implemented, syncopt achieves this with the following advantages:

lightweight
If a package is not to be installed local to a client then the burden is usually just two symlinks on the client.

optional
You can install packages with syncopt or with the vendor's packaging scheme, or both!

permits trial installs
Under syncopt you can install multiple versions of the same package for trial or legacy purposes.

centralised
The default package version is controlled by a symlink on the master host; change that and all the clients will follow suit next time they run syncopt.

customisable
Clients can control which packages are local and also override which version is their default for a given package.

It presumes you have a ``large'' main machine which can have an instance of everything installed on it and that your other machines generally have mostly-local core installs of the OS and main packages and probably run most of the optional stuff from the main server via NFS or other network filesystem.

Naming Issues

We will approach the last criterion first, since it's crucial to having things work smoothly in the long run. The vendor namespace tends to be /usr and its children - /usr/bin, etc. So, we elect not to install there. Likewise, the ``user namespace'' or ``local customisation namespace'' tends to be /usr/local and its children. Although our users could use a /opt style scheme, /usr/local is familiar and has its own conventions.

Accordingly, I've taken a leaf out of Sun's book http://www.sun.com/ and used /opt for my syncopt scheme. This doesn't actually fly in the face of Sun, as Solaris packages are installed with names of the form [A-Z]+[a-z0-9]+. Syncopt generally uses names of the form [a-z][a-z0-9]*-version, so we can play happily in Sun's field without conflict.

File Tree Organisation

That decision made, here is how we have things arranged at my workplace:

/u/syncopt
We have a special user called ``syncopt'' to store the master copies of everything to install in /opt. That way we can put the master repository wherever we like.

Under that directory is a subdirectory called common for the architecture independent stuff (config file, image/icon archives, scripts etc) and a subdirectory matching the $ARCH environment variable for the architecture dependent stuff (pretty well everything that gets compiled).

Here's the directory listing from ours:

        % ls -la /u/syncopt/.
        total 48
        drwxr-xr-x    8 root     bin          8192 Oct 12 14:45 .
        drwxr-xr-x   14 root     root         8192 Oct 11 15:05 ..
        drwxrwsr-x   17 root     technic      8192 Oct 14 11:56 common
        drwxrwsr-x    5 cameron  technic      8192 Sep  3 11:55 freebsd.x86.freebsd
        drwxrwsr-x    2 root     geeks          96 Jul 20  2000 redhat.ppc.linux
        drwxrwsr-x  184 cameron  geeks        8192 Oct 15 10:11 redhat.x86.linux
        drwxrwsr-x  155 root     geeks        8192 Sep 20 11:36 sun.sparc.solaris
        drwxrwsr-x    4 root     technic        96 Mar 13  2000 sun.sparc.sunos

/opt
/opt is a real, local, directory on every client machine. On machines which get most things from the server, this is a forest of symlinks pointing at the central copies in /u/syncopt. On machines with local copies of some things the local copies are real directories, exact images of those in the central opt directory and the non-local things are, of course, symlinks.

The purpose of the syncopt script itself is to make the /opt directory correctly configured with respect to /u/syncopt.

The core algorithm is simple and conservative: for every subdirectory found in /u/syncopt/arch or /u/syncopt/common, check the matching name in /opt. If it is missing, make it a symlink to the matching central item. If it is a directory, ensure the contents are an exact match for the central item, using rsync(1) http://rsync.samba.org/.

This behaviour can be overridden with the /opt/.syncopt configuration file as described below.

Using syncopt

Local Package Install

As a consequence, to make an instance of something local to a client machine (let's call it pkg, release version), remove the local symlink:

        $ rm /opt/pkg-version

Make a stub directory:

        $ mkdir /opt/pkg-version

Alternatively, just edit the .syncopt file and add this line:

        pkg-version local

Run syncopt:

        $ syncopt -x

That syncs everything. You can just do the new package like this:

        $ syncopt -x pkg pkg-version

which syncs the generic (unversioned) link and the version specific local directory.

Undoing a Local Package Install

To make a once-local copy remote, remove the local copy:

        $ rm -rf /opt/pkg-version

Also, if you edited the .syncopt file as above, remove that line.

Run syncopt:

        $ syncopt -x pkg pkg-version

Bringing a Client into Sync after a New Package Install

To set up a new client's /opt directory after a fresh install:

        $ mkdir /opt    # if necessary
        $ syncopt -x

/opt/.syncopt

The behaviour deduced from the presence or absence of a directory can be overridden with the /opt/.syncopt file, which contains line of the form:

        pkg version

to make version the default package version on this particular machine, or

        pkg local

to force a package to be local on this machine, or

        pkg-version local

to make a particular version local, or

        pkg nosync

to not run syncopt on it at all.

Installing packages for use with the syncopt scheme

A scheme such as this is naturally not useful without things to install this way.

For very small packages (usually only a single command and matching manual entry) it's often not worth bothering with /opt, instead installing them in the traditional way with the executable in /opt/bin and the manual in /opt/man and so forth.

For larger packages (netscape, emacs, vmware, the pbmtools, elm, mh, etc) the /opt comes into its own.

When building the package from source or installing a binary distribution, tell the package to install in /opt/package-version.

Usually version is the release version, but occasionally I tack extra info into it, such as the build platform where $ARCH is too vague. For example, I use redhat.x86.linux for RedHat Linux platforms. Unfortunately, these are not always binary compatible and so I might make the version be version-rh9 to indicate it was built on RedHat 9. In this way I can choose, say, a -rh7 build for RedHat 7 boxes.

For packages built with GNU autoconf [http://sourceware.cygnus.com/autoconf/] this usually is as simple as adding the --prefix option to the configure run:

        $ ./configure --prefix=/opt/package-version

Do the build as normal. If it is successful, set your umask to 2 for the install. This presumes a group exists with write privileges in /u/syncopt (ours is called ``geeks''), then make the master directory and link it to the local /opt:

        $ umask 2
        $ mkdir /u/syncopt/arch/pkg-version
        $ ln -s /u/syncopt/arch/pkg-version /opt/.

then install as normal:

        $ make install

Then adjust the permissions on the master to prevent accidental damage in the future:

        $ cd /opt/pkg-version   # should take you into /u/syncopt
        $ chmod -R a-w .

If that goes well, and this is to be the ``default'' version of the package, add the generic symlink:

        $ cd /u/syncopt/arch
        $ rm -f pkg
        $ ln -s pkg-version pkg

Then run syncopt or just make the same symlink in the local /opt by hand. In this way you can now talk about a generic /opt/package in shell scripts, /opt/pkg/bin in $PATH and so forth without having these things know about the version.

This has the added advantage that upgrades are done by installing the new version and switching the symlink in /u/syncopt. This way you can keep multiple versions around without conflict, which is often quite handy with unstable or experimental upgrades or for legacy uses.

It is quite important to never mention /u/syncopt outside of this scheme - by having the package and everything which uses it believe firmly in /opt, making things local or shuffling versions ``just works''.

Having done the central install in this way, saying:

        $ mkdir /opt
        $ syncopt -x

on every client machine suffices to finish things off. This can be put in a nightly cron job on each machine if you desire.

Complication

It all seemed so easy, didn't it? Well, I've glossed over a few issues. Let's peel back the paint:

What about $PATH?

Everything you've installed in /opt will most likely not be in your users' $PATH or $MANPATH, so they won't be able to simply type the name of the package and have it work.

There are three main approaches to correcting this situation:

Add a special path to $PATH
Edit your main /etc/profile to insert the relevant bits into your users' environment variables, eg:
        PATH=$PATH:/opt/package/bin
        export PATH

To keep central control of this kind of thing, here we keep this stuff a file in /opt/config/shell and simply source that from the /etc/profile. That way, aside from the initial edit of /etc/profile on each client machine (to add the source line), future package installs need only involve editing the central file in /u/syncopt/common/config/shell and rerunning syncopt.

Supply a small wrapper script
Another class of package is the package with one main command and a bunch of utility commands used only during a run of the main command. ELM [http://www.cis.ohio-state.edu/hypertext/faq/usenet/elm/FAQ/faq.html] and MH [http://www.faqs.org/faqs/mail/mh-faq/part1/preamble.html] are classic cases of this; users usually run the main command (elm or mh or xmh) and the many little utilities are run within the system. Nevertheless the ``stock'' install puts innumerable utility executables in the main bin directory of the victi^Wtarget machine, many with horribly generic names like ``next'' or ``forward''. It is far better that these packages keep their utilities squirrelled away in places like /opt/elm/bin. Instead, we put a single wrapper script in /opt/bin which inserts the /opt/package/bin directory at the front of the $PATH and then execs the real executable. In this way the main package is accessible as normal and the general utility names do not pollute the default command namespace. For example, the my elm wrapper:
        #!/bin/sh
        PATH=/opt/elm/bin:$PATH
        MANPATH=/opt/elm/man:$MANPATH
        export PATH MANPATH
        exec /opt/elm/bin/elm ${1+"$@"}

It is installed as /opt/bin/elm.

Note: users who truly wish the package's utilities in their default $PATH can naturally add /opt/package/bin to their own path in their .profile file.

Populate /opt/bin etc with links
Some packages have a single executable (like netscape) or genuinely have several executables with nice distinctive names (like netpbm [http://freshmeat.net/projects/netpbm/]). In that case the natural thing to do is to go to /opt/bin and add symlinks to the executable(s) and matching symlinks for the manual entries, eg:
        $ cd /opt/bin
        $ ln -s /opt/netscape/netscape
        $ ln -s /opt/netscape-4.79/netscape netscape-4.79

or

        $ cd /opt/bin
        $ ln -s /opt/netpbm/bin/* .
        $ cd /opt/netpbm-10.17/bin
        $ for bin in *
        > do ln -s /opt/netpbm-10.17/bin/$bin /opt/bin/$bin-10.17
        > done

Notice that we make both unversioned and version links so that users can run the default app or a specific version if needed.

What about per-machine dynamic files?

Some packages include their dynamic components in the install. For example, NNTPCache [http://www.nntpcache.org/] includes a var subdirectory to hold its news spool.

My approach with these is to make a matching /var/package subdirectory (or /var/spool/package is appropriate) on the client machines which will run the package, and to make the var subdirectory in the install directory a symlink to /var/package. This means that the package can continue to believe in its ``normal'' install structure and still correctly access the local, dynamic files. It also protects those files from syncopt by shifting them out of the /opt area.

if there's significant structure to the ``var'' directory I tend to move it sideways in the install tree:

        $ cd /u/syncopt/redhat.x86.linux/nntpcache-3.0.1
        $ mv var var.dist
        $ ln -s /var/spool/nntpcache var

Then on any client machine that will actually run it:

        $ mkdir /var/spool/nntpcache
        $ rsync -avHP /opt/nntpcache/var.dist/. /var/spool/nntpcache/.

and it's ready to go.

See Also

RedHat [http://www.redhat.com/]'s RPM [http://www.rpm.org/].

Debian [http://www.debian.org/.debian's apt [http://www.debian.org/doc/manuals/apt-howto/index].

Sun Solaris [http://www.sun.com/software/solaris/]' pkgadd [http://docs.sun.com/ab2/coll.47.8/SYSADV1/%40Ab2PageView/25509].