I’m setting up a virtualized development environment for webdev stuff, so that we can easily instantiate our web stuff for each developer. There are some fun DNS tricks to provide a nice interface to it, but I’ll get around to talking about those in a later post (since I haven’t built and configured Lighttpd 1.5, which I’m going to use to proxy requests). A more pressing and basic issue is package management on the jail instances.

Jails will probably be created on-demand — someone will want to do something, so they’ll click a button and boom! a jail is created and it has SSH running and all the software they need installed. Naturally, this means that building ports is not an option — everything will have to be installed via packages. Even then, fetching packages is really slow (and sometimes they’re not available). There are some other catches (“We need version X of this package because that’s what’s in production”). One solution for both is to build the packages locally, in a jail.

The next trick is getting them to serve properly. I added an extra line into that build script –

PACKAGE_SERVE_PATH=/usr/jails/bot-packages/usr/local/www/All/
find ${PACKAGE_DIR_PATH} -name '*.tbz' | xargs -I '{}' -n 1 cp '{}' ${PACKAGE_SERVE_PATH}

And have a jailed Lighttpd instance running in the bot-packages jail, serving /usr/local/www as the document root.

There’s a bitch of a catch here — you must put all your package tarballs in a directory named ‘All’. The pkg_install code is horrendous — when it processes the package dependencies, it takes the URL of the parent packages, lops off the filename and directory, then appends “All/” and the dependent package name. The logic is inconsistent with that of the behavior of pkg_add, so shit breaks.

You can fix this behavior with a patch I whipped up (but it’s easier to just work around the hacky system) –

--- usr.sbin/pkg_install/lib/url.c.orig 2009-03-26 19:56:12. +0000
+++ usr.sbin/pkg_install/lib/url.c      2009-03-26 20:41:44. +0000
@@ -57,7 +57,21 @@
        * to construct a composite one out of that and the basename we were
        * handed as a dependency.
        */
-       if (base) {
+       if (getenv("PACKAGESITE")) {
+               if (strlcpy(fname, getenv("PACKAGESITE"), sizeof(fname))
+                       >= sizeof(fname)) {
+                       return NULL;
+               }
+               if (strlcat(fname, spec, sizeof(fname))
+                       >= sizeof(fname)) {
+                       return NULL;
+               }
+               if (strlcat(fname, ".tbz", sizeof(fname))
+                       >= sizeof(fname)) {
+                       return NULL;
+               }
+       }
+       else if (base) {
           strcpy(fname, base);
           /*
            * Advance back two slashes to get to the root of the package

Anyway, once you’ve either got that patch applied to your base source (ugh) or have all your package tarballs served from a directory named “All/”, you can easily install them in the jails by setting the PACKAGESITE environment variable to the base URI of the package directory before running pkg_add -r -

root@test> setenv PACKAGESITE http://10.0.0.4/All/
root@test> pkg_add -r lighttpd-1.4.22
Fetching http://10.0.0.4/All/lighttpd-1.4.22.tbz... Done.
Fetching http://10.0.0.4/All/pcre-7.8.tbz... Done.

You can probably integrate this into ezjail’s flavor system, but I have to write an interface to manage the jail instances anyway, so I’ll probably just hack the package installation process into that.

Someone needs to rewrite pkg_install. The code is PIG DISGUSTING!

No comments

WordPress 2.7 can bite my fucking ass. STOP FUCKING EATING MY POSTS YOU REEKING PILE OF SHIT!

Anyway. I’ve ranted a lot about using jails as a means of isolating potentially vulnerable services from compromising one another, but there’s a lot more you can do with jails. I’m working on a system right now which provides disposable on-demand development environments: when a developer wants to do some work, they just instantiate a clean mirror of our production system, hammer away, then commit changes and throw away the environment.

One part of this system is maintaining all of the software between both the production environment and the jail instances in which the devel stuff runs — a perfect application for packages. For purposes of easy management, I just want to have a text file with a list of ports, ie

editors/vim
lang/python
www/apache22

Then have a script go through that and build all the appropriate packages which then get picked up and installed automatically when a jail is instantiated (via ezjail, of course). Once you’ve got all of the packages, you can just serve them out over HTTP (by setting PACKAGE_SITE) or dump them into an ezjail flavor.

As much as I love ezjail, it didn’t exactly make this easy. The first catch is that you can’t actually build a package within an ezjail-instantiated jail because of the games it plays with the ports system (ie, the ports makefiles want to write the package tarball to /usr/ports, which is mounted as read-only). I kind of wanted the packages to be written outside of the jail anyway, since I just want to delete the jail when everything’s finished. Just the place for some mountpoint games –

# Change ezjail's symlink to a mountable directory
rm $BUILD_JAIL_PATH/usr/ports
mkdir $BUILD_JAIL_PATH/usr/ports

mount -t nullfs $JAIL_PATH/basejail/usr/ports $BUILD_JAIL_PATH/usr/ports
mount -t unionfs $PACKAGE_DIR_PATH $BUILD_DIR_PATH/usr/ports

The other nasty bit is actually building the packages within a jail from within an unjailed script. All it should take is a couple of for loops to configure and then package everything –

for PORT in $PACKAGE_LIST ; do
    cd /usr/ports/$PORT
    make config-recursive
done

for PORT in $PACKAGE_LIST ; do
    cd /usr/ports/$PORT
    make BATCH=yes package-recursive clean
done

Unfortunately, the ezjail-admin console command uses execvf to spawn a jailed process rather than sh -c (which is probably the way the jail command it uses internally does it), so you have to explicitly pass it the sh -c stuff, followed by the argument.

And I couldn’t figure out how the fuck to quote the nice big chunk of code properly, so I took a slightly different route — just cat it all into a file within the jail, then execute that script directly with ezjail-admin console. It’s retarded, but it works.

[source]

6 comments