The random rantings of a concerned programmer.

Invoking mount(2) in FreeBSD 8.x

July 17th, 2011 | Category: Random

So I’m still writing Go bindings for a lot of common FreeBSD functionality. Yesterday I implemented a means to list all mounted filesystems, so today I’m writing the bindings to mount(2) to mount/umount them.

If you look at the man page for mount, you’ll see that the function signature looks like this:

    int mount(const char *type, const char *dir, int flags, void *data);

The void* should scare you.

I haven’t been able to dig up any information about what the fuck should be passed to it (granted, I haven’t looked very hard because, judging from the contents of src/sbin/mount_*/*.c in the FreeBSD sources, it’s been entirely superseded by nmount.

    int nmount(struct iovec *iov, u_int niov, int flags);

Poking around, struct iovec (eventually included from sys/uio.h) is defined as this:

struct iovec {
    void *iov_base;
    size_t iov_len;

Effectively, nmount takes an array of these structs which effectively form a flattened vector of (key, value) tuples. As far as I can tell, iov_base is always a NULL-terminated char*, and iov_len should be strlen(iov_base) + 1 (for the NULL terminator).

Unfortunately, the only hints that man 2 nmount gives us is

The following options are required by all file
           fstype     file system type name (e.g., ``procfs'')
           fspath     mount point pathname (e.g., ``/proc'')

     Depending on the file system type, other options may be recognized or
     required; for example, most disk-based file systems require a ``from''
     option containing the pathname of a special device in addition to the
     options listed above.

So far, the only way I’ve been able to find the actual options is to dig through mount_* sources and see what they use, but it’s pretty gross. Take, for example, the following two filesystems:

  • nullfs simply layers one vnode on top of another, effectively grafting one directory over another.
  • unionfs (roughly) does the same thing, but still lets you access the grafted-over directory in read-only mode (and can be configured to do cool shit like copy-on-write).

They’re pretty close, but let’s look at the arguments that each of them take:

  • fstype: “nullfs”
  • fspath: Path to the directory to graft over.
  • target: Path of the directory that’s being grafted onto another.

IMHO, "target" should be "from", bikesheds, et. al.

  • fstype: “unionfs”
  • fspath: Path to the directory where the unionfs will be mounted.
  • from: Same as “target”, above.
  • below: Makes “fspath” writable, “from” read-only (swaps default behavior)
  • errmsg: …I have no fucking idea, a char[255] which presumably is used as a buffer instead of errno?
  • …anything else passed as -oyour=mom passed to mount_unionfs?!

Maybe this is more a gripe that unionfs seems to be very shitty. And maybe I just haven’t found a nice magical table of options that every filesystem takes. But FFFFFF SERIOUSLY >:(


FreeBSD’s gmirror is a piece of shit

December 10th, 2010 | Category: Random

I do a lot of work on Fakku!, which is a weeaboo porn site. I swear the damn thing is cursed — it chews through HDDs at a daunting rate (though half of that, I suspect, is because the old host FDC, played dominos with the drives before installing them). Fakku’s main HDD (an SSD drive on top of that) blew up a month ago. Then a couple of weeks later the replacement drive they gave us also blew up. Naturally I was fucking through with their bullshit and migrated everything over to 100tb.

Instead of keeping with the monolithic-server-to-do-everything (8 cores and 16GB of RAM? fml), I decided it would be more cost effective to buy the shittiest servers they offer (4 core/8GB) and then balance the load between them. And then use gmirror to software RAID-1 the drives because restoring from backups takes bloody ages (separate datacenter, etc).


To install gmirror, you basically just load a kernel module in, create an array, then add the drives to an array. It’s kind of nice because you can create an array from an already-in-use drive, then add another drive to the array and it’ll work fine. This is helpful because sysinstall, FreeBSD’s installer, is an archaic piece of shit that only really supports plain vanilla installs on a single drive.

So I do all that jazz on the two machines and reboot them. One of them comes back up, the other one doesn’t. Fuck. So I read through the documentation, check the hardware specs, etc. The troubleshooting section has an “oh if it doesn’t work try compiling the kernel module in statically”. Guess what didn’t work. After dicking with the damn thing for a couple of hours I gave up on the second machine and just decided to run it without RAID.

Naturally, you’d assume having a RAID-1 setup would drastically increase your parallel read performance (and given we’re serving pornography, this is a pretty important figure). HAHA WELL YOU’D BE WRONG. Running a RAID-1 setup with gmirror does absolutely nothing to improve read throughput, which is fucking retarded.

But the fun doesn’t stop there.

A week after running the site on this setup, the RAID’d application server starts crashing randomly. It’s a soft crash — the machine is still running — you can ping it, you can do the first part of an SSH handshake — but anything involving disk I/O (validating SSH keys, requesting masturbation materials, etc) doesn’t work. Hard reboots don’t fix it — the data center tech has to go do something to it to power it back up (and they never tell you fucking anything). And there’s nothing in any fucking log about why it’s crashing. smartctl says the disks are peachy.

Whenever it’s brought back up, gmirror decides “SHIT A DRIVE IS DEGRADED MUST REBUILD THE ARRAY”. I’m not sure what the fuck it’s fucking doing but it’s probably related to the problem. God help you if the power cuts or anything bad happens while it’s rebuilding the array, too. I found the following advice on the FreeBSD mailing lists:

1) turn off gmirror
2) clear gmirror header on both providers
3) run fsck the other drive (not ad6, but the other used on mirror).
4) pray
5) after fsck will end it successfully (it should), create gmirror with the disk you checked
gmirror label
gmirror-name /dev/thedisk

6) reboot and start the system. should go well.
7) after system is running and not too much needing disk I/O, do
gmirror insert gmirror-name /dev/ad6
8) pray again, but with much less fear.
9) if gmirror will finish rebuild, all right.

Yeah… no thanks.

Fuck gmirror.


lighttpd, etc.

December 09th, 2007 | Category: Random

I made delicious ramen today. I finally managed to use the exact right amount of water to essentially steam the noodles so they both come out cooked and without the “soup” portion. I then seasoned them with some cumin, red pepper, Tabasco and salt (and msg) and it was delicious. Yum yum. Overcooked them a little bit, but overall pretty good and spicy.

Anyway, two points of interest for this post -

  1. I got Erlang running off two machines (one master, one node)! I kind of cheated and NFS mounted the master’s filesystem on the node by hand; I need to figure out how I’m going to export things like applications and stuff. I tried just booting the node from the root filesystem but that didn’t go over so well.
  2. lighttpd has built-in load-balancing. This, in addition to lighttpd already sounding fucking sweet, is just fucking awesome. I’m trying to install and configure it now.

I also went ahead and downloaded BProc just to see how many modifications were made to the Linux kernel to get it to work, and I lol’ed like a motherfucker. I doubt I’ll be able to get that shit working on FreeBSD for a long, long time, simply because I’m not at all familiar with the structure of FreeBSD’s kernel.

Ha ha ha oh wow.

No comments


December 07th, 2007 | Category: Random

FreeBSD 7.0-BETA4 is out; I’m wiping my sandbox machines right now and doing a fresh install (building world right now). This will give me a chance to work out the NFS issues I’ve been having.

With regards to that problem, I think I’m going to scratch the idea of giving each node persistent state, which kind of simplifies things. What I eventually want to have is a FreeBSD implementation of BProc, which is a set of kernel modifications in Linux to get a distributed process space. Going to take a lot of work to get anything anywhere close to useful, but hell, that’s what winter break is for, right?

Still kind of bummed out about the whole hardware issue, but I managed to repair the heatsink mounting stub I broke off one of my boards. Patched up fine with a couple dabs of superglue, which was kind of nice. And I got an email this morning about payment for some freelance work I did which’ll provide a nice $200 to spend on cases for the two boards I’m running naked.

Blah. It’ll all come together eventually <3

No comments