Bootstrapping Triton on whatever you can find

As part of my testing environment at work, I was provided some hardware to run Triton on. This isn’t the Joyent recommended hardware stack, but it meets many of the requirements, so it became a matter of getting it all up and running on this hardware. Other hardware was not a choice.

I had some advantages. For instance, all 3 servers were in the same configuration. I’m not yet trying to fix the mixed configuration challenge. It appears that will happen next week.

My method involves alterations to a Joyent official Triton image to allow it to function in my hardware stack. Triton uses what it finds on the headnode to boot the rest of the computers in the stack. I’ve used this to my advantage in altering the universe.

The first change I needed to make was to /kernel/drv/sd.conf. Using instructions from a SmartOS community member, I altered the boot media to have a boot time load of a replacement file on the ramdisk image. This was to support the annoyingly broken disk controller on the hardware I was using, which lies about the power management state of the disks.

The second change is that, basically, I either had too much ram, or not enough disk.

The servers I’m using are old VMware hosts. They weren’t supposed to have local disk on them. The drive cages let me load 8 disks, of which I have 146Gb SAS drives, which were extracted from other machines. Joyent’s disklayout tool looks at this configuration, and comes up with a zones pool of 4x 2 disk mirrors. That configuration results in just 584 Gigs of disk space. The system then fails to configure because the disk isn’t sufficiently larger than ram. This is because the Joyent way of flying Illumos is to reserve enough disk for all of ram as swap, thus making sure all ram has a spot on disk to go if necessary.

Thus, the second fix is to alter the setup tools to force the disk layout to raidz2, thus resulting in about 1.1Tb of useable disk. There’s no clean configuration option for this, but the file that needed to be altered does sit on the boot usb media directly, and is trivial to edit. At system startup, anything on the usb media is copied into the running system to execute.

After these two fixes are in place, Triton lit for me. The additional compute nodes used the modified variants that were present on the headnode’s usb key, thus in addition to getting the headnode up, all the compute nodes booted with the changes.

I expect that if you also have non Joyent certified hardware, you might have to make other alterations. Good luck.