gwax's rumblings

I have a very large RAID6 array (11TB) with an ext4 par­ti­tion that, due to par­tic­u­lar use cases, has become dis­gust­ingly frag­mented (~40% non-contiguous accord­ing to fsck). Sadly, as much as ext4 is designed to resist frag­men­ta­tion issues, my par­ti­tion has been hav­ing sub­stan­tial per­for­mance issues.

The ext4 defrag pro­gram e4defrag would be an ideal solu­tion to my prob­lems but it is not yet sta­ble enough for pro­duc­tion use.

Putting some thought into the mat­ter, I have came up with a tech­nique for defrag­ment­ing my par­ti­tion using only sta­ble tools. My process is very slow and requires sub­stan­tial peri­ods of down­time but pre­lim­i­nary results are good.

At present, I have 2TB of free space, which means that I can copy files off my frag­mented par­ti­tion and then copy them back to decrease the frag­men­ta­tion of indi­vid­ual files; alter­na­tively, the appli­ca­tion shake can be used to accom­plish a sim­i­lar result. However, the copy/recopy solu­tion will only work if the free space on my par­ti­tion is not frag­mented. Running e2freefrag I found that the free space on my par­ti­tion is mon­strously fragmented.

However, clever use of resize2fs can almost com­pletely defrag­ment the free space of a par­ti­tion. If you unmount the par­ti­tion, shrink it to a min­i­mum size and the expand it, the vast major­ity of the free space will be moved to a con­tigu­ous region at the end of the partition.

If the par­ti­tion is /dev/md1 and is mounted at /mnt/fragmented, your file sys­tem can be defrag­mented with the fol­low­ing set of commands:

$ umount /mnt/fragmented
$ fsck -f /dev/md1
$ resize2fs -M -p /dev/md1
$ resize2fs -p /dev/md1
$ mount /dev/md1 /mnt/fragmented
$ shake -S 0 /mnt/fragmented

This will result in some defrag­men­ta­tion of your par­ti­tion but it is likely that the process will need to be repeated mul­ti­ple times to achieve a sub­stan­tial degree of defragmentation.

I recently con­sid­ered the nam­ing of abyss and decided that the word oubli­ette made a bet­ter name. Oubliette is a much more appro­pri­ate descrip­tion in under­stand­ing the theme and pur­pose of the machine as a replace­ment for thevoid.

The name change hap­pened a while ago but I’m bring­ing it up now because oubli­ette is on my mind. Having just pur­chased six Western Digital 1.5TB dri­ves and another SATA con­troller, it is finally time to obso­lete the last rem­nants of thevoid. This actu­ally stirs some amount of sen­ti­men­tal­ity in me but, even though the hard­ware and oper­at­ing sys­tem are no longer that of thevoid, oubli­ette will carry the pur­pose and data for­ward. It is almost as though oubli­ette is my ghola thevoid.

For those with tech­ni­cal inter­est, the dri­ves will be arranged in an mdadm RAID6 array with luks dm-crypt. This should pro­vide me with 6TB, secure stor­age, and two dri­ves worth of failsafe.

I have finally decided, after 7 years of faith­ful ser­vice, that thevoid is due for decom­mis­sion­ing as my pri­mary desk­top. I love thevoid, she’s a great machine but at a 1.8GHz sin­gle core, 1GB of RAM, no USB2, no SATA, no PCI-E, no PCI-X and fans that are start­ing to make squealing/grinding noises, she’s fallen a lit­tle far behind the tech­nol­ogy curve. My gen­eral view on desk­tops is, and for a long time has been, to get a core setup with the most expand­abil­ity pos­si­ble and then incre­men­tally upgrade for as long as pos­si­ble; fol­low­ing that prin­ci­ple, I am replac­ing thevoid with abyss.

Abyss (cur­rent specs)

• Lian-Li PC-A77B case
• Tyan Tempest i5400PW (S5397) motherboard
• 2x Intel Xeon E5420 2.5GHz quad core CPUs
• 4x Kingston 1GB 667MHz FB-DIMM RAM
• 2x Western Digital 500GB SATA HD
• Lite-On SATA 20X DVD±R

Abyss (planned upgrades) (as funds are available)

• 4x Kingston 1GB 667MHz FB-DIMM RAM
• Some medium-to-high-end nVIDIA video card
• some medium-end multi-channel sound card
• Highpoint RocketRAID2240 16-channel SATA controller
• 3x Icy Dock 5-bay SATA backplane
• 15x 750GB HD

The hard­ware in abyss has been specif­i­cally cho­sen to have sup­port for Intel VT-x and VT-d vir­tu­al­iza­tion tech­nol­ogy so as to allow for as ver­sa­tile a machine as pos­si­ble. Abyss is already run­ning Xen with Gentoo Linux as the pri­mary, dom0, oper­at­ing sys­tem and Windows XP as a sec­ondary, domU, oper­at­ing sys­tem. My hope, after adding a video and sound card, is to install another Gentoo Linux and Windows XP oper­at­ing sys­tem, as well as a Windows Vista oper­at­ing sys­tem. Ultimately, the cur­rent Gentoo and Windows oper­at­ing sys­tems will always run in the back­ground, pri­mar­ily act­ing as head­less servers for under­ly­ing ser­vices, and for day to day usage I will be able to switch between Gentoo, XP and Vista domains based on my needs. In essence, I will have a dual-booting sys­tem of vir­tual hosts with direct access to my video and audio hardware.

Hopefully abyss, with some incre­men­tal upgrades, will serve my com­pu­ta­tion needs for the next five to ten years.

There is a right way and there are wrong ways to add dri­ves to a RAID5 array with mdadm. Annoyingly, I chose one of the wrong ways last week when I went to increase the hard drive space in thevoid. Thankfully, my mis­take has proven more both­er­some and time con­sum­ing than harmful.

Starting out last week, thevoid had 4x 500GB dri­ves in an mdadm RAID5 con­fig­u­ra­tion: /dev/hde1, /dev/hdg1, /dev/sda1, /dev/sdb1. This had me sit­ting on 1.5TB of space with one drive worth of fail­abil­ity, which was prov­ing insuf­fi­cient. In order to resolve my space issues, I ordered 3x 500GB dri­ves and pro­ceeded to install them when the arrived: /dev/hdb, /dev/hdc, /dev/hdd.

In my haste to have more space, I pro­ceeded to add them to the array and grow it:

mdadm --manage /dev/md1 --add /dev/hdb
mdadm --manage /dev/md1 --add /dev/hdc
mdadm --manage /dev/md1 --add /dev/hdd
mdadm --grow /dev/md1 --raid-devices=7

This process then got to work and took about 3–4 days to com­plete. I assume the time was on account of most of the dri­ves being IDE dri­ves, many shar­ing chan­nels and there being an awful lot of space. Once the grow­ing process was com­plete, I merely needed to resize the ext3 par­ti­tion on the array:

resize2fs /dev/md1

This then took an hour or so and I was sit­ting on 3TB of total space. The only part of the process dur­ing which my drive was inac­ces­si­ble was when I had to turn off the com­puter to phys­i­cally install my IDE drives.

Everything is per­fect now, right? Wrong. Guess who for­got that you should par­ti­tion dri­ves before using them? That’s right, me. I really wanted to put a linux raid autode­tect par­ti­tion on each of the dri­ves before adding them to the array. Not that it really does much harm to add the dri­ves straight to the array but it’s poor form and it might pose prob­lems that I am not aware of in the future so, clearly, it’s a thing that should be fixed. Thankfully, being RAID5, my array is able to lose dri­ves and still be fine, thus allow­ing the solu­tion of fail­ing, remov­ing, par­ti­tion­ing and adding each of the new dri­ves back into the array:

mdadm --manage /dev/md1 --fail /dev/hdb
mdadm --manage /dev/md1 --remove /dev/hdb

par­tion /dev/hdb and then:

mdadm --manage /dev/md1 --add /dev/hdb1

now wait until the array has rebuilt itself:

cat /proc/mdstat

and repeat for the other dri­ves. Overall, it’s not hard but it’s annoy­ing and it does take about 1/2 to 2/3 of a day per drive to fix. Thankfully, though, now the endeavor is done and my array works cor­rectly with no lost data and min­i­mal down time. Hooray for soft­ware RAID and hav­ing 3TB in a sin­gle place.

Oh, and you don’t want to for­get to update /etc/mdadm.conf after every step of the process of you could have some poten­tial problems.

I was talk­ing with my brother, who is not a com­puter geek, and dis­cussing vari­a­tions on Linux when he came up with an amaz­ing anal­ogy that was so bril­liant I had to share it with the rest of you. He said, “I guess it would be more like a blank piece of paper ver­sus a color by num­bers thing”. It’s exactly spot on, Ubuntu is the paint-by-numbers of Linux; it’s really easy to do the stuff it’s aimed at doing but going out­side the lines doesn’t look as good as if you’d used a more gen­eral option. Of course, while you Ubuntu users are all kinder­garten­ers, I’m busy ric­ing out my machines with Gentoo; I don’t actu­ally know which is a worse thing to be analo­gized to.

I’ve always been rather fond of try­ing new things that twist the way I think about and inter­act with things and com­put­ers are no excep­tion. Ever since I used The Typing of the Dead to learn how to touch type, I have taken an inter­est in key­board lay­outs and designs and the more time I spend typ­ing in my life, the more I come to under­stand the effects of typ­ing com­fort com­pounded over time. Prior to 2001, I was a ded­i­cated 2–4 fin­ger typ­ist, capa­ble of achiev­ing over 30-40wpm using what amounted to “hunt and peck” with­out the hunt­ing; it was essen­tially a suc­ces­sive off­set­ting solu­tion using mem­o­rized rel­a­tive posi­tions to guide my hands. Then, in 2001, I built my beloved thevoid and got a Microsoft Natural Keyboard Pro because it looked cool, had a USB hub and some neat pro­gram­ma­ble func­tion but­tons. The Natural key­board has the fairly stan­dard split key­board lay­out, which com­pletely broke my rel­a­tive posi­tion scheme–the gap in the mid­dle pre­vented cross-overs, which were inte­gral to the scheme. At that point I slowly began learn­ing how to touch type but later in the year, I dis­cov­ered The Typing of the Dead and that changed every­thing; in addi­tion to mak­ing speed typ­ing a game, it also included a very use­ful typ­ing tutor.

Having learned proper touch typ­ing, I had divorced myself from cross-overs and was able to enjoy the com­fort pro­vided by a split key­board. Additionally, at this time, I was liv­ing in a dor­mi­tory with my good friend, Riad, who swears by Kinesis ergonomic key­boards, which I must agree are really com­fort­able. The Kinesis key­board is prob­a­bly the most com­fort­able and ergonomic key­board solu­tion that I had encoun­tered prior to the jerry-rigged solu­tion that I’ve just devised (see below). The Kinesis key­boards, how­ever, have the huge dis­ad­van­tage of being really expensive.

From there, my key­board exper­i­ments lan­guished for a num­ber of years until one day, when I was bored, I put let­tered stick­ers on thevoid’s key­board keys and switched the lay­out to a Dvorak lay­out. Learning Dvorak was not entirely pain­less and I even­tu­ally gave it up because the posi­tions of the ‘[/{‘ and ’]/}’ keys made C/C++ pro­gram­ming inconvenient–this later turned out to be because I didn’t full learn to touch-type Dvorak. I have switched to Dvorak and back prob­a­bly half a dozen times since, get­ting bet­ter each time; some­times using Dvorak and QWERTY con­cur­rently on dif­fer­ent machines. At this point, I can switch between Dvorak and QWERTY with ease and I can say, with­out reser­va­tion, that Dvorak is much eas­ier, faster and more com­fort­able than QWERTY. At present, I am using QWERTY because some of my key­boards are not suited to Dvorak lay­outs and it makes my new con­fig­u­ra­tion more practical.

Recently, I’ve started to notice more so than before, how very uncom­fort­able it is to touch-type on an unsplit key­board; the arm and wrist con­tor­tion is ter­ri­ble. I was think­ing that I might do well to ask the IT depart­ment at work if I could get a split key­board but I’m much more the type to impro­vise an elab­o­rate solu­tion than walk 100 feet and ask some­one for some­thing. I asked myself what the ideal lay­out would be and decided that a split key­board solves the wrist con­tor­tion but it still requires the arms to be uncom­fort­ably tight in to the body. The solu­tion: two key­boards, one 45° left, one 45° right, mouse in the cen­ter; each hand uses half a key­board and it turns out to be really com­fort­able. If I want to adjust how one hand rests, I only need to adjust that one key­board. Sure it takes a lot of desk space but I have that in spades right now and it really com­ple­ments my multi-monitor setup. This is my akimbo solu­tion and I really like it; if you know how to touch-type, have the desk space and a spare key­board, I highly rec­om­mend giv­ing it a try. Having just checked with a small online test, I am aver­ag­ing about 60wpm and 96% accu­racy with my key­boards akimbo layout.

Also, just so we’re clear, I do know that akimbo is ety­mo­log­i­cally incor­rect but it is a lin­guis­tic muta­tion that I approve of.

I am–and have for quite a while been–a huge pro­po­nent of GNU Screen for the many-fold improve­ments that it pro­vides to ter­mi­nal and SSH ses­sions. One of the things about screen is that it runs in two processes, one is a head­less process that redi­rects ter­mi­nal out­put of appli­ca­tions to it (server process) and the other is a process which con­nects to the server process allow­ing inter­ac­tion with and the view­ing of out­put from appli­ca­tions in the server process (client process). The advan­tage to this two process approach is that you can run screen from an SSH con­nec­tion, dis­con­nect the client process, leave and then later recon­nect a new client process to the same server process you started ear­lier, thus allow­ing ses­sion per­sis­tence. It’s a won­der­ful appli­ca­tion with many other fea­tures that make my life easier.

In addi­tion to being fan­tas­ti­cally use­ful, screen appears to have been writ­ten by some­one with a sense of humor as it seems to con­tain rather a few Easter Eggs. One par­tic­u­lar Easter Egg that I recently came across is that if the server process dies while the client process is still con­nected, you meet the error message:

Suddenly the Dungeon col­lapses!! — You die…

It’s a cryp­tic and omi­nous error mes­sage that might put you off a lit­tle if you don’t know what’s going on. A quick googling will show that some have acci­den­tally mis­in­ter­preted the mean­ing; thank­fully I could guess what it was and check google for confirmation.