Extreme Home Network Makeover: IPv6 & hostapd



When constant.com made IPv6 available on its dedicated servers and shortly thereafter Comcast finally got around to lighting up IPv6 on my home connection, I inevitably had to begin playing with it. I had these main goals:

  • Distribute globally unique public addresses to every device on my home network
  • Keep using a home Linux server as my Internet gateway, configure it to do regular routing and forwarding without NATting for IPv6 traffic, and run a stateful firewall in place of hiding behind a NAT.
  • Serve websites on my dedicated server over IPv6
  • Bonus – see if I can rig up public DNS to return AAAA’s for hostnames on my home LAN.

I mostly prodded at it from time to time for months, with slow progress. It took quite awhile to sort out some real basic stuff: the public IPv6 address my home gateway acquired over DHCP was showing up as a /64, so I imagined that /64 subnet was available to me – that’s how it had worked with the /64 my dedicated server received from constant.com’s router advertisements. Many hours were spent sending traffic out with other source addresses in the /64 from my home gateway and into the void before I stumbled on DHCP-PD and figured out comcast was using it. After that discovery, it was pretty quick to get the ISC dhclient (the default on Ubuntu) to send out a request for prefix delegation, packet-sniff the offer that came back to figure out what actual /64 I could use for the other devices on my home network, and configure up dnsmasq to RA that. That netted my internal network devices a brief glimpse of IPv6 connectivity, until I restarted the gateway, or something. But, it was long enough for me to figure out that the nothing-special and no longer supported netgear wireless router I’d been using as my wifi access point was dropping more than 90% of IPv6 traffic, for some reason. So, I had to add “pick out a new wifi AP” to the list of things that needed doing. Boy, was that a rabbit-hole.

I’ve had a good 15 years of experience with all kinds of residential wireless routers not being very good. From basic models in the early days of 802.11b to the latest, (priced as) premium Apple AirPort Extreme, I’ve only ever gotten truly solid reliability from certain hardware revisions of the now obsolete Linksys WRT-54G. It seemed shortsighted to buy a new AP in late 2014 that didn’t do 802.11AC, but I wasn’t terribly enthused about picking up another device priced in the multiple-hundred$ that likely still would need the good old occasional power cycling, either. What if I could just have the Linux gateway machine itself be the AP, I wondered?

I did a respectable amount of research on Linux as a wifi access point. Hostapd was the obvious choice for running WPA/sending out beacon frame type stuff, but selecting a wireless card wasn’t as clear. There are many wifi radio chipsets with Linux drivers that also have some level of support for access point mode, although the ath10k Linux driver seemed to be the only as-yet available line of chipsets supporting both 802.11AC and access point mode. I found the Compex mini-PCIe card that seemed to be the only as-yet available consumer device using said chipset, found that you can basically only get it on eBay from some guy in Germany, realized that I’d still have to work out a scheme for external antennas, found a post from only a few months earlier on the ath10k developer’s mailing list mentioning someone had even made it run long enough to run a benchmarking suite before it crashed, and decided I’d be better off bailing on the 802.11AC idea.

Ath9k devices seemed a reasonable consolation, except for a note on ath9k’s wireless.kernel.org page that says it only supports seven simultaneous associated devices. We don’t have that many, but it’s not much room to grow. So, I picked up the Intel Centrino Advanced-N 6205 PCIe card from Microcenter. I fidgeted with hostapd and other configurations for a few days trying to get it to at least approximately resemble 802.11n performance, but didn’t succeed before its driver started crashing and needing to be re-insmodded. Back to Microcenter with that one (thanks for the no questions asked refund!)

Ultimately, I ended up with a wonderfully inexpensive Rosewill RNX-900PCE using the ath9k-compatible Atheros AR9380 chipset. This card’s been running now for a month or so with no issues, performing like an 802.11n access point, and, back to the main point, delivering IPv6 packets to my wireless devices.

But there was more trouble. My gateway now had an additional LAN interface – the wireless card. When a packet comes in for a particular host, how will the kernel know which of those two interfaces to forward it on?

Looked like the options were to either “bridge” the ethernet and wifi LAN interfaces, which I think involves layer 4 routing sending it to the bridged interface and then the bridged interface acting like a layer 2 switch implemented in software. Or, I could put the wifi card on its own, new subnet, and just have the routing rule be “destination address in new subnet range -> forward on wlan0.” Opting to not have yet another daemon process (the bridge) to keep running and configured, I went with the latter.

But there was more trouble. In order to have more than one IPv6 subnet and follow IETF recommendations, you need an IPv6 address space bigger than the /64 I’d been able to eek out of Comcast through the ISC dhcp client (since each subnet must be no smaller than a /64 in order to work correctly with stateless address autoconfiguration and stuff.)

A tremendously helpful Comcast employee surely considered way too smart by their management to ever be directly answering customer support requests through normal channels took the time to offer the handful of subscribers like me who’re nerdy enough to have these problems with some information about how to get a large enough delegated address space to run multiple subnets. There are a number of posts on comcast’s forums covering basically this subject matter, all seeming to have responses by “ComcastTusca”, but here’s the original and most comprehensive thread: http://forums.comcast.com/t5/Home-Networking-Router-WiFi/IPv6-prefix-size-and-home-routing/td-p/1495933. By switching from the ISC dhclient to dhcp6c (“WIDE dhcp”), and basing my configuration for that off of brodieb’s example, I was able to get my /60, and have my two lan interfaces be configured with their own /64’s each, automatically, when the /60 is delegated or re-delegated by Comcast.

That was about the end of the trouble. Another key element worth noting is dnsmasq’s RA “constructor” feature, which tells it to author appropriate router advertisements on each LAN interface based on that interface’s public IPv6 address information. Here’s the actual lines from my dnsmasq.conf:

dhcp-range=::,constructor:eth0,ra-stateless,ra-names
dhcp-range=::,constructor:wlan0,ra-stateless,ra-names

The “ra-names” bit is pretty cool too – read the manpage for the detailed description of the magic, but it tells it to go ahead and apply some trickery and heuristics to try to determine the ipv6 address that hosts running dual-stack IP have selected for themselves, ping that, and add it to the dns server side of dnsmasq as an AAAA if everything works. This is the basis for how you can look up the IPv6 address of internal hosts in my home network from anywhere in the world, for example mike-pc.lan.mbaynton.com, when they happen to be up. (Details I’m happy to publish on my blog, because setting up iptables on the gateway box so pinging it is about all you can do was mercifully not conceptually any different than with good old IPv4.)

Posted in Linux

CyberPower UPS OR700 with nut: secret sauce



After my last UPS (a dumb APC thing picked up from officemax) bit the dust, I got myself the CyberPower OR700. I was a wee bit leery of the CyberPower linux code, though, and opted to use more standardized software to manage the system shutdown. So, I gave NUT a try.

I let this puppet contrib module install and configure the nut client & server using pretty much the example manifest, and that went remarkably smoothly. (I used the usbhid-ups driver and had to add a udev rules file so the USB ups driver was allowed to do its thing, and specify that at least for now the nut server should just be “standalone” in /etc/nut/nut.conf; puppet did everything else; this stuff was easy to google & sort out.) Then I ran the recommended simulated low battery shutdown sequence test (upsmon -c fsd), and watched my system almost finish halting before the load power was killed. Hmm. I sniffed around a bit and decided the shutdown.resume command was being sent after the filesystems had been made readonly, so good enough, I figured.

The real trouble came when I did a “real” test by pulling the OR700 from the wall. At low battery, the system shutdown and load power was killed as seen in the simulation. But then, as in the simulated run where utility power was still available, after a few seconds the OR700 resumed power to the load (!) and the attached NAS dutifully began booting up. For those not following along, the UPS was unplugged and in a low battery state, but resumed power to the load anyway a few seconds after it received the shutdown.resume. Not good at all.

After much experimenting and many tests, the extra configuration from the defaults that did the trick were to add

  • ondelay = 0
  • offdelay = 100

to my /etc/nut/ups.conf. I was also pleased to find the aforementioned puppet module had parameters for those. Nice!

 

Posted in Uncategorized

Remote drush with only FTP and database access



Necessity

I manage a Drupal site as part of my job, which has taught me to appreciate the treat that is drush. With Drupal,  you really want ssh access to the webserver because it gives you the ability to use drush, the command-line administration utility for Drupal sites. Uninteresting tasks like module and core upgrades become much less tedious.

I also babysit the website for the Minnesota Valley Unitarian Universalist fellowship, a real simple Drupal site that lives on a basic shared hosting server. No ssh access here. Because doing drupal core updates solely over FTP sucks so much, finally today I sat down in earnest to see if I could somehow leverage the power of drush with such shared hosting sites.

Invention

There didn’t seem to be much accurate information about how to do it already, with top Google hits saying things like you need ssh and drush installed both locally and on the server (false and false), you can’t, and there was really nobody suggesting anything to the contrary especially when your local workstation is running Linux (here’s a workable method for Windows.) I stubbornly refused to believe them and came up with a method that has proven to work for upgrading modules and core successfully, and every other drush subcommand I have tried to date. This is a guide for Linux, and I’ll include the package names for Debian/Ubuntu, but other than that it should be generic to other distros too.

The basic idea is to get the site’s source code mounted as a filesystem on your local machine backed by your ftp server, install php on your local machine if needed, and get connected to the live site’s database server from your local machine. You then run drush directly on your local machine, but its changes impact the actual site. It’s a bit of work to get everything going the 1st time, but simplifies site management tons in the long run.

Warning

Before the step-by-step, two words of warning: 1. This method is for people that want to be lazy, but don’t mind if the computers take forever once you tell them what to do. This method’s performance is slow, due to curlftpfs. For my purposes I don’t care, but some might. 2. A somewhat faster and more reliable way involves copying your site’s source locally and synchronizing the real site to it after running commands that modify code, not a big deal with a few rsync commands. Even when you do it this way, you don’t need to get another instance of your site actually up and running locally, and you don’t need a clone of the database. I discuss this more in step 5.

Step-by-step Guide

1. Pick a Linux computer to run drush commands on, likely your desktop/laptop. This computer will need php with mysql and gd extensions installed, as well as curlftpfs (or something else that makes ftp servers mountable, if you have another preference.) On Debian/Ubuntu, run these commands to ensure you have all needed packages:

sudo apt-get install php5
sudo apt-get install php5-mysql
sudo apt-get install php5-gd
sudo apt-get install curlftpfs

Failure to install php5 ultimately produces a clear error message from drush that it failed to find php, but failure to install php5-mysql produces an error with inaccurate troubleshooting information when you run Drush, so make sure you have it. Lack of php5-gd will generate frequent warnings from drush, but can mostly be done without if you really don’t want to install it.

2. Install drush locally on the Linux computer you’ve chosen. When I put drush on a system I usually just untar the latest stable release from Drush’s github releases to somewhere like /usr/local/lib and symlink the drush shell script from the release to /usr/bin/drush, but drush is just php and shell scripts so you can untar and run it in your home directory or wherever as well if you like. I’ll skip the line-by-line commands for how to do these things; few readers will have never extracted a .tgz or .zip.

3. Mount your site somewhere on your local machine’s filesystem with curlftpfs. In this example I’ll mount to the path ‘/mnt/mvuuf_inmotion’; choose something appropriate to your site and create an empty directory there.

sudo mkdir -p /mnt/mvuuf_inmotion

Modifying the below command for your ftp server, username, password, and local mountpoint, run (as the regular user you plan to run drush commands as, not as root):

curlftpfs -o user="your-ftp-username:your-ftp-password,uid=$UID,gid=${GROUPS[0]}"  "ftp://your-ftp-server/" /mnt/mvuuf_inmotion

(ps, if you get an error about “fusermount: failed to open /etc/fuse.conf: Permission denied”, fix with ‘sudo chmod a+r /etc/fuse.conf’; if you get an error about “fusermount: user has no write access to mountpoint /mnt/mvuuf_inmotion”, fix with ‘sudo chown $UID:${GROUPS[0]} /mnt/mvuuf_inmotion’.)

You should now be able to cd into your local mountpoint, /mnt/mvuuf_inmotion in this example, and ls your site’s contents.

4. Remote database connectivity. You need to be able to connect to your site’s database  from your local computer using the credentials stored in sites/default/settings.php.

You should be confident that your hosting company supports such remote database connections before spending too much time on this; consult with them if you need to. Even when it is supported, many webhosts won’t actually allow connections from hosts other than the webserver unless you take special steps to allow it – again, consult with them if you need to. If your hosting provider gives you MySQL managed with cPanel, this is quite easy, just go to “Remote MySQL” under Databases and add your public IP address that your local machine makes connections out to the Internet with (as displayed at whatismyip.com.)

You should probably do a manual connection to the database to ensure this part works before proceeding. Assuming the database server software is MySQL and you’ve installed the mysql client locally, you can do that with

mysql -h [whatever.your.host.is] -u [username_from_settings.php] -p

where “whatever.your.host.is” would generally be a hostname given in your sites/default/settings.php, unless settings.php says “localhost” in which case you should try the domain name of your website. You’ll be prompted for a password, which should also be in the database settings in settings.php.

5. Try it out. From a terminal on your local machine, cd to the root directory of your site (where Drupal’s index.php file is) and try a simple drush command like drush status. (Note that when this runs successfully, drush status likes to list the site’s url as “http://default” even when a url is explicity given in your settings.php. This seems to be a drush bug but doesn’t seem to affect anything else; you can fix it anyway if you like with drush’s –uri option.)

Great! You now have drush set up for use with your remote site.

Optional: Working with a local copy of the code. If you’ve gotten this far, you have two choices: If everything works and you don’t mind how darned slow it is, you can choose to be done. However, if you can’t stand the slowness or if drush is even aborting during some operations with messages like “the MySQL server has gone away,” you might want to make a local copy of the site’s code on your computer, switch to that to run drush, then rsync the local copy back over to your curlftpfs mount after you do updates. (I needed to do this because the shared host had a quite short timeout on idle mysql connections, so doing everything over curlftpfs slowed drush runs down enough to break it.)

Here’s some sample rsync commands to create a local copy of your site’s code, and to sync the remote site with your local copy. Note that you should place your site in maintenance mode before beginning upgrades with this method, and exit maintenance mode only after the remote sync is complete.

Example command to create/sync existing local copy:

rsync -aP --exclude="sites/default/files" /mnt/mvuuf_inmotion ~/mvuuf_local

Example command to sync the remote site up with your local copy after it has changed:

rsync -aP --temp-dir=/var/tmp/rsync --exclude="sites/default/files" --delete ~/mvuuf_local /mnt/mvuuf_inmotion

(these commands assume you have your curlftpfs-mounted copy of the site at /mnt/mvuuf_inmotion and you want to store the local copy under your home directory in a folder called mvuuf_local; adapt to suit.) The specific options to rsync here are composed with some care, don’t change them unless you’ve given it careful thought.

The obligatory notice accompanying all drupal upgrade etc writeups: whenever you do things that make considerable changes to your site, whether with this remote drush method or any other, make backups first. Drush itself can do this for you if you like.

Posted in Drupal, Linux

ZFS it is



Quite a few months back, in a post about the lack of ZFS block pointer rewrite, I mentioned that I’d begun investigations on whether I should migrate my home file server to a more modern filesystem. At that time, I already knew a few things about ZFS, but said that I wasn’t prepared to actually use it at home because it lacked the ability to add disks to a raid set. Well, the move to a new filesystem is now underway, and despite the lack of raidset reconfiguration, ZFS it is anyway.

So how did that happen?

When it came down to it, I basically had three options: ZFS, btrfs, or punt and do nothing for now, in hopes that either ZFS would support adding drives in future or btrfs would become more stable in future. I set up a test server with five old/shelved 320gb disks, and installed Ubuntu 14.04 with ZFS on Linux and btrfs tools. Then I set about evaluating both of them live & in person.

Btrfs Evaluation

I was really hoping my evaluations with btrfs would be positive – sure, it’s not “officially stable” yet, but that’ll come with time, and it pretty much does everything feature-wise. So I created a btrfs filesystem in its raid5-like layout on my 5 320gb drives, and queried both df and btrfs-tools for space available. 1.5TiB, it said. Ok, so that’s a little odd, since there can’t be more than 4x320gb available, but moving on…

I copied 20 gb or so over to it fine, then pulled one of the five disks from the online system and checksummed the 20 gb against the original. All ok. Then I went hunting for some kind of indication from btrfs tools that my filesystem was in a degraded state and action was needed. There were plenty of indications of trouble in syslog, but coming from an unhappy disk controller driver that’d lost contact with a disk. The closest I could come in btrfs was through ‘btrfs device stats,’ which showed a handful of read and (strangely, since I was reading the data to checksum it) more write errors on the device removed. Given that btrfs is designed to detect and correct such errors on disks, if I saw a small number of read/write errors to a device in a real-world scenario where I believed all disks were present, it wouldn’t be clear to me what if any action I should take. So strike one, screwy space accounting, strike two, lack of clear reporting of degraded state.

Next, I went ahead and filled the filesystem up to see what those df numbers would do as the filesystem approached the sensible, 4x320gb definition of full. What I ended up getting was, as I wrote more data to it, the Available value in df decreased faster than the Used value increased, so that by the time I’d written ~4x320gb, the Used value showed 1.2T, but the Avail value showed just a few GiBs. I decided I could live with that; I know how much actual space I have, so just remember not to rely on df / btrfs tools for accurate space available values on half-full filesystems basically.

Then, I pulled a different disk from the one I had originally removed, and wrote another GB or two. I forget if the first kernel panics happened during the writes or when I started trying to read it back, but either way I ended up with an unstable system that didn’t even shutdown without hanging and hung on reads to some of the btrfs files. In a last-ditch chance for btrfs to redeem itself, I hard reset the system and let it do its offline repair operation before mounting the filesystem again. This resulted in my last file disappearing completely, but didn’t prevent the system from panicking / becoming unstable again. Strike 3.

This left either ZFS or punt.

ZFS Evaluation

ZFS’s administration tools are a treat in comparison to btrfs’s. I easily created a raidz zpool of my five disks and then a couple of ZFS filesystems with different options enabled. Space reporting through any tool you like (df, zfs list, zfs get all, zpool list, zpool get all) made sense, and I saw some real potential in being able to do things like say “dedupe in this filesystem which I would like mounted where my photos directory used to be” (not so much data that the dedupe table will swamp my RAM, and I have a bad habit of being messy and having the same photo copied off cameras more than once), or say “run transparent gzip compression on this filesystem which I would like mounted where my log files go”, or “use lz4 compression and the ARC for my coding/music/wife’s stuff, but don’t compress and never cache data coming from the videos filesystem,” since video data will almost never be accessed more than once in time proximity and my tests proved it hardly compresses at all. Since filesystems all share the space of the single zpool they reside on as needed, there’s no administrative overhead to resizing filesystems or volumes to make all this happen (unless you want…there’s a filesystem quota you can use.)

I put ZFS through the same paces as btrfs in terms of disk-pulling. The difference was it handled it without issue, and not only did it give me detailed information about the pool’s degraded status (through zpool status) after yanking a drive, it even gave me in the output suggested actions to take.

Basically I concluded ZFS kicks butt.

I still have one major conceptual beef with ZFS, but it doesn’t apply to installations of my size, so whatever.

“Wait & See” Evaluation

The third choice was to keep adding more data to my md-raid/ext4 configuration for, realistically, at least a few years, until ZFS added the one missing feature or btrfs’ multiple device support matured. The rub with that is, there’s surely never going to be a way to transform in-place an md-raid to a btrfs or ZFS raid, so the migration further down the road would involve buying my entire storage utilization over again in blank disks and copying. But, that’s the same thing I’d end up doing if I moved to ZFS now and added a second RAID set to the zpool years down the road. So, I concluded, why not start getting the feature and integrity assurance benefits of ZFS today.

Posted in Big Storage @ Home

Drupal views join on multiple columns



Drupal views UI is nice from time to time, once you’ve spent lots of time poking at it and know it’s quirks. The rest of the time, what you need can’t be done via the UI, and you are plunged into the realms of programmatic customization whose APIs may or may not be properly documented. Today, I had to fix the query behind a view I’d inherited which had been constructed mostly through the UI and had a bad join, because the UI only lets you specify a single column from the left and right tables as defining the relationship. Determining and writing the correct SQL manually took 5 minutes, but getting views to duplicate it took reading the docs – and then the views code – for the rest of the afternoon. Here’s what I finally figured out is the most simple / direct way to specify joins against multiple columns with views:

1. Add a join_handler to your relationship in hook_views_data

Chances are if you need to be joining on multiple columns, you are using a custom data model and have described it to views with hook_views_data already. (If not, you’ve probably got something different going on and this post may not help you. see my addendum at the bottom.) A basic “relationship” aka SQL join on only one column is defined something like this in hook_views_data:

  $data['tablea']['some_field_from_tableb'] = array(
    'title' => t('Some field from table B'),
    'relationship' => array(
      'base' => 'tableb',
      'base field' => 'a_id',
      'field' => 'id',
      'handler' => 'views_handler_relationship',
    ),
  );

This’ll get you SQL along the lines of

... FROM tablea LEFT JOIN tableb ON tablea.id = tableb.a_id

But if you want SQL along the lines of

... FROM tablea LEFT JOIN tableb ON tablea.id = tableb.a_id AND tablea.instance = tableb.instance

then try the following php:

  $data['tablea']['some_field_from_tableb'] = array(
    'title' => t('Some field from table B'),
    'relationship' => array(
      'base' => 'tableb',
      'base field' => 'a_id',
      'field' => 'id',
      'handler' => 'views_handler_relationship',
      'join_handler' => 'my_fixed_up_join',
    ),
  );

2. Code my_fixed_up_join

Then, code up my_fixed_up_join, which should be a class extending views_join. You can get quite carried away with that if you want, but in the unlikely event that you’re of the mindset that we are simply trying to add an extra condition to a join in an SQL query here, and defining a new class at all in order to achieve this is already a bit much, here’s a quick and dirty version that’s “good enough” if you only have a few views that use it:

class my_fixed_up_join extends views_join
{
  public function build_join($select_query, $table, $view_query) {
    $select_query->addJoin('INNER', $this->table, $table['alias'], "table_alias_a.id = ${table['alias']}.a_id AND table_alias_a.instance = ${table['alias']}.instance");
  }
}

This class needs to live in a file that drupal’s auto-including tendrils have already sucked in; it won’t happen automatically by virtue of any file naming conventions. I suggest sticking it in the same file the related hook_views_data lives in.

You’ll also need to clear drupal’s caches so it reruns your hook_views_data.

Addendum – altering field joins

If your view is of content or other fieldable entities, and you need to tweak a join for one of the fields, this can also be done by writing an extension to the views_join class — you just have to tell Views about it in a different way.

In hook_views_query_alter:

function MYMODULE_views_query_alter(&$view, &$query) {
// logic to filter down to just the view we want to be modifying, such as:
if($view->name == 'whatever-my-view-is') {
  if(!empty($query->table_queue['table_alias_views_assigned_to_the_table_whose_join_needs_altering']['join'])) {    $query->table_queue['table_alias_views_assigned_to_the_table_whose_join_needs_altering']['join'] = new my_fixed_up_join();
  }
}

In my case, it was sufficient to pass no arguments to the my_fixed_up_join constructor, and have it just kick out a hardcoded string of SQL conditions:

 public function build_join($select_query, $table, $view_query) {
    $join_sql = <<<JOINSQL
   (field_collection_item_field_data_field_sessions.item_id = field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_id  AND field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_type = 'field_collection_item')
OR (
    node.nid = field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_id AND field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_type = 'node'
  )
  AND (
    field_collection_item_field_data_field_sessions__field_data_field_event_date.deleted = '0'
  )
JOINSQL;

    $select_query->addJoin('LEFT', $table['table'], $table['alias'], $join_sql);
  }

Why would you need to do this? In the example above, I was displaying several types of content in the view, and some of them attach the field directly while others are associated to the field through a “relationship” to another entity. The views UI will let you add the field as a member of the content, and will let you add a different field as a member of the related entity, but under the hood it joins to the field data table twice, creates two result columns, and you won’t be able to do things like sort on that field. What you really want is one join to the field data table with an OR condition, so the field comes up as a single column.

Posted in Drupal

Drupal webforms submissions by form_key



With the ease of entry for  basic use and API for extensibility, the Drupal webforms module is an indispensable tool. One snag with it, though, is that wherever it exposes the results of a submission in code, the submitted values are just expressed in a big 0-based numerically indexed array. Using this $submission->data array directly would make for hard-to-read and fragile code, and there doesn’t seem to be a function provided in webforms to give you the submitted data in an associative array.

Creating my own function to generate an associative array of results fortunately wasn’t that bad though, and seems like a valuable enough thing to make note of here.

/**
 * Returns an associative array of the submission data, instead of the
 * numerically indexed $submission->data
 *
 *  Inspired by http://drupal.stackexchange.com/questions/23607/how-do-i-access-webform-components-labels-on-the-congratulations-page
 *
 * @param int $nid The node id of the webform
 * @param int $sid The submission id.
 */
function webform_submission_data_keyed($nid, $sid) {
  $data = array();
  $node = node_load($nid);

  module_load_include('inc', 'webform', 'includes/webform.submissions');
  $submission = webform_get_submission($nid, $sid);

  foreach($node->webform['components'] AS $key => $component) {
    if(isset($submission->data[$key])) {
      $data[$component['form_key']] = $submission->data[$key];
    }
  }

  return $data;
}

This’ll give you a multidimensional associative array with the “machine name” of each field as keys, and arrays as values. The subarrays are 0-based numeric indicies to one or more response values the user selected.

If you wanted the human names as keys, use $data[$component[‘name’]] = $submission->data[$key].

Posted in Drupal

Crazy scheme of the week: melding heterogeneous filesystems



At work, we have an interesting problem. The primary storage system at MSI is from Panasas, a vendor that specializes in high aggregate I/O performance from a single filesystem to thousands of Linux compute nodes. Besides having a stunningly high CPU-to-disk ratio (read: being expensive), a big part of Panasas’ performance strategy is to remove the bottleneck that could easily occur at filer heads in traditional NAS when thousands of compute nodes are all asking for I/O ops. They do this by outsourcing much of the NAS head’s role to the compute nodes themselves, leaving only a thin metadata server that puts compute nodes into direct contact with the storage blades that contain the files being accessed.

The problem is a large amount of the data housed on our Panasas system doesn’t really need the privilege of residing on such a high-speed system, either because it is owned by non-HPC users who are only accessing it from one node at a time, or (the bigger problem) because nobody’s looked at it in 2 years, and they’re not likely to again anytime soon. And unfortunately, for all the things Panasas does, HSM is not currently one of them.

We could easily whip up a separate filesystem on a denser SAN or something, but offloading all the data at uber-rest to it would not be transparent to the users who currently have data resident on the Panasas. Doesn’t mean it can’t happen, but not ideal.

An alternative that I’ve been kicking around is to create a pseudo-filesystem that only actually stores metadata, and delegates I/O to one of several other filesystems. The idea is basically that the Linux kernel provides a well-defined API to filesystems for operations at the file level which every mountable filesystem in Linux conforms to. It is a common denominator whether your filesystem is a FAT-16 floppy disk, an ext4 local hdd, or even a DirectFlow Panasas filesystem. So, it ought to theoretically be possible to write another piece of code that knows how to consume the same API, but isn’t the kernel itself…and in fact also implements this API.

This pseudo-filesystem would be mounted by the kernel, and it in turn would be configured to mount any other filesystems of the administrator’s choice. Then, when asked for a file, the pseudo-fs would query a metadata server of its own to determine which underlying filesystem the file resides on, and then simply pass through all further interactions on that file handle to the API methods of the appropriate underlying filesystem.

With a scheme like this, one additional step is added when opening a file, but (importantly), the pseudo-fs would not be expected to introduce a measurable performance impact on all subsequent I/O to opened files, since native performance charactaristics of the underlying filesystems would be preserved. In my case, data transfer from storage to compute node would still happen using Panasas’ proprietary DirectFlow protocol if the data was on the high-speed Panasas system.

Clearly, completing this would be a very ambitious undertaking, but so far I haven’t discovered any fundamental reasons why this system wouldn’t work, and if such a thing existed, it might prove to be a unique open-source solution to HSM across any combination of heterogeneous storage systems.

Fortunately, it feels like a project with a pretty manageable, progressive roadmap. A feasible and likely very instructive first step would be to simply implement a kernel module providing a mountable filesystem that in fact passes all calls through to some single underlying filesystem.

Now we just have to see if this ever becomes more than my latest crazy scheme of the week.

Posted in Uncategorized

Newegg good service shoutout



Just a quick well-deserved post plugging newegg for taking care of me so well. I ordered some open-box hdd’s, clearly labeled as having a refund-only, no replacement policy. One of the disks I received was refurbished. I called to say no, that’s not what I ordered, suspecting a refund would be the only remedy offered. A refund would have bummed me out a little since I ordered 3 disks and ideally that’s how many I wanted at such an uncommonly good price. But, when I told that to the representative (who did initially just offer me a refund), he bypassed several standard policies to ship a replacement disk – before I had returned the refurb so that I would get one before their open-box stock ran out – shipped it overnight, and sent me a prepaid return label as well. The correct product arrived tonight, and all three disks are now original and show 0 hours previously powered on. Way to go Newegg!

Posted in Product Reviews

ZFS block pointer rewrite status, 2013



Through one part idle internet window shopping and one part good luck, I came across a  supply (now sold out) of the disks I use in my main RAID at home, nearly 50% off the next lowest price I’d ever seen for them–open-box (but not refurb) disks from a reputable online retailer with only a few months already consumed on Seagate’s 5 year warranty.  This seemed to me by far the best deal I’d seen on these disks in 5 years of looking, so I immediately bought some.

So now I am at one of those rarely encountered times where I have sufficient unused hard disk capacity to create a whole second copy of my current stash of bits if I wanted, and that gives me the flexibility to switch to another disk/filesystem/volume management arrangement if I want.

I always assume that at some point the amount of data I’ve accumulated will grow too large to allow for another opportunity to start from scratch like this, and so even though I’ve tried to choose wisely in selecting a storage arrangement in the past, and as a result could feasibly grow my current MD-based RAID out to 22 TB or so, I still want to make sure there isn’t something better out there that I could switch to since I do have the chance to do it now. Plus, the last time I investigated storage options was three years ago (which put me on my current 2TB-per-member MD RAID with ext4), so I took another look.

There are lots of things I like in ZFS, such as the added data integrity safeguards, snapshots to protect me from the fat-fingered rm -rf *, and with support for remote mirroring, it would definitely be worth another look at nearly-realtime backups to my hosted server (something I previously rejected due to the lack of snapshots.) A less unique but still important feature for long-term scalability is logical volumes, as I think a single RAID with > 12 drives would be a stretch even for the relatively light load my home array experiences.

But, as many a home user before me can tell you, ZFS has one huge feature hole for nerd@home scale operations: it is not possible to drop in another disk or two onto an existing RAID set. Enterprise users will drop another rack of raid sets into an existing zpool, which is possible with ZFS and makes sense. But do that after buying a handful of disks, and you’ll have a bunch of little RAID sets with wasted parity disks everywhere. RAID reconfiguration is something that’s been possible with MD RAID since I fired up my first array in 2003, and became possible as on online operation some years afterwards. It’s a feature I’ve used several times that I’m not comfortable giving up.

So, I dug into the state of development of this for ZFS, and was pleased to find some pretty current and comprehend-able information from Matt Ahrens, co-founder of ZFS way back when it was at Sun (awesomely, he seems to still be involved in ZFS through employment elsewhere.) The short summary sounds like achieving this via “block pointer rewrite” will almost surely never happen, because any code that correctly updates all necessary data structures would, in Matt’s opinion, introduce code that is too confusing/ugly/unmaintainable to encourage further ZFS development of other kinds. The YouTube video from October 2013 that I found this on is a great watch, if you’re interested in more detail. He also says, however, that there may be less ambitious approaches than block location rewrite to achieve the similar task of removing a device from a pool, so perhaps the more targeted task of adding a disk to a RAID-Z could also be tackled another way. Something I might try to learn more about?

First though, I need to research ZFS basics a bit more to find out why I shouldn’t just build a zpool of one or two vdevs (configured from ZFS’s point of view as plain disks) that actually happen to be MD RAID devices. Would it work to just let MD RAID do what it does best, and let ZFS do what it does best on top of it?

(Update 1/2014: the above md/zfs scheme would be ill-advised because ZFS implements a nontraditional “RAID” that includes couplings to the filesystem level, providing advantages such as self-healing data in the event of a parity/data disagreement, since the “RAID” subsystem and filesystem can compare notes to see if the data aligns with ZFS’s filesystem-level hashing (meaning the parity must be bad) or does not align with ZFS’s filesystem-level hash, meaning the data should be reconstructed from parity.)

Posted in Big Storage @ Home

JavaScript’s Object.keys performance with inheritance chains



Backstory: (You can cut to the chase) I’m working on a JavaScript library in which I want to identify any changes a method has made to properties of the object it is applied to, preferably in a more intelligent way than cloning all the properties and comparing them all before and after invocation. This can be done by applying the method in question to an empty object x inheriting from the true/intended object, and looking for any properties in x after the method is applied.

But, how does one efficiently look for the properties in x? In JS implementations supporting ECMAScript 5, Object.keys looks like a promising candidate. The function is decidedly O(n), but I wanted to be sure that with ES5, “n” is the properties of just my object, and not properties of all objects in the inheritance chain, like the ES3 polyfill.

(The chase:) My test on jsperf.com shows that yes, in browsers with ES5, Object.keys doesn’t waste time iterating properties of the inheritance chain and discarding them.

See also a simple html page with some primitive timings, which basically says the same thing:

<html>
<head>
<script>
function BaseClass() {}
for (var i = 0; i < 10e4; i++) {
  BaseClass.prototype["p" + i] = i;
}

function SubClass() {
	this.thing = 'x';
}
SubClass.prototype = BaseClass.prototype;

var objInTest = new SubClass();


function polyfillKeys(obj) {
  var output = [];
  for (var key in obj) {
    if (obj.hasOwnProperty(key)) {
      output.push(key);
    }
  }
  return output;
}
// could do Object.keys = polyfillKeys

var start = new Date();
var keys = Object.keys(objInTest);
console.log("Native Object.keys: " + (new Date() - start));

start = new Date();
var keys = polyfillKeys(objInTest);
console.log("Polyfill way: " + (new Date() - start));
</script>
</head>
<body></body>
</html>
Posted in JavaScript
← Older Entries Newer Entries →