eworldproblems
  • Home
  • About
  • Awesome Ideas That Somebody Else Already Thought Of
  • Perl defects
  • Books & Resources
Follow

Reset a crashed migration in Drupal 8



I’ve been writing migration code for my wife’s WordPress blog to Drupal 8, using the migrate framework. This framework attempts to track if a particular migration is currently running, but does not always unset that flag if it crashed with a PHP fatal error (which will happen when you are writing new code.) Subsequent attempts to run the migration will produce “Migration your_migration_name is busy with another operation: Importing”.

To unstick it in this case, you can run the following in drush:

drush php-eval 'var_dump(Drupal::keyValue("migrate_status")->set('your_migration_name', 0))'
Posted in Drupal

Handling GET-method submissions with Drupal 7’s Forms API



Remarkably, Drupal 7’s FAPI treats form submissions using the GET method as something of an afterthought. “Just register a URL with wildcards for that!” is the usual refrain from the Drupal devout when the less indoctrinated express their shock over this state of affairs. That approach, however, has several limitations over the W3C standard Way To Do It, such as if your form /has/twenty/optional/fields/that/all/need/a/position/in/the/url, or if your goal is to have Drupal serve response pages for requests formed by standards-complaint tools.

Fortunately though, it is possible. Getting it to work at all requires a multi-line recipe; things become even trickier if you want Drupal to handle submissions coming from another web framework or a static html file. I was faced with this prospect when I needed to reimplement a single form in a more complex legacy application. The frontend form needed to stay as-is because that’s where users look for it, but Drupal had all the backend pieces to make the magic happen in < 100 lines, so I figured I would just point the existing form’s method to a custom module in Drupal. Here’s a generalized sample of what all is needed in such cases, derived from combined googling and interactive debugging through core:

  1. Build the form
    $form_state = array(
      'method' => 'get',
      'always_process' => true, // if support for submissions originating outside drupal is needed
    );
    $form = drupal_build_form('name_of_my_form', $form_state);
    

    Returning $form at this point would suffice as the entirety of a page callback. (Note that because of the needed $form_state, you have to drupal_build_form instead of drupal_get_form, and the standard trick of just registering drupal_get_form as your page callback in hook_menu isn’t an option.)

  2. Write the form callback
    Despite the fact that you already told Drupal you want a GET form in the $form_state, you’ve gotta do so in $form in your form callback as well:

    function name_of_my_form($form, &$form_state) {
       $form['#method'] = 'get';
       $form['#token'] = 'false';
       ... // describe your form as usual
       return $form;
    }
    
  3. Break the redirect loop and otherwise handle the submission in the form submission callback
    So, you’ve made a form that trips the submission callback when you access it and loads up the values specified via GET into Drupal’s usual form_state structures. But wait! Drupal’s standard thing is to redirect somewhere following a form submission, and your GET url looks like a form submission to it now, so endless redirect loop ahoy. This can be killed off in a variety of ways, but I’ll mention two that I think should cover most cases.

    If you examine the submitted values and find the submission to be incomplete, such as if the user visited the registered path without providing any GET parameters, you can simply set $form_state['no_redirect'] = true in your submit handler.

    If you wish to actually process a submission and display results on the page, here’s how I did it: instead of setting no_redirect, set $form_state['rebuild'] = true. This implicitly stops a redirect from occurring, and also results in two calls to your form callback during the request – one to provide Drupal with your form information for validation/submission processing, and a second after the submission handler has been run. The results of the second invocation are what is rendered to the page. So, it’s easy enough to stick some results data in your $form_state in the submit callback, then look for it in the form callback and add some additional elements to the form’s render array to show them.

Of course, 'always_process' => true explicitly lowers Drupal’s CSRF shields, so be sure to confirm for yourself that your GET form really isn’t modifying the state of your application.

Also, beware the magic q: Drupal uses the URL param ‘q’ when pretty URLs are disabled, and treats it as the string to be routed whenever it is present as a parameter, whether pretty URLs are on or off. So, it’s unavailable for use as the name of an input on your form.

Posted in Drupal

More super ubuntu tips: udev-based NIC naming & lvm silicon_medley_raid_member



I had a pretty booked weekend, but decided to see if I could slam an Intel Atom SoC-based board into my home server anyway. Plan of attack: take a snapshot of OS lvm volume in case things went south, mount snapshot to double-check it, shut down, swap out system boards, resume business as usual.

Actual result: Still on the old board, after taking the root volume snapshot, I was unable to mount the thing, because mount detected the volume as of type “silicon_medley_raid_member”  rather than the ext4 filesystem it was. I’d never seen that before, but mount -t ext4 fixed it; I examined the mounted snapshot which seemed fine and ignored that strange hiccup.

Upon power-up on the new board, the kernel failed to mount the real root filesystem, so the boot sequence halted before pivot_root with an error (mount: cannot mount /dev/mapper/ssd–vg-root on /root: no such device) and dumped me in busybox on the initramfs.

Now. given that a mount without explicit fs type had failed on the snapshot, I thought it possible that somehow the type bits of the root filesystem itself had gotten corrupted, and that was why the boot-time mount was failing as well. How to confirm and/or fix? Thanks to this blog post, I got answers to both. Saved me a bundle of trouble! Since it seems to always involve LVM, and neither I nor anyone else who has been bit by this has a clue what happened, I’m calling software bug.

With my machine booting again (35 watts lower!) I just had to tackle the NIC names, so the new NIC plugged to my lan was still eth0 etc, making all my configurations and firewall rules and such “just work.” I happily pulled up the MAC addresses with lshw -class network, and swapped out the old values with the new in /etc/udev/rules.d/70-persistent-net-rules. Aaand it didn’t work. So, what the heck.

Searching through dmesg, I discovered the igw driver sanely brought the NICs up as eth0 through eth3, and then udev came along and arbitrarily renamed them em1 through em4. No idea where it got that from. After quite a bit of prodding and searching, I found this suggestion which I am not sufficiently esteemed to upvote on that site. The default contents of the Ubuntu udev network interface rules file requires that the interface being renamed already be named eth*. That corresponds to

KERNEL="eth*",

in the file. Seems udev selects em* as the name and sets it before it bothers to read the rules file, and thus it never renames them again to the desired values. Removing that constraint, I was again fully up and running.

I’m grateful these folks documented the respective magic incantations, leaving substantial portions of my weekend for more fun things. The whole thing was maybe 4 or 5 hours, not as bad as I’d feared. Next time I build up enthusiasm to break stuff, I’ll tackle the link-aggregating switch I picked up…

Posted in Linux

Extreme Home Network Makeover: IPv6 & hostapd



When constant.com made IPv6 available on its dedicated servers and shortly thereafter Comcast finally got around to lighting up IPv6 on my home connection, I inevitably had to begin playing with it. I had these main goals:

  • Distribute globally unique public addresses to every device on my home network
  • Keep using a home Linux server as my Internet gateway, configure it to do regular routing and forwarding without NATting for IPv6 traffic, and run a stateful firewall in place of hiding behind a NAT.
  • Serve websites on my dedicated server over IPv6
  • Bonus – see if I can rig up public DNS to return AAAA’s for hostnames on my home LAN.

I mostly prodded at it from time to time for months, with slow progress. It took quite awhile to sort out some real basic stuff: the public IPv6 address my home gateway acquired over DHCP was showing up as a /64, so I imagined that /64 subnet was available to me – that’s how it had worked with the /64 my dedicated server received from constant.com’s router advertisements. Many hours were spent sending traffic out with other source addresses in the /64 from my home gateway and into the void before I stumbled on DHCP-PD and figured out comcast was using it. After that discovery, it was pretty quick to get the ISC dhclient (the default on Ubuntu) to send out a request for prefix delegation, packet-sniff the offer that came back to figure out what actual /64 I could use for the other devices on my home network, and configure up dnsmasq to RA that. That netted my internal network devices a brief glimpse of IPv6 connectivity, until I restarted the gateway, or something. But, it was long enough for me to figure out that the nothing-special and no longer supported netgear wireless router I’d been using as my wifi access point was dropping more than 90% of IPv6 traffic, for some reason. So, I had to add “pick out a new wifi AP” to the list of things that needed doing. Boy, was that a rabbit-hole.

I’ve had a good 15 years of experience with all kinds of residential wireless routers not being very good. From basic models in the early days of 802.11b to the latest, (priced as) premium Apple AirPort Extreme, I’ve only ever gotten truly solid reliability from certain hardware revisions of the now obsolete Linksys WRT-54G. It seemed shortsighted to buy a new AP in late 2014 that didn’t do 802.11AC, but I wasn’t terribly enthused about picking up another device priced in the multiple-hundred$ that likely still would need the good old occasional power cycling, either. What if I could just have the Linux gateway machine itself be the AP, I wondered?

I did a respectable amount of research on Linux as a wifi access point. Hostapd was the obvious choice for running WPA/sending out beacon frame type stuff, but selecting a wireless card wasn’t as clear. There are many wifi radio chipsets with Linux drivers that also have some level of support for access point mode, although the ath10k Linux driver seemed to be the only as-yet available line of chipsets supporting both 802.11AC and access point mode. I found the Compex mini-PCIe card that seemed to be the only as-yet available consumer device using said chipset, found that you can basically only get it on eBay from some guy in Germany, realized that I’d still have to work out a scheme for external antennas, found a post from only a few months earlier on the ath10k developer’s mailing list mentioning someone had even made it run long enough to run a benchmarking suite before it crashed, and decided I’d be better off bailing on the 802.11AC idea.

Ath9k devices seemed a reasonable consolation, except for a note on ath9k’s wireless.kernel.org page that says it only supports seven simultaneous associated devices. We don’t have that many, but it’s not much room to grow. So, I picked up the Intel Centrino Advanced-N 6205 PCIe card from Microcenter. I fidgeted with hostapd and other configurations for a few days trying to get it to at least approximately resemble 802.11n performance, but didn’t succeed before its driver started crashing and needing to be re-insmodded. Back to Microcenter with that one (thanks for the no questions asked refund!)

Ultimately, I ended up with a wonderfully inexpensive Rosewill RNX-900PCE using the ath9k-compatible Atheros AR9380 chipset. This card’s been running now for a month or so with no issues, performing like an 802.11n access point, and, back to the main point, delivering IPv6 packets to my wireless devices.

But there was more trouble. My gateway now had an additional LAN interface – the wireless card. When a packet comes in for a particular host, how will the kernel know which of those two interfaces to forward it on?

Looked like the options were to either “bridge” the ethernet and wifi LAN interfaces, which I think involves layer 4 routing sending it to the bridged interface and then the bridged interface acting like a layer 2 switch implemented in software. Or, I could put the wifi card on its own, new subnet, and just have the routing rule be “destination address in new subnet range -> forward on wlan0.” Opting to not have yet another daemon process (the bridge) to keep running and configured, I went with the latter.

But there was more trouble. In order to have more than one IPv6 subnet and follow IETF recommendations, you need an IPv6 address space bigger than the /64 I’d been able to eek out of Comcast through the ISC dhcp client (since each subnet must be no smaller than a /64 in order to work correctly with stateless address autoconfiguration and stuff.)

A tremendously helpful Comcast employee surely considered way too smart by their management to ever be directly answering customer support requests through normal channels took the time to offer the handful of subscribers like me who’re nerdy enough to have these problems with some information about how to get a large enough delegated address space to run multiple subnets. There are a number of posts on comcast’s forums covering basically this subject matter, all seeming to have responses by “ComcastTusca”, but here’s the original and most comprehensive thread: http://forums.comcast.com/t5/Home-Networking-Router-WiFi/IPv6-prefix-size-and-home-routing/td-p/1495933. By switching from the ISC dhclient to dhcp6c (“WIDE dhcp”), and basing my configuration for that off of brodieb’s example, I was able to get my /60, and have my two lan interfaces be configured with their own /64’s each, automatically, when the /60 is delegated or re-delegated by Comcast.

That was about the end of the trouble. Another key element worth noting is dnsmasq’s RA “constructor” feature, which tells it to author appropriate router advertisements on each LAN interface based on that interface’s public IPv6 address information. Here’s the actual lines from my dnsmasq.conf:

dhcp-range=::,constructor:eth0,ra-stateless,ra-names
dhcp-range=::,constructor:wlan0,ra-stateless,ra-names

The “ra-names” bit is pretty cool too – read the manpage for the detailed description of the magic, but it tells it to go ahead and apply some trickery and heuristics to try to determine the ipv6 address that hosts running dual-stack IP have selected for themselves, ping that, and add it to the dns server side of dnsmasq as an AAAA if everything works. This is the basis for how you can look up the IPv6 address of internal hosts in my home network from anywhere in the world, for example mike-pc.lan.mbaynton.com, when they happen to be up. (Details I’m happy to publish on my blog, because setting up iptables on the gateway box so pinging it is about all you can do was mercifully not conceptually any different than with good old IPv4.)

Posted in Linux

CyberPower UPS OR700 with nut: secret sauce



After my last UPS (a dumb APC thing picked up from officemax) bit the dust, I got myself the CyberPower OR700. I was a wee bit leery of the CyberPower linux code, though, and opted to use more standardized software to manage the system shutdown. So, I gave NUT a try.

I let this puppet contrib module install and configure the nut client & server using pretty much the example manifest, and that went remarkably smoothly. (I used the usbhid-ups driver and had to add a udev rules file so the USB ups driver was allowed to do its thing, and specify that at least for now the nut server should just be “standalone” in /etc/nut/nut.conf; puppet did everything else; this stuff was easy to google & sort out.) Then I ran the recommended simulated low battery shutdown sequence test (upsmon -c fsd), and watched my system almost finish halting before the load power was killed. Hmm. I sniffed around a bit and decided the shutdown.resume command was being sent after the filesystems had been made readonly, so good enough, I figured.

The real trouble came when I did a “real” test by pulling the OR700 from the wall. At low battery, the system shutdown and load power was killed as seen in the simulation. But then, as in the simulated run where utility power was still available, after a few seconds the OR700 resumed power to the load (!) and the attached NAS dutifully began booting up. For those not following along, the UPS was unplugged and in a low battery state, but resumed power to the load anyway a few seconds after it received the shutdown.resume. Not good at all.

After much experimenting and many tests, the extra configuration from the defaults that did the trick were to add

  • ondelay = 0
  • offdelay = 100

to my /etc/nut/ups.conf. I was also pleased to find the aforementioned puppet module had parameters for those. Nice!

 

Posted in Uncategorized

Remote drush with only FTP and database access



Necessity

I manage a Drupal site as part of my job, which has taught me to appreciate the treat that is drush. With Drupal,  you really want ssh access to the webserver because it gives you the ability to use drush, the command-line administration utility for Drupal sites. Uninteresting tasks like module and core upgrades become much less tedious.

I also babysit the website for the Minnesota Valley Unitarian Universalist fellowship, a real simple Drupal site that lives on a basic shared hosting server. No ssh access here. Because doing drupal core updates solely over FTP sucks so much, finally today I sat down in earnest to see if I could somehow leverage the power of drush with such shared hosting sites.

Invention

There didn’t seem to be much accurate information about how to do it already, with top Google hits saying things like you need ssh and drush installed both locally and on the server (false and false), you can’t, and there was really nobody suggesting anything to the contrary especially when your local workstation is running Linux (here’s a workable method for Windows.) I stubbornly refused to believe them and came up with a method that has proven to work for upgrading modules and core successfully, and every other drush subcommand I have tried to date. This is a guide for Linux, and I’ll include the package names for Debian/Ubuntu, but other than that it should be generic to other distros too.

The basic idea is to get the site’s source code mounted as a filesystem on your local machine backed by your ftp server, install php on your local machine if needed, and get connected to the live site’s database server from your local machine. You then run drush directly on your local machine, but its changes impact the actual site. It’s a bit of work to get everything going the 1st time, but simplifies site management tons in the long run.

Warning

Before the step-by-step, two words of warning: 1. This method is for people that want to be lazy, but don’t mind if the computers take forever once you tell them what to do. This method’s performance is slow, due to curlftpfs. For my purposes I don’t care, but some might. 2. A somewhat faster and more reliable way involves copying your site’s source locally and synchronizing the real site to it after running commands that modify code, not a big deal with a few rsync commands. Even when you do it this way, you don’t need to get another instance of your site actually up and running locally, and you don’t need a clone of the database. I discuss this more in step 5.

Step-by-step Guide

1. Pick a Linux computer to run drush commands on, likely your desktop/laptop. This computer will need php with mysql and gd extensions installed, as well as curlftpfs (or something else that makes ftp servers mountable, if you have another preference.) On Debian/Ubuntu, run these commands to ensure you have all needed packages:

sudo apt-get install php5
sudo apt-get install php5-mysql
sudo apt-get install php5-gd
sudo apt-get install curlftpfs

Failure to install php5 ultimately produces a clear error message from drush that it failed to find php, but failure to install php5-mysql produces an error with inaccurate troubleshooting information when you run Drush, so make sure you have it. Lack of php5-gd will generate frequent warnings from drush, but can mostly be done without if you really don’t want to install it.

2. Install drush locally on the Linux computer you’ve chosen. When I put drush on a system I usually just untar the latest stable release from Drush’s github releases to somewhere like /usr/local/lib and symlink the drush shell script from the release to /usr/bin/drush, but drush is just php and shell scripts so you can untar and run it in your home directory or wherever as well if you like. I’ll skip the line-by-line commands for how to do these things; few readers will have never extracted a .tgz or .zip.

3. Mount your site somewhere on your local machine’s filesystem with curlftpfs. In this example I’ll mount to the path ‘/mnt/mvuuf_inmotion’; choose something appropriate to your site and create an empty directory there.

sudo mkdir -p /mnt/mvuuf_inmotion

Modifying the below command for your ftp server, username, password, and local mountpoint, run (as the regular user you plan to run drush commands as, not as root):

curlftpfs -o user="your-ftp-username:your-ftp-password,uid=$UID,gid=${GROUPS[0]}"  "ftp://your-ftp-server/" /mnt/mvuuf_inmotion

(ps, if you get an error about “fusermount: failed to open /etc/fuse.conf: Permission denied”, fix with ‘sudo chmod a+r /etc/fuse.conf’; if you get an error about “fusermount: user has no write access to mountpoint /mnt/mvuuf_inmotion”, fix with ‘sudo chown $UID:${GROUPS[0]} /mnt/mvuuf_inmotion’.)

You should now be able to cd into your local mountpoint, /mnt/mvuuf_inmotion in this example, and ls your site’s contents.

4. Remote database connectivity. You need to be able to connect to your site’s database  from your local computer using the credentials stored in sites/default/settings.php.

You should be confident that your hosting company supports such remote database connections before spending too much time on this; consult with them if you need to. Even when it is supported, many webhosts won’t actually allow connections from hosts other than the webserver unless you take special steps to allow it – again, consult with them if you need to. If your hosting provider gives you MySQL managed with cPanel, this is quite easy, just go to “Remote MySQL” under Databases and add your public IP address that your local machine makes connections out to the Internet with (as displayed at whatismyip.com.)

You should probably do a manual connection to the database to ensure this part works before proceeding. Assuming the database server software is MySQL and you’ve installed the mysql client locally, you can do that with

mysql -h [whatever.your.host.is] -u [username_from_settings.php] -p

where “whatever.your.host.is” would generally be a hostname given in your sites/default/settings.php, unless settings.php says “localhost” in which case you should try the domain name of your website. You’ll be prompted for a password, which should also be in the database settings in settings.php.

5. Try it out. From a terminal on your local machine, cd to the root directory of your site (where Drupal’s index.php file is) and try a simple drush command like drush status. (Note that when this runs successfully, drush status likes to list the site’s url as “http://default” even when a url is explicity given in your settings.php. This seems to be a drush bug but doesn’t seem to affect anything else; you can fix it anyway if you like with drush’s –uri option.)

Great! You now have drush set up for use with your remote site.

Optional: Working with a local copy of the code. If you’ve gotten this far, you have two choices: If everything works and you don’t mind how darned slow it is, you can choose to be done. However, if you can’t stand the slowness or if drush is even aborting during some operations with messages like “the MySQL server has gone away,” you might want to make a local copy of the site’s code on your computer, switch to that to run drush, then rsync the local copy back over to your curlftpfs mount after you do updates. (I needed to do this because the shared host had a quite short timeout on idle mysql connections, so doing everything over curlftpfs slowed drush runs down enough to break it.)

Here’s some sample rsync commands to create a local copy of your site’s code, and to sync the remote site with your local copy. Note that you should place your site in maintenance mode before beginning upgrades with this method, and exit maintenance mode only after the remote sync is complete.

Example command to create/sync existing local copy:

rsync -aP --exclude="sites/default/files" /mnt/mvuuf_inmotion ~/mvuuf_local

Example command to sync the remote site up with your local copy after it has changed:

rsync -aP --temp-dir=/var/tmp/rsync --exclude="sites/default/files" --delete ~/mvuuf_local /mnt/mvuuf_inmotion

(these commands assume you have your curlftpfs-mounted copy of the site at /mnt/mvuuf_inmotion and you want to store the local copy under your home directory in a folder called mvuuf_local; adapt to suit.) The specific options to rsync here are composed with some care, don’t change them unless you’ve given it careful thought.

The obligatory notice accompanying all drupal upgrade etc writeups: whenever you do things that make considerable changes to your site, whether with this remote drush method or any other, make backups first. Drush itself can do this for you if you like.

Posted in Drupal, Linux

ZFS it is



Quite a few months back, in a post about the lack of ZFS block pointer rewrite, I mentioned that I’d begun investigations on whether I should migrate my home file server to a more modern filesystem. At that time, I already knew a few things about ZFS, but said that I wasn’t prepared to actually use it at home because it lacked the ability to add disks to a raid set. Well, the move to a new filesystem is now underway, and despite the lack of raidset reconfiguration, ZFS it is anyway.

So how did that happen?

When it came down to it, I basically had three options: ZFS, btrfs, or punt and do nothing for now, in hopes that either ZFS would support adding drives in future or btrfs would become more stable in future. I set up a test server with five old/shelved 320gb disks, and installed Ubuntu 14.04 with ZFS on Linux and btrfs tools. Then I set about evaluating both of them live & in person.

Btrfs Evaluation

I was really hoping my evaluations with btrfs would be positive – sure, it’s not “officially stable” yet, but that’ll come with time, and it pretty much does everything feature-wise. So I created a btrfs filesystem in its raid5-like layout on my 5 320gb drives, and queried both df and btrfs-tools for space available. 1.5TiB, it said. Ok, so that’s a little odd, since there can’t be more than 4x320gb available, but moving on…

I copied 20 gb or so over to it fine, then pulled one of the five disks from the online system and checksummed the 20 gb against the original. All ok. Then I went hunting for some kind of indication from btrfs tools that my filesystem was in a degraded state and action was needed. There were plenty of indications of trouble in syslog, but coming from an unhappy disk controller driver that’d lost contact with a disk. The closest I could come in btrfs was through ‘btrfs device stats,’ which showed a handful of read and (strangely, since I was reading the data to checksum it) more write errors on the device removed. Given that btrfs is designed to detect and correct such errors on disks, if I saw a small number of read/write errors to a device in a real-world scenario where I believed all disks were present, it wouldn’t be clear to me what if any action I should take. So strike one, screwy space accounting, strike two, lack of clear reporting of degraded state.

Next, I went ahead and filled the filesystem up to see what those df numbers would do as the filesystem approached the sensible, 4x320gb definition of full. What I ended up getting was, as I wrote more data to it, the Available value in df decreased faster than the Used value increased, so that by the time I’d written ~4x320gb, the Used value showed 1.2T, but the Avail value showed just a few GiBs. I decided I could live with that; I know how much actual space I have, so just remember not to rely on df / btrfs tools for accurate space available values on half-full filesystems basically.

Then, I pulled a different disk from the one I had originally removed, and wrote another GB or two. I forget if the first kernel panics happened during the writes or when I started trying to read it back, but either way I ended up with an unstable system that didn’t even shutdown without hanging and hung on reads to some of the btrfs files. In a last-ditch chance for btrfs to redeem itself, I hard reset the system and let it do its offline repair operation before mounting the filesystem again. This resulted in my last file disappearing completely, but didn’t prevent the system from panicking / becoming unstable again. Strike 3.

This left either ZFS or punt.

ZFS Evaluation

ZFS’s administration tools are a treat in comparison to btrfs’s. I easily created a raidz zpool of my five disks and then a couple of ZFS filesystems with different options enabled. Space reporting through any tool you like (df, zfs list, zfs get all, zpool list, zpool get all) made sense, and I saw some real potential in being able to do things like say “dedupe in this filesystem which I would like mounted where my photos directory used to be” (not so much data that the dedupe table will swamp my RAM, and I have a bad habit of being messy and having the same photo copied off cameras more than once), or say “run transparent gzip compression on this filesystem which I would like mounted where my log files go”, or “use lz4 compression and the ARC for my coding/music/wife’s stuff, but don’t compress and never cache data coming from the videos filesystem,” since video data will almost never be accessed more than once in time proximity and my tests proved it hardly compresses at all. Since filesystems all share the space of the single zpool they reside on as needed, there’s no administrative overhead to resizing filesystems or volumes to make all this happen (unless you want…there’s a filesystem quota you can use.)

I put ZFS through the same paces as btrfs in terms of disk-pulling. The difference was it handled it without issue, and not only did it give me detailed information about the pool’s degraded status (through zpool status) after yanking a drive, it even gave me in the output suggested actions to take.

Basically I concluded ZFS kicks butt.

I still have one major conceptual beef with ZFS, but it doesn’t apply to installations of my size, so whatever.

“Wait & See” Evaluation

The third choice was to keep adding more data to my md-raid/ext4 configuration for, realistically, at least a few years, until ZFS added the one missing feature or btrfs’ multiple device support matured. The rub with that is, there’s surely never going to be a way to transform in-place an md-raid to a btrfs or ZFS raid, so the migration further down the road would involve buying my entire storage utilization over again in blank disks and copying. But, that’s the same thing I’d end up doing if I moved to ZFS now and added a second RAID set to the zpool years down the road. So, I concluded, why not start getting the feature and integrity assurance benefits of ZFS today.

Posted in Big Storage @ Home

Drupal views join on multiple columns



Drupal views UI is nice from time to time, once you’ve spent lots of time poking at it and know it’s quirks. The rest of the time, what you need can’t be done via the UI, and you are plunged into the realms of programmatic customization whose APIs may or may not be properly documented. Today, I had to fix the query behind a view I’d inherited which had been constructed mostly through the UI and had a bad join, because the UI only lets you specify a single column from the left and right tables as defining the relationship. Determining and writing the correct SQL manually took 5 minutes, but getting views to duplicate it took reading the docs – and then the views code – for the rest of the afternoon. Here’s what I finally figured out is the most simple / direct way to specify joins against multiple columns with views:

1. Add a join_handler to your relationship in hook_views_data

Chances are if you need to be joining on multiple columns, you are using a custom data model and have described it to views with hook_views_data already. (If not, you’ve probably got something different going on and this post may not help you. see my addendum at the bottom.) A basic “relationship” aka SQL join on only one column is defined something like this in hook_views_data:

  $data['tablea']['some_field_from_tableb'] = array(
    'title' =&gt; t('Some field from table B'),
    'relationship' =&gt; array(
      'base' =&gt; 'tableb',
      'base field' =&gt; 'a_id',
      'field' =&gt; 'id',
      'handler' =&gt; 'views_handler_relationship',
    ),
  );

This’ll get you SQL along the lines of

... FROM tablea LEFT JOIN tableb ON tablea.id = tableb.a_id

But if you want SQL along the lines of

... FROM tablea LEFT JOIN tableb ON tablea.id = tableb.a_id AND tablea.instance = tableb.instance

then try the following php:

  $data['tablea']['some_field_from_tableb'] = array(
    'title' =&gt; t('Some field from table B'),
    'relationship' =&gt; array(
      'base' =&gt; 'tableb',
      'base field' =&gt; 'a_id',
      'field' =&gt; 'id',
      'handler' =&gt; 'views_handler_relationship',
      'join_handler' =&gt; 'my_fixed_up_join',
    ),
  );

2. Code my_fixed_up_join

Then, code up my_fixed_up_join, which should be a class extending views_join. You can get quite carried away with that if you want, but in the unlikely event that you’re of the mindset that we are simply trying to add an extra condition to a join in an SQL query here, and defining a new class at all in order to achieve this is already a bit much, here’s a quick and dirty version that’s “good enough” if you only have a few views that use it:

class my_fixed_up_join extends views_join
{
  public function build_join($select_query, $table, $view_query) {
    $select_query-&gt;addJoin('INNER', $this-&gt;table, $table['alias'], &quot;table_alias_a.id = ${table['alias']}.a_id AND table_alias_a.instance = ${table['alias']}.instance&quot;);
  }
}

This class needs to live in a file that drupal’s auto-including tendrils have already sucked in; it won’t happen automatically by virtue of any file naming conventions. I suggest sticking it in the same file the related hook_views_data lives in.

You’ll also need to clear drupal’s caches so it reruns your hook_views_data.

Addendum – altering field joins

If your view is of content or other fieldable entities, and you need to tweak a join for one of the fields, this can also be done by writing an extension to the views_join class — you just have to tell Views about it in a different way.

In hook_views_query_alter:

function MYMODULE_views_query_alter(&$view, &$query) {
// logic to filter down to just the view we want to be modifying, such as:
if($view->name == 'whatever-my-view-is') {
  if(!empty($query->table_queue['table_alias_views_assigned_to_the_table_whose_join_needs_altering']['join'])) {    $query->table_queue['table_alias_views_assigned_to_the_table_whose_join_needs_altering']['join'] = new my_fixed_up_join();
  }
}  

In my case, it was sufficient to pass no arguments to the my_fixed_up_join constructor, and have it just kick out a hardcoded string of SQL conditions:

 public function build_join($select_query, $table, $view_query) {
    $join_sql = <<<JOINSQL
   (field_collection_item_field_data_field_sessions.item_id = field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_id  AND field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_type = 'field_collection_item')
OR (
    node.nid = field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_id AND field_collection_item_field_data_field_sessions__field_data_field_event_date.entity_type = 'node'
  )
  AND (
    field_collection_item_field_data_field_sessions__field_data_field_event_date.deleted = '0'
  )
JOINSQL;

    $select_query->addJoin('LEFT', $table['table'], $table['alias'], $join_sql);
  }

Why would you need to do this? In the example above, I was displaying several types of content in the view, and some of them attach the field directly while others are associated to the field through a “relationship” to another entity. The views UI will let you add the field as a member of the content, and will let you add a different field as a member of the related entity, but under the hood it joins to the field data table twice, creates two result columns, and you won’t be able to do things like sort on that field. What you really want is one join to the field data table with an OR condition, so the field comes up as a single column.

Posted in Drupal

Drupal webforms submissions by form_key



With the ease of entry for  basic use and API for extensibility, the Drupal webforms module is an indispensable tool. One snag with it, though, is that wherever it exposes the results of a submission in code, the submitted values are just expressed in a big 0-based numerically indexed array. Using this $submission->data array directly would make for hard-to-read and fragile code, and there doesn’t seem to be a function provided in webforms to give you the submitted data in an associative array.

Creating my own function to generate an associative array of results fortunately wasn’t that bad though, and seems like a valuable enough thing to make note of here.

/**
 * Returns an associative array of the submission data, instead of the
 * numerically indexed $submission->data
 *
 *  Inspired by http://drupal.stackexchange.com/questions/23607/how-do-i-access-webform-components-labels-on-the-congratulations-page
 *
 * @param int $nid The node id of the webform
 * @param int $sid The submission id.
 */
function webform_submission_data_keyed($nid, $sid) {
  $data = array();
  $node = node_load($nid);

  module_load_include('inc', 'webform', 'includes/webform.submissions');
  $submission = webform_get_submission($nid, $sid);

  foreach($node->webform['components'] AS $key => $component) {
    if(isset($submission->data[$key])) {
      $data[$component['form_key']] = $submission->data[$key];
    }
  }

  return $data;
}

This’ll give you a multidimensional associative array with the “machine name” of each field as keys, and arrays as values. The subarrays are 0-based numeric indicies to one or more response values the user selected.

If you wanted the human names as keys, use $data[$component[‘name’]] = $submission->data[$key].

Posted in Drupal

Crazy scheme of the week: melding heterogeneous filesystems



At work, we have an interesting problem. The primary storage system at MSI is from Panasas, a vendor that specializes in high aggregate I/O performance from a single filesystem to thousands of Linux compute nodes. Besides having a stunningly high CPU-to-disk ratio (read: being expensive), a big part of Panasas’ performance strategy is to remove the bottleneck that could easily occur at filer heads in traditional NAS when thousands of compute nodes are all asking for I/O ops. They do this by outsourcing much of the NAS head’s role to the compute nodes themselves, leaving only a thin metadata server that puts compute nodes into direct contact with the storage blades that contain the files being accessed.

The problem is a large amount of the data housed on our Panasas system doesn’t really need the privilege of residing on such a high-speed system, either because it is owned by non-HPC users who are only accessing it from one node at a time, or (the bigger problem) because nobody’s looked at it in 2 years, and they’re not likely to again anytime soon. And unfortunately, for all the things Panasas does, HSM is not currently one of them.

We could easily whip up a separate filesystem on a denser SAN or something, but offloading all the data at uber-rest to it would not be transparent to the users who currently have data resident on the Panasas. Doesn’t mean it can’t happen, but not ideal.

An alternative that I’ve been kicking around is to create a pseudo-filesystem that only actually stores metadata, and delegates I/O to one of several other filesystems. The idea is basically that the Linux kernel provides a well-defined API to filesystems for operations at the file level which every mountable filesystem in Linux conforms to. It is a common denominator whether your filesystem is a FAT-16 floppy disk, an ext4 local hdd, or even a DirectFlow Panasas filesystem. So, it ought to theoretically be possible to write another piece of code that knows how to consume the same API, but isn’t the kernel itself…and in fact also implements this API.

This pseudo-filesystem would be mounted by the kernel, and it in turn would be configured to mount any other filesystems of the administrator’s choice. Then, when asked for a file, the pseudo-fs would query a metadata server of its own to determine which underlying filesystem the file resides on, and then simply pass through all further interactions on that file handle to the API methods of the appropriate underlying filesystem.

With a scheme like this, one additional step is added when opening a file, but (importantly), the pseudo-fs would not be expected to introduce a measurable performance impact on all subsequent I/O to opened files, since native performance charactaristics of the underlying filesystems would be preserved. In my case, data transfer from storage to compute node would still happen using Panasas’ proprietary DirectFlow protocol if the data was on the high-speed Panasas system.

Clearly, completing this would be a very ambitious undertaking, but so far I haven’t discovered any fundamental reasons why this system wouldn’t work, and if such a thing existed, it might prove to be a unique open-source solution to HSM across any combination of heterogeneous storage systems.

Fortunately, it feels like a project with a pretty manageable, progressive roadmap. A feasible and likely very instructive first step would be to simply implement a kernel module providing a mountable filesystem that in fact passes all calls through to some single underlying filesystem.

Now we just have to see if this ever becomes more than my latest crazy scheme of the week.

Posted in Uncategorized
← Older Entries Newer Entries →

Recent Posts

  • Reset connection rate limit in pfSense
  • Connecting to University of Minnesota VPN with Ubuntu / NetworkManager native client
  • Running nodes against multiple puppetmasters as an upgrade strategy
  • The easiest way to (re)start MySQL replication
  • Keeping up on one’s OpenSSL cipher configurations without being a fulltime sysadmin

Categories

  • Computing tips
    • Big Storage @ Home
    • Linux
  • dev
    • devops
    • Drupal
    • lang
      • HTML
      • JavaScript
      • PHP
    • SignalR
  • Product Reviews
  • Uncategorized

Tags

Apache iframe malware performance Security SignalR YWZmaWQ9MDUyODg=

Archives

  • June 2018
  • January 2018
  • August 2017
  • January 2017
  • December 2016
  • November 2016
  • July 2016
  • February 2016
  • January 2016
  • September 2015
  • March 2015
  • February 2015
  • November 2014
  • August 2014
  • July 2014
  • April 2014
  • February 2014
  • January 2014
  • October 2013
  • August 2013
  • June 2013
  • January 2013
  • December 2012
  • November 2012
  • September 2012
  • August 2012
  • July 2012

Blogroll

  • A Ph.D doing DevOps (and lots else)
  • gavinj.net – interesting dev blog
  • Louwrentius.com – zfs@home with 4x the budget, other goodies
  • Me on github
  • My old edulogon.com blog
  • My old GSOC blog
  • My wife started baking a lot
  • Now it's official, my wife is a foodie

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

EvoLve theme by Theme4Press  •  Powered by WordPress eworldproblems