Automatic security updates and a support lifecycle: the missing links to Backdrop affordability



I saw Nate Haug and Jen Lampton give their Backdrop CMS intro talk last weekend at the Twin Cities Drupal Camp. They have already done much to identify and remedy some big functionality and UX issues that cause organizations to leave Drupal for WordPress or other platforms, but you can read more about that elsewhere or experience it yourself on Pantheon.

Their stated target audience is organizations who need more from their website than is WordPress’s primary use case, but still don’t need all the capability that Drupal 8’s software engineering may theoretically enable — everyone doesn’t need a CMS that supports BigPipe or compatibility with multiple heterogeneous data backends.

That positions them squarely in the same space in the same target market as Squarespace — but whereas Squarespace is a for-profit business, Backdrop is open-source software, and affordability to the site owner is so important to the project that it gets a mention in Backdrop’s single-sentence mission statement.

Simplified site building for less-than-enterprise organizations is a crowded space already. The Backdrop philosophy identifies a small area of this market where a better solution is conceivable, but I think a big reason so much emphasis is given to goals and philosophy on the Backdrop website and in their conference session is that Jen and Nate recognize their late entry in this crowded market leaves little room for Backdrop to miss in achieving its goals. Backdrop’s decision makers definitely need to keep a clear view of the principles the project was founded for in order for Backdrop to positively differentiate itself.

Affordability is one potential differentiator, and I am personally happy to see it’s one already embodied by Backdrop’s promises of backwards compatibility and low server requirements. But, frankly, WordPress has got those things already. An objective evaluation of Backdrop’s affordability in its current form would put it on par with, but not appreciably better than a more established competitor. But there is hope, because:

Current CMSs don’t give site owners what’s best for them

To make converts, Backdrop could truly differentiate itself with a pledge to offer two new things:

  1. Security backports for previous releases, with a clearly published end-of-support date on each release, so site owners can address security issues with confidence that nothing else about their site will change or stop working.
  2. A reference implementation for fully automated updates, so that installed sites stay secure with zero effort.

Here’s why.

Let’s assume there’s a certain total effort required to create and maintain that highly customized website described by Backdrop’s mission statement. Who exerts that effort is more or less transferable between the site builder and the CMS contributors. On one extreme, the site owner could eschew a CMS entirely and take on all the effort themselves. Nobody does this, because of the colossal duplication of exertions that would result from all the sites re-solving many of the same problems. Whenever a CMS takes over even a small task from site builders, a mind-boggling savings in total human hours results.

Consider this fact in the context of CMS security. From a site owner’s perspective, here’s the simplest-case current model in Drupal, Backdrop, WordPress, and other popular open-source CMS’s for keeping the sites they run secured over time. This model assumes a rather optimistic site owner who isn’t worried enough about the risk of code changes breaking their site to maintain a parallel development environment; organizations choosing to do this incur even more steps and costs.

  1. A responsible individual keeps an eye out for security updates to be released. There are myriad ways to be notified of them, but somebody has to be there to receive the notifications.
  2. When the individual and/or organization is ready, an action is performed to apply the latest version of the code, containing the security update as well as non-security changes the developers have made since this site was last updated. Sometimes this action is as simple as clicking a button, but it cannot be responsibly automated beyond this point due to the need to promptly perform step 3.
  3. The site owner browses around on their site, checking key pages and functions, to find any unintended consequences of the update before their customers do. (For non-enterprise, smaller sites, let’s assume an automated testing suite providing coverage of the highly customized features does not exist. Alternatively, if it did exist, there would be a cost to creating it that needs to be duplicated for each instance of the CMS in use.)
  4. In practice, the developers usually did their job correctly, and the update does not have unintended consequences. But sometimes, something unexpected happens, and a cost is incurred to return the site to its past level of operation, assuming funds are available.

Once a site has been developed, unsolicited non-security changes to the code rarely add business value to the organization operating the site. In the current model, however, changes are forced on organizations anyway as a necessary component of maintaining a secure site, merely because they are packaged along with the security update. In my opinion, the boldface observation above ought to be recognized as one of the principles guiding Backdrop’s philosophy. In the classic model, the CMS avoids a small task of backporting the security fix to past releases and the work is transferred to site owners in the form of the above steps. That’s expense for the site owner, and in total it is multiplied by each site the CMS runs — a much larger figure than offering a backport would have amounted to.

This is a clear shortcoming of current offerings, and Backdrop’s focus on affordability makes it a ripe candidate for breaking this mold. Not to mention the value proposition it would create for organizations evaluating their CMS options. Heck, make a badge and stick it on backdropcms.org/releases:

Supported

3 years

Stable

Backdrop could guarantee security updates will not disrupt the sites they run; the competition could only say “A security update is available. You should update immediately. It is not advisable to update your production site without testing the update first, because we introduced lots of other moving parts and can’t make any guarantees. Good luck.”

That’s something I think developers and non-technical decision makers alike can appreciate, and that would make Backdrop look more attractive. Don’t want to pay monthly dues to Squarespace? Don’t want to pay periodic, possibly unpredictable support fees to a developer? Now you can, on Backdrop.

The above case that a software support lifecycle would make site maintenance more affordable to site owners does not even begin to take into consideration the reality that many sites simply are not updated in a timely fashion because the updates aren’t automated. If you are not an enterprise with in-house IT staff, and you are not paying monthly premiums to an outfit like Squarespace or a high-end web host with custom application-layer firewalls, history shows a bot is pretty guaranteed to own that site well before you get around to fixing it. Exploited sites are in turn used to spread malware to their visitors, so adding automated updates to the CMS that can be safely applied, rapidly, without intervention would have a far-reaching overall impact on Internet security.

But how achievable is this?

Isn’t extended support boring to developers volunteering their time?

Yes, probably. Top contributors might not be too psyched to do backports. But just as in a for-profit development firm, developers with a range of abilities are all involved in creating open-source software. Have top contributors or members of the security team write the fix and corresponding tests against the latest release, and let others merge them back. The number of patches written for Drupal core which have never been merged has I think demonstrated that developer hours are eminently available, even when the chance of those hours having any impact is low. Propose to someone that their efforts will be reflected on hundreds or thousands of sites across the Internet in a matter of days, and you’ll get some volunteers. Novice contributors show up weekly in #drupal-contribute happy to learn how to reroll patches as it is. Security issues might be slightly more tricky in that care needs to be taken to limit their exposure to individuals whose trust has been earned, but this is totally doable. Given the frequency of core security releases in Drupal 7, a smaller pool of individuals known personally by more established community members could be maintained on a simple invite-only basis.

Some interesting possibilities exist around automating attempts to auto-merge fixes through the past releases and invoke the test suite, but rigging up this infrastructure wouldn’t even be an immediate necessity.

Also, other projects in the wider, and even PHP FOSS world show us it can be done. Ubuntu made software lifecycles with overlapping supported versions famous for an entire Linux distribution (though most all of the other distros also pull it off with less fanfare, chalking it up as an implicit necessity), and it’s even been embraced by Symfony, the PHP framework deeply integrated into Drupal 8. While Drupal adopted  Symfony’s software design patterns and frequently cites this as one of Drupal 8’s strengths, they didn’t adopt Symfony’s software lifecycle practices. In this regard, Drupal is increasingly finding itself “on the island.” Hey, if Backdrop started doing it, maybe Drupal would give in eventually too.

What about contributed modules?

I would argue that a primary strategy to handle the issue of contrib code should be to reduce the amount of contrib code.  This fits well with a Backdrop initiative to put the things most people want in the core distribution of the product.  A significant number of sites — importantly, the simplest ones that are probably least inclined to think about ongoing security — would receive complete update coverage were backports provided only for core. The fact that contrib is there in CMS ecosystems is sometimes cited as a reason security support lifecycles would not be possible, but it’s no excuse not to tackle it in the CMS’s core.

Contrib will always be a major part of any CMS’s ecosystem though, and shouldn’t be left out of the opportunity to participate in support lifecycles.  I would propose that infrastructure be provided for contrib developers to clearly publish their own support intentions on their project pages. Then, when a security issue is disclosed in a contrib module, the developer would identify, by means of simple checkboxes, the versions that are affected. There would be no obligation to actually produce a patched version of old releases identified as vulnerable, however, regardless of previously published intentions. This would have two effects: A) the developer would be reminded that they committed to do something, and therefore might be more likely to do it, and B) sufficient data would be available to inform site owners of a security issue requiring their attention if the contrib module chose not to provide a backported fix. Eventually, the data might also be used as a statistic on project pages to aid site builders in selecting modules with a good support track record.

Aren’t automated updates inherently insecure?

No, although some web developers may conflate performing an automatic update with the risks of allowing code to modify other code when it can be invoked by untrusted users. A reference implementation of an automatic updater would be a separate component from the CMS, capable of running with a different set of permissions from the CMS itself.

Brief case study: “Drupalgeddon”

Here’s the patch, including its test coverage, that fixed what probably proved to be the most impactful security vulnerability in Drupal 7‘s history to date:

drupalgeddon

The fix itself is a one-line change in database.inc. Security patches are, as in this case, often very small and only have any impact on a site’s behavior in the face of malicious inputs. That’s why there’s value in packaging them separately.

Drupal 7.32, the version that fixed this vulnerability, was released in October 2014. A

git apply -3 drupalgeddon.patch

is able to automatically apply both the database.inc and database_test.test changes all the way back to Drupal 7.0, which was released almost four years earlier in January 2011. Had the infrastructure been in place, this fix could have been automatically generated for all earlier Drupal versions, automatically verified with the test suite, and automatically distributed to every Drupal 7 website with no real added effort on the part of the CMS, and no effort on the part of site owners. Instead, in the aftermath, tremendous time energy and money was expended by the site owners that were affected or compromised by it, with those that didn’t patch in a matter of hours facing the highest expenses to forensically determine if they were compromised and rectify all resulting damages.

You better believe botnet operators maintain databases of sites by CMS, and are poised to use them to effectively launch automated exploits against the bulk of the sites running any given CMS within hours of the next major disclosure. So, unless the CMSs catch up, it is not a matter of if this will happen again, but when.

The only way to beat the automation employed by hackers is for the good guys to employ some automation of their own to get all those sites patched. And look how easy it is. Why are we not doing this.

Final thoughts

A CMS that offered an extended support lifecycle on their releases would make site ownership more affordable and simpler, and would improve overall Internet security. Besides being the right thing to do, if it made these promises to its users, Backdrop would be able to boast of a measurable affordability and simplicity advantage. And advantages are critical for the new CMS in town vying for market share in a crowded and established space.

Posted in dev, Drupal, PHP

HAProxy “backup” and “option redispatch”



I’m testing out my new infrastructure-in-progress for redundant web hosting, described in my last post. The quick outline of the components again, is a total of four VMs:

  • Two small & inexpensive instances in a tier 4 datacenter running Apache Traffic Server + HAProxy.
  • One bigger, rather spendy backend VM in a tier 4 datacenter running Apache, webapp code, databases, etc. Call it hosting1.
  • One VM roughly equivalent in specs to the backend VM above, but running on a host machine in my home office so its marginal cost to me is $0. Call it hosting2.

The thinking is that hosting1 is going to be available 99%+ of the time, so “free” is a nice price for the backup VM relative to the bandwidth / latency hit I’ll very occasionally take by serving requests over my home Internet connection. Apache Traffic Server will return cached copies of lots of the big static media anyway. But this plan requires getting HAProxy to get with the program – don’t EVER proxy to hosting2 unless hosting1 is down.

HAProxy does have a configuration for that – you simply mark the server(s) you want to use as backups with, sensibly enough, “backup,” and they’ll only see traffic if all other servers are failed. However, when testing this, at least under HAProxy 1.4.24, there’s a little problem: option redispatch doesn’t work. Option redispatch is supposed to smooth things over for requests that happen to come in during the interval after the backend has failed but before the health checks have declared it down, by reissuing the proxied request to another server in the pool. Instead, when you lose the last (and for me, only) backend in the proper load balance pool, requests received during this interval wait until the health checks establish the non-backup backend as down, and then return a 503, “No servers are available to handle your request.”

I did a quick test and reconfigured hosting1 and hosting2 to both be regular load balance backends. With this more typical configuration, option redispatch worked as advertised, but I now ran the risk of traffic being directed at hosting2 when it didn’t need to be.

Through some experimentation, I’ve come up with a configuration that my testing indicates gives the effect of “backup” but puts both servers in the regular load balance pool, so option redispatch works. The secret, I think, is a “stick match” based on destination port — so matching all incoming requests — combined with a very imbalanced weight favoring hosting1.

Here’s a complete config that is working out for me:

global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  log  10.2.3.4 local0
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  stats  socket /var/lib/haproxy/stats
  user  haproxy

defaults
  log  global
  maxconn  8000
  option  redispatch
  retries  3
  stats  enable
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen puppet00
  bind 127.0.0.1:8100
  mode http
  balance static-rr
  option redispatch
  retries 2
  stick match dst_port
  stick-table type integer size 100 expire 96h
  server hosting1 10.0.1.100:80 check weight 100
  server hosting2 10.0.2.2:80 check weight 1

I tested this by putting different index.html’s on my backend servers and hitting HAProxy with cURL every 0.5 seconds for about an hour, using the `watch` command to highlight any difference in the data returned: 

watch -x --d=permanent -n 0.5 curl -H 'Host: www.mysite.com' http://104.156.201.58  --stderr /dev/null

It didn’t return the index.html from hosting2 a single time, until I shutdown apache on hosting1 after about an hour.

There’s a small amount of magic here that I unfortunately don’t understand, but will resign myself to being happy that it works as desired. Once failed over to hosting2, I would kind of expect the stick table to update and cause everything to stay with hosting2 even after hosting1 comes back. In actuality, it returns to hosting1. So cool, I guess.

Posted in devops

Top gotchas: Creating virtual machines under KVM with virt-install



After a stint building Stacy’s new blog whilst contributing to Drupal 8, I’ve been directing my free development time to more ops-like concerns for the past month or so. I moved to a cheaper parking facility at work and plan to direct the savings to bankroll the expansion of my own little hosting platform. And, I’m going virtualized.

This is a big shift for me, as I’ve gotten many years of trusty service without much trouble from the “single dedicated server with OS on bare metal” model, only experiencing a handful of minutes of downtime a month for major security updates, while watching several different employers flail around keeping all the pieces of their more complex virtualized environments operational.

On paper, the platform I’ll be moving to should be more resilient than a single server — the cost savings of virtualization plus infusion of extra cash will enable me to run two redundant webserver instances behind a third load balancing VM, with another (fourth) load balancing VM that can take over the public IP addresses from the first load balancer at a moment’s notice (thanks to the “floating IP” or “reserved IP” features that the better VPS providers have been rolling out recently). The provider I’m going with, Vultr, has great customer service and assures me that my instances will not share any single points of failure. Thus, I should have full redundancy even for load balancing, and I’ll be able to take instances down one at a time to perform updates and maintenance with zero downtime.

Alas, even on VMs, full redundancy is expensive. I could have just bought sufficiently small instances to host everything at Vultr and stay within my budget, but I’m betting I’ll get better performance overall by directing all traffic to a beefed-up instance at Vultr (mostly superior to the old hardware I’m currently on) under nominal conditions, and have the load balancer fail over to a backend server on hardware in my home office for the short periods where my main Vultr instance is unavailable. My home office isn’t the Tier 4 DuPont Fabros facility in Piscataway, NJ that the rest of the instances will reside in, but it is server grade components with a battery backup that hasn’t seen a power failure yet, and the probability of a simultaneous outage of both webservers seems very low.

The two backend webservers need to be essentially identical for sanity’s sake. However, as hinted in other posts such as the one where I installed an Intel Atom SoC board, I’m a bit obsessive about power efficiency, and there was no way in hell I was running a separate machine at home to serve up a few minutes of hosting a month. So, I needed to figure out KVM.

As near as I can ascertain having completed this experience, there are one or two defaults you always need to override if you want a VM that can write to its block device and communicate on a network.

  1. You probably need to create an ethernet bridge on the host system, onto which you attach your guests’ virtualized NICs. Examples abound for configuring the bridge itself in various linux distros, but none seem to mention with clarity that the host kernel needs to be told not to pump all the ethernet packets traversing the (layer 2!) bridge through the (layer 3!) IPTables. I’m puzzled whose idea this was, but someone made this thing called bridge-nf and apparently talked their way into it being default ‘on’ in the Linux kernel. Maybe that would be nice if you wanted run suricata on a machine separate from your main router, or something, but yeah otherwise I’d recommend turning this off: 
    cd /proc/sys/net/bridge
for f in bridge-nf-call-*; do echo 0 > $f; done

    …and if the bridge should pass tagged vlan traffic, snag bridge-nf-filter-vlan-tagged too. See http://unix.stackexchange.com/questions/136918/why-does-my-firewall-iptables-interfere-in-my-bridge-brctl/148082#comment-218527 for more detail. If you have strict iptables rules on your host, your guests won’t get any network connectivity — and on the flipside, while troubleshooting that problem I inadvertently made packets on the theoretically isolated bridge start appearing on a totally different network segment when I tested a blanket-accept rule on the host iptables’ FORWARD chain. I’m trying to isolate my hosting-related vlan and subnet from my plain-old home Internet connection sharing vlan and subnet, and to get this result was just wrong. Bridge-nf is scary stuff; turn it off.

  2. If you opt to use lvm volumes as the backing for your VM’s disks, virt-install provides a handy syntax to cause it to make and manage the logical volume along with the VM: 
    virt-install --disk pool=vm-guest-vg,size=80,bus=virtio ...

    where vm-guest-vg is a volume group you’ve set aside for VM disks. This says, make me a new 80G logical volume in the vm-guest-vg volume group, associate it to this instance, and attach it as a disk on the guest through the optimized virtio bus. If you do this, your VM will experience many I/O errors. You must also add “sparse=false”:

    virt-install --disk pool=vm-guest-vg,size=80,bus=virtio,sparse=false ...

    and then, finally, your VM hosting environment will be reasonably usable to a guest operating system.

For completeness/my own reference later on, an entire virt-install command:

virt-install -n 'the-instances-name' --virt-type kvm --hvm --autostart -r 4096 --vcpus=4 --os-type=linux --os-variant=ubuntutrusty --cpu host --disk pool=vm-guest-vg,size=80,bus=virtio,sparse=false --network bridge=hostingbr,model=virtio --graphics vnc -c "/data/public/Program Installers/ubuntu-14.04.3-server-amd64.iso"

There’s still much work ahead of me to build this entire infrastructure out — some custom scripting will likely be needed to pull off automated IP address transfers between the load balancers, and I’m shooting to use Apache Traffic Server as the HTTPS terminator/caching proxy, because it seems to have been doing the HTTP/2 thing for longer than alternatives like nginx, and HTTP/2 is cool. I’ll do a follow-up post after it’s all been up and running for awhile and we’ll see if going virtualized was even a good idea…

And, stay tuned, once I’m done with that, I have plans to further reduce my home infrastructure’s power consumption by losing the bulky full ATX PSU in my main server, which burns about 10 watts minimum, and go to a little laptop-sized DC transformer. If I put the hot data on an SSD and move the big PSU and spinning disks (which are the only things that need that much power, and only when spinning up) to a second machine that just acts as an iscsi target with wake-on-mac, I think I can keep the big disks and their PSU in standby the vast majority of the time. Fun times, fun times.

Posted in Linux

Pinterest hover buttons and picturefill



Pinterest hover buttons, those little “Pin It” buttons that appear when mousing over images on Pinterest-integrated sites, don’t work if your site is using HTML5 responsive images (the picture element and srcset attribute) in the way recommended when using the picturefill polyfill for legacy browser compatibility.

The reason is that the JavaScript Pinterest asks you to include on your site, “pinit.js,” assumes any image that should receive a hover button has a nonempty src attribute, while your markup written for optimal use of picturefill excludes the src attribute, relying solely on the srcset attribute instead. This way, legacy browsers don’t unnecessarily load a version of the image that may turn out not to be needed.

Of note, Drupal 8’s Responsive Images module, as themed by default, issues an srcset but no src attribute for this reason. Voila, no Pinterest joy.

Turns out a tiny modification to pinit.js (actually, pinit_main.js; pinit.js is merely a loader) fixes this issue, so long as you’re also comfortable specifying which particular version of the image you want pinned in your img tags using Pinterest’s data-pin-media custom data attribute. I’ve recently opened a pull request on github to try and get this into their hosted version of the JavaScript, but meanwhile you can grab my repository from github and host it yourself as well. Update: the above PR was merged; the stock pinterest script now supports the below example:

Once running this JavaScript, your markup can be

<img srcset="/my-pic.png" data-pin-media="http://mysite.com/my-pic.png" />

…and you should be good to go!

I’ve also got a really quick module for Drupal 8 that adds data-pin-media to all responsive images at https://github.com/mbaynton/pinterest_responsive, which I’ll see about making as a drupal.org project if/when my JS gets merged.
Update: Released as pinterest_hover for Drupal 8.x

Posted in Uncategorized

Reset a crashed migration in Drupal 8



I’ve been writing migration code for my wife’s WordPress blog to Drupal 8, using the migrate framework. This framework attempts to track if a particular migration is currently running, but does not always unset that flag if it crashed with a PHP fatal error (which will happen when you are writing new code.) Subsequent attempts to run the migration will produce “Migration your_migration_name is busy with another operation: Importing”.

To unstick it in this case, you can run the following in drush:

drush php-eval 'var_dump(Drupal::keyValue("migrate_status")->set('your_migration_name', 0))'
Posted in Drupal

Handling GET-method submissions with Drupal 7’s Forms API



Remarkably, Drupal 7’s FAPI treats form submissions using the GET method as something of an afterthought. “Just register a URL with wildcards for that!” is the usual refrain from the Drupal devout when the less indoctrinated express their shock over this state of affairs. That approach, however, has several limitations over the W3C standard Way To Do It, such as if your form /has/twenty/optional/fields/that/all/need/a/position/in/the/url, or if your goal is to have Drupal serve response pages for requests formed by standards-complaint tools.

Fortunately though, it is possible. Getting it to work at all requires a multi-line recipe; things become even trickier if you want Drupal to handle submissions coming from another web framework or a static html file. I was faced with this prospect when I needed to reimplement a single form in a more complex legacy application. The frontend form needed to stay as-is because that’s where users look for it, but Drupal had all the backend pieces to make the magic happen in < 100 lines, so I figured I would just point the existing form’s method to a custom module in Drupal. Here’s a generalized sample of what all is needed in such cases, derived from combined googling and interactive debugging through core:

  1. Build the form 
    $form_state = array(
  'method' => 'get',
  'always_process' => true, // if support for submissions originating outside drupal is needed
);
$form = drupal_build_form('name_of_my_form', $form_state);

    Returning $form at this point would suffice as the entirety of a page callback. (Note that because of the needed $form_state, you have to drupal_build_form instead of drupal_get_form, and the standard trick of just registering drupal_get_form as your page callback in hook_menu isn’t an option.)

  2. Write the form callback
    Despite the fact that you already told Drupal you want a GET form in the $form_state, you’ve gotta do so in $form in your form callback as well:

    function name_of_my_form($form, &$form_state) {
   $form['#method'] = 'get';
   $form['#token'] = 'false';
   ... // describe your form as usual
   return $form;
}
  3. Break the redirect loop and otherwise handle the submission in the form submission callback
    So, you’ve made a form that trips the submission callback when you access it and loads up the values specified via GET into Drupal’s usual form_state structures. But wait! Drupal’s standard thing is to redirect somewhere following a form submission, and your GET url looks like a form submission to it now, so endless redirect loop ahoy. This can be killed off in a variety of ways, but I’ll mention two that I think should cover most cases.

    If you examine the submitted values and find the submission to be incomplete, such as if the user visited the registered path without providing any GET parameters, you can simply set $form_state['no_redirect'] = true in your submit handler.

    If you wish to actually process a submission and display results on the page, here’s how I did it: instead of setting no_redirect, set $form_state['rebuild'] = true. This implicitly stops a redirect from occurring, and also results in two calls to your form callback during the request – one to provide Drupal with your form information for validation/submission processing, and a second after the submission handler has been run. The results of the second invocation are what is rendered to the page. So, it’s easy enough to stick some results data in your $form_state in the submit callback, then look for it in the form callback and add some additional elements to the form’s render array to show them.

Of course, 'always_process' => true explicitly lowers Drupal’s CSRF shields, so be sure to confirm for yourself that your GET form really isn’t modifying the state of your application.

Also, beware the magic q: Drupal uses the URL param ‘q’ when pretty URLs are disabled, and treats it as the string to be routed whenever it is present as a parameter, whether pretty URLs are on or off. So, it’s unavailable for use as the name of an input on your form.

Posted in Drupal

More super ubuntu tips: udev-based NIC naming & lvm silicon_medley_raid_member



I had a pretty booked weekend, but decided to see if I could slam an Intel Atom SoC-based board into my home server anyway. Plan of attack: take a snapshot of OS lvm volume in case things went south, mount snapshot to double-check it, shut down, swap out system boards, resume business as usual.

Actual result: Still on the old board, after taking the root volume snapshot, I was unable to mount the thing, because mount detected the volume as of type “silicon_medley_raid_member”  rather than the ext4 filesystem it was. I’d never seen that before, but mount -t ext4 fixed it; I examined the mounted snapshot which seemed fine and ignored that strange hiccup.

Upon power-up on the new board, the kernel failed to mount the real root filesystem, so the boot sequence halted before pivot_root with an error (mount: cannot mount /dev/mapper/ssd–vg-root on /root: no such device) and dumped me in busybox on the initramfs.

Now. given that a mount without explicit fs type had failed on the snapshot, I thought it possible that somehow the type bits of the root filesystem itself had gotten corrupted, and that was why the boot-time mount was failing as well. How to confirm and/or fix? Thanks to this blog post, I got answers to both. Saved me a bundle of trouble! Since it seems to always involve LVM, and neither I nor anyone else who has been bit by this has a clue what happened, I’m calling software bug.

With my machine booting again (35 watts lower!) I just had to tackle the NIC names, so the new NIC plugged to my lan was still eth0 etc, making all my configurations and firewall rules and such “just work.” I happily pulled up the MAC addresses with lshw -class network, and swapped out the old values with the new in /etc/udev/rules.d/70-persistent-net-rules. Aaand it didn’t work. So, what the heck.

Searching through dmesg, I discovered the igw driver sanely brought the NICs up as eth0 through eth3, and then udev came along and arbitrarily renamed them em1 through em4. No idea where it got that from. After quite a bit of prodding and searching, I found this suggestion which I am not sufficiently esteemed to upvote on that site. The default contents of the Ubuntu udev network interface rules file requires that the interface being renamed already be named eth*. That corresponds to

KERNEL="eth*",

in the file. Seems udev selects em* as the name and sets it before it bothers to read the rules file, and thus it never renames them again to the desired values. Removing that constraint, I was again fully up and running.

I’m grateful these folks documented the respective magic incantations, leaving substantial portions of my weekend for more fun things. The whole thing was maybe 4 or 5 hours, not as bad as I’d feared. Next time I build up enthusiasm to break stuff, I’ll tackle the link-aggregating switch I picked up…

Posted in Linux

Extreme Home Network Makeover: IPv6 & hostapd



When constant.com made IPv6 available on its dedicated servers and shortly thereafter Comcast finally got around to lighting up IPv6 on my home connection, I inevitably had to begin playing with it. I had these main goals:

  • Distribute globally unique public addresses to every device on my home network
  • Keep using a home Linux server as my Internet gateway, configure it to do regular routing and forwarding without NATting for IPv6 traffic, and run a stateful firewall in place of hiding behind a NAT.
  • Serve websites on my dedicated server over IPv6
  • Bonus – see if I can rig up public DNS to return AAAA’s for hostnames on my home LAN.

I mostly prodded at it from time to time for months, with slow progress. It took quite awhile to sort out some real basic stuff: the public IPv6 address my home gateway acquired over DHCP was showing up as a /64, so I imagined that /64 subnet was available to me – that’s how it had worked with the /64 my dedicated server received from constant.com’s router advertisements. Many hours were spent sending traffic out with other source addresses in the /64 from my home gateway and into the void before I stumbled on DHCP-PD and figured out comcast was using it. After that discovery, it was pretty quick to get the ISC dhclient (the default on Ubuntu) to send out a request for prefix delegation, packet-sniff the offer that came back to figure out what actual /64 I could use for the other devices on my home network, and configure up dnsmasq to RA that. That netted my internal network devices a brief glimpse of IPv6 connectivity, until I restarted the gateway, or something. But, it was long enough for me to figure out that the nothing-special and no longer supported netgear wireless router I’d been using as my wifi access point was dropping more than 90% of IPv6 traffic, for some reason. So, I had to add “pick out a new wifi AP” to the list of things that needed doing. Boy, was that a rabbit-hole.

I’ve had a good 15 years of experience with all kinds of residential wireless routers not being very good. From basic models in the early days of 802.11b to the latest, (priced as) premium Apple AirPort Extreme, I’ve only ever gotten truly solid reliability from certain hardware revisions of the now obsolete Linksys WRT-54G. It seemed shortsighted to buy a new AP in late 2014 that didn’t do 802.11AC, but I wasn’t terribly enthused about picking up another device priced in the multiple-hundred$ that likely still would need the good old occasional power cycling, either. What if I could just have the Linux gateway machine itself be the AP, I wondered?

I did a respectable amount of research on Linux as a wifi access point. Hostapd was the obvious choice for running WPA/sending out beacon frame type stuff, but selecting a wireless card wasn’t as clear. There are many wifi radio chipsets with Linux drivers that also have some level of support for access point mode, although the ath10k Linux driver seemed to be the only as-yet available line of chipsets supporting both 802.11AC and access point mode. I found the Compex mini-PCIe card that seemed to be the only as-yet available consumer device using said chipset, found that you can basically only get it on eBay from some guy in Germany, realized that I’d still have to work out a scheme for external antennas, found a post from only a few months earlier on the ath10k developer’s mailing list mentioning someone had even made it run long enough to run a benchmarking suite before it crashed, and decided I’d be better off bailing on the 802.11AC idea.

Ath9k devices seemed a reasonable consolation, except for a note on ath9k’s wireless.kernel.org page that says it only supports seven simultaneous associated devices. We don’t have that many, but it’s not much room to grow. So, I picked up the Intel Centrino Advanced-N 6205 PCIe card from Microcenter. I fidgeted with hostapd and other configurations for a few days trying to get it to at least approximately resemble 802.11n performance, but didn’t succeed before its driver started crashing and needing to be re-insmodded. Back to Microcenter with that one (thanks for the no questions asked refund!)

Ultimately, I ended up with a wonderfully inexpensive Rosewill RNX-900PCE using the ath9k-compatible Atheros AR9380 chipset. This card’s been running now for a month or so with no issues, performing like an 802.11n access point, and, back to the main point, delivering IPv6 packets to my wireless devices.

But there was more trouble. My gateway now had an additional LAN interface – the wireless card. When a packet comes in for a particular host, how will the kernel know which of those two interfaces to forward it on?

Looked like the options were to either “bridge” the ethernet and wifi LAN interfaces, which I think involves layer 4 routing sending it to the bridged interface and then the bridged interface acting like a layer 2 switch implemented in software. Or, I could put the wifi card on its own, new subnet, and just have the routing rule be “destination address in new subnet range -> forward on wlan0.” Opting to not have yet another daemon process (the bridge) to keep running and configured, I went with the latter.

But there was more trouble. In order to have more than one IPv6 subnet and follow IETF recommendations, you need an IPv6 address space bigger than the /64 I’d been able to eek out of Comcast through the ISC dhcp client (since each subnet must be no smaller than a /64 in order to work correctly with stateless address autoconfiguration and stuff.)

A tremendously helpful Comcast employee surely considered way too smart by their management to ever be directly answering customer support requests through normal channels took the time to offer the handful of subscribers like me who’re nerdy enough to have these problems with some information about how to get a large enough delegated address space to run multiple subnets. There are a number of posts on comcast’s forums covering basically this subject matter, all seeming to have responses by “ComcastTusca”, but here’s the original and most comprehensive thread: http://forums.comcast.com/t5/Home-Networking-Router-WiFi/IPv6-prefix-size-and-home-routing/td-p/1495933. By switching from the ISC dhclient to dhcp6c (“WIDE dhcp”), and basing my configuration for that off of brodieb’s example, I was able to get my /60, and have my two lan interfaces be configured with their own /64’s each, automatically, when the /60 is delegated or re-delegated by Comcast.

That was about the end of the trouble. Another key element worth noting is dnsmasq’s RA “constructor” feature, which tells it to author appropriate router advertisements on each LAN interface based on that interface’s public IPv6 address information. Here’s the actual lines from my dnsmasq.conf:

dhcp-range=::,constructor:eth0,ra-stateless,ra-names
dhcp-range=::,constructor:wlan0,ra-stateless,ra-names

The “ra-names” bit is pretty cool too – read the manpage for the detailed description of the magic, but it tells it to go ahead and apply some trickery and heuristics to try to determine the ipv6 address that hosts running dual-stack IP have selected for themselves, ping that, and add it to the dns server side of dnsmasq as an AAAA if everything works. This is the basis for how you can look up the IPv6 address of internal hosts in my home network from anywhere in the world, for example mike-pc.lan.mbaynton.com, when they happen to be up. (Details I’m happy to publish on my blog, because setting up iptables on the gateway box so pinging it is about all you can do was mercifully not conceptually any different than with good old IPv4.)

Posted in Linux

CyberPower UPS OR700 with nut: secret sauce



After my last UPS (a dumb APC thing picked up from officemax) bit the dust, I got myself the CyberPower OR700. I was a wee bit leery of the CyberPower linux code, though, and opted to use more standardized software to manage the system shutdown. So, I gave NUT a try.

I let this puppet contrib module install and configure the nut client & server using pretty much the example manifest, and that went remarkably smoothly. (I used the usbhid-ups driver and had to add a udev rules file so the USB ups driver was allowed to do its thing, and specify that at least for now the nut server should just be “standalone” in /etc/nut/nut.conf; puppet did everything else; this stuff was easy to google & sort out.) Then I ran the recommended simulated low battery shutdown sequence test (upsmon -c fsd), and watched my system almost finish halting before the load power was killed. Hmm. I sniffed around a bit and decided the shutdown.resume command was being sent after the filesystems had been made readonly, so good enough, I figured.

The real trouble came when I did a “real” test by pulling the OR700 from the wall. At low battery, the system shutdown and load power was killed as seen in the simulation. But then, as in the simulated run where utility power was still available, after a few seconds the OR700 resumed power to the load (!) and the attached NAS dutifully began booting up. For those not following along, the UPS was unplugged and in a low battery state, but resumed power to the load anyway a few seconds after it received the shutdown.resume. Not good at all.

After much experimenting and many tests, the extra configuration from the defaults that did the trick were to add

  • ondelay = 0
  • offdelay = 100

to my /etc/nut/ups.conf. I was also pleased to find the aforementioned puppet module had parameters for those. Nice!

 

Posted in Uncategorized

Remote drush with only FTP and database access



Necessity

I manage a Drupal site as part of my job, which has taught me to appreciate the treat that is drush. With Drupal,  you really want ssh access to the webserver because it gives you the ability to use drush, the command-line administration utility for Drupal sites. Uninteresting tasks like module and core upgrades become much less tedious.

I also babysit the website for the Minnesota Valley Unitarian Universalist fellowship, a real simple Drupal site that lives on a basic shared hosting server. No ssh access here. Because doing drupal core updates solely over FTP sucks so much, finally today I sat down in earnest to see if I could somehow leverage the power of drush with such shared hosting sites.

Invention

There didn’t seem to be much accurate information about how to do it already, with top Google hits saying things like you need ssh and drush installed both locally and on the server (false and false), you can’t, and there was really nobody suggesting anything to the contrary especially when your local workstation is running Linux (here’s a workable method for Windows.) I stubbornly refused to believe them and came up with a method that has proven to work for upgrading modules and core successfully, and every other drush subcommand I have tried to date. This is a guide for Linux, and I’ll include the package names for Debian/Ubuntu, but other than that it should be generic to other distros too.

The basic idea is to get the site’s source code mounted as a filesystem on your local machine backed by your ftp server, install php on your local machine if needed, and get connected to the live site’s database server from your local machine. You then run drush directly on your local machine, but its changes impact the actual site. It’s a bit of work to get everything going the 1st time, but simplifies site management tons in the long run.

Warning

Before the step-by-step, two words of warning: 1. This method is for people that want to be lazy, but don’t mind if the computers take forever once you tell them what to do. This method’s performance is slow, due to curlftpfs. For my purposes I don’t care, but some might. 2. A somewhat faster and more reliable way involves copying your site’s source locally and synchronizing the real site to it after running commands that modify code, not a big deal with a few rsync commands. Even when you do it this way, you don’t need to get another instance of your site actually up and running locally, and you don’t need a clone of the database. I discuss this more in step 5.

Step-by-step Guide

1. Pick a Linux computer to run drush commands on, likely your desktop/laptop. This computer will need php with mysql and gd extensions installed, as well as curlftpfs (or something else that makes ftp servers mountable, if you have another preference.) On Debian/Ubuntu, run these commands to ensure you have all needed packages:

sudo apt-get install php5
sudo apt-get install php5-mysql
sudo apt-get install php5-gd
sudo apt-get install curlftpfs

Failure to install php5 ultimately produces a clear error message from drush that it failed to find php, but failure to install php5-mysql produces an error with inaccurate troubleshooting information when you run Drush, so make sure you have it. Lack of php5-gd will generate frequent warnings from drush, but can mostly be done without if you really don’t want to install it.

2. Install drush locally on the Linux computer you’ve chosen. When I put drush on a system I usually just untar the latest stable release from Drush’s github releases to somewhere like /usr/local/lib and symlink the drush shell script from the release to /usr/bin/drush, but drush is just php and shell scripts so you can untar and run it in your home directory or wherever as well if you like. I’ll skip the line-by-line commands for how to do these things; few readers will have never extracted a .tgz or .zip.

3. Mount your site somewhere on your local machine’s filesystem with curlftpfs. In this example I’ll mount to the path ‘/mnt/mvuuf_inmotion’; choose something appropriate to your site and create an empty directory there.

sudo mkdir -p /mnt/mvuuf_inmotion

Modifying the below command for your ftp server, username, password, and local mountpoint, run (as the regular user you plan to run drush commands as, not as root):

curlftpfs -o user="your-ftp-username:your-ftp-password,uid=$UID,gid=${GROUPS[0]}"  "ftp://your-ftp-server/" /mnt/mvuuf_inmotion

(ps, if you get an error about “fusermount: failed to open /etc/fuse.conf: Permission denied”, fix with ‘sudo chmod a+r /etc/fuse.conf’; if you get an error about “fusermount: user has no write access to mountpoint /mnt/mvuuf_inmotion”, fix with ‘sudo chown $UID:${GROUPS[0]} /mnt/mvuuf_inmotion’.)

You should now be able to cd into your local mountpoint, /mnt/mvuuf_inmotion in this example, and ls your site’s contents.

4. Remote database connectivity. You need to be able to connect to your site’s database  from your local computer using the credentials stored in sites/default/settings.php.

You should be confident that your hosting company supports such remote database connections before spending too much time on this; consult with them if you need to. Even when it is supported, many webhosts won’t actually allow connections from hosts other than the webserver unless you take special steps to allow it – again, consult with them if you need to. If your hosting provider gives you MySQL managed with cPanel, this is quite easy, just go to “Remote MySQL” under Databases and add your public IP address that your local machine makes connections out to the Internet with (as displayed at whatismyip.com.)

You should probably do a manual connection to the database to ensure this part works before proceeding. Assuming the database server software is MySQL and you’ve installed the mysql client locally, you can do that with

mysql -h [whatever.your.host.is] -u [username_from_settings.php] -p

where “whatever.your.host.is” would generally be a hostname given in your sites/default/settings.php, unless settings.php says “localhost” in which case you should try the domain name of your website. You’ll be prompted for a password, which should also be in the database settings in settings.php.

5. Try it out. From a terminal on your local machine, cd to the root directory of your site (where Drupal’s index.php file is) and try a simple drush command like drush status. (Note that when this runs successfully, drush status likes to list the site’s url as “http://default” even when a url is explicity given in your settings.php. This seems to be a drush bug but doesn’t seem to affect anything else; you can fix it anyway if you like with drush’s –uri option.)

Great! You now have drush set up for use with your remote site.

Optional: Working with a local copy of the code. If you’ve gotten this far, you have two choices: If everything works and you don’t mind how darned slow it is, you can choose to be done. However, if you can’t stand the slowness or if drush is even aborting during some operations with messages like “the MySQL server has gone away,” you might want to make a local copy of the site’s code on your computer, switch to that to run drush, then rsync the local copy back over to your curlftpfs mount after you do updates. (I needed to do this because the shared host had a quite short timeout on idle mysql connections, so doing everything over curlftpfs slowed drush runs down enough to break it.)

Here’s some sample rsync commands to create a local copy of your site’s code, and to sync the remote site with your local copy. Note that you should place your site in maintenance mode before beginning upgrades with this method, and exit maintenance mode only after the remote sync is complete.

Example command to create/sync existing local copy:

rsync -aP --exclude="sites/default/files" /mnt/mvuuf_inmotion ~/mvuuf_local

Example command to sync the remote site up with your local copy after it has changed:

rsync -aP --temp-dir=/var/tmp/rsync --exclude="sites/default/files" --delete ~/mvuuf_local /mnt/mvuuf_inmotion

(these commands assume you have your curlftpfs-mounted copy of the site at /mnt/mvuuf_inmotion and you want to store the local copy under your home directory in a folder called mvuuf_local; adapt to suit.) The specific options to rsync here are composed with some care, don’t change them unless you’ve given it careful thought.

The obligatory notice accompanying all drupal upgrade etc writeups: whenever you do things that make considerable changes to your site, whether with this remote drush method or any other, make backups first. Drush itself can do this for you if you like.

Posted in Drupal, Linux
← Older Entries