Wednesday, April 8, 2009

Agile Testing: Experiences deploying a large-scale infrastructure in Amazon EC2

Nice article from OpenX guy about automating deployments at scale. Via Nilesh.

Sunday, November 2, 2008

THE COCOTRON

MIT-licensed project to implement sufficient Cocoa frameworks on top of an abstracted cross-platform Obj-C layer (including patched gcc toolchain) in order to make a viable Xcode -> Windows cross-compiler. Only Foundation is in place yet but apparently active work on CoreGraphics and AppKit is underway, along with some on CoreData. Seems it's got some way to go but looks good.

Monday, July 14, 2008

php-fpm: a smoother PHP/FastCGI process manager

I’ve been running PHP apps with the standard PHP FastCGI server behind nginx for a couple of years now, and in that time have worked up a set of tools to manage PHP processes with multiple configuration profiles. This has been based around my slightly hacked version of Alexey Kovyrin’s PHP–FCGI spawn script, along with a chkconfig–compatible init script I’ve written as a front–end to control it, taking in some simple per–profile configuration in /etc/sysconfig/php-fcgi.

Various contributors to the English nginx mailing list have posted in that time singing the praises of Andrei Nigmatulin’s php-fpm, a patch to the PHP source which adds “FastCGI Process Management” to the standard php-fcgi binary. It’s apparently in use in some pretty heavily–loaded sites, and I’ve had it in mind to check it out, but as my setup has been stable (if not exactly full–featured) over the last few projects, I haven’t had a huge impetus to get and do it. Today, however, I finally had enough downtime to check it out, and now I’m wishing I’d done it earlier.

My existing method basically worked like this:

· init script reads configuration file, with one “profile” per line detailing location of php.ini file, interface and port to run on, number of child processes to run, number of requests to serve before a re–spawn, etc.;

· init script uses these details to construct a command–line to call Alexey’s spawn script, once for each profile;

· spawn script constructs environment and command–line argument list to spawn a set of PHP processes for each profile entry.

This has worked fine, but as so often with these things, the init script became a bit unwieldy with additions over time, and is still unable to do anything elegant like graceful restarts on the PHP daemons.

php-fpm addresses these issues and more. The “profile” configuration is put into a sensible, clear configuration file (php-fpm.conf) which allows you to specify a number of named PHP process “pools”, each with its own detailed FastCGI server and PHP configuration.

The documentation’s somewhat light, and mostly in Russian, but it has all you need to get going, and the configuration file is easy to read. Once you’ve configured the pools you want (in my case sets of named dev/stage/live setups on different ports, so as to keep include_paths — and therefore library code — properly staged), you just need to run php-cgi --fpm. From there on, you can send various signals to the master process including SIGQUIT for a graceful stop, SIGUSR1 to cycle log files, and SIGUSR2 for a graceful reload/restart. The master process ID is stored in $PHP_PREFIX/logs/php-fpm.pid.

I’ll probably write a simple chkconfig/init wrapper to send these signals to the master using e.g. /etc/init.d/php-fpm graceful, but that’s about all I’ll need to do in order to replicate and extend my existing setup.

Not only does this simplify and tidy up my PHP–FCGI setup enormously, it also adds a number of convenient extra points, including IP–restriction and a nice fix for the “empty error” page problem. Intuitively it “feels” a lot more solid, and I’m looking forward to trying it out on the next suitable project. Nice bit of Russian coder humour there on the “extra points” page, too. Thanks, Andrei!

Sunday, July 13, 2008

Setting up “yum” on RedHat Enterprise Linux

I just found myself in the situation of needing to install a load of software on a RHEL 4 box which had not had up2date set up. “Simple,” I thought, “it’s RPM-based, so just install yum and all will be well. yum’s nicer than up2date anyway, so”.

A cursory Google threw up this guide to doing just that, but I found that the list of RPMs provided was incomplete, possibly due to the age of the article. With the duplicates removed, package versions matched, and downloads sourced from the up-to-date CentOS and pbone, the set now installed without dependency problems, but left a non-functioning yum installation:

Setting up Install Process
Setting up repositories
not using ftp, http[s], or file for repos, skipping - Null is not a valid release or hasnt been released yet
Cannot find a valid baseurl for repo: update
Error: Cannot find a valid baseurl for repo: update

I hadn’t delved into yum’s config or repository setup much before, as on most non-RHEL rpm-based distributions it tends to work out of the box; I’d added other repositories, notably Dag Wieers’, but not looked at the format much. Imagine my delight on realising that now was my chance.

A first glance at the repository definition in /etc/yum.repos.d/CentOS-Base.repo suggested trying to use the baseurl entry which is commented out by default, rather than the mirrorlist. No joy, but it gave a pretty obvious clue:

http://mirror.centos.org/centos/Null/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Cannot open/read repomd.xml file for repository: update
failure: repodata/repomd.xml: from update: [Errno 256] No more mirrors to try.
Error: failure: repodata/repomd.xml: from update: [Errno 256] No more mirrors to try.

Got it? That nasty Null is caused because the repository definition file uses a variable $releasever which, as man 5 yum.conf tells us, is taken from the currently installed version of the package named as distroverpkg in /etc/yum.conf - which by default is centos-release (presumably taken from the RHEL redhat-release convention). Thus the only step necessary to get it working was to install the centos-release package. The baseurl entry in the repository definition could be commented out again.

So here, for your edification, and as an »aide-memoire« for me, is my list of the packages required to get yum working correctly on RHEL4:

centos-release
centos-yumconf
python-elementtree
python-sqlite
python-urlgrabber
sqlite
yum
yum-metadata-parser

Joy.

Wednesday, November 28, 2007

Netcat - a couple of useful examples

Easy piping in and out of sockets

Wednesday, April 25, 2007

How to protect base yum when installing rpmforge

Friday, November 3, 2006

exim4 / mysql HOWTO

How to set up exim4 with MySQL (virtual users), AMaViS, spamassasin, SMTP-Auth, Mailman, ...

Tuesday, October 31, 2006

White Box Enterprise Linux - Home

Sunday, October 29, 2006

SliceHost

Tuesday, September 26, 2006

EC2 server with Fedora Core 5