Using Movable Type’s Publishing Queue
Movable Type features a new capability that enables publishing your blog in the background. Depending on your specific circumstances, requirements and system configuration, the Publish Queue may help you reduce the time your users (both authors/administrators and site commenters) wait while Movable Type publishes files to your file system, and improve overall system performance.
System Requirements
- There are no special system requirements for running Movable Type’s Publishing Queue.
- To execute
run-periodic-taskson a predefined schedule, users will need access to a scheduling framework like cron in Unix or Window’s scheduled task framework.
Using Publish Queue
Using the Publish Queue requires you to:
- Configure your blog to use the queue
- Run the periodic tasks script as a cron, or as a daemon
Configure your blog to use the Publish Queue
Each blog in your system can be configured to use the Publish Queue. To configure your blog to use the Publish Queue:
Movable Type 4.2 and Movable Type 5
Edit the template you want to be published in the background via Publish Queue.
Select “Via Publish Queue” under the Publishing pull down menu under “Template Options.”
Once you have configured your blog to use the Publish Queue, the system will no longer automatically publish pages when a user creates a new entry, leaves a comment or does any other action that requires Movable Type to publish files. Instead, jobs for those tasks will be placed in the job queue.
Running the run-periodic-tasks script
Jobs are processed from the queue with the script run-periodic-tasks. This is shipped with Movable Type in the tools directory.
» See Setting up run-periodic-tasks.pl
Running multiple instances of the Publish Queue
Movable Type allows you to setup multiple instances of the run-periodic-tasks daemon. This allows administrators to more easily scale the capacity and speed of the Publish Queue. There are a few things to note when running multiple instances of the Publish Queue:
- Users are free to run multiple instances of
run-periodic-taskson a single machine, as well as running multiple instances across multiple servers. - When running multiple instances concurrently, it is recommended that at least one daemon be instructed to process jobs in a more random manner by using the
--randomlyflag onrun-periodic-tasks(supported in Movable Type 4.2 and later).
How the Publish Queue works
The Publish Queue consists of two key components:
- The publishing job queue
- The publishing worker(s)
The publishing job queue
If you have enabled and configured Publish Queue, when a user saves an entry, leaves a comment, or performs any action that would publish new content to your blog, the system puts that publish action into a queue. Jobs in this queue are then processed by one or more publishing worker(s).
The publishing job queue is managed in your Movable Type database in tables prepended with mt_ts.
The publishing worker(s)
The publishing worker is launched via the script run-periodic-tasks and is responsible for processing jobs in the queue. Parameters can be passed to run-periodic-tasks when it is invoked to modify and fine tune its behavior according to your specific needs.
Behind the scenes: The Schwartz
The publish queue and its two primary components — the job queue and the worker — is actually an open source set of tools built and maintained by Six Apart called The Schwartz.
The Schwartz is taglined “a reliable job queue system” (and named after an engineering manager at Six Apart) and was originally developed as a generic job processing system for Six Apart’s hosted services. It is used in production today on TypePad, LiveJournal and Vox for managing tasks that can be performed by the system without user interaction.
The Schwartz follows a very simple request cycle:
A user (or the application itself) makes a request which the application decides to put into the Schwartz job queue. (In the case of Movable Type’s publish queue, this would be a file publish request.)
The application registers a Schwartz job in the Schwartz database. In Movable Type, Schwartz jobs are stored in the Movable Type database with the table prefix
mt_ts.A worker, typically running as a daemon, grabs one or more Schwartz jobs off of the queue in a relatively random manner. In Movable Type, a worker is managed through the script
run-periodic-tasks. Therefore multiple workers can be deployed by spawning multiple instances of therun-periodic-tasksscript. Starting in Movable Type 4.2, jobs are pulled off the queue according to their assigned priority. Users wishing to process jobs random can use the--randomlyflag on `run-periodic-tasks.pl script.**The worker completes its assigned work, and when it is complete removed the job from the queue completely. If the worker fails in completing the task then the job is requeued to be processed again at some point in the future (typically 5 minutes later).
Rob Staveley on October 19, 2009, 10:47 a.m. Reply
We found that FastCGI was timing out, because publishing a complicated set of index templates in one of our blogs was taking more than 30 seconds.
A knee-jerk response would have been to increase the idle-timeout to allow more than 30 seconds publishing, because our set up was as follows:
However, the right thing to do was to move towards publishing via the Publish Queue (TheSchwartz). It is unreasonable to expect a blog editor to wait for more than 30 seconds for a response to a click. Increasing idle-timeout would simply have moved the problem elsewhere (e.g. to a HTTP/408 on a proxy server or to our editor’s patience). Using TheSchwartz to do the publishing in the background has made the system feel responsive.
Veteran Windows programmers are comfortable with the idea of UI threads and worker threads. The web is slowly catching up with the desktop experience. Background publishing and AJAX are both steps in the right direction.
dlc on April 5, 2010, 5:50 p.m. Reply
I’m running this under SMF on solaris 10 using the following manifest:
<?xml version='1.0'?> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <service_bundle type='manifest' name='export'> <service name='application/the-schwartz' type='service' version='0'> <instance name='movable-type' enabled='true'> <exec_method name='start' type='method' exec='/usr/local/bin/perl -I/mt/lib -I/mt/extlib /mt/tools/run-periodic-tasks --daemon --sleep 60 &' timeout_seconds='0'> <method_context working_directory='/mt'> <method_credential user='daemon' group='daemon'/> </method_context> </exec_method> <exec_method name='stop' type='method' exec=':kill -15' timeout_seconds='30'> <method_context/> </exec_method> </instance> </service> </service_bundle>Save this to an xml file, and run:
And it’s running: