Xgrid is Apple's rather ahead-of-its-time grid computing platform. Platform is perhaps a bit generous in this case, though, as it's a somewhat incomplete, confusingly documented and poorly disseminated system. Neat nevertheless.
I'll save giving an overview of Xgrid here as Apple give an overview on their developer pages. Also extremely good background in general, though specifically aimed for the VisualHub video program, is Don McAllister's free Podcast on the topic here.
What I shalll try and do here, however, is to fill in some gaps and mention some of the traps for the unwary. I'll also provide some links to the better resources out there in webland.
To give a flavour of what Xgrid could be used for, I'll mention some useful tasks and divide them into categories.
1. Embarrassingly Parallel CPU Tasks
This means that the task can be split into a number of independent tasks that are completely independent of the others. Furthermore, they only require number crunching of the computers rather than using other resources like files systems.
Examples in this category are things like Montecarlo simulations, pi calculations (only when the algorithms are appropriate), the Mandelbrot demonstration (Xgrid Mandelbrot.app from the Developer Tools' Examples folder), password cracking etc.
2. Embarrassingly parallel but Resource Dependent Tasks
The tasks can all happen simultaneously as above. However, in this case there is significant resource requirements of the grid computers, most often the filesystem or network.
The classic example here is video conversion or ripping. In these instances a large volume of network traffic will need to take place in order to pass files to each computer. This adds some complexity and careful consideration to the design of the Xgrid architecture.
A useful commercial software package that does video conversion in batches using Xgrid is VisualHub.
3. Partially Dependent Tasks
The tasks can't operate entirely independently and need to communicate back to each other somehow. It is assumed that nonetheless an algorithm is known that reduces the total execution time even though it won't be as significant as an embarrassingly parallel algorithm.
The classic group of this type of distributed algorithm are those that use Pascal's triangle-like problems where each vector in a row is dependent upon two cells of the previous row.
See this tutorial on MPI and Pascal's
Triangle at Daugerresearch
These pages will concentrate on the first two types of
tasks as they are the ones for which Xgrid is most
suited.
Example 1
Let's take a most obvious choice of EP (embarrassingly parallel) tasks in IT security - brute-force code breaking and password cracking.
In these scenarios a trial and error approach is needed and the cracker needs to run over all possible choices in the search space to see if it is the correct one. As soon as a correct choice is found, the search is over.
It is obvious that, for example, a dictionary attack on an encrypted password is EP. You can hand off all words starting with 'a' to computer one, those with 'b' to computer two and so on. There is no dependency on the work being performed by any computer with any other, though it may be acceptable to terminate the processes on the remainder when one of the grid computers discovers the solution. However, that's not the real point about their independence.
Now we don't have to have 26 computers available in order to break down the workload for this. We can hand out all a,b,c to the first computer, d,e,f,g to the second and break them down however we like. We could be smart and figure out the likelihood of w,x,y,z as being small and so bundle up more according to their statistical frequency in the English language. The point is simply that we can partition the "space" however we like.
Note: Xgrid is only granular to the task level - it can't divide up the work it is sent into smaller parcels itself, so if you have a large domain to search, it is up to the user to partition this effectively. A good rule of thumb is to make your packages of Xgrid tasks small enough so that a lower spec'd machine is not going to be a bottleneck in the process. With eight tasks and four computers it is possible that the lowest performing computer only just finishes its first task and starts on the second when the other three computers have finished two each. Thus the processing time is almost double what it could be if the domain was broken into smaller parcels. Obviously the circumstances determine whether this can be done or not and converting four videos is hard to split into smaller tasks than four.
To start off our example Xgrid application, let's take the even simper task of searching for a matching binary key to an encrypted hash.
To perform this task we want to implement the following sort of process for Xgrid:
[section under development - coding examples to follow]
The first thing you will need is the Server Admin Tools 10.4.7 from Apple Support. These are management applications of various services that are standard on Mac OS X Server but not on the desktop version (the one most users will have). The item of interest in the package is the Xgrid Admin application.
There is an easy way and
a hard way to get the various services configured and
started. We will focus on the easy way using a free
third party download called Xgridlite. You can download
it from this site or directly here. Although free, the author
accepts PayPal donations. Since the software is so useful,
you may wish to use this service.
Xgridlite is a preference pane plugin that provides the
most useful Xgrid functions of setting up the
controller and passwords.
A useful feature is the
Reset button, which can sets things straight when they
go awry - as oft they will with Xgrid.
Because activities are happening on remote computers
and it is very difficult to determine what is going on
and debugging, another extremely useful utility is the
Xgrid Sample application that comes with the free
Developer Tools for OS X. The current application can
be downloaded here from some Xgrid resources
at geekmeetsgirl.net, though if you have the tools it
is best to compile this yourself.
Getting Started
To get started you will need to enable at least one (and preferably two) computer(s) as Xgrid agents. THis is done through the Sharing panel of Preference Pane. You will tick the Xgrid box and choose Configure...
It is a good idea to
choose a password for the agents. This is for security
because any person on the network will be able to
execute commands on any of the Xgrid agent computers
without password protection.
Note that as of this time VisualHub's Xgrid
implementation requires that you disable passwords. Do
so at your own risk.
Let's assume you have added in password ABC to each
computer's Xgrid configuration as above.
Now you need to set the controller so that it knows
these passwords. using Xgridlite in the preference
pane, click "Provide password to agents" and set the
password to be the same - ABC.
You should also set a password for the controller. I
set mine to be the same as the agents for simplicity.
Distributed Tiger: Xgrid Comes of Age by Drew McCormack
Xgrid at Wikipedia - the classic source for information on all things technical
Xgrid at Apple - sales pitch on the Mac OS X Server pages
Technology brief from Apple - PDF with more detail