Intoduction to Mindhive

First, What is This Thing Called?

mindhive is the name of a shared computing and data storage facility at the McGovern Institute for Brain Research at MIT. It is often referred to as the "mindhive cluster." Sometimes people refer to it as "the BA's" - each individual server in the system is named after a Broadmann Area; for example, ba1.mit.edu is one of the individual servers that makes up the mindhive cluster.

If you'd like to know what is under the hood of the cluster, read this. If you'd like to learn how to log in and start using the cluster, this page is just for you.

Web Site != Cluster

This web site is not part of the cluster, per se. Some people have gotten confused in the past, thinking that it was necessary to log into the web site itself, somehow, in order to do computation. This is not the case - this web site, even though it is named mindhive.mit.edu, is a read-only web site designed to document the actual compute cluster.

How Do I Get Started?

Unfortunately, mindhive is not available to everyone at the McGovern Institute. In fact, it is  just for use by a handful of labs who originally contributed financially to the hardware and salary for a system admin: Gabrieli and Kanwisher being the two primary financial contributors, as well as the Saxe and DiCarlo labs. Unless you're in one of those labs, sorry - regrettably, your lab will need to re-invent the wheel when it comes to IT resources.

Before you apply for an account on mindhive, you should make sure you've gotten an MIT Athena account first. Although an MIT account is not strictly necessary - mindhive has its own authentication system that is independent from MIT - we use your User ID number from Athena in your account here. This prevents certain sorts of issues that can happen later down the line. You can learn more about getting an Athena account at http://ist.mit.edu/support/accounts/information .

Once that's out of the way, you need to find out which research groups' data you will need to be able to access. Ask your supervisor or lab admin, or another RA/TA/PostDoc/Student. You might already have an account if you've arrived at this page from our introductory email message!

Finally, send email to the following email address:

mibr-admin@techsquare.com

Be sure to include your MIT Athena username (not your MIT ID number, which you should not reveal to anyone...), your supervisor's name, and the research groups that you'll need data access for.

Once that all happens, your account will be created, and you'll be ready to read the next section in this ostentatious manual...






Being a Good Citizen

Putting the "Shared" in "Shared Computing"

The term "cluster", as it is generally used in computing circles, means a whole bunch of computers that are all tied together with a fast backbone and that all act together to solve huge computational problems. Typical uses for a traditional cluster in this sense include things like nuclear fusion simulations, massive search algorithms, encryption-cracking routines, and many other very nerdworthy goals. However, neuroscience hasn't quite caught up yet with some of the rest of the computing world, and many programs we use are inherently interactive in nature - they require a human to run some commands, check the output, make adjustments, rerun the commands, view images, think about it all, and so forth. So a traditional cluster isn't really terribly useful to most of our users. A newer breed of users has come along, though - users who do in fact both need the higher power afforded by traditional computer clusters and who also have the savvy to write their jobs in the Procrustean way that clusters enforce on users. Putting both sorts of users into the same computation fishbowl is not generally a good idea! But funding is always scarce, so in an attempt to create a dual-purpose system, what we have is an unusual combination of interactive-login servers and a thing called a batch system (known as Sun Grid Engine, or SGE for short - though this will soon be called something else, since Oracle acquired Sun and will be rebranding everything as they go). The mindhive cluster is shared amongst many research groups at MIT. Some system administrators have extremely tight control over what sorts of jobs can be run on a system, what kinds of software can be used, and so on. Often, systems that have Draconian admins go unused since no one can get any work done! Mindhive is more of a communal system that requires that people who share it are mindful of one another. Although we've put as many checks and balances in place to keep one person from being able to dominate the entire cluster, it's still possible. You can read the hairy details at "On Compute Power" . So this section is dedicated to cluster etiquette, and how to be a good citizen. Almost everyone who uses the cluster is considerate. Those who are not, however, after repeated polite requests to shape up, will be ejected from the cluster.'Nuff said.

Cluster Do's:

  1. Do check the load of the servers to see which is the least-used when you want to log in. Use the cluster load widget on the home page of this web site. Lower numbers mean less load.
  2. Do use the Sun Grid Engine (SGE) system to submit multiple jobs - SGE is designed to keep the servers from melting down under extreme load and also to prevent memory exhaustion.
  3. Do proactively monitor the amount of memory your jobs are using by running the htop program - if a server begins to run out of physical memory, it will start to use the hard drive as supplementary working memory, and this is very, very bad.
  4. Do let the rest of the cluster users know if your group has a major deadline coming up if you'll need extra cycles on the cluster. You can send mail to the mindhive mailing list to do so.

Cluster Don'ts:

  1. Don't run jobs on all the servers simultaneously UNLESS you are using SGE to submit the jobs. Running homegrown batch jobs or using iPython or brute force to commandeer the entire cluster is not OK.
  2. Don't consume vast amounts of disk space. Each lab has its own allocation of disk space, which is finite. If you fill up your own lab's share, your labmates might hunt you down. Some of the older disk shares like /users, /u2, and /g2 are shared by everyone - so if you overindulge, you can fill up the filesystem and cause other other lab's analyses to crash. Hell hath no fury like a PI whose lab is stalled during a deadline crunch due to a full filesystem...

Logging In - Command Line

Choose an Access Method

The mindhive servers are in a locked closet in a cooled room, so people don't just sit in front of them and type away; they log into them remotely. There are several ways to do this.

Using SSH

One popular way to log into a remote Linux server is via SSH, which is short for Secure SHell. SSH looks like an old-fashioned text console, maybe a bit like the one David Lightman used in Wargames. You run an SSH Client on your computer - whether your computer runs Mac OS, Microsoft Windows, some flavor of Linux, GNU/Hurd, iOS, or SkyNet. This is usually a fairly nondescript piece of software. Most operating systems except Microsoft Windows have this program built-in. If you use Windows, you can download the MIT standard software suite, SecureCRT and SecureFX from the IST Software Distribution site at http://ist.mit.edu/services/software/securecrtfx/6x . SSH is purely text-based, so it is used for running commands that don't require fancy graphics, e.g., visualization software like Freesurfer. It is, however, pretty good for doing things like moving files around, running programs that don't need a GUI (like bedpostx or recon-all), or checking out the load on a server. If you are running MacOS or Linux, running SSH is pretty easy. Just open up a command terminal, and type the following: ssh username@baN.mit.edu ... where "username" is, of course, your own username, and the N is the number of one of the servers. Currently, these are the available servers for login:

Great, But I Use Windows.

Doh! Well, running ssh is a little more painful on Windows, but can be done. As previously noted, you'll need to download and install the SecureCRT/SecureFX from IST. Once that is done, you can launch SecureCRT, and enter the hostname of the server to which you'd like to connect, your username, and your password. If enough people have trouble using SecureCRT, I'll add a HOWTO here...

And Now What?

Once you are logged in, a bunch of default settings will get initialized for you, depending on which lab you said you were working for primarily when you requested your account. This is all controlled through a facility called dot-mindhive. You can read more about this in the "Customizing your login files" page of this manual. You can run any of the thousands of Linux commands available to you from your new shell, but this is outside the scope of this manual. Chances are, though, at some point you are going to want to run graphical programs like Matlab, SPM, Freesurfer, AFNI, and so forth. The next section explains how to get a graphical desktop session running.

Logging in - Graphical Desktop Interface

Give Me GUI!

Despite the gnarly sounding title of this page, it's probably what you will need to learn to do, first and foremost. Getting a remote desktop of your own on the compute servers will let you do all sorts of things, like run Matlab, FreeSurfer, Python, and so forth. Some people are comfortable just logging into a server via a command-line interface like SSH, but many prefer to see a more user-friendly interface. For this, I strongly suggest you use the software package "NX" - it requires the fewest steps to connect and is generally better than the other options. If someone tells you that you should use something called VNC instead, tell him or her that they will have to arm wrestle you first.

Remote Access using NX Client

NX is a client/server system that will allow you to connect via a graphical interface to a server. To get started, you need to download the appropriate client for your operating system...

Once you've installed NX Client, you need to set up a connection profile. This just means entering a name for the connection (can be anything, but I suggest using the hostname of the remote computer, e.g., ba1.mit.edu as well as the remote hostname (probably the same as the connection name, e.g., ba1.mit.edu.

Note that there is a slider that lets you choose the sort of network connection you have. I tend to find that for desktops that are on MITNet, the "LAN" option gives best results, while wireless connections of any sort or wired connections from off-campus tend to do better with the "WAN" setting. If you are on a really slow connection, you might need to ratchet this setting down to find an optimum balance between snazziness and function.

IMPORTANT... Read the next bit carefully, since most of the questions I get about failed NX connections stem from having KDE instead of GNOME set up in the connection profile.... Next, you need to set up your Desktop session to use Gnome, instead of the default KDE (KDE is not currently installed on the cluster). Just select Gnome from the drop-down box as seen in the following figure.

Now you will get a login prompt; use your regular hive username and password.

If you're running Microsoft Windows, you'll probably need to answer a bunch of annoying messages about unblocking nxssh and other apps.

The very first time you connect to this host, you'll probably also be prompted to trust the remote site's key.

After all that, you'll be set. Once you connect, you'll get a standard Gnome desktop. Note that NX works a little differently from VNC, in that you don't tend to run tons of tunnels at once - in fact, if you try to connect to a host where you already have a desktop open, your previously-open client session will close and you'll be connected to your existing desktop. This is meant to be a feature, not a bug.