What is /scratch?

There's been some recent interest in using /scratch: what is it, what's it for, how do I use it? This document addresses these questions.

What's a filesystem?

To understand what /scratch is, and what it's for, you need to understand a little about UNIX filesystems. Each hard-drive in a UNIX system is generally divided into partitions, and each partition is usually "mounted" as filesystems to a particular directory. If you do 'df -k' on the command line, you'll see a list like this, showing the filesystems (i.e., disk partitions) and where they are mounted to:

> df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2               256665     94623    148789  39% /
/dev/sda5              1035660     62172    920880   7% /tmp
/dev/sda6              5162796   2128068   2772472  44% /var
/dev/sda7             10325748   1770040   8031188  19% /usr
/dev/sda8             16421236     32864  15554200   1% /scratch
dogma:/d0            336467496 275803320  57245864  83% /d0

So, for example, '/dev/sda5' is a partition consisting of 1035660 1k-blocks (i.e., about 1 gigabyte of space). It's currently 7% full, and it can be accessed by going to /tmp.

The first five lines above refer to "local" filesystems. These are partitions on one of the hard drives that sit physically in the llama's box. They're equivalent to the hard drives you might have on your home computer.

What's a remote filesystem?

The last line refers to a remote filesystem called 'dogma:/d0'. This syntax, with the colon ':' means that it refers to a partition that lives on a machine called 'dogma', in a partition called 'd0'. What's dogma? That's the name of the Roth Lab fileserver. It's a specialized machine whose job is simply to run a huge set of hard drives, reading from and writing to them as necessary. You can see that it's a huge filesystem: about 330GB. Since it's 83% full, there's a good chance it will have been expanded by the time you read this.

What's dogma, and why do we need it?

The files for every user on the system are stored here. (You may think your files live somewhere else, such as /home. But if you do 'ls /home', you'll see that it's just a link to '/d0/home'.) That means every time you read a file, or do 'ls' on a directory, every time you write something to a file, or compress one, you're generating a request to dogma. This request travels over a network from the llama, or one of the dolly nodes, to be processed on the dogma, and the results, if any, are returned to the requesting machine.

This is a great system in many regards: all files are stored centrally, facilitating backups and administration in general. However, its drawback is the network traffic it creates, and the limitations on simultaneous file access: if too many people are trying to access too many files at once, some of them will have to wait until the others are done. In operational terms, this means that interactive users on the llama will often feel that the llama's response has degraded, and yet when they run 'top', there appears to be no great CPU load on the llama.

How do I do file access without going through dogma?

How do we get around this situation? That's where /scratch comes in! Each and every node (llama and all the dollies) has a filesystem called /scratch. It has a number of important properties:

  1. It's big. On the llama, it's about 16GB, making it about 15 times the size of /tmp. On the dollies, it's significantly larger than /tmp.
  2. It's always local.

That the first one is a good thing needs no clarification. But what does it mean that a filesystem is local? It means that file-access operations are handled by the same machine where they're issued. If your job is running on dolly09, and you attempt to write to a file in /scratch, then dolly09 will do it. No request will be sent to dogma, dolly09 will not have to wait for the result to come back from dogma, it's handled by dolly09. This is good news for you (because your program runs faster, since there's no network traffic, and no waiting around for a response), and for your fellow llama-users (because you are not loading dogma with file-access requests).

Being local also means that the filesystem is inaccessible from other machines. So, /scratch on dolly09 is completely independent of /scratch on dolly08 or dolly10. That's something to keep in mind when using the queuing system.

You've convinced me. How do I use /scratch?

If you're running jobs that do a lot of file access (reading, writing, etc), consider modifying your programs so that their first step is to copy the files to /scratch, before processing them. Once all the processing is done, they can be copied back to your home directory. If you're really done with them, you should consider deleting them from /scratch as well, so that others can use the space. If you're not done, you can leave them there to be used again.