There's been some recent interest in using /scratch: what is it, what's it for, how do I use it? This document addresses these questions.
To understand what /scratch is, and what it's for, you need to understand a little about UNIX filesystems. Each hard-drive in a UNIX system is generally divided into partitions, and each partition is usually "mounted" as filesystems to a particular directory. If you do 'df -k' on the command line, you'll see a list like this, showing the filesystems (i.e., disk partitions) and where they are mounted to:
> df -k Filesystem 1k-blocks Used Available Use% Mounted on /dev/sda2 256665 94623 148789 39% / /dev/sda5 1035660 62172 920880 7% /tmp /dev/sda6 5162796 2128068 2772472 44% /var /dev/sda7 10325748 1770040 8031188 19% /usr /dev/sda8 16421236 32864 15554200 1% /scratch dogma:/d0 336467496 275803320 57245864 83% /d0
So, for example, '/dev/sda5' is a partition consisting of 1035660 1k-blocks (i.e., about 1 gigabyte of space). It's currently 7% full, and it can be accessed by going to /tmp.
The first five lines above refer to "local" filesystems. These are partitions on one of the hard drives that sit physically in the llama's box. They're equivalent to the hard drives you might have on your home computer.
The last line refers to a remote filesystem called 'dogma:/d0'. This syntax, with the colon ':' means that it refers to a partition that lives on a machine called 'dogma', in a partition called 'd0'. What's dogma? That's the name of the Roth Lab fileserver. It's a specialized machine whose job is simply to run a huge set of hard drives, reading from and writing to them as necessary. You can see that it's a huge filesystem: about 330GB. Since it's 83% full, there's a good chance it will have been expanded by the time you read this.
The files for every user on the system are stored here. (You may think your files live somewhere else, such as /home. But if you do 'ls /home', you'll see that it's just a link to '/d0/home'.) That means every time you read a file, or do 'ls' on a directory, every time you write something to a file, or compress one, you're generating a request to dogma. This request travels over a network from the llama, or one of the dolly nodes, to be processed on the dogma, and the results, if any, are returned to the requesting machine.
This is a great system in many regards: all files are stored centrally, facilitating backups and administration in general. However, its drawback is the network traffic it creates, and the limitations on simultaneous file access: if too many people are trying to access too many files at once, some of them will have to wait until the others are done. In operational terms, this means that interactive users on the llama will often feel that the llama's response has degraded, and yet when they run 'top', there appears to be no great CPU load on the llama.
How do we get around this situation? That's where /scratch comes in! Each and every node (llama and all the dollies) has a filesystem called /scratch. It has a number of important properties:
That the first one is a good thing needs no clarification. But what does it mean that a filesystem is local? It means that file-access operations are handled by the same machine where they're issued. If your job is running on dolly09, and you attempt to write to a file in /scratch, then dolly09 will do it. No request will be sent to dogma, dolly09 will not have to wait for the result to come back from dogma, it's handled by dolly09. This is good news for you (because your program runs faster, since there's no network traffic, and no waiting around for a response), and for your fellow llama-users (because you are not loading dogma with file-access requests).
Being local also means that the filesystem is inaccessible from other machines. So, /scratch on dolly09 is completely independent of /scratch on dolly08 or dolly10. That's something to keep in mind when using the queuing system.
If you're running jobs that do a lot of file access (reading, writing, etc), consider modifying your programs so that their first step is to copy the files to /scratch, before processing them. Once all the processing is done, they can be copied back to your home directory. If you're really done with them, you should consider deleting them from /scratch as well, so that others can use the space. If you're not done, you can leave them there to be used again.