Disk Quotas, File Storage and Archiving
Your quota is the amount of space that your files are allowed to take up on the Institute homes server machines. If everyone was allowed to write as many files and as much data as they wanted, then the available disk space would soon get used up. Also if one person (accidentally) filled the entire disk everyone would be inconvenienced/could suffer file loss or corruption. It's very easy to leave large files lying around forgotten, so the Institute computers implement quotas which ensures the usage is monitored and if limits are reached that the individual is notified so they can check to see if there is a problem or if the growth is as expected.
How big is my quota?
To see what your quota is at the moment, and how much of it you have used up, use the quota
command on the Linux machines, e.g.
$ quota
/home/bloggs 3.93G ( 36% ) used of 10.80G quota
There are various tools you can use to assess you disk usage to see where your largest files are located. One such tool is the command du
(disk usage). Using the command du -Sh ~/ | sort -h
we can list the disk usage of each directory in your account and then sort the results numerically to give you a clear picture of your largest directories (as they will appear at the bottom of the output).
To list the size of all contents of a directory, you can do something similar to the above, but substitute ~/
for ~/<directory>/
.
If you prefer a graphical tool then you could try the Disk Usage Analyser under Applications -> System Tools.
Help! I'm over my quota!
Every night a quota check is run which emails anyone who is over quota.
A summary of the warning information is also seen by system administrators and in many cases they will act to raise quotas without the need for you to do anything.
If you are nearing your quota limit, and you don't get an email telling you it has been raised, you need to do something about it reasonably quickly, as once you completely run out of space you will be unable to write to your home directory.
Although there is information below about reducing your disk usage and ways to store your files in different location, please do talk to IT support staff or email @email to discuss your needs or the advice below if you have any concerns or queries, etc., as any data storage is a critical consideration and minimising the risk of data loss whilst making the most appropriate use of the options is key.
Reducing your quota usage and Archiving
To get your disk usage back below your soft limit, you can do one or more of the following:
- Delete unwanted files (e.g. using the
rm
command or the graphical file manager). If you use a graphical file manager you may also need to remember to empty your trash/wastebasket afterwards as many graphical tools often just move deleted files rather than remove them in the first instance. - Compress any files that you aren't currently using, but still want to keep (e.g. use gzip or bzip2, or the graphical file manager or other graphical archiving tools.
Be careful only to delete files that you are absolutely sure you don't want, as you may not be able to get them back (see help with restoring files if you do make a terrible mistake and delete something important).
As well as reducing the size of the files you have you can also consider storing different files in different places according to their importance and/or regular use.
- Home directory: accessible from all machines, backed up nightly to departmental and central university backup system, quota limits imposed
- /scratch partition: local to each Linux machine, typically 50-350GB of space, this may not be backed up and only stored on a single consumer grade disk at all so all data may be lost in the event of a hardware failure and should be reproducible.
- removable media: e.g. USB memory sticks (2GB to 256GB of space typically)
Setting up a personal directory on a local scratch disk
The scratch disk on each Linux machine is writeable by any user. It is an ideal location for large data files that could be recreated or for archives of old files or PDFs of articles etc.
Do not in general use the scratch space for very important files that cannot be easily restored from other sources or recreated.
As mentioned above the scratch directory on each machine is configured to backup to the central university service. There are over 350 machines in the department backing up in this manner. On any given backup we might see 0-5% of these machines fail to backup the scratch partition successfully. This may be for any number of reasons, e.g. the machine was mistakenly off, the machine was faulty, the central university backup system failed, a file was changing as the backup ran, general networking problems etc.
To ensure you can easily identify your files and control access to them the recommended appraoch is to create a directory within scratch of your own and then set the permissions on it as appropriate, e.g.
mkdir /scratch/myusername
chmod 700 /scratch/myusername
where you would replace myusername
by your actual username. The above creates the directory and sets the default permissions on it so only you have access to it (i.e. just like your home directory). You can now create/move/copy files to this location as appropriate.
Note you could equally do the above using the graphical file manager. Start the file manager at your home directory and us the Up button to climb the directory tree to the top and then double click on the scratch folder to enter it. Next right click the window and choose "Create Folder" from the menu to create a folder. As in the above example it is recommended you name the folder with your username. Having created the folder right click it and choose "Properties" to enter the folder properties window. Now click on the permissions tab and uncheck the permissions for group and other users so that only the owner has permissions.
Central scratch
Whilst the local scratch disk typically provides the most space for individuals extra files suitable for that location there are times when it is convenient to have those files central to the system so they can be easily accessed from many machines or shared with other users. For this purpose we do have a 20TB central scratch space. A large per person quota is imposed on central scratch to ensure no one user can fill it up or a run away process can exhaust all the space and cause problems. Note central scratch is not backed up!
University Research File Service (RFS)
If your storage need is part of a research project/group then you may wish to consider obtaining space on the university RFS service (up to 20TB free for a project). One can then mount this RFS storage from any machine as required.
I think I need a bigger quota
If you have deleted all unnecessary files, compressed any large files that you aren't currently using, have made sensible use of scratch partitions (e.g. for large data files that could be recreated or for archives of old files or PDFs etc), made use of OneDrive or RFS where appropriate, but still need more space, then just email IT support to ask for more quota. Please be aware though the maximum quota provided is typically ~150GB.
In summary
Location | Type | Typical quota/size | Backups |
---|---|---|---|
Home directory (/home/$USER ) | Network | 60GB - 150GB | Nightly |
Local scratch (/scratch/$USER )[only for reproducible data] | Local (per machine) | 50GB - 350GB+ | Sometimes |
Central scratch (/mi/share/scratch/$USER )[only for reproducible data] | Network | ~200GB | Never |
Research File Service (RFS) | Network | 100GB - 20TB+ | Daily |
External USB drive | Local (portable) | 2GB - 256GB+ | N/A |