Skip to content

High performance computing for Digital Humanities

This workshop is an introduction to using High-Performance Computing (HPC) systems, using an AWS-hosted HPC instance as an example.

By the end of the workshop you should be able to:

  • Find and load available software
  • Use Python virtual environments
  • Understand what a job scheduler does and why this is important for HPC systems
  • Use job scheduler commands to submit jobs and find out information about them
  • Submit different types of jobs
  • Use Singularity containers

What is an HPC system?

An HPC system is a term that usually describes a clustered network of computers. The computers in a cluster typically share a common purpose, and are used to accomplish tasks that might otherwise be too big for any one computer. Those computers might share the same, or similar hardware specifications, are usually connected via high speed networks and are backed by a fast, network filesystems.

At a very high level, those computers can be divided into the following categories:

  • Login nodes
  • Compute nodes
  • Storage nodes
  • Management nodes

Important

Login nodes should only be used for submitting jobs and running simple tasks, such as editing of your job scripts, etc

Compute node diagram

Example HPC systems

National HPC, funded by research councils:

Regional HPC, run by groups of research institutions:

Local HPC, run by individual research institutions:

  • University of Manchester Computational Shared Facility (CSF);
  • King's Computational Research, Engineering and Technology Environment (CREATE).

See https://www.hpc-uk.ac.uk/facilities/ for a list of UK HPC facilities that accept external users.

Prerequisites

In order to join this training workshop, you must have a terminal application installed on your computer. If you use MacOS or Linux, you will have one available by default. If you use Windows, you can use PowerShell - some commands will differ from those used on Mac/Linux.

If you are working through these materials on your own, outside a workshop, you will need access to a HPC system.

References

The material in this course was inspired by / based on the following resources