Skip to main content
Print

Containers in HPC: Enhancing Performance and Portability

Introduction to Containers

Containers are a software package that bundles together all the dependencies, libraries, and other runtime components that are required when running an application. Containers are designed to be portable and self-contained, which means that they can be deployed on any system. A container’s information is stored in an image file, which is a prepackaged environment that contains all the necessary components to run an application. One of the major draws to containers is that images are shareable and can be distributed across different systems, organizations, and a wide range of other environments. When the container is launched, it runs on top of the existing system architecture, not requiring a separate kernel like a virtual machine (VM) would. This means that it shares the resources of the system versus having to use a hypervisor and provision resources that will be separated for the VM from the host system. Containers can be managed in many ways including through the docker/singularity command line and through orchestration tools such as Kubernetes and Portainer. There are command line and graphical user-interface (GUI) options.

Why Use Containers in HPC?

There a re many Benefits of Containers in HPC

  • Small Size

Due to the container only containing the necessary programs and services required to run, containers have a relatively small size when it comes to storage, often under 500 MB. This is a dramatic reduction in storage comparable to the storage and system requirements of a VM.

  • High Performance

Containers can help improve the workflow and runtime environment for applications as they help to optimize the runtime environment for applications. Additionally, they help to reduce overhead by running only the necessary components for the application.

  • Reproducibility

Because container images are prepackaged software environments, this means that they contain a predetermined list of software. This allows a container to perform and deliver the same environment regardless of the host system and when the container is run.

  • Shareability

Because container images are prepackaged, it is easy to share the image with other users through hubs and repositories such as Dockerhub and Github. This allows for uniformity in work environments and simplifies working on projects as a group. Users can easily pull container images from public repositories as well as set up and pull from private repositories for more security in your container images.

  • Easy to Learn and to Operate

Through services such as singularity and docker, users can easily interact with and run container images through the command line using simple commands. This simplifies the process of running certain applications by removing the aspect of requiring the user to set up their own environment. Users can also manage containers with a GUI through services such as Portainer.

Key Technologies

Several technologies support containerization in HPC environments:

  • Docker: A popular platform for building, deploying, and managing containers. Suitable for a wide range of applications.
  • Singularity/Apptainer: Designed specifically for HPC and scientific computing, optimized for performance with features like GPU acceleration.
  • Kubernetes: An open-source container orchestration service that automates deployment, scaling, and management of containers.
  • Portainer: An open-source user interface for managing Docker containers, offering a GUI for ease of use.
Use Cases in HPC

Containers are versatile and can be applied in various HPC scenarios:

High-Performance Computing (HPC):

Containers can be used in HPC environments to provide a consistent environment for data analysis workloads and complex simulations. Containers in the HPC environment can also reduce overhead and improve resource utilization when it comes to workflows in the HPC.

Application Deployment:
Containers can be used to deploy applications and services to multiple different environments from on premise, to the cloud, and to edge devices. Containers allow administrators to deploy and manage many complicated applications and services by providing a consistent runtime environment.

Microservices:

Containers are able to be deployed to manage micro service architecture. This means that complex applications are able to be broken down into smaller, more independent deployable services. This provides a lightweight and efficient runtime environment for services, allowing them to better scale while maintaining flexibility and reliability.

DevOps:

Containers can be used to help provide a consistent and standardized environment for building, testing, and deploying applications that are developed by software developers. By allowing software developers to quickly share and iterate through application development, developers are able to work more efficiently as a team.

Data Science

Containers can be used to package and distribute data science workflows. Containers can also be used to package and distribute machine learning models. This helps to increase portability and reproducibility by encapsulating different libraries and other dependencies which allows the workflow process to be simplified.

Containers that are preconfigured in Dockerhub or other sources for HPC

  • OpenFOAM
  • TensorFlow
  • GROMACS
  • NAMD
  • R
  • Apache Hadoop
  • Apache Spark
  • MATLAB
  • PyTorch
  • PostgreSQL and other SQL databases
  • Anaconda
  • BWA
  • BioContainers

References material and additional information:
[1] Docker, Use containers to Build, Share and Run Your Applications,

https://www.docker.com/resources/what-container/, 2024.  Accessed 5/12/24

[2] Singularity, Introduction, https://docs.sylabs.io/guides/2.6/user-guide/introduction.html, 2024. Accessed 5/12/24

[3] J. Koch and W. Hao, “Practical Applications of Edge Computing to Accelerate Cloud Hosted Web Content,” 2022 IEEE World AI IoT Congress (AIIoT), 2022, pp. 256-263, doi: 10.1109/AIIoT54504.2022.9817313.

[4] Kubernetes, Kubernetes, https://kubernetes.io/, 2024. Accessed 5/16/24

[5] Portainer, Portainer, https://www.portainer.io/, 2024. Accessed 5/16/24

[6] S. Singh and N. Singh, “Containers & Docker: Emerging roles & future of Cloud technology,” 2016 2nd International Conference on Applied and Theoretical Computing and Communication Technology (iCATccT), 2016, pp. 804-807

[7] Docker, Docker Hub, https://hub.docker.com/ , Accessed 5/22/24

Table of Contents