Overview
ParaTools Pro for E4S™1 - the Extreme-scale Scientific Software Stack E4S2 hardened for commercial clouds and supported by ParaTools, Inc. provides a platform for developing and deploying HPC and AI/ML applications. It features a performant remote desktop environment (based on DCV) on the login node and compute nodes interconnected by a low-latency, high bandwidth network adapter based on AWS Elastic Fabric Adapter (EFA). ParaTools Pro for E4S™ features a suite of over 100 HPC tools built using the Spack3 package manager and the proprietary MVAPICH MPI tuned for EFA. It features ready to use HPC applications (such as OpenFOAM, LAMMPS, Xyce, CP2K, deal.II, GROMACS, Quantum Espresso) as well as AI/ML tools based on Python (such as NVIDIA NeMo™, TensorFlow, PyTorch, JAX, Horovod, Keras, OpenCV, matplotlib and supports Jupyter notebooks) and the Codium IDE. New packages can be easily installed using Spack and pip and are accessible on the cluster compute and login nodes. It may be used for developing the next generation of generative AI applications using a suite of Python tools and interfaces.
E4S has built a unified computing environment for deployment of open-source projects. E4S was originally developed to provide a common software environment for the exascale leadership computing systems currently being deployed at DOE National Laboratories across the U.S. Support for ParaTools Pro for E4S™ is available through ParaTools, Inc. This product has additional charges associated with it for optional product support and updates.
This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Office of Advanced Scientific Computing and Research (ASCR), under SBIR Award Number DE-SC0022502 ("E4S: Extreme-Scale Scientific Software Stack for Commercial Clouds").
Note: This product contains repackaged and tuned open source software (e.g., E4S, Spack and AI/ML tools like NVIDIA NeMo™, Horovod, JAX, Keras etc.) which is configured and linked against a proprietary MVAPICH MPI implementation specifically developed and tuned for EFA.
Highlights
- Full E4S™ and Machine Learning stacks built and optimized for AWS EFA and ParallelCluster
- A proprietary MVAPICH2 implementation that offers lower latency and higher throughput than default OpenMPI implementations for pre-installed applications and user installed applications
- HPC software environment managed using the Spack package manager
Details
Typical total price
$2.934/hour
Features and programs
Financing for AWS Marketplace purchases
Pricing
Free trial
- ...
Instance type | Product cost/hour | EC2 cost/hour | Total/hour |
|---|---|---|---|
t2.large | $0.99 | $0.093 | $1.083 |
t2.xlarge | $0.99 | $0.186 | $1.176 |
t2.2xlarge | $0.99 | $0.371 | $1.361 |
t3.nano | $0.99 | $0.005 | $0.995 |
t3.micro AWS Free Tier | $0.99 | $0.01 | $1.00 |
t3.small | $0.99 | $0.021 | $1.011 |
t3.medium | $0.99 | $0.042 | $1.032 |
t3.large | $0.99 | $0.083 | $1.073 |
t3.xlarge | $0.99 | $0.166 | $1.156 |
t3.2xlarge | $0.99 | $0.333 | $1.323 |
Additional AWS infrastructure costs
Type | Cost |
|---|---|
EBS General Purpose SSD (gp3) volumes | $0.08/per GB/month of provisioned storage |
Vendor refund policy
Refund Policy: The standard refund policy outlined here will be followed: https://docs.thinkwithwp.com/marketplace/latest/userguide/refunds.html
Any additional refund enquiries can be sent to support@paratools.com and will be considered individually on a case by case basis.
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
64-bit (x86) Amazon Machine Image (AMI)
Amazon Machine Image (AMI)
An AMI is a virtual image that provides the information required to launch an instance. Amazon EC2 (Elastic Compute Cloud) instances are virtual servers on which you can run your applications and workloads, offering varying combinations of CPU, memory, storage, and networking resources. You can launch as many instances from as many different AMIs as you need.
Version release notes
AWS ParallelCluster compatibility: pcluster v3.11.1 E4S release: 24.05 Inter-node communication requirement: EFA compatible instances Default MPI: mvapich2-x-aws OS: Ubuntu 22.04
CRITICAL:
- Some versions prior to v2024.05.16.1904-pcluster-3.9.1-e4s-24.05-amd64 are exposed to the Log4Shell (CVE-2021-44228) vulnerability. (Vulnerable log4js versions were getting pulled in as a dependency of hive.) Please upgrade as soon as possible to this version or a version equal to or newer than v2024.05.16.1904-pcluster-3.9.1-e4s-24.05-amd64 to mitigate this vulnerability.
- Recent changes to AWS EFA instances have broken the MVAPICH2-X-AWS installation on older versions of this product. Please upgrade to this version or a version equal to or newer than version v2024.01.25.1225-pcluster-3.8.0-e4s-23.11-amd64 to ensure proper operation of MVAPICH2-X-AWS and the installed software stack.
IMPORTANT:
- To use this product with AWS ParallelCluster, you must use the same version of parlallel cluster, v3.11.1
- The version of MVAPICH2-X-AWS installed requires compute node instances to have EFA support for inter-node communication. If you do not plan to run multi-node MPI jobs you can safely disregard this. The head node may be a non-EFA instance type, but the compute nodes MUST be EFA capable. To get a list of instance types that are EFA compatible you can use the AWS CLI:
Updates in this version:
- Upgrade parallel cluster version to v3.11.1
- Improvements to NeMo installation/environment
- Kubernetes 1.30 installed
ParaTools Pro for E4S (TM) provides a turn-key cloud HPC solution with the E4S Software stack and machine learning tools and SDKs installed and optimized to run on AWS. The key software features and versions installed are:
- Ubuntu 22.04, X86_64
- DCV remote access installed
- Codium IDE
- Kubernetes
- E4S 24.05 CPU applications installed
- NVIDIA NeMo bare-metal install v2.0.0rc0
- MVAPICH2-X-AWS (EFA is required for inter-node communication, optimized for CPU, not CUDA aware, but GDR support will be availablie in future versions)
- Python/Machine learning tools, SDKs, Frameworks etc.
- TensorFlow, PyTorch, JAX, Horovod, Keras, OpenCV, OpenAI, Scikit-Learn
- numpy, scipy, matplotlib, pandas, opencv-python
- NCCL 2.19.3
- jupyter notebook
- pywaggle
The full list of E4S applications installed via Spack is as follows:
-- linux-ubuntu22.04-x86_64 / gcc@11.4.0 -------------- adios2@2.10.0 adios@1.13.1 alquimia@1.1.0 aml@0.2.1 amrex@24.04 arborx@1.6 argobots@1.2 ascent@0.9.2 axom@0.9.0 boost@1.79.0 bricks@2023.08.25 butterflypack@2.4.0 cabana@0.6.0 caliper@2.10.0 chai@2024.02.0 charliecloud@0.37 conduit@0.9.1 cp2k@2024.1 cusz@0.3.1 darshan-runtime@3.4.4 darshan-util@3.4.4 datatransferkit@3.1.1 dealii@9.5.1 dyninst@13.0.0 e4s-alc@1.0.2 e4s-cl@1.0.4 ecp-data-vis-sdk@1.0 exago@1.6.0 faodel@1.2108.1 flecsi@2.2.1 flit@2.1.0 flux-core@0.61.2 fortrilinos@2.3.0 fpm@0.10.0 gasnet@2023.9.0 ginkgo@1.7.0 globalarrays@5.8.2 glvis@4.2 gmp@6.2.1 gotcha@1.0.6 gptune@4.0.0 gromacs@2024.1 h5bench@1.4 hdf5-vol-async@1.7 hdf5-vol-cache@v1.1 hdf5-vol-log@1.4.0 hdf5@1.12.2 hdf5@1.12.3 hdf5@1.14.3 heffte@2.4.0 hpctoolkit@2024.01.1 hpx@1.9.1 hypre@2.31.0 kokkos-kernels@4.3.00 kokkos@4.3.00 laghos@3.1 lammps@20230802.3 lbann@0.104 legion@24.03.0 libcatalyst@2.0.0-rc4 libnrm@0.1.0 libpressio@0.95.1 libquo@1.4 libunwind@1.6.2 loki@0.1.7 magma@2.8.0 mercury@2.3.1 metall@0.25 mfem@4.6.0 mgard@2023-12-09 mpark-variant@1.4.0 mpich@4.2.1 mpifileutils@0.11.1 nccmp@1.9.1.0 nco@5.1.9 nekbone@17.0 netcdf-fortran@4.6.1 netlib-scalapack@2.2.0 nrm@0.1.0 nvhpc@24.3 omega-h@9.34.13 openfoam@2312 openmpi@4.1.6 openpmd-api@0.15.2 papi@7.1.0 papyrus@1.0.2 parallel-netcdf@1.12.3 paraview@5.12.0 parsec@3.0.2209 pdt@3.25.2 petsc@3.21.0 phist@1.12.0 plasma@23.8.2 plumed@2.9.0 precice@3.1.1 pruners-ninja@1.0.1 pumi@2.2.8 py-cinemasci@1.7.0 py-deephyper@0.6.0 py-h5py@3.11.0 py-jupyterhub@1.4.1 py-libensemble@1.2.2 py-petsc4py@3.21.0 py-warpx@23.08 qthreads@1.18 quantum-espresso@7.3.1 raja@2024.02.0 rempi@1.1.0 scr@3.0.1 slate@2023.11.05 slepc@3.21.0 strumpack@7.2.0 sundials@7.0.0 superlu-dist@8.2.1 superlu@5.3.0 swig@4.0.2-fortran sz3@3.1.7 sz@2.1.12.5 tasmanian@8.0 tau@2.33.2 trilinos@13.0.1 trilinos@15.1.1 umap@2.1.0 umpire@2024.02.0 unifyfs@2.0 upcxx@2023.9.0 variorum@0.7.0 veloc@1.7 vtk-m@2.0.0 vtk-m@2.1.0 wannier90@3.1.0 xyce@7.8.0 zfp@0.5.5Additional details
Usage instructions
The 1-Click Security Group opens port 22 only so that you can access your instance via SSH using login 'ubuntu', you may change this later.
For software development and basic usage:
- Launch the ParaTools Pro for E4S (TM) AMI via 1-Click
- On the 'EC2 Launch an Instance' page pick the key pair you will use to login
- On the 'EC2 Launch an Instance' page optionally edit the network settings by pressing the edit button. Adjust the firewall rules if needed to ensure ssh access and enable Auto-assign public IP if you plan to access the instance remotely from a non-AWS IP address.
- Click 'Launch Instance'
- Find your running instance in the EC2 Instances section of the EC2 dashboard, and connect to the instance via SSH using the key pair you previously selected by picking the instance and pressing the connect button.
For more advanced usage, including launching an ParaTools Pro for E4S (TM) cluster with AWS ParallelCluster, please see:
Resources
Support
Vendor support
For general support questions, please email support@paratools.com
Paid support contracts and custom AMIs and computing environments are available. Please see https://paratoolspro.com/ for additional details.
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
