We have 34 contributed talks lined up from users and contributors all over the world.

Schedule

• The full schedule is online here.

The sessions are:

Wednesday, May 7:

• Welcome and overview

• Building Spack from members of the core team

• Collaborations Using Spack

• Lightning Talks

• Cloud + Benchmarking + Containers

• Developer Workflows: Challenges and Lessons Learned

• Site Deployment Stories

• DevOps for Large Applications

We couldn’t have hoped for a better set of talks – thanks to all of you who submitted. We’re looking forward to sitting back and hearing what you’re doing with Spack. We’ll be taking notes.

Registration

SUM 2025 will be hosted as part of the first annual HPSF Conference. If you’re interested in attending, visit the registration site. Register before March 21 for the early bird discount! Hope to see you in Chicago!

We hope you can join us for the first ever Spack User Meeting (SUM) in Chicago, IL on May 7th and 8th, 2025, as part of the first High Performance Software Foundation (HPSFcon) conference.

What to Expect

• Updates from Spack developers We’ll give a talk on the State of the Spack ecosystem, and we’ll have technical updates on the latest features and release road map.
• User presentations We hope to feature many presentations from Spack users on their experiences and on what they’ve built with Spack.
• Poster Session There will also be a poster session at where you can showcase your work and interact with others.
• Networking Opportunities SUM is hosted in conjunction with HPSFCon, so you can meet folks from other HPC communities – including other HPSF projects.

Who Should Attend:

Anyone in the Spack community: users, developers, dev-ops, system administrators, user support staff, or anyone interested in learning more.

Why Attend SUM / HPSFcon:

We think it’s time we gave our users a forum to present what they’re building with Spack. This the first ever in-person Spack user meeting, and we’re hoping to learn about all the things you’re building. So far, the Spack community has gotten together at BOF sessions at conferences, exchanged comments on GitHub, and has been very active on Slack. The inaugural SUM meeting will give us an even higher bandwidth, interactive venue to exchange ideas and to get a better sense of the needs of the community.

SUM is also part of HPSFcon, which is the first-ever conference of the High Performance Software Foundation, bringing together the community of developers and users of open-source software for high-performance computing. This is a unique opportunity to learn about the latest trends and technologies in HPC and connect with leaders in the field.

Call for Submissions:

We are seeking presentations and lightning talks for SUM.

20-minute presentations: Share your experiences using Spack - any and all spack-related topics are welcome!

• User experience reports with Spack
• Best practices for using Spack in different environments
• Tools, applications, and other software you’ve built on top of Spack
• Best practices and deployment stories
• Unique use cases for Spack
• War stories and other problems you’ve encountered
• Suggestions for features and improvements
• Integration of Spack into DevOps pipelines, applications, and unique environments
• and, yes, all your pain points – we want to know how we can improve Spack!

Submission Deadline (extended to): March 2, 2025

Please submit your abstracts through the HPSFcon submission system.

Registration:

Registration for HPSFcon is open now. Register on the official HPSFcon website.

We look forward to seeing you in Chicago!

Frequently Asked Questions (FAQ)

Q: What if I only want to attend the Spack User Group (SUM) meeting?

A: You still need to register for HPSFcon. There is no separate registration process for the SUM meeting.

Q: Is there a virtual attendance option for the SUM meeting?

A: The 2025 Spack User Meeting will be held in-person at HPSFcon in Chicago. This provides a valuable opportunity for face-to-face interaction with Spack developers and other users in the community.

We do plan to record talks and make them available online, after the event, but we encourage you to attend in person.

Spack was designed as a from-source package manager, but today users can stop waiting for builds. Spack has been able to create binary build caches for several years, but the onus of building the binaries has been on users: application teams, deployment teams at HPC facilities, etc.

Today, enabled by some of the changes in Spack v0.18, and helped by the folks at AWS, Kitware, and the E4S Project we’re starting down the path towards building binaries for everything in Spack. Right now, we’ve got binaries for the following OS’s/architectures:

You can also browse the contents of these caches at cache.spack.io.

If you already know what you’re doing, here’s what you need to try out the 0.18 binary release:

spack mirror add binary_mirror  https://binaries.spack.io/releases/v0.18
spack buildcache keys --install --trust

If you aren’t familiar with Spack binaries, we’ve got instructions for using these binaries on graviton2 nodes below. Note that you do not need to have the Spack v0.18 release checked out; you can use this cache from the latest develop branch, too.

What’s different about Spack binaries?

Binary packaging isn’t new, and .rpm and .deb-based distributions have been around for years, and conda is very popular in the Python world. In those distributions, there is typically a 1:1 relationship between binary package specs and binaries, and there’s a single, portable stack that’s maintained over time.

Traditionally, HPC users have avoided these types of systems, as they’re not built for the specific machine we want to run on, and it’s hard to generarte binaries for all the different types of machines we want to support. GPUs and specific microarchitectures (e.g., skylake, cascadelake, etc.) complicate matters further.

Spack binaries are more like those used by Nix or Guix – they are caches of builds, not specially prepared binaries, and installing a package from binary results in essentially the same type of installation as building it from source. For determinism, Packages are deployed with the dependencies they built with. One difference with Spack is that we store considerably more metadata with our builds – the compiler, microarchitecture target (using archspec), flags, build options, and other metadata. We can use a single package to build all these targets, and we can generate many optimized binaries from a single package description. Here’s what that looks like:

With Spack, we’re trying to leverage our portable package DSL not only for portable builds, but for optimized binaries as well. We can generate many optimized binaries from the same package files and targets, and we can maintain many portable software stacks using the same package descriptions. This reduces the burden on maintainers, as we do not need to support many thousands of recipes – only as many recipes as we have packages. And it is easy to assemble new software stacks on the fly using Spack environments. Spack binaries are also relocatable so you can easily install them in your home directory on your favorite HPC machine, without asking an admin for help.

CI for Rolling releases

To support combinatorial binaries, we are trying to push as much work upstream as possible. We don’t want to have to spend a week preparing a new release, so we’ve built cloud automation around these builds. For every PR to Spack, we rebuild the stacks we support, and we allow contributors to iterate on their binary builds in the PR. We test the builds on different architectures and different sites, and if all builds changed or introduced by the PR pass, we mark the PR ready for merge. This allows us to rely on the largest possible group of package maintainers to ensure that our builds keep working all the time.

One issue with this approach is that we can’t trust builds done in pull requests. We need to know if the PR builds work, but we may not know all contributors, and we cannot trust things built before our core maintainers approve PRs. We’ve solved this by bifurcating the build.

Every PR gets its own binary cache, and contributors can iterate on builds before any maintianer has a chance to look at their work (though we do obviously have to approve CI for first-time contributors). Contributors can work in parallel on their own stacks, and per-PR build caches enable them to work quickly, rebuilding only chnanged packages.

Once a maintainer looks at the PR and approves, we rebuild any new binaries introduced in a more trusted environment. This environment does not reuse any binary data from the PR environment, which makes us sure that bad things from an earlier PR commit won’t slip in through binaries. We do rebuilds for develop and release branches, and we sign in a separate environment from the build so that signing keys are never exposed to build system code. Finished, signed rebuilds are placed in a rolling develop binary cache or in a per-release binary cache. You can find the public keys for the release at spack.github.io/keys

Expanding the cache

This is only the tip of the iceberg. We are currently running 40,000 builds per week in Amazon AWS to maintain our binaries, with help from the E4S project to keep them working. Kitware is ensuring that our cloud build infrastructure can scale. You can see stats from the build farm at stats.e4s.io.

We will be expanding to thousands more in the near future. Our goal is to eventually cover all Spack packages, and to make source builds unnecessary for most users and eliminate the long wait times between deciding on a simulation to run and your ability to start a scientific simulation. Expect more compilers, more optimized builds, and more pacakges.

Spack aims to lower the burden of maintaining a binary distribution and make it easy to mix source builds with binaries. Packages available in the binary mirror will install from binaries, and Spack will seamlessly transition to source builds for any packages that are not available. Users will see approximately 20X installation speedups for binary packages, and performance improvements for binary packaging are in the works.

Running Spack binaries in the cloud

On a brand new amazon-linux image, you can have optimized HPC applications installed in only 7 commands. (If you already run Spack, it’s only 3 additional commands). You can do this on a ParallelCluster instance if you want to run these codes in parallel.

Let’s say you want to install gromacs.

First we need to get Spack installed. Spack has a couple low-level dependencies it needs to be able to do anything, including a C/C++ compiler. Those aren’t on your image by default, so you will need to install them with yum. sudo yum install -y git gcc gcc-c++ gcc-gfortran

Now you can clone Spack from github and check out the latest release. Spack will bootstrap the rest of what it needs to install the binary packages. You’ll also want to setup the Spack shell integration so Spack is in your path and all its features are available git clone https://github.com/spack/spack.git cd spack git checkout v0.18.0 . share/spack/setup-env.sh

Now you can configure Spack to use the pre-built binary caches. You can either point it to develop or releases/v0.18.0

spack mirror add binary_mirror  https://binaries.spack.io/develop
spack buildcache keys --install --trust

You can compare the public key that Spack downloads against the ones available at https://spack.github.io/keys/ for an alternate source of truth to validate source of the binaries.

Now you can download optimized HPC application binaries

spack install gromacs

Spack will install from binaries for any package spec that is available, and will aggressively reuse binaries for dependencies. You can see what’s available with Spack

spack buildcache list --allarch

Anything that’s not available will still reuse the available packages for dependencies, and will seamlessly fail over to building from source for any components for which binaries are not available.

10 minutes in and your application is already available, now you can go on to do the science you set out to do.

Spack and its own dependencies

Since its earliest releases, Spack has had software requirements, and we’ve tried to restrict them to very basic system requirements that you’d find on most machines. A compiler, patch, make, etc. are commonly found on at least Linux and macOS systems, and we’ve expected users to make them available themselves.

With Spack v0.16, we introduced an option to use a new concretizer (dependency solver), which is based on Clingo under the hood. Clingo is a very powerful Answer Set Programming system – it lets us solve the complex constraint problems among Spack packages – all the versions, conflicts, feature requirements, optional dependencies, and compiler/target compatibility issues are now handled by Clingo (more here). Using Clingo, we’ve been able to handle much more complex Spack environments, and we’ve been able to fix issues where the old, greedy concretizer would fail on concretizable specs, or where it would just make incorrect decisions. The new concretizer is more maintainable and allows us to develop features rapidly. It will enable us to more aggressively reuse dependencies and to model more complex software relationships in the future.

We’re making the new concretizer the default

In Spack v0.17, we will make the new concretizer the default, but that means that Clingo will be a prerequisite, and it is not something most people will have installed by default. However, we want Spack to be just as usable out of the box after this change. In particular, we want spack install to work immediately after you clone Spack, just like it did before.

And we’re bootstrapping it from binaries

What we decided is that from v0.17 on Spack will, by default, install some of its dependencies from a public buildcache of portable binaries. The source code we use to create this buildcache is open and hosted on GitHub. The bootstrapping procedure will be transparent – the first concretization you do will just be slightly slower while the binary packages are fetched from the buildcache and verified by their SHA256 checksum. It looks like this:

$ spack find -b
==> Showing internal bootstrap store at "/home/spack/.spack/bootstrap/store"
==> 0 installed packages

$ spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
zlib@1.2.11%gcc@11.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell

$ spack find -b
==> Showing internal bootstrap store at "/home/spack/.spack/bootstrap/store"
==> 2 installed packages
-- linux-rhel5-x86_64 / gcc@9.3.0 -------------------------------
clingo-bootstrap@spack  python@3.6

This default should preserve the “clone and run” simplicity in setting up Spack that you’re used to.

What if I don’t want to use Spack’s binaries?

For those of you who do not want to use our public binaries, you can opt out of bootstrapping, and Clingo will be built from source instead. To disallow bootstrapping from binaries, but still permit Spack to bootstrap from sources, just run:

$ spack bootstrap untrust github-actions
==> "github-actions" is now untrusted and will not be used for bootstrapping

To completely disable any bootstrapping, you can run:

$ spack bootstrap disable

If you do this, you’ll need to ensure for yourself that Spack’s prerequisites are properly installed. You can use this option, for example, if you know that Clingo is already installed (e.g., using pip) in the python you’re using to run Spack.

For more details on how bootstrapping will work, you can check out the pull request #22720. Also keep an eye on our permanently-pinned issue for other high-impact changes arriving soon in Spack develop.

• Spack BoF (runtime 1:00:40): This “birds of a feather” gathering details major developments in Spack releases, collaborative work with the E4S team, roadmap for future development, and results from our community survey. The session was led by Todd Gamblin, Greg Becker, Tammy Dahlgren, Peter Scheibel, Richarda Butler, and Sergei Shudler.
• Using Spack to Accelerate Developer Workflows (runtime 6:14:42): This tutorial focuses on developer workflows, covering covered installation, package authorship, Spack’s dependency model, and Spack environments and configuration. Participants can learn new skills in this tutorial, even if they have participated in Spack tutorials in the past. Presenters were Todd Gamblin, Greg Becker, Peter Scheibel, Tammy Dahlgren, and Rob Blake. See also Spack’s tutorial docs.

Spack has been around for a while, and we’ve always felt like we have a pretty good sense of the community through channels like GitHub, Slack, and our Google group. However, the project is getting larger, and we wanted to better understand the community’s needs with more structured feedback. Spack is funded by the NNSA ASC Program and the U.S. Exascale Computing Project (ECP), and we wanted to understand the different needs of ASC, ECP, and other types of users.

So, this year, we ran our first ever Spack user survey. Read on to see the results.

About the survey

The survey has 26 multiple-choice questions and 6 longer-form questions. It was open from September 28 to October 9, and there were 169 respondents. We advertised to the broadest audiences we knew how to reach. Specifically, we advertised the survey through:

• Our Google group (402 members);
• Slack (~900 members);
• Twitter (~1,100 followers);
• the ECP-wide mailing list (ECP is ~1,000 people); and
• the El Capitan Center of Excellence (COE) mailing list.

There’s probably a lot of overlap between these lists, so this may not be as wide an audience as it would seem. It’s probably biased towards U.S. users, as the people on the mailing lists are very U.S.-centric. So, while the sample is by no means scientific, we know at least that it covers a lot of Spack users.

Data

The results are summarized here, but you can get the full data set and the scripts we used to generate these charts here. There are a lot of results – let us know if you find anything interesting that we missed.

Thanks!

Thanks to the 169 users who filled out the survey, both for this feedback and for your continuing contributions to Spack! This project wouldn’t be possible without the community!

Demographics

In this section of the survey, we tried to understand the composition of the Spack community.

ECP and Spack

The first question we asked was whether respondents were part of ECP.

ECP represents just over a third of the community (~35%, or 61 of 169 respondents). It includes people from both sides of the U.S. Department of Energy – NNSA laboratories like LLNL, LANL, and Sandia; and Office of Science, laboratories like ANL, ORNL, LBL, PNNL, BNL, and others. Under ECP, the Spack team is focusing on delivering a software stack for the first U.S. exascale machines, which includes upcoming systems like:

• LBL’s Perlmutter: AMD CPU / NVIDIA GPU (pre-exascale)
• ANL’s Aurora: Intel CPU / Intel GPU
• ORNL’s Frontier: AMD CPU / AMD GPU
• LLNL’s El Capitan: AMD CPU / AMD GPU

All of these are HPE/Cray systems, but the hardware is quite diverse (particularly the GPUs). We wanted to see whether the users at the bleeding edge of HPC had significantly different needs from the Spack community as a whole. So, in most of the following sections, we present responses for Spack and for ECP separately.

What kind of user are you?

We asked users what their role was at their organization.

Spack was originally targeted at user support teams and system administrators, but by far the largest parts of the community are end users (scientists/researchers) installing software on HPC machines (35%) and software developers (41%). System administrators were ~11% of the overall community, and user support staff were only 8%. In ECP, this is even more pronounced – only a small fraction of respondents identified as system administrators, and developers were nearly 43% of the ECP user base.

Part of the absence of sysadmins may be that at DOE labs, administrators don’t typically handle installation of user software – that’s left to dedicated support teams who engage with users. Admins (at least in DOE) tend to focus on keeping the machines running and managing the host OS underneath Spack.

If you compare this to EasyBuild’s latest survey (slide 13), you’ll see that the composition of the communities is very different. In EasyBuild’s similar survey, only 3% of the respondents identified as developers, and only 9% were scientists. User support and admins were 26% and 53% of the EasyBuild user base, respectively.

Where do you work?

We next asked users where they work.

The community as a whole is diverse. Slightly less than a third (31%) are from Universities. 37.6% are from DOE NNSA and Office of Science laboratories (more than all of ECP – so parts of DOE not in ECP are included). Other public research labs make up 18% of Spack users, and ~13% are from private companies and cloud providers. Within ECP, a large majority of users (76%) were from DOE labs, but there was still some participation from public labs and universities.

Comparing again with EasyBuild’s survey (slide 13), we can see that EasyBuild has a much smaller percentage of users from national computing centers (13% vs. 37%), and a larger percentage of users from universities (55%). It is hard to tell exactly how the proportions compare, as EasyBuild’s survey provided a “university research group” option, while in our survey that is likely spread across the “University HPC center” and “public research lab” categories.

What country are you in?

Just under 2/3 of Spack users are in the United States, and nearly exactly 2/3 are from North America when our two Canadian respondents are included. 27% are from Europe, 5% from Asia, and there was one respondent each from the Middle East (Saudi Arabia) and South America (Argentina). Within ECP (which is a U.S. DOE project), the proportion is much higher – nearly 97% are from the U.S.

What are your primary application areas?

These results are mostly as we expected – most Spack users (~80%) are doing traditional HPC and simulation. In the broader community around 30% are doing computer science research, but within ECP around 50% are doing CS research. In ECP, AI and bioinformatics were noticeably less emphasized than in the broader Spack community. Interestingly, compiler testing was the 7th most popular application area outside ECP, but the 4th most popular inside ECP. Only one user reported using Spack for web applications.

How did you find out about Spack?

Both in and outside ECP, about half of Spack users hear about the tool via word of mouth. This result made us pretty happy – we think it means that users are very willing to recommend Spack to friends. After word of mouth, 22% people heard of Spack because it was used at their site. Within ECP this was slightly more likely at 30%.

24% of users heard about Spack from outreach activities: tutorials, BOF sessions, and presentations.

How long have you been using Spack?

Spack usage has been growing over time, and the number of users who join the community each year is increasing. Most of the overall community has been using Spack for less than two years. We also see this effect in the number of contributors to the project on GitHub. In 2018, after the project had been publicly available for 4 years, there were around 300 contributors. 2 years later, there are nearly 700.

Within ECP, the community is older – most people started using Spack in ECP 2-3 years ago. There has been less new adoption since then, which we attribute to Spack’s rapid adoption in ECP. Spack caught on quickly there, people have continued to use it, and there is not a large influx of users into ECP. The population stays mostly the same over time.

Have you contributed to Spack?

Around 75% of users who responded have contributed to Spack in some way or another, and most of those (60%) have contributed a package. Nearly 40% of users are active on Slack, and nearly 40% have filed issues on GitHub.

It’s notable that Slack discussions are far more active than the Spack mailing list – users seem to want to engage live rather than sending emails back and forth.

Not many users (~10%) have contributed to documentation, but even this small amount helps the project – this is around 16 people.

There don’t seem to be any special takeaways here for ECP, except that ECP users seem to be slightly more likely to contribute a package (nearly 70% have done so).

Spack Usage

Having characterized the user base, we moved on to finding out how they use Spack.

What version(s) of Spack do you use?

Spack users like to be on the bleeding edge, and it shows up the versions of Spack they use. Just under 60% of users are using the develop branch of Spack, and the number is higher (~65%) in ECP. The next most popular version was 0.15, which was the latest Spack release at the time of this survey.

Comparatively few users were on older versions, though a very small number of people were using 0.10. A lot has happened since then – 0.10 was released in January 2017, and there were around 1,000 packages then (vs. 5,000 now). We hope the users still on 0.10 will upgrade – both for features and for package fixes.

What OS do you use Spack on?

Spack is targeted at HPC, so it’s no surprise that nearly 100% of users are using it on Linux. What users sometimes forget is that Spack also works on macOS. Around 35% of all users run Spack on their macs, and over half the users within ECP are also running it on macOS. A small number of users (<10%) are running Spack within the Windows Subsystem for Linux. We don’t test there, but we’re told that Spack works fine in the WSL environment.

If we look at the responses in more detail, we can see the specific Linux distributions that users are running. The most popular, by far, are CentOS and Red Hat. In ECP, Red Hat is especially popular. Ubuntu is the next most popular after these, then macOS, then other Linux distributions and WSL. SuSE is more popular within ECP than in the broader community, likely because it is the Linux distribution that most Cray systems are built on.

How many software installations have you done with Spack in the past year?

Almost 28% of the community has done over 200 software installations, and over 12% have done over 1,000 installs. The ECP numbers are similar to the general population, but the distribution is slightly shifted towards larger numbers of installations.

What Python version(s) do you use to run Spack?

Support for Python 2 ended January 1, 2020, but nearly a year later, around 40% of Spack users are still using Python 2.7. 4 users are even using Python 2.6. Python 2.7 is still the system Python version on many operating systems, including Red Hat 7 and CentOS 7, and on Red Hat and CentOS 6 (which are still used at some sites) the default is 2.6.

While many projects can pick their version of Python, Spack is often the tool people use to install newer versions of Python, and we don’t want to make users use another installer just to install Spack’s dependencies. We want it to work out of the box, so we try to make Spack work with the system Python everywhere.

How bad would it be if Spack dropped support for Python 2.6?

We asked whether it would be OK for us to drop Python 2.6, and we found that there are still around 4 hold-outs who really need Spack to work on Python 2.6, and over 20% of people would be at least mildly inconvenienced by this change. For the time being, we’ll keep supporting Python 2.6, but you can probably expect its deprecation to be announced sometime within the next year, as the last few Red Hat 6 installations dwindle.

How bad would it be if Spack only worked with Python 3?

Eventually, we’d like to drop Python 2 entirely, but with 40% of users still using 2.7, and with 36% of users likely to be bothered by the shift, we’ll hold off on dropping 2.7, as well. Interestingly, while ECP users were less likely than the broader community to completely oppose dropping 2.6, they were more likely than the community to oppose dropping 2.7.

How do you get installed Spack packages into your environment?

The most common way to use Spack packages is still through modules, and module usage seems to be split about evenly between Lmod and TCL modules, with some overlap. We were surprised to see that the spack load command is the second most popular way to use Spack packages, only a few percentage points behind modules.

We don’t have data on past usage of spack load, but we’ve tried to make it easy to use everywhere, and this may have caused its usage to increase. In particular, in earlier Spack releases, spack load required modules in order to work (it was a thin layer around module load). As of Spack 0.14 in early 2020, it only requires Spack’s own environment support – so you can easily load a one-off package on your mac or personal Linux box where you don’t already have modules installed.

After spack load, around 35% of users are making use of Spack environments to load groups of packages together. Like spack load, Spack environments require no support from the module system – they work regardless of where Spack is deployed and provide a more portable alternative to loading environment modules.

Which Spack features do you use?

While Spack environments ranked below modules for simply getting packages into PATH, environments are actually the most widely used single feature of Spack (at least on our list here). Around 2/3 of users say they use environments. environments can be used to add packages to PATH, to maintain a list of dependencies via spack.yaml, to version a spack.yaml environment in a repo, to do combinatorial builds, to reproduce builds with spack.lock, to configure and run CI pipelines, and to build container images.

Looking Ahead

We wanted to get a sense of what users will need from Spack in the coming year, so we asked about upcoming architectures, Spack features, and events.

Which CPUs do you expect to use with Spack in the next year?

Nearly everyone in the Spack community plans to run on Intel CPUs in the next year, and around 80% expect to use Spack to build for AMD systems. Just over 40% of users will run on ARM and just under 40% will run on Power. Within ECP, the percentage of users that want to run on any of the non-Intel CPUs is larger – as you might expect, ECP is targeting a more diverse set of architectures. There were many more ECP users who expected to run on Power than in the broader community, likely because the current top two U.S. systems, Summit and Sierra, are Power machines.

We can’t draw a fair comparison with EasyBuild on this question, as EasyBuild’s survey asked users what CPUs they were currently using rather than what they expected to be using in the next year, and their survey was done a year ago, and things are changing fast in HPC. So, take it with a grain of salt, but the difference is still worth mentioning. In the EasyBuild survey (slide 26), the vast majority of users were similarly running on Intel machines. But, less than 20% were using AMD chips, less than 5% were using Power, and only one user reported using ARM. It’s likely their numbers for AMD and ARM will increase on the next survey.

Which GPUs do you expect to use with Spack in the next year?

All users in the Spack community expect to run on GPUs in the next year, and over 90% plan to build for NVIDIA GPUs. Around half of the overall community expects to build for AMD GPUs, and around 30% expect to build for Intel GPUs. Within ECP, the percentage of users planning to run on NVIDIA GPUs is very slightly lower, likely because NVIDIA will not be the GPU on any of the initial three exascale machines. Aurora will be an Intel GPU system and both Frontier and El Capitan will use AMD GPUs. As you might expect, 80% of ECP users expect to use AMD GPUs and over half expect to use Intel GPUs.

GPU usage in the EasyBuild community was similarly high – 96.5% of EasyBuild users were compiling software for GPUs – so it’s pretty clear that GPUs have become pervasive in HPC.

Which compilers do you expect to use with Spack in the next year?

As you might expect given the CPU and GPU results, Spack users anticipate using a very wide range of compilers. gcc is still king, with almost 100% of users expecting to use it, and LLVM and Intel compilers are next on the list.

Interestingly, only around 60% planned to use nvcc, and a bit more than 40% planned to use NVIDIA’s HPC compilers. Given that over 90% of users said they expected to build on NVIDIA GPUs, we’re tempted to explain the discrepancy by saying that a large percentage of users expect to use NVIDIA GPUs not through CUDA directly, but through GPU-optimized libraries or through compiler capabilities like OpenMP offload. The same can probably be said for AMD and Intel GPUs – the percentage of users who plan to run with compilers specifically intended for these GPUs was consistently lower than the number of users who anticipated using them.

Rank upcoming Spack features by importance

We asked respondents to rank some planned and not-yet-planned Spack features by importance: “not important”, “slightly important”, “somewhat important”, “very important”, and “critical”. The two most frequently ranked “critical” (and also the top two features by average score) were reusing external installs and the new concretizer. These are related, as the new concretizer is needed to reuse existing installations.

After those, the most important features were better compiler flag handling and better support for developers. Separate concretization of build dependencies (i.e., using gcc for packages like CMake even if the user asked that the main package be built with the Intel compilers) was next on the list, followed by language virtuals (ability to depend on cxx, c, or fortran and have that resolve to a compiler and runtime library), automatic package maintainer notifications on GitHub, and build testing.

The features rated least important were build testing, publicly available optimized binary packages, package testing, cloud integration for Spack, and Windows support, with the last three rated significantly less important than all the others.

Every feature was listed as “critical” by at least some users, but there are some clear preferences here that we’ll be trying to tailor our efforts to. We already shipped the new concretizer as an experimental feature in Spack v0.16.0, and we’ve already merged a number of fixes for it in Spack v0.16.1. Separate concretization of build dependencies and reusing existing installs are both modifications that we’ll need to make to the new concretizer, and we’ve already started looking into how we can provide them. Better developer support, language virtuals, maintainer notifications, better build testing, and package testing are already milestones for 2021.

The feature that stands out that we haven’t yet worked into our plans is better compiler flag handling. Based on this survey we’re going to see if we can work that into our schedule for 2021, as well.

In addition to the community-wide averages above, we looked at whether different segments of the community rated features differently. On the right above, we split out average feature ratings by workplace, and on the left we split them out by job type.

Overall, the rank order of features was similar across different workplaces and job types. Reusing existing installs and the new concretizer were consistently at the top of everyone’s list, and the lowest-rated features had low ratings across the board. Industry users prioritized cloud integration noticeably higher than other groups, and user support staff placed a much higher value on package testing than other job types (perhaps because they are involved in more package testing efforts at their sites). Managers and ASCR labs rated separate build dependencies lower than other groups. Other than these outliers there were not significant deviations from the overall order of preference.

There are some noticeable trends across groups. System administrators, user support staff, and industry users tended to rate features as less important across the board. It’s hard to know how to interpret this – it could mean that they’re happy with the existing capabilities of Spack, or that these particular improvements aren’t their top priorities.

If we had a (virtual) workshop on Spack, would you attend?

We’ve thought about having a Spack user meeting for a while, and we had actually started planning for an inaugural Spack User Meeting earlier this year. That fell apart when the pandemic hit. Other similar tools have had good luck with meetings like this (e.g., NixCon and the EasyBuild User Meeting), so we asked users what they thought of a potentially virtual meeting:

Just over half of users (over 85 people) said they would attend, and 17 said they’d be willing to give a presentation. That seems like more than enough for an initial Spack meeting, so expect us to announce something for 2021.

Getting Help

We’re interested in making it easier to learn about Spack, so we asked people how they’re doing it now.

Have you done a Spack tutorial?

We were surprised to see that over 60% of all our users have done a Spack tutorial. Since 2016, We’ve been doing Spack tutorials at conferences like Supercomputing, ISC, and PEARC, and this year we had over 125 attendees at our virtual Spack tutorial on AWS. This seems to show that tutorials have been a very effective form of outreach, even if they aren’t the main way people first hear about Spack (per our earlier question). At the very least, they likely contribute to the high rate of contribution in the community.

How do you get help with Spack when you need it?

Users go to the docs more than any other place for help with Spack. Slack, as mentioned above, is also very popular – 50% of users use it to get help. We were happy to see that around 40% of users get help from their coworkers, and when we looked further at this data, those who got help from coworkers were not confined to big laboratories – they came from all the types of workplaces that we considered.

How often do you consult the Spack documentation?

Users consult the documentation reasonably frequently – weekly to monthly for most. A small fraction (14%) check it daily. ECP users check the documentation less frequently on average than the community as a whole, but ECP users have also been using Spack for longer, and are likely more familiar with it.

If there were commercial support for Spack, would you or your organization buy it?

We don’t have any plans to provide commercial support for Spack at the moment, but it’s nice to know that 23%, or 39 users and their organizations, might be willing to pay for it. That’s a fairly large percentage of users willing to pay for support for an open source product.

Quality of Spack

We wrapped up the multiple choice part of our survey with a final question asking users to rate the quality of Spack.

How would you rate the overall quality of Spack, its community, docs, and packages?

Similar to our question on features above, users were asked to rate different parts of Spack as “horrible”, “bad”, “ok”, “good”, and “excellent”. We split the results out by workplace and job:

Responses were positive on average for all categories. Only 3.5% of users responded negatively for the quality of Spack overall (cf. 2% for EasyBuild, slide 60). Only 5% responded negatively for any aspect. Consistently, the highest-rated aspect was the community, which is great, because Spack wouldn’t be sustainable without its community. Just after the community was Spack itself.

While both the community and Spack averaged “good” or higher overall, the docs and packages had the lowest average ratings. While some users have praised the documentation, a lot of documentation has accumulated and it likely needs to be organized better. Spack targets many different kinds of users, and there is not just one workflow. We’ve gotten a lot of requests to provide clearer how-to guides for common site deployment and developer workflows, which is something we plan to work on over the next year.

Spack packages are a harder problem. The package DSL is part of what makes Spack unique – there is one template for each package, and the same template lets you build any version or configuration of a package. This makes it easier to port Spack packages to new systems, but it also makes the testing surface for Spack packages very large. We think we are still on the right path, for several reasons:

• Together with Kitware, we’ve built up a sophisticated CI system using the support for pipelines built into Spack environments.
• We’ve hooked up our GitLab instance to Spack’s main GitHub repository, and within the next couple of weeks, we’ll be testing a subset of Spack builds on every pull request.
• These are the same builds used to produce ECP’s Extreme Scale Scientific Software Stack (E4S).

One of our priorities for 2021 under ECP is hardening these builds and testing Spack packages on a wide range of platforms, and now that the new concretizer is in Spack, we expect to be able to steer Spack configurations towards well-tested ones, based on builds in our pipeline and under E4S. So, we expect package stability to get much better over the coming year, and we’re hoping that it will show up in next year’s survey responses.

Longer answers

We asked 6 long answer questions. If you want, you can read them all in the data repository. The number and length of responses to these was overwhelming, and we haven’t come up with a great way to summarize them, but reading them all gave us a great picture of what people in the community are up to. We’ve picked a few responses per question and quoted them below.

Tell us briefly about your use case and your usual Spack workflow.

• I am using Spack to build the software environment for our users on our University’s centralized HPC system.

• Using Spack to build complex applications natively. Moving the build into containers, if possible with spack containerize.

• We use spack to provide 3 consistent entry points into our nuclear physics software environment: cvmfs, build_caches, containers.

• Rather heterogeneous cluster with an environment per architecture. Currently having lots of fun packaging bioinformatics tools.

• Support Spack in Fugaku

• I’m a math library developer and use spack to build third party libraries e.g., blas, lapack, MPI, hypre, SuperLU_MT, SuperLU_DIST, PETSc, and Trilinos both for developing on my laptop and for continuous integration on a dedicated workstation.

• We use spack to install most facility-provided software on OLCF HPC machines.

• Distributed build system with Spack environments

What about Spack helps you the most?

• The community

• Greg Becker responding to my questions on Slack.

• How Spack handles installation of multiple versions of the same app.

• How Spack should be Linux OS agnostic (yet to be tested) so we can experiment with offering other distros for users.

• Absolute flexibility, especially compared to, i.e., nix. And dependency handling, which I never want to do manually again.

• I’m still in dependency hell, but Spack took me from the 7th circle (violence - for the violence I’d like to commit against my keyboard while building things) to the 3rd circle (gluttony - for the voracious appetite I now have for spack-installed packages and the indulgent number of dependencies they require).

• Well-defined package specifications and solid concretization.

• The concretizer (despite some issues) is the most helpful aspect of Spack. It allows for automatic dependency management and reproducibility.

What are the biggest pain points in Spack for your workflow?

• surprising re-concretizations, updating environments and removing old packages

• inter dependencies with many variants makes a huge/complicated package file

• Better parallel building of environments. I think Slurm integration could be very good to have.

• Right now, the time it takes to concretize in our deployment with ~2000 packages already in the database.

• Not having language virtual dependencies makes it harder to have language polyfills for newer features when the compiler doesn’t support them. It also is hard to say what compiler versions you support

• c++ language standard dependencies, build dependency blow-up

• It seems that we should specify external package path every time so it would be great if Spack can detect preinstalled libraries.

What’s the biggest thing we could do to improve Spack over the next year?

• Keep doing outreach efforts, videos, tutorials, hackathons, whatever to spread the voice more.

• Python as a virtual dependency

• The complaint I still have to field from people is “I tried to build a simple package, and Spack built Python and CMake and and and and…” so I think better deciphering of externals (which I know you’re working on as we speak) would be good.

• QA: less features but really solid CI on tagged releases, including packages.

• Cross compiler support, new concreteizer

• Documentation organisation, examples, and explicit API listing of all internal functionality.

• New concretizer, maintainer bot

• Build lots of buildcaches for each site to speed up builds. It could be nice to have a cloud repository could be AWS, GCP, where spack host all the buildcaches.

• Fully-working backtracking concretizer

• Don’t lose momentum.

Are there key packages you’d like to see in Spack that are not included yet?

• Probably new build system support like Julia / Golang

• I would like to be able to contribute one day adding the Uintah software suite

• moose

• WRF. I’m aware that it’s now included in develop branch.

• None that we haven’t been able to rapidly write for ourselves.

Do you have any other comments for us?

• This project makes me enjoy coming into work.

• Yes, I like spack, can’t do without it now.

• It would be great if https://github.com/spack/spack-configs included more DOE machines and were updated more frequently. If the administrators at computing centers provide these files somewhere it does not seem to be well advertised to users.

• Pretty much every year my biggest victory with Spack is “they fixed the thing that was biggest gripe about Spack last year,” which is a sign of a really good job listening to users, so keep that up.

• Keep up the good work. I will continue using and supporting Spack for many years to come.

• Keep being awesome! Spack is my favorite tool of the last 5 years!

• It’s easy to create a real tangle of versions and special builds. I think the project should work to maintain good communication with the application developers, so that standard or common/expected versions and dependency sets can be more clearly identified

• Spack is extraordinary and blows away all past attempts to bring sanity to HPC software. I encourage you to offer commercial support. Please have support tiers that allow us to select an appropriate level of support.

Be sure to follow @spackpm on Twitter for updates!

Mon., November 9

• 10:00am - 2:00pm EST (live)
  Our tutorial runs over two half-days. Managing HPC Software Complexity with Spack: Part 1 kicks off on Monday. Be sure to check out our tutorial documentation.

  The tutorial provides a thorough introduction to Spack’s capabilities: installing and authoring packages, integrating Spack with development workflows, and using Spack for deployment at HPC facilities. Attendees will leave with foundational skills for using Spack to automate day-to-day tasks, along with deeper knowledge for applying Spack to advanced use cases.

Tue., November 10

• 10:00am - 2:00pm EST (live)
  The second day of the tutorial: Managing HPC Software Complexity with Spack: Part 2

Wed., November 18

• 11:30am - 12:45pm (live)
  Join the Spack Community BOF and ask our developers anything. The core team will give updates on the community, new features, and the roadmap for future development. We will poll the audience to gather valuable information on how Spack is being used, and will open the floor for questions. All are invited to provide feedback, request features, and discuss future directions. Help us make installing HPC software simple!