Researchers update supercomputing infrastructure that powers research at Notre Dame
Twice each year, the High Performance Computing (HPC) team at the Center for Research Computing (CRC), and a ten-person team work 150–200 hours to ensure that Notre Dame’s research computing is stable, up-to-date, and as fast as possible.
No downtime
The CRC schedules two maintenance windows each year to make important updates while minimizing the impact of the outages on students and staff. The dates include the first working weekend in January and the Commencement weekend in May. Together, the Operations and User Support teams begin planning for these outages six to eight weeks in advance.
The Operations team focuses on firmware, hardware, and system software while the User Support team manages user applications and performance testing. User-facing login nodes receive weekly systems software and security updates, but our compute nodes are only updated during maintenance weekends with the rare exception made for critical security patches. As such, operating system (OS) and firmware updates across our 1,400+ servers are an outage necessity. Even a minor OS version update requires both teams to test dozens of different hardware and software combinations to look for application incompatibilities, missing features, or noticeable performance degradation. Upgrading major OS versions—such as moving from RHEL 8 to RHEL 9—can sometimes result in retiring entire servers if key hardware support has been dropped.
In the weeks leading up to the outage, both teams converge on a set of updates and then begin working on the timeline for the maintenance weekend. By carefully mapping out tasks, meticulously planning, and creating an ordered checklist, the odds of having to extend the outage due to unforeseen circumstances can be minimized.
Down to business
Outages begin around 6:00 a.m. on Friday. Systems are cleared of users and running tasks and allow only essential staff access for the remainder of the outage. The first task involves shutting down all servers and non-essential infrastructure to apply new firmware and security updates. Even with automation and a team of 5-7 people on hand, this step alone can take several hours.
Once most of the hardware updates are finished, the system's software updates or Operating System (OS) rebuilds can begin. Each server rebuild can take 10-25 minutes to complete and often requires multiple reboots. Fortunately, the onsite team can rebuild 50–100 servers at once. The challenge begins, however, with servers that either do not accept the firmware updates, will not rebuild with the new OS, or in some cases, do not turn back on. With almost 1,500 servers containing thousands of hard drives and memory DIMMS that can fail, network cables that can work loose, fan bearings that can burn out, and power supplies that can overheat, there are many things to check and many things that can go wrong.
The Operations team commonly works late into Friday evening to have as many systems as possible online and ready for User Support to begin their work in the morning.
The User Support team takes over the bulk of the work on Saturday as they begin updating user applications and tools. Even for minor OS updates, this involves going through more than 100 different software modules to check for consistency and expected behavior. During major OS upgrades or default compiler updates, dozens of software packages may need to be recompiled from scratch.
Once the software stack has been verified, an intensive validation and verification process begins across all active servers. Suites of benchmark tests are run and compared to previous results to look for performance anomalies. This ensures that the system or software changes do not adversely affect user results. At this stage, it is common to find servers without obvious hardware or software failures that need extra attention. In addition, the Operations team continues working on various tasks and updates backend infrastructure hardware such as networking routers, license servers, and facility power and cooling.
Access granted
Assuming the process goes according to plan, Sunday mornings are reserved for last-minute updates or work on problematic servers. The team will also conduct any additional work required on login nodes and backend infrastructure systems. Once everything appears to be working as expected, the team reopens access to users and shares a follow-up with a summary of tasks completed over the weekend.
Originally published by crc.nd.edu on April 16, 2024.
atLatest Research
- NDTL shares aerospace research with local high school studentsMark H. Ross, senior research scientist at NDTL Propulsion & Power, recently stepped out of the lab and into a local high school. Ross, who received his Ph.D. in Aerospace Engineering from Notre Dame in 2014, visited the computer science and engineering magnet…
- Notre Dame senior wins Best Publication Image AwardNotre Dame senior Kevin Armknecht of the Berthiaume Institute for Precision Health has been recognized by ND’s Integrated Imaging Facility (NDIIF) with the Best Publication Imaging Award, based on images he created in a recent Nanoscale Advances publication. Armknecht, a pre-professional studies…
- A&L and engineering faculty to create new curriculum for responsible computingFaculty from the College of Arts and Letters and the College of Engineering are working together to integrate responsible computing instruction across the undergraduate curriculum, to better equip students to think critically and thoughtfully about technology. The project, “Computing, Culture, and Society: A Community-based, Intersectional Approach to Responsible Computing Across the Curriculum,” is led by Katherine Walden and Karla Badillo-Urquiola.
- Keough School establishes two new doctoral programsNotre Dame’s Keough School of Global Affairs has established two new doctoral programs in sustainable development and peace studies. The peace studies and sustainable development programs will enable doctoral students in the Keough School to examine from different perspectives the intersection of poverty, the environment, violent conflict and peace. Both programs will enroll students beginning in fall 2025.
- Robert Norton earns A&L Research Achievement Award for his transformative impact on German intellectual history“My interest is in the invisible, or at least the hard-to-discern, currents of thought that inform large cultural phenomena," Robert Norton said. "My scholarly career is devoted to understanding the nature of the impact and what lies underneath and what was lost.”
- ISLA Funds Twelve Projects Examining “Technology and the Common Good” through 2023-24 Annual Research Theme Grant ProgramLast spring, the Institute for Scholarship in the Liberal Arts announced “Technology and the Common Good” as its annual research theme for AY 2023-2024. Taking to heart the notion that technology is a subject of inquiry not only for science and engineering but also the liberal arts, ISLA invited proposals…