The Problem: Summarizing XRootD Stash Data

Our goal was straightforward: aggregate XRootD stash access logs (xrd-stash*) into a daily summary index (osdf-summary-{year}). This involved grouping by several file path components, server details, and user domains, then calculating sums, averages, and counts of metrics like filesize, read, and write.

Here is a snippet of the initial (problematic) transform configuration:

{
  "transform": {
    "transform_id": "osdf-summary-2022",
    "description": "OSDF summary transform for year 2022",
    "source_index": "xrd-stash*",
    "target_index": "osdf-summary-2022",
    "page_size": 1000,
    "groups": [
      {
        "date_histogram": {
          "source_field": "@timestamp",
          "target_field": "@timestamp",
          "calendar_interval": "1d"
        }
      },
      {
        "terms": {
          "source_field": "dirname1.keyword",
          "target_field": "dirname1"
        }
      }
    ],
    "aggregations": {
      "filesize_sum": { "sum": { "field": "filesize" } },
      "filesize_avg": { "avg": { "field": "filesize" } }
    }
  }
}

The Symptoms: Generic Errors and Timeouts

The transform job kept failing with a rather unhelpful message in its metadata:

{
  "status": "failed",
  "failure_reason": "Failed to index the documents",
  "stats": {
    "pages_processed": 96,
    "documents_processed": 89737708,
    "documents_indexed": 96000,
    "index_time_in_millis": 44733,
    "search_time_in_millis": 1715612
  }
}

Notice the high search_time_in_millis compared to index_time_in_millis. This was a critical clue that the aggregation phase was struggling.

Further attempts to debug with _explain or custom composite aggregation queries often resulted in:

• 502 Bad Gateway / timed_out: The query was too resource-intensive for the cluster to handle.
• illegal_argument_exception: Missing value for [after.date_histogram]: A mismatch in how the after_key was structured versus the sources in the composite aggregation.
• illegal_argument_exception: Invalid value for [after.site], expected comparable, got [null]: The transform was getting stuck on null values within its grouping keys.

The Debugging Journey and Discoveries

Through a series of focused queries and iterative refinements, we uncovered several interconnected issues.

1. Composite Aggregation Challenges and the “Ghost Key”

Our composite aggregation debugging queries kept failing. This was traced to:

• Syntax mismatches: names in the after key must exactly match the names defined in sources (for example, @timestamp must match @timestamp).
• null values in after_key: terms aggregations can fail when after_key includes null, unless handled explicitly.

Then came the key finding: a direct search for documents matching the transform’s after_key yielded zero results. The transform was trying to resume from a state that no longer existed in source data.

2. The Real Culprit: Unparsed “Garbage” Data

An inverse query (documents missing expected fields) revealed records like:

{
  "_index": "xrd-stash-ilm-000037.reindexed",
  "_id": "cAqUoH4BOTrVvgqCSyKq",
  "_source": {
    "message": "GET / HTTP/1.1\n",
    "@timestamp": "2022-01-28T12:06:20.222Z",
    "host": "ec2-3-110-169-111.ap-south-1.compute.amazonaws.amazonaws.com",
    "tags": ["_grokparsefailure"]
  }
}

These were logs that failed parsing and were actually web traffic hitting the server, not XRootD stash operations. They lacked key transform fields like logical_dirname, filesize, and server.

When the transform encountered enough of these records, grouping keys became null. Combined with malformed or very long field values, the composite aggregation became unstable and hit timeouts.

3. Precision for PetaByte-Scale Data

Not a crash cause, but still important: float is not precise enough for large sums at petabyte scale.

Solution: use double for sums/averages and long for counts.

The Ultimate Solution: Resilience and Precision

The final, robust fix used multiple changes together.

1. Stop and Delete Stale State

Stop the transform and delete the target index to clear bad transform/index state.

POST _plugins/_transform/osdf-summary-2022/_stop
DELETE osdf-summary-2022

2. Recreate Index with Explicit High-Precision Mappings

PUT osdf-summary-2022
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "dirname1": { "type": "keyword" },
      "logical_dirname": { "type": "keyword" },
      "filesize_sum": { "type": "double" },
      "filesize_avg": { "type": "double" },
      "filesize_count": { "type": "long" },
      "doc_count": { "type": "long" }
    }
  }
}

3. Add Intelligent Filtering in data_selection_query

• Exclude _grokparsefailure events.
• Require existence of critical grouping fields.
• Add script guards against empty or oversized keyword values.

"data_selection_query": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": f"{year}-01-01T00:00:00Z",
            "lt": f"{year + 1}-01-01T00:00:00Z"
          }
        }
      }
    ],
    "must_not": [
      { "term": { "tags": "_grokparsefailure" } }
    ],
    "filter": [
      { "exists": { "field": "logical_dirname.keyword" } },
      {
        "script": {
          "script": "doc['logical_dirname.keyword'].size() > 0 && doc['logical_dirname.keyword'].value.length() < 1000"
        }
      }
    ]
  }
}

4. Reduce page_size

Lowering page_size from 1000 to 50 significantly reduced memory pressure per composite aggregation page and helped avoid 502 Bad Gateway failures.

5. Restart the Transform

After recreating the index and updating the transform definition, restart the job.

Conclusion

By combining explicit mappings, stronger filtering, smaller pagination, and a reset of stale transform state, the transform ran reliably and produced accurate summaries without repeated failure loops.

This debugging story reinforced a key lesson: robust pipelines are not just about handling valid data, but actively excluding invalid or malformed records before they poison downstream aggregation logic.

This is a topic about which I've written at length in the past on my blog, and as I read Dr. Reed's latest post (and the Riding the Wave paper that motivated it), I found myself agreeing with a many of his positions but disagreeing with some others.

My own background is in the world at the center of Dr. Reed's writing: traditional HPC for scientific computing at the national scale. However, my outlook has also been colored by the years I spent at Microsoft supporting massive-scale supercomputing infrastructure for training frontier models and the days I now spend at VAST, steeped in the wider enterprise AI market. This undoubtedly results in an unusual lens through which I now view Dr. Reed's position, and I couldn't help but mark up his essay with my own notes as I read through it.

In the event that my perspective--that of an HPC-turned-AI infrastructure practitioner--is of interest to anyone who found Dr. Reed's latest essay as engaging as I did, I've shared them below.

New Maxim Two: Energy and data movement, not floating point operations, are the scarce resources.

This has been true long before exascale in the HPC world. This is not a new maxim. Ironically, it is in the AI world that this maxim is relatively new; as inference overtakes training as the predominant consumer of GPU cycles, we are seeing widespread shortages of DRAM because of the extreme demand for HBM and the memory bandwidth it provides.

New Maxim Three: Benchmarks are mirrors, not levers. Benchmarks rarely drive technical change. Instead, they are snapshots of past and current reality, highlighting progress (or the lack thereof), but they have little power to influence strategic directions.

Benchmarks drive technical change amongst technology providers who act without conviction. The tech industry is full of companies who are blindly chasing consumer demand, and these companies design entire product lines to achieve high benchmark results with the mistaken belief that those benchmarks are a reasonable proxy for actual productivity. And even worse, many buyers (especially in lower-sophistication markets like enterprise) also believe that benchmarks, by virtue of being designed by community organizations who have ostensibly thought deeply about performance, are a good proxy for productivity, make purchasing decisions around these same benchmarks.

The net result is that a bad set of benchmarks can create and sustain an entire economy of buyers and sellers who think they are buying and selling something useful, when in fact they are wasting resources (time, energy, and COGS) because none of them actually understand what really drives productivity within their organizations.

Fortunately, the HPC community is generally savvier than enterprises, and most national computing centers now recognize that HPL is simply not a meaningful yardstick. While it used to be good for convincing politicians and other non-technical funders that good work was being done, the discourse around AI has squarely put R_max in the ground as a meaningful metric. Politicians now understand "hundreds of thousands of GPUs" or "gigawatts," neither of which require a benchmark like HPL to prove.

Also, as an aside, I find it ironic that a paper with Jack Dongarra listed as an author is now saying HPL is a snapshot of the past. I've heard that he is the reason that HPL results achieved using emulated FP64 are not allowed on Top500. Despite achieving the required residuals through more innovative means than simply brute-forcing a problem through FP64 ALUs, using techniques like the Ozaki scheme were deemed incompatible with the purpose of Top500. Which is to say, I think he's the reason why HPL and Top500 has been reduced to a benchmark that reflects outputs (hardware FP64 throughput) rather than outcomes (solving a system of equations using LU decomposition).

New Maxim Four: Winning systems are co-designed end-to-end—workflow first, parts list second.

…

In HPC, we must pivot to funding sustained co-design ecosystems that bet on specific, high-impact scientific workflows

I don't agree with this. Funding sustained co-design is just swimming upstream with more conviction.

The real way forward is to find ways to align scientific discovery with the way the technology landscape is moving. This means truly riding the wave and accepting that scientific discovery may have to turn to completely different techniques that achieve their desired precision and validation through means that may render obsolete the skills and expertise some people have spent their careers developing.

Consider the scaffolding of end-to-end workflow automation; a rich ecosystem of technologies exists in the enterprise and hyperscale worlds that have been used to build extreme-scale, globally distributed, resilient, observable, and high-performance workflows that combine ultra-scalable analytics engines with exascale data warehouses. However, realizing these capabilities in practice requires fundamentally rethinking the software infrastructure on which everything is built. The rigidities of Slurm and the inherent insecurities of relying on ACL- and kernel-based authentication and authorization need to be abandoned, or at least understood to be critically limiting factors that the HPC community chains itself to.

To make this very specific, consider a bulk-synchronous MPI job running across a hundred thousand GPUs; if one node fails, the whole job fails. The "swimming upstream with more conviction" way of solving this problem is to pay a storage company to build a faster file system, pay some researchers to develop a domain-specific checkpoint library that glues the MPI application to platform-specific APIs, and pay SchedMD to automate fast restart based on these two enhancements. Fund all three projects under the same program, and it is arguably a "co-designed end-to-end workflow."

Riding the wave would be something different though: instead of requiring a job requeue and full restart from checkpoint upon job failure, treat the entire job as an end-to-end workflow. If a node fails, the job doesn't stop; it just transitions into a recovery state, where the orchestrator gives it a new node on which the job runtime can rebuild the state of the dead node using distributed parity or domain-specific knowledge. A fast file system is completely unnecessary for failure recovery. But the application developers would have to abandon the model of an application being a single process invocation in favor of the application being a system whose state evolves with the underlying hardware.

Slurm can't do any of this, because Slurm is tied to the MPI model of parallel execution which assumes nothing ever fails. Which is to say, I think co-design should be deferred until a time that the HPC community first recognizes that, so long as they continue to approach end-to-end co-design as an HPC problem to be solved by HPC people using HPC approaches, they will continue to swim upstream regardless of how much co-design they do.

New Maxim Five: Research requires prototyping at scale (and risking failure), otherwise it is procurement. A variant of our 2023 maxim, prototyping – testing new and novel ideas – means accepting the risk of failure, otherwise it is simply incremental development. Implicit in the notion of prototyping is the need to test multiple ideas, then harvest the ones with promise. Remember, a prototype that cannot fail has another name – it’s called a product.

The idea is right, but the title is wrong. Prototyping at scale is the wrong way to think about developing leadership supercomputing capability. The largest commercial AI infrastructure providers do not prototype at scale. Instead, they frame their thinking differently: anything done at scale is production, and if it doesn't work, make it work.

In practice, this means foregoing years-long acceptance test processes and beating up suppliers over hundred-page-long statements of work. Instead, they accept the reality that they share the responsibility of integration with their suppliers, and if things go sideways, they are working with partners who will not walk away when times get tough.

National-scale supercomputing has always been this way in practice, but the HPC community likes to pretend that it isn't. Consider Aurora: if that system wasn't a prototype-at-scale, I don't know what is. That system's deployment and operations was and remains fraught, and it is built on processors and nodes that were cancelled as products before the system even entered production. Yet the theatrics of acceptance testing went on, Intel got paid something, and we all pretend like Aurora just like Frontier or Perlmutter.

AI doesn’t prototype at scale; they just take a risk because the next breakthrough can't wait for every "i" to be dotted and "t" to be crossed. If a hyperscale AI system is a failure, that’s fine. The demand for FLOPS is sufficiently high that it will be utilized by someone for something, even if that use generates low-value results rather than the next frontier model that it was meant to build. The same is true for systems like Aurora; it's not like these systems sit idle, even if they don't live up to their original vision.

And rest assured, AI systems prove to be bad ideas just like HPC systems do. The difference is scale: there are multi-billion-dollar AI supercomputers in existence that were obsolete before they even came online, because the problem they were designed to solve became irrelevant in the years it took to build them. But what was really lost? A bit of money and a little time. The GPUs are still used for day-to-day R&D or inferencing, and the time lost was made up for in lessons learned for the systems that followed.

All the big AI systems are prototypes, because AI workloads themselves are continually evolving prototypes. As a result, the line between prototype and production become blurry, if not meaningless.

All too often, in scientific computing, our gold is buried in disparate, multi-disciplinary datasets. This needs to change; we must build sustainable, multidisciplinary data fusion.

This is so easy to say, but it always feels empty when it is said. What’s stopping this data fusion? I don’t think it’s willpower or resources. It’s just really difficult to figure out what good any of it would be within a standard theory-based modeling framework. Making productive use of fused multimodal data (meshes, particles, and discrete observations, for example) requires multimodal, multiphysics models. And such models are really expensive relative to the insights they deliver.

To me, this means the challenge isn't in getting the world's scientific data to hold hands and sing kumbaya; it's accepting that there's limited value in actually doing this data fusion unless you're willing to also take on more approximations within the models that use them so that the net return--science per dollar--comes out as a net positive over today's physics-based, single-mode scientific models.

The AI community accepts that wholly empirical models are much less interpretable but can much more readily turn multimodal data into results in a meaningfully faster, most resource-efficient way. for example the Aurora model and how it took all sorts of disparate climate datasets to develop an incredibly efficient forecasting tool. In a minute on a single GPU, it produces forecasts of comparable quality to what would take hours across multiple GPUs using a physics-based model. And it achieves this efficiency by having trained on a diverse collection of gridded 3D atmosphere data and tabular data that was fused.

The only problem, of course, is that the model is much less interpretable than a physics-based model. If the Aurora model's forecast is off, forecasters mostly have to shrug and move on with life. But for the purposes of solving the scientific problem at hand (predicting the weather a few days out), that may be good enough.

Governments must now treat advanced computing as a strategic utility, requiring a scale of coordination and investment that rivals the Manhattan Project or the Apollo program.

Manhattan Project and the Apollo mission had distinct goals with a defined "lump of work" required to achieve them. They are not comparable. Computing is a commodity, and it’s a far fairer comparison to liken it to oil or gas reserves. And even then, exactly what good are these computing reserves or capabilities really? Is it one big supercomputer, or many small ones? What are the range of problems that such a strategic utility would be called upon to solve?

In the AI game, advanced computing is certainly a pillar of competitiveness, but it is not necessarily the most limiting one. DeepSeek showed us that ingenuity and massive computing are two orthogonal axes towards developing new capabilities. They showed that, although you can spend a ton of money on GPUs to train a new frontier model, you can also be a lot more clever about how you use much fewer GPUs to do the same thing. And the ratio of people to capital that resulted in DeepSeek-R1 arguably showed that investing in innovation, not just datacenter buildout, has a much higher return on investment.

In the context of the above statement, I think governments would do far better to treat its innovators as a strategic asset and worry less about issuing press releases that lead with how many thousands of GPUs they will deploy. For every thousand GPUs to be deployed on government land in the US this year, how many government researchers, architects, and visionaries have headed out the door and are never coming back?

I am talking to teams where CI is a load-bearing piece of infrastructure. Teams where 20 or 50 or 200 engineers push code daily. Teams where a broken CI pipeline doesn’t mean one person waits a few extra minutes; it means a queue of pull requests backs up, a deploy window gets missed, and product timelines slip. Teams where CI time is measured in engineering-hours-lost-per-week and has a line item on somebody’s OKRs.

It also leans heavily on one of my favorite papers, “Build systems à la carte” by Mokhov et al. From the discussion of that paper:

The real takeaway is not that bash is bad. It’s that the design space of build systems has structure, and that structure has been studied, and that the properties you care about (minimality, correctness, support for dynamic dependencies, cloud caching, early cutoff) correspond to specific architectural choices that live at a level of abstraction bash cannot express. When you write a build pipeline in bash, you are either implementing one of the twelve cells in the Mokhov-Mitchell-Jones matrix (poorly, by hand, with strings and exit codes), or you are living in the busy cell and rebuilding everything every time.

It’s a long read but a good one, go check it out.

“Operations” is not a dirty word, a synonym for toil, or a title for people who can’t write code. May those who shit on ops get the operational outcomes they deserve.

You should absolutely go read the full piece, as well as Charity’s earlier post on the Honeycomb blog: You had one job: Why twenty years of DevOps has failed to do it.

Below find several pull quotes from the post itself, because there were just too many to choose from.

The difference between “dev” and “ops” is not about whether or not you can write code. Dude, it’s 2026: everyone writes software.

The difference between dev and ops is a separation of concerns.

The hardest technical challenges and the long, stubborn tail of intractable problems have always been on the infrastructure side. That’s why we work so hard to try not to have them—to solve them by partnerships, cloud computing, open source, etc. Anything is better than trying to build them again, starting over from scratch. We know the cost of new code in our bones.

As I have said a thousand times: the closer you get to laying bits down on disk, the more conservative (and afraid) you should be.

The difference between dev and ops isn’t about writing code or not. But there are differences. In perspective, priorities, and (often) temperament.

I touched on a number of these in the article I just wrote on feedback loops, so I’m not going to repeat myself here.

The biggest difference I did not mention is that they have different relationships with resources and definitions of success.

Infrastructure is a cost center. You aren’t going to make more money if you give ten laptops to everyone in your company, and you aren’t going to make more money by over-spending on infrastructure, either. Great operations engineers and architects never forget that cost is a first class citizen of their engineering decisions.

Operational rigor and excellence are not, how shall I say this…not yet something you can take for granted in the tech industry. The most striking thing about the 2025 DORA report was that the majority of companies report that AI is just adding more chaos to a system already defined by chaos. In other words, most companies are bad at ops.

Because when the people training these models justify why they’re worth it, they appeal to pretty extreme outcomes. When Dario Amodei wrote his essay Machines of Loving Grace, he wrote that he sees the benefits as being extraordinary: “Reliable prevention and treatment of nearly all natural infectious disease … Elimination of most cancer … Prevention of Alzheimer’s … Improved treatment of most other ailments … Doubling of the human lifespan.” These are the benefits that the CEO of Anthropic uses to justify his belief that LLMs are worth it. If you think that these risks sound fanciful, then I might encourage you to consider what benefits you see LLMs as bringing, and then consider if you think the risks are worth it.

From Carlini’s recent talk/article on Are large language models worth it?

The entire article is well worth reading, but I was struck by this bit near the end. LLM researchers often dismiss (some of) the risks of these models as fanciful. But many of the benefits touted by the labs sound just as fanciful!

When we’re evaluating the worth of this research, it’s a good idea to be consistent about how realistic — or how “galaxy brain” — you want to be, with both risks and benefits.

The trick is to read plainly.

The key word in Artificial General Intelligence is General. That’s the word that makes this AI unlike every other AI: because every other AI was trained for a particular purpose and, & even if it achieved it in spectacular fashion, did not do anything else. Consider landmark models across the decades: the Mark I Perceptron, LeNet, AlexNet, AlphaGo, AlphaFold … these systems were all different, but all alike in this way.

Language models were trained for a purpose, too … but, surprise: the mechanism & scale of that training did something new: opened a wormhole, through which a vast field of action & response could be reached. Towering libraries of human writing, drawn together across time & space, all the dumb reasons for it … that’s rich fuel, if you can hold it all in your head.

It’s important to emphasize that the open-ended capability of these big models was a genuine surprise, even to their custodians. Once understood, the opportunity was quickly grasped … but the magnitude of that initial whoa?! is still ringing the bell of this century.

I’m extreme in this regard: I think 2020’s Language Models are Few-Shot Learners marks the AGI moment. In that paper, OpenAI researchers demonstrated that GPT-3 — at that time, the biggest model of its kind ever trained — performed better on a wide range of linguistic tasks than models trained for those tasks specifically. A more direct title might have been: This Thing Can Do It All?!

“AGI” is such a misused, ill-defined term that I honestly don’t find it too useful… but it’s hard to argue with Sloan’s argument here! Certainly if you showed current LLMs to someone from 20 years ago, or even 10, they’d seem like wild science fiction.

It also reminds me of a quote from Asimov on the definition of “artificial intelligence” and how the goal posts move as new achievements are retrospectively deemed as “not AI”:

[artificial intelligence is] a phrase that we use for any device that does things which, in the past, we have associated only with human intelligence

(via Nicholas Carlini)

So. Do we have AGI? Do we even meaningfully have AI? What would we have to see for the general consensus to agree they had been achieved?

Anyway, they are mostly marketing terms at this point. But it can still be interesting to think about them.

Thoughts from a dog walk listening to the Sloan article using ElevenReader.

Services I run over Tailscale, just for myself, include:

• An RSS feed reader
• A personal git forge
• An IRC bouncer
• A (poorly maintained) wiki
• JupyterLab
• Open WebUI for playing with local LLMs on a GPU workstation
• SSH to a powerful workstation, hosted at home but without complex configs

And probably a few things I’ve forgotten! It’s really just very neat. Sure I could do it all with manual Wireguard configs. But Tailscale just makes the underlying primitive much more ergonomic.

Anyway, back to programming. I have a single suggestion for you, my friend. Whatever you believe about what the Right Thing should be, you can't control it by refusing what is happening right now. Skipping AI is not going to help you or your career. Think about it. Test these new tools, with care, with weeks of work, not in a five minutes test where you can just reinforce your own beliefs. Find a way to multiply yourself, and if it does not work for you, try again every few months.

Yes, maybe you think that you worked so hard to learn coding, and now machines are doing it for you. But what was the fire inside you, when you coded till night to see your project working? It was building. And now you can build more and better, if you find your way to use AI effectively. The fun is still there, untouched

From Don’t fall into the anti-AI hype

In particular, I appreciate the detailed discussion of using different profilers and performance-analysis tools at different levels to determine how to optimize scheduling to improve two key goals: providing high average FPS while keeping 99th-percentile FPS as low as possible, e.g. to prevent UI stuttering. Optimizing for battery usage is also important, as the Steam Deck was one of the main targets for this work.

The key finding that came out of his analysis is perhaps somewhat obvious: a single high-level action, such as moving a character on-screen and emitting a sound based on a key-press event, requires that many tasks work together. Some of the tasks are threads in the game process, but others are not because they are in the game engine, kernel, and device drivers; there are often 20 or 30 tasks in a chain that all need to collaborate. Finding tasks with a high waker or wakee frequency and prioritizing them is the basis of the LAVD scheduling policy.

As always with LWN there’s good coverage not only of the talk itself, but also the Q&A following the session and ideas from the audience on tooling and other improvements.

Phoronix also covered a different talk from the same conference (I think) on how Meta is using the LAVD scheduler as the basis for a new default scheduler used on their fleet.

I haven’t had a chance to watch this talk yet (video linked from the article) but I’m very interested in the idea that the same concepts might be useful to a hyper scaler as well as a device like a Steam Deck.

At the IBM booth, we showcased how IBM LSF can schedule and orchestrate a hybrid quantum–classical workflow across IBM Quantum systems and classical x86 compute. The demo featured the Sample-based Quantum Diagonalization (SQD) workflow, to estimate the ground-state energy of a Hamiltonian representing a molecular system. SQD is part of the IBM Qiskit add-ons.

Before diving into the details on what was demonstrated at SC25, and how LSF was used to manage the workflow, I would like to acknowledge that this work was supported by the Hartree Center for Digital Innovation, a collaboration between UKRI-STFC and IBM. The demonstration was created in close collaboration with Vadim Elisseev and Ritesh Krishna from IBM Research, alongside Gábor Samu and Michael Spriggs from IBM. Additionally, this post does not aim to provide an in-depth look at SQD itself. Rather the focus is on how LSF can manage hybrid quantum-classical workflows across a heterogeneous environment comprised of both quantum and classical resources.

Hybrid workflows are not new

For three decades, we have seen the use of accelerators in HPC to drive performance—from GPUs to FPGAs and other specialized architectures. Effective scheduling of tasks in these heterogeneous environments has always been a key consideration for efficiency, scalability—and to maximize the ROI in commercial HPC environments. As resource topologies grow more complex, scheduling must account for characteristics such as connectivity, latency, and dependency constraints across increasingly diverse infrastructures. Quantum Processors (QPUs) are now making their appearance as complementary resources within HPC workflows, aim at challenges such as specific optimization problems, many-body physics and quantum chemistry.

Demo details

The IBM LSF cluster was deployed on IBM Cloud using the LSF Deployable Architecture, which rapidly deploys and configures a ready-to-use HPC environment. IBM Research provided integration components for LSF in the form of esub and jobstarter scripts. These scripts enable LSF to query the cloud-based IBM Quantum Platform to determine which QPUs are available for a given user account and meet the qubit requirements specified at job submission. The list of eligible QPUs is then sorted by queue length, and the system with the shortest queue is selected as the target for the quantum circuit. These integration scripts (esub and jobstarter) are intended to be made open source at a later time.

The LSF environment was deployed on IBM Cloud using the LSF Deployable Architecture v3.1.0:

• LSF 10.1.0.15
• RHEL 8.10
• IBM Cloud profile bx2-16x64 (compute hosts)

The IBM Qiskit package versions used:

• qiskit v2.2.1
• qiskit-addon-sqd v0.12.0
• qiskit-ibm-runtime v0.43.0

The SQD Python program is available as part of the IBM Qiskit Add-ons (see details here). For this demonstration, the original monolithic SQD script was refactored into four smaller Python programs—each representing a distinct step in the workflow. These steps map directly to LSF jobs, enabling orchestration of the workflow across the quantum and classical HPC resources as shown in the architecture diagram (Figure 1):

• Stage 1 map the inputs to a quantum problem.
• Stage 2 optimizes the problem for quantum hardware execution—this is where the circuit is transpiled and optimized for the target QPU
• Stage 3 executes the circuit on the QPU using Qiskit primitives
• Stage 4 performs post-processing and returns the result in the desired classical format

For this demonstration, we used IBM LSF Application Center—a web-based interface for job submission and management. LSF Application Center supports application templates, which simplify job submission by providing predefined forms. Templates were created for both the SQD workflow and the Jupyter Notebook application, which is used to visualize the workflow results.

Demo execution steps

• We start by using the SQD template to submit an instance of the SQD workflow (Figure 2) which is used to calculate an approximate ground-energy state of the nitrogen molecule (N2). The submission form is customized to let users specify the script for each step of the workflow and specify the desired number of qubits required on the QPU for the quantum circuit. This parameter is used by LSF to select the appropriate quantum system from the available resources. Note that jobs are submitted to LSF with a done dependency condition, ensuring that each stage runs only after the previous one completes successfully. Stage 2 begins after Stage 1, Stage 3 follows Stage 2, and Stage 4 executes once Stage 3 has finished

• Next, we submit an instance of the Jupyter Notebook to monitor the workflow initiated in Step 1. This notebook is designed for this demonstration to visualize the status of each workflow step, displaying results as they successfully complete. Figure 3 shows the Jupyter submission form.

• The Workload view in the LSF Application Center can be used to monitor the progress of each job within the workflow. Additionally, the Jupyter Notebook instance can be accessed here via the provided hyperlink. Figure 4 shows the workload view in LSF Application Center. This shows a list of jobs in the LSF system.

• As each stage of the SQD workflow completes, the Jupyter Notebook displays the corresponding output in new browser tabs. This includes qubit coupling maps for the QPUs available on the IBM Quantum Platform for the specific account, a diagram of the circuit mapped to the selected QPU, readings from the QPU, and a plot of the estimated ground-state energy of the N2 molecule.

• Given that demo environment was built using the LSF Deployable Architecture, IBM Cloud Monitoring is automatically configured. It provides a dashboard for the underlying cloud infrastructure, including detailed hardware metrics. In addition, an LSF Dashboard is available through IBM Cloud Monitoring, showing overall cluster metrics such as total jobs, job status, and queue distribution, along with scheduler performance trends over time. IBM Cloud Monitoring infrastructure view and LSF dashboard are shown in Figure 5.

A video recording of the end-to-end demonstration can be found here.

Conclusions

This demo marked a milestone by demonstrating that IBM Spectrum LSF can seamlessly orchestrate quantum and classical compute resources for a unified workflow. This example demonstrates a practical approach to integrating quantum capabilities into an existing HPC environment running IBM LSF.

This capability lays the foundation for hybrid computing pipelines that integrate emerging quantum hardware into established HPC environments. As organizations adopt these architectures and tools mature, we can expect production-grade workflows tackling complex problems across domains. The future of HPC is not a choice between classical or quantum—it is their convergence, working together to unlock new computational possibilities.

The topic of scheduling for hybrid quantum-classical environments will be the subject of an upcoming paper “On Topological Aspects of Workflows Scheduling on Hybrid Quantum - High Performance Computing Systems” by Vadim Elisseev, Ritesh Krishna, Vasileios Kalantzis, M. Emre Sahin and Gábor Samu.