The important theme here is that I think they’re best for tasks where the work is expensive but the validation is cheap and easy.

Prose writing

I don’t use LLMs to draft any kind of prose writing, whether for work documents like design documents or email, or for the occasional fiction writing I do (i.e., notes for an RPG campaign).

I generally write for two main purposes:

• Because I’m using the writing to structure my own thoughts on a topic and help evolve them
• Because I want to then communicate those thoughts to others

Neither of those purposes are improved by having a machine write for me! As Leslie Lamport has said:

If you’re thinking without writing, you only think you’re thinking.

The only place where LLMs have been helpful in my writing process is for limited proofreading help. I never use them to directly revise or create new text, but I’ve found them helpful when I ask things like “read this document and identify any inconsistencies or gaps that may need to be addressed”.

Sometimes they flag so-called problems I intended to put there, but often they find a case where I revised an argument and then forgot to update a paragraph further back. That’s the kind of thing that’s nice to catch before I share with others.

In any case, it’s worth noting that I never use LLMs for blogging or social media. It would certainly defeat the purpose of “thinking out loud”! If anything here seems half-baked, well, I’m the only baker.

Code

I use LLMs for limited code generation purposes, especially for relatively mechanical code.

I generally don’t open a coding harness until I already have some idea of my planned architecture, technology choices, and the implementation steps needed to get there. I get really suspicious of “vibe coding” where someone starts with a really limited prompt like “give me a service that exposes this database over HTTP”, giving the LLM too much discretion over architecture.

However, once I have a design and implementation plan, I’m perfectly happy letting the LLM write a bunch of mechanical functions calling REST APIs, parsing JSON, or arguing with the database. Mentally I class this as an evolution from past code generation tools like protobuf generators. A matter of degree rather than kind, at least from a workflow perspective.

I will say even for this, LLMs still need close supervision and review. Unlike a conventional code generator, they try to guess what else you might want and do extra work… which is where they often go wrong.

Data visualization

This is sort of a subset of coding, but highest-leverage use I get from LLMs is honestly doing dataviz in Jupyter notebooks. I find writing matplotlib scripts especially tedious, and it’s honestly incredibly helpful to just say “create a histogram of this dataset with x bin increment and annotations at the median and p95 values”.

Importantly, I never delegate the analysis process to the LLM or ask it to fetch datasets from elsewhere. I really only trust it to make graphs.

Current LLMs don’t seem to hallucinate or make wild mistakes at this kind of task, but also it’s a task that is quite easy to check for accuracy.

Search

I use conventional search engines (DuckDuckGo or Google) for most everyday queries, where I’m confident I can find a single-digit number of helpful results in the first few pages of links. A lot of my everyday queries look like “python datetime documentation” or “babylon 5 streaming service”.

On the other hand, I use LLMs connected to search tools for “needle in a haystack” queries. Or really, “find and connect 10 different needles”.

For example, if I would normally expect to spend 30+ minutes fiddling with my search engine keywords and reading lots of results in detail… this is a perfect case for delegating an LLM to go run those searches for me and sift through the results. This has been especially useful for me in a work context searching through large document databases.

A typical LLM “search” for me might be: “I’m looking for the list of supported OSFP transceivers compatible with network card X. There doesn’t seem to be a clear document that lists these in detail but I’ve seen this kind of data scattered amongst several slide decks from the sales team in the past. These might be stored in either Google Drive or Sharepoint, and you probably need to check both. If you find multiple conflicting sources I want information and links from both.”

Crucially, this is the kind of search that is expensive to run but cheap to check. If the LLM comes back to me with two slide decks and a spreadsheet, it’s pretty straightforward for me to look at those and check if they have the information I want. And it’s much, much faster than finding and reading 50 different documents myself. Not to mention less tedious.

These things are not for everyone!

Computing, and especially software development, are really in the sweet spot for applicability for this tech. We construct our work using a tool that looks a lot like language, but is nevertheless machine-checkable and more functional than expressive. There’s a reason so many programmers use this tech — it’s aimed squarely at our work!

I can’t imagine LLMs being terribly useful for any kind of author or artist, except in some analytical sense. For example, Charlie Stross comments in a post on his non-use of LLMs that:

I’d quite like a tool (running entirely locally on my own hardware, with no cloud service and no copyright-thieving grifters making bank on it via subscription fees) that digests a manuscript and derives a scene-by-scene timeline, that I could then query interactively and use to plan my next round of edits. Being able to map out where and when each protagonist and minor character shows up, and see a frequency distribution heat map of names in the manuscript, would be useful.

But such a tool would be useful to me in the same way a spelling checker is useful—as a decision-support tool, not as a substitute for doing the hard work (and having a copy of the Oxford English Dictionary on the shelf). The value of such a tool is considerably less than the value of a well-trained brain that can do the entire job the hard way, if necessary.

I’m also skeptical that LLMs will work well in the near future in fields like law, where much of the work is language-based but the work is not machine-checkable to any reliable degree. Especially when there are high consequences for mistakes.

Ethics

Plenty of folks’ objection to LLMs isn’t functional at all, of course, but ethical. Even in cases where these tools are useful, they’d argue they shouldn’t be used.

(Internet commenters are wont to lump it all together, but eh, I’m not overly bothered by sloppy arguments on social media.)

I have more sympathy with some of these arguments than others, and I hardly think this is a simple matter. But I do think actual solutions are going to be complex and fact-specific, and I think addressing them is going to be more a matter of policy wonkery than moral imperatives.

Resource usage is, to my mind, a real concern — but one with so much variation that it’s hard to say anything in a blanket fashion.

I know way too much about datacenters to think they’re all alike, even the ones running GPU clusters. Each facility is different in terms of power, cooling, efficiency, and local impact. A datacenter running off the local grid and paying for infrastructure upgrades is very different from xAI running its cluster using gas-fired peaker plants and ignoring permitting rules.

I’m actually not over-bothered by datacenter moratoriums! I don’t think the industry will be damaged all that much by having to slow down. But I see them as a method to buy time to write real regulations around infrastructure build-outs, cost sharing, local pollution and noise regs, and permitting approvals. Not as a realistic way to limit the growth of the “AI” industry in the long term.

Copyright is an area where I’m honestly more on the side of the model trainers than not. I remain an old school copyright-minimalist remix-culture techie of the early-2000s school, and I struggle to see any principled argument that model training isn’t transformative fair use.

I do, however, think that the big labs should be held accountable for non-transformative distribution-based enforcement. For example, when Anthropic straight-up pirated ebooks for training data, I would have liked to see them pay the statutory damages of $30,000 per work, and not settle for a tenth of that.

And if LLMs are found to directly distribute copyrighted content — e.g., regurgitating substantial segments of books or news articles — I think they should face similar damages for that. Again, perfectly straightforward infringement.

Concentration of wealth and power is the area where I have the most genuine concern, and I suspect this is what actually drives a lot of folks’ concern around both copyright and resource usage. It makes me really nervous to see how concentrated the economy is getting in a few companies, and the existence of fascist billionaires with great political power is a travesty to be combated by anyone who believes in democracy.

That said, I don’t think copyright or zoning laws are going to prove effective tools to combat these folks, though I understand reaching for the tools you’ve got. I’m much more in favor of anti-corruption laws, antitrust, and confiscatory tax regimes. And all of the above seem equally unlikely to make much progress in the near future, to all our detriment.

In any case, even if all the so-called AI labs went out of business today, the models that exist today are going to continue to do so, and are sufficient for all my purposes. So I almost exclusively use open-weights models and preferably run on my own hardware that lives in my own spaces.

This also helps mitigate climate and resource concerns, insofar as I’m not running any gas-fired peaker plants to power my house! Though from a strict per-query power perspective, my local setup is less efficient since I’m not batching queries or sharing infra with anyone else.

This balance is a double-edged sword though, and I found myself spread thin amidst trying to attend conference sessions, catching up with colleagues, and supporting the business™. I didn't see as much of the program as I'd have liked, but a few noteworthy themes still stood out to me.

Foremost, HPC has moved to the final stage of grief with respect to AI: acceptance. The "us vs. them" mentality of past conferences is largely gone, replaced now with a lingering question of purpose. What does the HPC community want to be now that it cannot define itself by having the biggest GPU clusters? What is on the horizon for scientific computing that we've been ignoring because of AI? And how do we work with AI technologies to get there faster?

Nobody had answers to these questions, and I'm not even sure attendees realized they were being asked. But this uncertainty was pervasive, resulting in a technical program that felt scattered. There was a bit of optimism which took the form of more mature, deep discourse around the ways in which AI can accelerate discovery. At the same time, many sessions retreated to the same old safe and comfortable topics; many of the same old broad challenges and opportunities (like non-von Neumann architectures and disaggregated memory) kept popping up throughout the week.

There was one big surprise during the week though--that of the new Chinese, all-CPU LineShine supercomputer appearing at the top of the Top500 list--and it was an amazing distillation of the uncertainties across the HPC community. It forces us to think about a wide range of existential questions:

• Is it OK to care about HPC more than AI? LineShine is an FP64-first system that is not trying to masquerade as an AI machine.
• Does leadership in HPC necessarily require accelerators and heterogeneity? LineShine is a CPU-only, homogeneous architecture.
• How do we reconcile the HPC community's history of openness and collaboration with the demands of sovereignty, supply chain competition, and the optics of leadership?

Beyond LineShine, China also flexed its sophistication with a new all-flash parallel file system developed by Sugon and debuted at the top of the IO500, beating out Argonne's massive DAOS system. Though it didn’t get the same headline coverage as compute, this storage system indicates that China is no longer trying to simply fill gaps left by export controls; it is building its own best-in-class HPC hardware and software stack for its own domestic uses.

As always, it's hard for me to find the best way to structure a post that captures both these broad themes and all the little interesting bits across the week. So I figure I’d just jump around the interesting things I saw in no particular order.

• LineShine!
  • LineShine's mostly-Chinese architecture
  • Chinese cores
  • Chinese memory
  • Chinese networking
  • China wants to make the HPC world a better place
  • The geopolitical message
• Not caring about AI is OK
• But pretending to care about AI remains pervasive
  • Misconception 1: AI is just going to wait for hardware to solve their problems
  • Misconception 2: AI for HPC is a singular area of research
• Open or closed?
  • AI doesn't want free
  • And HPC isn't really that open
• Sovereign is the next big thing for everyone (except the US)
  • Sovereignty is forcing modernization
  • Sovereignty is an opportunity
• CPUs are back...maybe?
• Storage stuff
  • ParaStor
  • Only bad AI requires many IOPS
• Maybe HPC and AI really are different
• Take-away themes
• Appendix: The personal stuff
  • ISC is two conferences
  • The makings of a keynote
  • Dumb things to avoid at ISC

LineShine!

The most exciting news of the week for me was China's [[LineShine]] supercomputer, which debuted at the top of the Top500. It was interesting for several reasons:

1. It is homogeneous and uses Arm CPUs exclusively to achieve its FP64 performance. This opens up the possibility that GPUs aren't the only path forward.
2. It uses components that are all Chinese-designed, and many of them appear to be Chinese-fabricated too. The blacklisting of Chinese supercomputing centers that prevents them from buying US-designed chips appears to have succeeded in forcing China to develop its own supercomputing supply chain that is (in many ways) at least as good as what the rest of the world can design and deploy.
3. China has notably abstained from submitting to the Top500 since 2016, and they came back this year with a message of peace and love.

Here’s what I gathered throughout the week about the system and its implications.

LineShine's mostly-Chinese architecture

The basic LineShine hardware architecture has been covered in the media for a few months now, as some of the Gordon Bell prize submissions performed on this system appeared on arxiv a few months ago. There are a few interesting technical details to infer from these sources which I've detailed in my Digital Garden's LineShine page, but let's talk about why they might matter based on the slides that Yutong Lu (卢宇彤) shared at the Top500 session.

Chinese cores

LineShine's LX2 processors are dual-die, with each die having its own I/O chiplet, core cluster chiplets, HBM controllers, DDR controllers, and 800G NIC. Given its nominal frequency of 1.55 GHz and 304 cores per socket, its 60.3 TFLOPS FP64 with ARMv9's matrix extensions (SME) cleanly resolves into each core being able to do 128 FLOPS (or 64 FMACs) per cycle. This works out to 512-bit vectors, which was supported by a cartoon showed by Lu.

Unlike GPUs, which use a bunch of threads to cooperatively compute a matrix multiplication, SME gives each thread its own “ZA tile register” to fill up using vector-vector outer products. This makes SME a little more forgiving when matrices are small and there isn't as much parallelism, but I'd imagine it also makes it less energy-efficient than a tensor/matrix core on a per-FLOP basis. Good for weirdly shaped HPC problems, but perhaps less so for huge transformer tensors.

The most interesting thing about LX2 is what wasn't said though--who fabricated it, and what lithography was applied. Given its chiplet architecture, some part of the manufacturing or integration may rely on foreign fabs.

Chinese memory

LineShine attaches HBM and DDR to each CPU, and it allows the HBM to act as a huge cache for the DDR through dedicated DMA engines. This seems conceptually similar to how Knight's Landing CPUs could run their MCDRAM in cache mode, which turned out to be a very easy and effective way to get its memory bandwidth benefits without burdening programmers.

The size (32 GB) and performance (4 TB/s) of each LX2's eight HBM stacks suggests it's older HBM2e, which happens to be the standard that CXMT (the Chinese memory fabricator) may have begun shipping in 2025. Seeing as how LineShine came online at the end of 2025, this adds up.

LineShine also uses some unidentified form of DDR, though one of the early papers benchmarked it at 125 GB/s per die. Given each die has four memory controllers and NUMA domains, this matches up with DDR5: either domestically produced DDR5-4800(ish) with four channels, or foreign-produced DDR5-8000.

Like the CPU lithography and HBM spec, the exact DDR specification has been explicitly omitted, with one LineShine paper giving plenty of detail about a different Chinese exascale system (having 8-channel DDR5-6400), but LineShine simply having "DDR memory" in the next paragraph.

Chinese networking

LineShine's LingQi network is also Chinese-made. Its topology is cost-optimized and likely build on 32-port 100G switches, well within the capability of Chinese manufacturing. Each compute node is advertised as having 1.6 Tbit/s of injection bandwidth, but Lu shared a network diagram that clarifies that this is really two (likely independent) 800G injection ports into independent planes:

The only way I could get the math to work with Lu’s network diagram and constraints are if the network looks like this:

• Each compute node is attached to its own switch(!), and those leaf switches have non-blocking connectivity to in-rack (actually in-"frame") spines. This is all copper.
• Each compute node frame uses optical to connect into dedicated network frames with a 2:1 taper. These network frames implement an all-copper L3 and L4 switching layer with a 3.81:1 taper.

The most likely exact configuration results in 32-port, 100G switches where

• L1 has 2,944 switches with all-copper up and down (8D, 8U)
• L2 has 1,472 switches with copper down, optical up and a 2:1 taper to minimize optics (16D, 8U)
• L3 has 512 switches with optical down, copper up, and a 23:6 taper (23D, 6U). This feels weird, though. Maybe L3 uses a different switch radix?
• L4 has 192 switches with copper down (16D)

The end result is a steep 7.67:1 taper but with a very low use of optics. It's also all achievable using 32x100G switching ASICs which should be well within the capabilities of Chinese domestic foundries. I don't know how good Chinese transceivers are, but the choice to minimize optics could have been driven by both cost and minimal reliance on foreign-made parts.

This taper is also good enough to run HPL. Since this is an all-CPU system, there is no pesky host-device memcpy in the loop to drag down the HPL efficiency. They used OpenMPI with a custom driver for this LQLink interconnect, and its collectives are almost certainly aware of the high locality effects of staying within the nonblocking L2 domain.

China wants to make the HPC world a better place

One of Lu's concluding statements was that LineShine is open for international collaboration. And while I would normally be cynical about such an empty statement, I spoke with a non-Chinese researcher who was already running on LineShine and--much to my surprise--was finding it productive.

Lu and others were also openly bragging that they have authored 14 Gordon Bell submissions for SC'26 based on work performed on LineShine, and three of those papers are Gordon Bell finalists. By all accounts, the system works, and China wants the world to judge LineShine for themselves.

The geopolitical message

James Lin (林新华), the vice director of Shanghai Jiao Tong University’s HPC center, gave a much more unvarnished perspective on LineShine's Top500 entry later in the week, where he presented what I believe to be the best presentation of the conference.

He led with this magnanimous slide:

In his eyes, China breaking its ten-year absence on Top500 is good for the entire HPC community worldwide:

1. It brings international recognition to China for its technical achievements - this is undebatable.
2. It restores credibility to Top500 - mostly true. There are still giant AI systems missing from the list, but if we accept that Top500 is really about the best supercomputers for modeling and simulation, this is 100% true.
3. It will cause the US to panic at visibly losing supremacy, bringing more HPC funding to the U.S. Department of Energy. Absent a crystal ball, I would wager this is absolutely true. And the timing to debut this system right before the US’s big semiquincentennial birthday bash makes it sting extra.

Lin went on to present another slide that put to words exactly what everyone has been thinking for the past decade: the US had essentially turned the Top500 list into a hitlist.

Any Chinese supercomputer center that dared to list on the Top500 was banned from being allowed to buy American CPUs or GPUs.

China’s reappearance at the top of the Top500 list sends a clear message that this no longer matters; being banned from buying US technologies has been ineffective in stopping China from achieving world-leading computational capability. China was able to design, fabricate, and deploy the world's fastest supercomputer for simulation, without any weird accelerators, using a hardware stack and supply chain that is predominantly domestic. No other country can claim that, not even the US.

However, what was understated is how much open-source software (derived from US-funded work, no less!) was used to turn a pile of Chinese-made silicon into a functional supercomputer. The software stack appears to rely on a mix of open-source software like PyTorch, OpenBLAS, and OpenMPI (the latter two being used for the HPL run) and Huawei software (KML BLAS, Hyper MPI). Similarly, the Chinese-developed network appears to expose a UCX provider, an open interface whose original founders include (surprise!) U.S. Department of Energy.

In bringing this message to ISC, China is painting a picture for the HPC community about the spirit of collaboration in furtherance of science: while the US government has been banning exports of its hardware, banning the use of its AI models by foreign nationals, and generally rolling back commitments international collaborations, China is throwing open its doors and inviting all comers to try out its home-grown technologies.

Of course, I don't know whether China's intentions are true or if they're just using the current political climate as an performative opportunity to show the greatest contrast with the US on an international stage. But to a non-political scientist, the optics of "we want to collaborate even if it brings us punishment" are hard to dismiss outright.

I'm also not sure who all is included in "China" here. The New York Times reported,

Dr. Dongarra, who wrote a detailed report on the new system, was told while visiting China that the system had been made without government funding, so the designers felt it was permissible to submit tests for the Top500 ranking, he said.

There was no mention at ISC of who sponsored the design and manufacture of this machine if not the Chinese government. LineShine's LX2 processors, despite being Arm-based, are unlikely to appear in any other systems given how bespoke some of its features (like the on-package LQLink NIC) are. In addition, the story I heard is that the designers had to fight with Chinese bureaucrats to allow the HPL results to be submitted to Top500.

This tells me that the voices of Lu and Lin may reflect that of the HPC community in China rather than the Chinese government; they are not necessarily one and the same. The researchers may genuinely want to collaborate internationally even if the government does not. And mind you, this is not unique to China; a US colleague intimated that everything his institution publishes is scrutinized by a government watchdog. Alarm bells ring if papers written with any hint of collaboration with Chinese research centers.

That all said, one announcement at ISC gives me a pang of doubt that China’s relationship with the HPC community may really change: the Top500 is being transitioned to ACM SIGHPC under the governance of representatives from the US, Japan, Europe, the Middle East, and Japan--but not China. As such, China remains as much an outsider going forward as in the past.

Not caring about AI is OK

This ISC felt like the first time that the conference program has come to accept the fact that HPC and AI, despite some overlap of technologies, are not the same thing.

There was no shortage of sessions about AI, agents, and the usual rigamarole. But there were also sessions that were not about AI as well. In fact, the conference opened with a keynote that was about as indifferent to AI as a keynote could be in this age.

Martin Schulz gave the conference opening keynote titled "HPC: A Heterogeneous Future," and he managed to say "AI" only ten times across his entire 40-minute presentation--and nine of those were within the first three minutes. He even managed to avoid saying "AI" for two minutes on a three-minute slide titled "AI Has Shifted the Center of Gravity." Amazing.

Given that I said "AI" more times at breakfast than the conference's entire opening keynote, I got the strong message that ISC is an HPC conference, not an AI conference, and it's OK if you don't want to talk about AI. So what did Schulz talk about? Quantum computing.

Schulz's keynote was quite broad, and came away with two (perhaps unintentional) realizations:

1. Quantum computing seems to be less about high-performance computing and more about using weird computers to solve physics problems, which is also what motivates most (non-AI) HPC today. The challenges of quantum computing (hardware engineering, algorithms) overlap with many challenges of parallel computing, but I don't think quantum and HPC are as complementary as HPC-turned-quantum researchers try to make them. The fact that they are mashed together at conferences like ISC seem to reflect that the same kind of people like both HPC and quantum.
2. It seems pretty easy for smart people to quickly pivot from traditional HPC to quantum computing. I've known of Schulz because of his work on the MPI Forum, so I was expecting a keynote on something a little closer to that. Instead, he gave a survey of quantum computing research he's touched over the past few years, and it all sounded pretty impressive to me. So I guess if you are getting tired of parallel computing but don't care about AI, consider caring about quantum instead.

I was also glad to realize that I'm not the last HPC person to be a quantum expert. Rio Yokota, next year's ISC chair, admitted to having not studied quantum much during the closing session. This was echoed in private conversation by a couple long-time HPC folks who told me that they were using the conference as an opportunity to figure out where their gaps in understanding quantum are. If I was smarter, maybe I would've done the same. But it seems like I still have time, because there are still plenty of HPC experts in the community who haven't gotten on the quantum bandwagon yet.

On the topic of AI indifference though, the second day's morning plenary followed a similar pattern as the first. Amanda Randles spoke about digital twins which--despite having considerable backing by NVIDIA--are overwhelmingly built on physics-based models rather than empirical ones. Randles only said "AI" four times and "machine learning" six times in her entire 37-minute keynote, and by comparison, said "simulation" 16 times. Again, a strong message that not everything at ISC has to be about AI.

Finally, LineShine was probably the biggest proof point that everything doesn't have to be about AI. While it can do hardware-accelerated 16-bit math, it has no 8-bit support; it is very much a traditional 64-bit machine for modeling and simulation. Someone in China decided that spending hundreds of millions on a machine for physics simulation instead of AI training was worth the investment.

But pretending to care about AI remains pervasive

Just as HPC experts are now becoming quantum experts, many HPC experts are also pivoting to become AI experts (myself included!). And as I noticed at SC'25, the level of discourse around AI within the HPC community is rapidly maturing. Panels about how bad ChatGPT is at generating MPI code are no longer passing peer review; instead, their place was taken by talks about how to fine-tune models to generate better Fortran this year.

That said, the AI sessions I attended still felt like a step back from where I thought this community was based on the talks at SC'25 last year. Two misconceptions perpetuated throughout the week and are worth addressing:

Misconception 1: AI is just going to wait for hardware to solve their problems

I attended a panel on advanced memory architectures where a panelist's position was that future memory technologies for AI should be designed to reflect the fact that AI (inferencing...) is a workflow that has distinctly separate compute-bound and memory bandwidth-bound stages. Through a hardware lens, his argument was reasonable: memory bandwidth is the harder problem to solve, so if you design a node architecture around that problem rather than the compute, you can create a node that's optimal for more of the workflow.

The problem is that this hardware lens assumes that AI model developers just sit around waiting for hardware people to tape out new chips to solve their problems. This is not the case.

Although he didn't name it outright, the AI workflow that this panelist was describing is inferencing of transformer models, which is notoriously costly. Inferencing transformers is the reason why we hear about gigawatt datacenters being built for billions of dollars in the news. It follows that the entire AI industry has billions of dollars of financial incentives to solve this memory bandwidth problem more quickly than the time it takes design a new memory architecture from a blank sheet of paper.

Instead of solving this problem with hardware, the AI community is iterating on software and algorithms to reduce the burden of inferencing by designing non-transformer models that simply do not have such severe memory bandwidth problems. It took Intel seven quarters to go from formally announcing its HBM CPU at ISC'21 to shipping it to its first customers in 2023. By comparison, in the seven quarters that followed ChatGPT's release, the following fundamental inferencing algorithm advancements were made:

1. Speculative decoding came out, reducing the pressure on memory bandwidth
2. PagedAttention came out, reducing the pressure on HBM capacity
3. FlashAttention-2 came out, improving compute utilization
4. Prefill/decode disaggregation came out, improving compute and memory bandwidth utilization

To assume that memory bandwidth will always be the biggest challenge for AI reflects ignorance of the pace at which AI algorithms are advancing. Just as techniques like mixtures of experts dramatically reduced the HBM capacity required to inference transformer models a few years ago, we're seeing non-attention or linear attention methods rapidly replacing the memory-bandwidth-heavy quadratic attention that transformers have historically used.

Like HPC, AI tends to adapt to whatever hardware is widely available. Designing future memory architectures to solve today's problems might be a fun thought exercise, but anything too exotic or bespoke will be irrelevant by the time it's ready to run real workloads.

Misconception 2: AI for HPC is a singular area of research

There were a lot of talks about using AI in HPC across the main program and workshops, but many of them mistook "AI for science" to include using AI for any aspect of HPC or scientific computing. I don't think this is right; instead, there are really two orthogonal parts to AI in HPC:

1. AI for productivity: Using AI to allow researchers to do more of the same things faster, and
2. AI for science: Using AI to allow researchers to tackle new classes of problems that were previously intractable

I felt like AI for productivity got a disproportionate amount of attention at ISC this year. Many of those talks felt incremental or like claims of "me too" that just showed how to translate commercial AI productivity tools to HPC-specific work. For example, this slide on "key elements for an accelerated discovery loop" was shown at a BOF on agentic AI for HPC:

However, you could take this exact same slide and (excepting the "HPC ready" requirement) drop it into any domain--commercial, scientific, or whatever else--and it'd still be true.

I also sat in on on the 2nd LLM4HPC workshop on Friday, hoping to see how language models were improving science, only to find that the entire workshop was focused on using LLMs to generate scientific code or, at best, simulation input decks. On one hand, I get it: code generation is where the world is finding AI most useful today, and it only makes sense for scientific programmers to explore how tools like Claude Code can simplify their daily grind. But on the other hand, Claude Code is not AI for science. It is AI for productivity, cast in the context of people who support science.

AI for science is the area with much more opportunity for impact; these are cases where AI (whether they be surrogate models, agentic data exploration, or other techniques) turn intractable problems into solvable ones. AI for science involves new models that operate directly on numerical data or agents that analyze data rather than run parameter sweeps. And, unlike at SC25 last year, I just didn't see much of it at ISC. It felt like the majority of people presenting on AI were reaching for the low-hanging fruit of, essentially, how AI can automate scientific drudgery. This is useful, but it isn't terribly intellectual.

Open or closed?

An unexpected theme that I kept tripping over during the week was the growing tension between openness and closedness across the HPC ecosystem. While HPC and open source have historically gone together like peanut butter and jelly, a couple of interactions during the week made me reconsider whether this remains true.

AI doesn't want free

The first interaction happened off-site at one of my private meetings with a large HPC center, where I was told that my employer's proprietary nature was not desirable compared to our open-source competitors. This feedback didn't surprise me. But when I brought this up with one of my open-source competitors, I was surprised to hear that they get the exact opposite feedback: being open-source is undesirable, because nothing good is free. That is, while large HPC centers may prefer open source, industry (particularly enterprise AI) prefers proprietary because it "feels" like higher quality.

Since enterprises tend to pay premium prices for software and support, this creates a closed system of rapid advancement. Software sophistication can rapidly iterate because it's generating lots of revenue, allowing it to improve faster than open source. And because it's improving faster, it becomes more desirable, driving more revenue, and so on. The only way for open source to keep up is with a broad, committed base of developers. And in the HPC world, there simply aren't enough user-developers to maintain a level of software quality that can compete with the deep pockets of AI investors.

I haven't thought through how broadly this applies across the software used in HPC and AI, but it certainly is true in the storage world. If you look at where the most rapid and exciting advances are being made in storage for HPC and AI, it's not in open source; it's in proprietary file systems and object stores. AI companies may start with open-source Ceph or community Lustre, but they invariably graduate to proprietary systems when it becomes easier to solve their problems with money rather than time and engineering effort.

And HPC isn't really that open

At the LLM4HPC workshop on Friday, there were two talks that also made me realize that there is another systematic reason why HPC is being left out of the greatest advancements being realized by AI today: it's not actually as open, so frontier models and tools just aren't very good at doing HPC-specific work.

The first talk was straightforward in principle: create a set of coding and optimization tests that involved Fortran and porting OpenACC to OpenMP, then see how different open and closed models perform. And the result wasn't terribly surprising: today's leading LLMs are still pretty bad at writing Fortran.

What was unexpected is that open models optimized for code, which are becoming quite good at general programming, are significantly worse than proprietary frontier models like Claude Opus and GPT-5. And even worse, purpose-built models for Fortran coding were the worst of all. Why is this?

The latter finding (Fortran-specific models are junk) actually made perfect sense to me. The world's best AI model builders are not building Fortran-specific models, so conversely, a model built for Fortran is probably old, poorly trained, and not keeping up with the state of the art in training models.

But the former finding (open models are worse than frontier models, but all models are bad) was more interesting. It's generally accepted that larger models are better at cross-lingual transfer learning than smaller models. That is, a big model can see an algorithm written in C++ and figure out how to implement it in Fortran better than a smaller model when all else is equal. So, massive (proprietary) models will probably generate better Fortran than smaller (open) models even though they've both trained on the same amount of Fortran simply because they're better at generalizing to Fortran.

Unfortunately, much of the world's highest-quality Fortran code is not represented in these models' training corpora. That code is often proprietary and in private or air-gapped repositories because it is responsible for computations that inform defense or national security. Because of this, some of the best examples of HPC code are not very open at all.

The talk that followed was essentially a prescription of how to address this problem: how to fine-tune open models using non-public, HPC-specific source code. It was specifically describing a new method for reward modeling that allowed a model being fine-tuned to get the most out of every snippet of HPC-specific code it was shown, and the outcomes were amazingly good in some cases.

What I really took away from these talks, though, is that

1. The obscurity and "closedness" of HPC applications is why the HPC community doesn't get as many benefits from new coding models as, say, the Python developer community. The fact that much of the world's Fortran is locked up and not open source means the improvements we do see are the result of cross-lingual transfer learning, which is not very efficient compared to simply training a model on good Fortran.
2. Fine-tuning an open model by exposing it to proprietary, HPC-specific examples can turn a mediocre coding model that struggles with Fortran or OpenMP into a strong one. And because it's just fine-tuning, it's possible to get this HPC coding model with a limited amount of Fortran and a limited number of GPUs compared to creating a coding model from scratch, which remains too costly for all but the largest AI companies.

This leads me to envision a future where every large HPC site has its own fine-tuned coding model that has been post-trained on internal, closed-source HPC application source so that it has the general coding knowledge of a leading open model and the HPC-specific experience learned from examining proprietary code. Of course, this requires that each HPC site also have some expertise in fine-tuning their own coding models and integrating those models into popular agentic coding suites (like Claude Code) running on-prem. At present, these skills are too scarce for every center to hire, but perhaps in the future they become as commonplace as knowing how to run an MPI job is today.

What isn't clear to me is how much HPC centers will be willing to pay for the ability to fine-tune their own coding models. As described earlier, this is all squarely AI for productivity, not AI for science. When an HPC center's mission is to support science, what is the relative importance of using GPUs to generate scientific insights versus using GPUs to write boilerplate Fortran code faster?

Sovereign is the next big thing for everyone (except the US)

This idea of fine-tuning models on proprietary HPC code touches on another theme that seemed very big at ISC this year: sovereign AI infrastructure. And like any good conference buzzword, it wasn't immediately clear that many attendees really understood what it meant.

Andrew Jones posted the following cynical take on sovereign HPC infrastructure at the outset of the conference:

And he's not wrong; of the twenty sessions in the ISC technical program that self-identified as being related to "sovereignty in AI" in any way, 60% were vendor pitches. And as someone who works for a company selling AI technologies, I can confirm: every country buying their own GPU infrastructure instead of sharing it is great for business.

However, the single strongest argument for sovereign AI happened a week before ISC started on June 12: the US government declared that foreign nationals could no longer use Anthropic's most powerful models, Fable 5 and Mythos 5. While this was probably a mild annoyance to most of us who had just gotten started using it, one conversation I had during ISC made me realize how broad the implications were: the US government demonstrated that it could cut any nation off from all the AI models being produced by the world's leading AI companies.

Think: what would happen to you or your work if OpenAI, Anthropic, xAI, and every other frontier model lab simply disappeared tomorrow? It may be a mild annoyance to you, but how much time would you lose by having to go back to writing every line of code by hand, writing every e-mail or waste-of-time report by hand, and sifting through every Google search by hand? How many apps and companies would suddenly have nothing to offer?

Multiply that by the working population of a country, then imagine how much worse that would be in a year or two as our economies become increasingly dependent on AI to sustain productivity. Cutting off a country (or all countries) from AI would have devastating economic effects at minimum. And if AI ever becomes integral to parts of national security, defense, or public safety, losing it could be a genuine national emergency. It's not a stretch to say that cutting off an entire country from leading models could be like cutting off an entire country from the Internet.

Sovereign AI infrastructure started as a response to this hypothetical threat, and after the June 12 reclassification of Anthropic's models by the US, the threat and urgency became real for other nations. The sovereign AI infrastructure being funded in the United Kingdom, Canada, and across Europe is less about being competitive in developing AI capabilities, and now more about protecting against weaponizing the current US monopoly over frontier AI models.

Although governments around the world are committing billions to building sovereign AI infrastructure, the consensus I heard at ISC is that these sovereign AI supercomputers are not intended to eliminate nations' dependencies on US AI infrastructure and models completely. Rather, they are being built as an emergency reserve of capability so that, if a country was to lose its access to US-made frontier models, they would have enough sovereign models and sovereign GPUs to support the AI workloads required by critical infrastructure--applications like defense and energy.

For HPC centers, it seems like "sovereign AI" as a rallying cry is good for everyone. Weicheng Huang from the National Center for High-Performance Computing Taiwan quantified the benefit it's brought his center:

And of course, since those AI infrastructure dollars flow directly to companies selling AI infrastructure, it's no wonder that every AI company's CEO is unabashedly saying that every country needs its own sovereign AI. Like I said before, sovereign AI is good for business.

Sovereignty is forcing modernization

Less cynically though, this push for sovereign AI is also forcing long-overdue modernization of security, privacy, and governance policies across HPC. Since national HPC centers are often the only places where expertise in operating GPU infrastructure at scale is concentrated, these HPC centers are now being approached by industry asking for AI infrastructure that meets compliance standards far beyond anything open science has ever needed. In a sense, these non-US HPC centers are now being asked to provide the services offered by AI clouds.

This has been interesting for me to see, since I spent three years at Microsoft trying to convince HPC centers that they need to pay attention to this exact issue. HPC's beloved walled gardens (where any user who can log in is implicitly trusted to behave) are dangerously outdated. Now that sovereign AI is becoming critical though, the most progressive centers within the HPC community are going all-in. This was the first ISC where I didn't have to explain that multitenancy is more than usernames and passwords. Instead, HPC architects came to the table with strong opinions on how user-provided keys should be managed.

I think the HPC community is still at least one generation of supercomputers away from being as secure as the AI clouds are today, but everyone I talked to at ISC seemed to be convinced that they need to get there soon.

Sovereignty is an opportunity

I've commented in the past that the HPC community has seemed adrift since passing the exascale milestone, but it felt like there was a clear next big goal emerging at ISC this year. Sovereign AI checks many of the right boxes: it's a capital-intensive, big-iron problem; it brings plenty of challenges across applications and infrastructure for researchers to work on; it has broad societal impact if successful; and the world all seems to agree that it needs to be done.

The funny thing is, this unifying theme and its associated funding exists across the entire HPC community with the exception of one country: the US. The US has no need for sovereign AI, because its willingness to exert export controls over HPC technologies and its mercurial stance towards other nations leading in HPC is the reason other nations see the urgent need for sovereign AI infrastructure.

This has given rise to a perceptible split in how the HPC community is focusing its efforts towards the future:

Europe, Asia, and the Commonwealth are structuring programs to rapidly deploy large-scale ($500M+) AI infrastructure. These are often being tied up with public-private partnerships to either (a) access the money required to build massive supercomputers or (b) to accelerate domestic AI innovation and develop AI talent. Significant HPC research effort is being directed at closing the gaps between the tools that HPC has historically used and the new demands of AI workflows on compute, networking, and storage.

The United States is pursuing a largely unfunded mandate to jam AI into scientific discovery and realize dramatic increases in productivity. It has an epic name ("The Genesis Mission"), but there has been less-than-epic clarity, structure, or funding driving the community forward. As a result, the first Genesis funding opportunity for $293M of research funding is alleged to have resulted in 10,000 proposals. That's enough to buy every research team exactly one GPU.

It will be interesting to see how this split evolves. Will SC maintain the high concentration of interest around sovereign AI we saw at ISC? Or will the Genesis Mission narrative of spraying AI all over science win out?

CPUs are back...maybe?

LineShine's exascale all-CPU architecture was a surprise to many, especially the conference organizers who opened ISC with a plenary that asserted that "heterogeneity is inevitable." Less than an hour later, Yutong Lu was on the same stage, explaining that a homogeneous architecture was an explicit design point of the supercomputer at the top of Top500.

Horst Simon called out the deep irony explicitly and posed the question during the Top500 presentation: "Will the fact that it's an all-CPU machine mean something for the future of CPU versus GPU computing?"

It's hard to answer this question purely on the basis of LineShine, because whether CPUs or GPUs are the ideal architecture for scientific computing has never been strictly about technology. Rather, it's an economic question. Is there enough of a market for FP64-optimized chips to justify a company like Intel, AMD, or NVIDIA establishing and sustaining a line of processors or accelerators that are optimized for FP64? The fact that none of them are doing this (except AMD, sort of) tells you the answer is "no."

I think LineShine is a terrible indicator of whether CPUs have an economic future in HPC, because its LX2 CPUs are purpose-built, non-commercial parts designed for a one-off giant system. It's not clear who paid for LineShine (one anecdote says it was built without government funding--dubious), but nothing about it indicates LX2 will be commercialized into a product whose profits will fund a follow-on chip. There are signs that LineShine has Huawei DNA (e.g., its user environment provides Huawei's BLAS implementation), but Huawei takes no credit for any part of it.

But when you take economics and profit out of the picture (as the Chinese government is able to do), the question becomes more interesting: is the LX2 CPU (or any CPU with matrix extensions) architecturally competitive with a GPU?

At its surface, LX2 looks an awful lot like a GPU. Each core has a big matrix tile of registers and can execute instructions that perform fused multiply-accumulate ops that fill this matrix, and there are hundreds of these cores. But unlike a GPU, every core is independent; each of the 304 cores can work on an entirely different arithmetic function and work within its own vector and matrix registers, reducing the need to think about coordinating warps or wavefronts. Although I'm no chip expert, I would guess that LX2 and its matrix extensions are less space- and power-efficient than a GPU for GEMMs, trading the absolute power-performance benefit of a GPU for the ability to work on less-coherent and scaled-down problems more efficiently.

It's also hard to do an apples-to-apples comparison of LX2 and a GPU to see where a CPU with matrix extensions could be competitive with a GPU, because China remains a few years behind the state of the art in their semiconductors. At the surface, LX2 is comparable to a Hopper-era GPU (see my LX2 and H100 pages for sources):

*LX2 has no native 8-bit floating point matrix support, but neither does Huawei's Ascend 910C, China's leading AI accelerator.

But its support for FP64 matrix operations clearly comes at the cost of lower-precision 32/16/8-bit matrix operations, making this a solution that is not very compelling in a world where scientific applications are finding ways to exploit 32-bit (and lower) precision. Similarly, it's hard to argue that this CPU-based approach is ideal for mixed HPC/AI workloads when its low-precision performance is so far from what GPUs' tensor/matrix cores can deliver in a comparable power envelope. And if you look at the Chinese press surrounding LineShine, their hero number for AI inferencing (578 tokens per second) is limited to decode-only performance, which is a function of HBM bandwidth and not FLOPS.

Finally, Dongarra, Hoefler, and Matsuoka wrote an opinion piece called "Do We Still Need GPUs?" shortly after ISC that ostensibly adds to the discussion. Their title is misleading though, because the piece is less about answering the question posed by Horst and more a collection of Claude-fueled napkin math that demonstrates how Arm CPUs with matrix extensions could be competitive with GPUs for sparse models that are already inefficient on GPUs.

So, to answer Horst's question: maybe CPUs (or, CPU matrix extensions) can be relevant for the future, but choosing CPUs over GPUs will limit your opportunity to also run low-precision algorithms with the highest performance-per-watt efficiency.

And this is all only examining matrix performance, which itself is not the prevailing motif in scientific applications that still need FP64. Little has been said about LX2's vector implementation beyond the width being 512 bits, so we'll have to wait until more papers emerge based on LineShine before we can tell if its CPUs are only useful as FP64 matrix accelerators, or if its vector performance points in a direction that is competitive with the value that FP64-vector-optimized GPUs like MI355X will deliver.

Storage stuff

Most of the storage-related discussion at ISC felt incremental as well. Lustre and DAOS are slowly implementing features that have been on their roadmaps for years. Vendors are turning the crank on new Lustre appliances. And HPE seems to be taking DAOS seriously enough to productize it and release the world's first DAOS appliance. But none of this was terribly surprising.

However, there was one surprise in the storage world at ISC: Sugon's new ParaStor F9000, a Chinese-developed all-flash parallel file system and appliance that unseated DAOS from the top of the IO500 list.

ParaStor

China has long submitted weird research file systems to IO500, and some of them were so absurdly gamed that the IO500 committee had to split the list into separate "research" and "production" lists to distinguish real file systems from science experiments. Despite that, IO500 remains essentially a metadata performance leaderboard which has allowed the big DAOS system at Argonne to consistently rank 10x higher than any other system on the list.

Sugon finally broke the record with its own flashier (ha!) massive-scale all-flash system entry that showed over twice the bandwidth and metadata performance of Argonne's DAOS. It didn't do this by just deploying more flash, though; it actually used fewer servers (442 vs. 642) and fewer SSDs (5K vs. 10K) than the DAOS submission it beat. In addition, this ParaStor submission came with a couple of details that surprised me:

1. ParaStor is a real POSIX(ish) file system, so had to solve the hard problem of getting high metadata performance while simultaneously adhering to POSIX. DAOS does not do this, and instead used its non-standard file-like API (DFS) to get its IO500 score.
2. ParaStor is a real product, and Sugon had full racks of it on display at their booth to prove it.
3. ParaStor's implementation reflects a modern, production-minded architecture with a lot of bells and whistles that open-source solutions like DAOS and Lustre lack.

Point 3 was the most interesting to me, and the IO500 submission (and a Chinese-language press piece released during ISC) disclosed a lot of details that I wouldn't expect of a file system I never heard of a month ago. Specifically, it's a shared-nothing parallel file system that uses Lustre-like 2U24 dual-controller HA chassis with standard 14+2 Reed-Solomon erasure codes for data. Like Lustre, it uses a custom kernel client, and it supports transport over RDMA.

It does everything you'd expect from an all-flash file system; it binds together groups of CPU cores, NICs, NUMA nodes, and SSDs to minimize latency and variability, and metadata is triplicated to avoid the synchronous overheads of calculating and updating parity for tiny I/Os. It supports min/max quality of service policies on bandwidth and IOPS, and it also allows full host bypass using "XDS," Sugon's equivalent to GPUDirect Storage.

Some of the details of the file system sounded very WEKA-like to me, though. Like WEKA, its client supports using client-local SSDs and RAM to prefetch and cache pieces of the file system, improving small-file performance. It also supports "intelligent data tiering" from its flash layer to a hard drive layer, though the IO500 submission only tested against all-flash.

Some of the stated capabilities seem a little odd though, so until independent testing of ParaStor emerges from either China or other nations buying Sugon supercomputers, I'm a little skeptical of how fully baked some of these features are.

For example, the 14+2 erasure code on data does not map neatly to the 12-drive servers they use, leading me to believe that erasure coding is done across servers. That has latency implications, and it also obviates the need for the dual-controller HA enclosures they're using; implementing HA at both the intra-chassis and inter-chassis levels is costly and complex, telling me their EC might not work as well as it should. In addition, supporting performance minimums in quality of service sounds great on paper, but it is notoriously difficult to implement in practice. I am dubious that they've cracked that nut in a reliable way this quickly.

That all aside though, Sugon did have three racks of their ParaStor F9000 on the show floor.

I didn't get a chance to talk to anyone at the booth to ask questions, so all I know is what the booth placard said and what was disclosed in ParaStor's IO500 submission:

• The 42U rack is HDD-based and intended for capacity-optimized data warehousing and backup.
• The 36U and 26U are flash appliances. The 36U targets traditional HPC and AI workloads, while the 26U is targeted at "enterprise business systems."
• The 2U24 ParaStor F9000 enclosures have two controllers, each with a 64-core ("CISC," so AMD?) CPU, 12x 15.36 TB NVMe drives, and 4x 400G Chinese-made InfiniBand NICs.
• It scales up to at least 221 enclosures (442 servers), for a total of 72.3 PiB raw capacity (5,304 15.36 TB NVMes). The formatted capacity of this system was 63 PiB, which is almost exactly the 14+2 EC overhead. It is unclear how they store metadata and internal data structures, since there's effectively no capacity set aside for it.
• The ior-easy tests break down to 80 GB/s/server for reads and 73 GB/s/server writes. This is uncannily high and represents over 6 GB/s/SSD for both reads and writes.

The bandwidth-per-enclosure (160 GB/s and 146 GB/s) is extremely competitive with the latest Lustre appliances (190 GB/s and 140 GB/s) that were being touted during the week:

But the metadata performance they claimed only seems achievable if they are using advanced data structures to index the namespace and implement redirect-on-write. Since they didn't disclose anything about how they're handling metadata though, I suspect they are cutting corners around reliability and durability to achieve the performance they posted. And as anyone who's supported production parallel storage, going fast on an empty system is the easiest part. Keeping the system running predictably and reliably day after day is much harder.

Only bad AI requires many IOPS

There is an oft-repeated assertion that AI workloads generate many small I/Os, and therefore an AI-optimized file system requires an extreme level of IOPS. Yet, given everything I've done in the world of large-scale AI, I have never seen this requirement bear out. And on a more fundamental level, I cannot find any step within a model's training-inferencing lifecycle that should require accessing data in a truly unpredictable way. So this year, I made it a point to ask around: exactly what is the AI workload that is requiring these IOPS?

The answer was almost always "well, I don't actually know" or "you'd have to ask my users."

I only met one person who could back up this assertion, and his reasoning was disappointing: there's nothing intrinsic to AI that actually requires small I/Os, but most of the people trying to run AI workloads on supercomputers today have no idea what they're doing. Their applications' I/O patterns are not optimized for parallel storage, so supporting high IOPS for AI is more about coddling a new wave of inexperienced users than supporting a fundamentally new workload. The corollary I took away is that HPC seems to be willing to spend money on hardware (IOPS) to deal with problems that could be fixed in software (bad user code). As with my observations on designing new memory architectures to address the needs of today's transformers, this is not a tradeoff often made in the commercial world.

That said, there was an anecdote repeated at a couple of BOFs that was noteworthy: AI agents are really good at bringing down Lustre file systems. Like inexperienced users in need of coddling, today's AI agents appear to be treating parallel file systems like laptop file systems, and they do everything a bad user would do like creating bazillions of files or repeatedly walking the namespace. But unlike a bad human, these bad agents are very good at parallelizing work, and they cannot be phoned up by user services and told to stop.

So, I remain unconvinced that a supercomputer built for AI really needs a storage subsystem that delivers tons of IOPS. I've certainly worked on productive AI supercomputers that didn't have them. But I do think AI is creating software problems faster than traditional HPC file systems can fix them, and a storage system designed for a few well-behaved humans (like Lustre) is probably not a good near-term solution for a compute workload that attracts many poorly behaved robots.

Maybe HPC and AI really are different

A big benefit of the HPC community coming to terms with AI's effect on the industry is that some of the bright minds in HPC are starting to ask really interesting, existential questions about long-held assumptions. Just because HPC has used a certain tool for decades doesn't mean it's the right tool, and AI's disruptive effects might be dislodging these old ideas, creating room for better ideas to develop.

The most interesting such example I heard was posed by James Lin at his HPC Around the World: Asia Pacific talk. After he was done talking about how good LineShine is for US supercomputing, he described this fascinating project happening at Shanghai Jiao Tong University:

What would a supercomputer designed for AI instead of humans look like?

As discussed in the previous section, today's agents are quite supercomputer-ignorant and do the same dumb things that novice human users do. But models also do smart things that humans do not, like read documentation. It wouldn't be hard for a well-designed agentic system to discover what does and doesn't work well on a supercomputer, then ensure that the jobs it runs always strive to interact with compute, networking, storage, orchestration, and other services in a superhumanly optimal way.

Lin didn't get into the details of what he envisions such a system to be, but this idea really got me thinking about all of the weird software and interfaces we use to interact with supercomputers that are designed for human convenience rather than efficiency. There's tons of things that we could strip away, and probably a bunch of things we should add.

Parallel file systems and middleware like MPI-IO are probably unnecessary if users can index their own data objects (rather than relying on a file system hierarchy) and choose the optimal concurrency and I/O sizes for a storage subsystem. Conversely, HPC is still reliant on a lot of human-friendly imperative interfaces like the terminal and Slurm scripts rather than robot-friendly APIs and declarative manifests.

I admittedly came to ISC with my own opinions on this, since I was invited to speak at the Advancing Autonomous Scientific Discovery (A2SD) Workshop and presented the case that Slurm is the wrong tool for autonomous and agentic orchestration. I won't rehash my talk here, but I was pleased to find that other speakers shared similar sentiments: autonomous systems don't interact with HPC infrastructure in the same way that people do, so a system specifically designed to support agent-driven workflows instead of coddling humans might look very different than today's supercomputers.

Take-away themes

As is often the case, the week of ISC was over much too soon, and there wasn't enough time to see everything I wanted to see. From the parts that I did get to attend though, I left the week with a few distinct impressions.

Foremost, the HPC community is finding equilibrium with AI. As shown by the first two keynotes, you can still be in HPC if you don't care about AI. And LineShine's focus on FP64 at the cost of lower precisions is proof that leadership in HPC can happen independently of leadership in AI. At the same time, AI is accelerating the pace at which the HPC community is willing to revisit long-held dogma about the way things should be, and there are a lot of interesting new non-AI ideas on the horizon that are being effected by AI.

At the same time, AI is reshaping the strategic priorities driving national HPC investment. Outside of the US (and China), who are already controlling their own frontier models, countries around the world are pursuing sovereign AI infrastructure and calling on their domestic HPC experts to build the capability to run and fine-tune frontier models domestically. As global productivity increasingly depends on frontier AI, the ability to serve that dependency without relying on other nations is becoming a necessary mitigation to the risk that access to frontier models be restricted through political pressure or export controls.

There's still a lot of uncertainty and unclear direction about the future across the HPC community, and the diversity of topics covered in this year's ISC program reflected that. But we may have seen the first signs of direction nucleating now that the community is no longer blindly thrashing about with respect to AI, and I'm looking forward to seeing if any of these themes crystallize into concerted, high-impact efforts in the year ahead.

Appendix: The personal stuff

I wrote the rest of what follows mostly for me, so you won't miss anything technical if you stop reading here. Promise.

ISC is two conferences

This was my seventh year attending ISC, and as I've written in past conference retrospectives, I feel like I see less and less of these conferences the older I get. Part of this is because I spend more time catching up with the growing list of friends and colleagues I've made over the years--an enviable position that I can't complain about. But this year, it felt like I was attending two conferences at the same time: one was the same ISC I've always been attending, and the other is the ISC that exhibitors attend.

The "ISC for exhibitors" conference that my coworkers were attending, largely in the exhibit hall and off-site, had all the logistical chaos of a big conference: finding last-minute meeting spaces, tracking down missing people, and unwinding once it shut down with the marketing team were all part of that experience. But I also went to the "ISC for practitioners" conference that I've always attended, running between the session rooms in the CCH, finding time to look at the research posters, and preparing slides for a presentation the next day.

I don't know how many people straddle both conferences like I do, but I suspect it's not many. Anecdotally, I think I was also the only person from my company with a technical program badge; the other couple dozen of us were there as exhibitors or to meet with specific customers. This created a bit of tension across the week for me, because it was often assumed (reasonably) that I was exclusively attending the "ISC for exhibitors" conference and didn't have any other demands on my time. And yet, I paid the staggering €2,270 to attend the technical program as an industry participant, so I felt an obligation to get the most out of that as well.

The conclusion I reached is that attending both conferences--ISC for exhibitors and ISC for practitioners--is difficult to make worthwhile. ISC doesn't leave room for attendees to do both halves of the conference. Either you attend ISC for exhibitors and pay your €440 exhibitor registration fee, or you attend ISC for practitioners and pay €2,270 technical program/workshop fee. If you try to do both, you get the worst of both worlds--registration costs 5x more, yet you have half as much time to make use of it. In addition, paying €2,270 for a conference that everyone else at the company paid €440 for raises eyebrows in the finance department.

By the numbers, ISC 2026 officially had 4,035 registrants, and over 60% self-identified as being from industry. While many industry attendees undoubtedly came as practitioners rather than exhibitors, the fact that all of ISC's keynotes are held in a room that could only seat a quarter of the total registrants suggests the majority of ISC registrants aren't actually there for the technical program. "ISC for exhibitors" may be the bigger conference, yet it felt like there was a big financial disincentive for exhibitors to dip their toe in the technical program.

So as not to be wholly unproductive in my complaining about this, perhaps there is room for a new type of conference pass for those of us who want to do both; acknowledge the fact that exhibitors subsidize the technical program and carve out a rate for them to attend the technical program that sits somewhere between academia and industry. I assume that many of the costs of full registration (like meals) are unnecessary for those of us attending both halves anyway. For example, I didn't get to each a single conference-provided lunch because I always had off-site obligations with customers at that coveted opening in the schedule. Surely there's a way to pass those sorts of savings on so that corporate bean counters don't have quite as big a reason to balk at the cost of attending ISC.

The makings of a keynote

I was recently invited to deliver my first conference keynote (a great honor!) which represents a new stage in my career. Having only ever given research talks and educational lectures, I went to ISC with an eye towards scrutinizing all the conference and workshop keynote speakers to determine the essential ingredients for a great keynote. What I came away with was the realization that many keynote speakers don't seem to think very hard about what they are going to talk about. Instead, they give what I consider the "program manager smorgasbord" talk.

In academia, there's a boilerplate talk whose sole purpose is to impress your program manager--the person responsible for deciding whose research gets funded and whose doesn't. The point of those talks isn't to inspire the audience or offer deep insight as much as it is to dazzle the audience with big words and fancy graphics in abundance. In marketing, we call this brand marketing, and its point is to make someone feel a certain way about you or your work.

In academia, brand marketing manifests in slide decks full of standalone quad charts that each explain a specific paper you wrote. Turning a stack of quad charts into a presentation is just a matter of finding a story that makes it sound like all your papers are building towards some fantastic outcome. Because they're really a show-and-tell rather than a concrete problem-hypothesis-results-CTA though, these smorgasbord talks make for bad keynotes. The only take-away is "this person does a lot of work," and there's no deeper insight that can be drawn since each piece of work is treated so superficially.

Despite this, many (but not all!) of the keynotes I sat in throughout the week turned out to be program manager smorgasbord talks. Some speakers did a nice job of motivating the grand challenge up-front with inspiring videos or graphics, but they invariably shifted into a series of slides with the predictable format of

• Here's the paper title
• Here's a key take-away plot or graphic
• Here's the list of students' names who wrote the paper
• Here is where you can find the PDF online

Some speakers had a story that carried through series of these slides, but some also unabashedly devolved into saying "and in this paper, we ..." or "and this slide shows our work on ..." I found myself wondering if anyone enjoys sitting through these types of talks, because they're too superficial for technical audiences yet too jargony for general audiences. Do speakers not reflect on their audience when they are invited to give a keynote?

Although I didn't leave ISC with any new insights on how to craft a great keynote, I did leave with a reinforcement on what to avoid:

• Don't just present a superficial overview of all the papers you've written. Nobody but your program manager cares, and the audience is usually too broad to appreciate more than one or two of those papers. The audience as a whole will have a hard time remaining enthusiastic as you essentially talk about yourself for a half hour.
• At the same time, don't devote the entire keynote to general knowledge that anyone can find on the Internet. Some of the best presentation advice I ever got was from Eli Dart, who once told me, "tell them something they won't hear from anyone else." This mantra is great, because it forces you to think about both your unique perspective and your audience's experiences.
• Don't stroll down memory lane without a meaningful point. I've seen greybeards do this, and I suspect it results from their most productive days being history. While war stories are fun in moderation, HPC conferences often have a lot of students and early-career attendees who simply do not care about a computer that existed when they were in diapers. Focus on why a lesson learned from an old computer remains relevant to their current professional lives instead.
• Don't stray too far out of your lane. If you find yourself quoting The Information or other rumor mills as the backbone of your thesis, you probably don't actually know what you're talking about. Keynote speakers tend to be experts in something, so it follows that the audience wants to hear about your expertise, not whatever theories you've found on the Internet. This also avoids the risk that someone in the audience actually knows the truth behind the rumors and can see that you're not speaking credibly.

I write all this being fully aware that I have never given a real keynote to thousands of people before, and I risk bombing my own keynote that's coming up. But if that happens, you can believe that I'll add those learnings to the above list.

Dumb things to avoid at ISC

I am not immune from doing dumb things at conferences, and this year was no different. The biggest dumb thing I did was decide to have a renal biopsy six days before I flew to Hamburg. I don't recommend doing that.

Nothing bad happened and I was fully functional by the time I had given my first presentation on Tuesday, but I had not counted on a minor medical procedure largely wiping out my productive work week leading up to ISC. As a result, I landed in Hamburg feeling underprepared for the talks I had to give and the people I needed to meet. However, as my wife often reminds me, I always feel underprepared going into conferences, and I didn't get boo'ed off any stages once I got there.

This is all to say, I learned not to underestimate the psychological effects of having a medical procedure done in the week before ISC. Even if the doctor says getting on the flight and going to the conference is medically fine, it added a lot of stress and anxiety to the time leading up to an already-stressful week at ISC.

The other unwelcome stressor this year was navigating intercontinental travel and a full conference schedule with Invisaligns. I've managed to go 18 months without complaining about this in any of my conference recap posts, but seeing as how this will be the last major conference I attend while undergoing orthodontic treatment, I'll say: conferences are not kind to people with dietary restrictions and who cannot just snack on random food opportunistically.

I've been documenting my lessons learned about navigating work-related travel with Invisaligns elsewhere, but for ISC specifically, 60-minute lunch breaks are not long enough to have a social meal and deal with orthodontics. Since I started Invisalign treatment at the end of 2024, I've been noticing a lot of conferencegoers with them (or permanent braces) and am incredibly sympathetic. They add another layer of stress and complexity over just about every activity, from presenting to meals to receptions. I will be glad when I am rid of them, and I will be forever cognizant of how much harder conferences can be for people who have to be mindful of what and how they eat.

The Problem: Summarizing XRootD Stash Data

Our goal was straightforward: aggregate XRootD stash access logs (xrd-stash*) into a daily summary index (osdf-summary-{year}). This involved grouping by several file path components, server details, and user domains, then calculating sums, averages, and counts of metrics like filesize, read, and write.

Here is a snippet of the initial (problematic) transform configuration:

{
  "transform": {
    "transform_id": "osdf-summary-2022",
    "description": "OSDF summary transform for year 2022",
    "source_index": "xrd-stash*",
    "target_index": "osdf-summary-2022",
    "page_size": 1000,
    "groups": [
      {
        "date_histogram": {
          "source_field": "@timestamp",
          "target_field": "@timestamp",
          "calendar_interval": "1d"
        }
      },
      {
        "terms": {
          "source_field": "dirname1.keyword",
          "target_field": "dirname1"
        }
      }
    ],
    "aggregations": {
      "filesize_sum": { "sum": { "field": "filesize" } },
      "filesize_avg": { "avg": { "field": "filesize" } }
    }
  }
}

The Symptoms: Generic Errors and Timeouts

The transform job kept failing with a rather unhelpful message in its metadata:

{
  "status": "failed",
  "failure_reason": "Failed to index the documents",
  "stats": {
    "pages_processed": 96,
    "documents_processed": 89737708,
    "documents_indexed": 96000,
    "index_time_in_millis": 44733,
    "search_time_in_millis": 1715612
  }
}

Notice the high search_time_in_millis compared to index_time_in_millis. This was a critical clue that the aggregation phase was struggling.

Further attempts to debug with _explain or custom composite aggregation queries often resulted in:

• 502 Bad Gateway / timed_out: The query was too resource-intensive for the cluster to handle.
• illegal_argument_exception: Missing value for [after.date_histogram]: A mismatch in how the after_key was structured versus the sources in the composite aggregation.
• illegal_argument_exception: Invalid value for [after.site], expected comparable, got [null]: The transform was getting stuck on null values within its grouping keys.

The Debugging Journey and Discoveries

Through a series of focused queries and iterative refinements, we uncovered several interconnected issues.

1. Composite Aggregation Challenges and the “Ghost Key”

Our composite aggregation debugging queries kept failing. This was traced to:

• Syntax mismatches: names in the after key must exactly match the names defined in sources (for example, @timestamp must match @timestamp).
• null values in after_key: terms aggregations can fail when after_key includes null, unless handled explicitly.

Then came the key finding: a direct search for documents matching the transform’s after_key yielded zero results. The transform was trying to resume from a state that no longer existed in source data.

2. The Real Culprit: Unparsed “Garbage” Data

An inverse query (documents missing expected fields) revealed records like:

{
  "_index": "xrd-stash-ilm-000037.reindexed",
  "_id": "cAqUoH4BOTrVvgqCSyKq",
  "_source": {
    "message": "GET / HTTP/1.1\n",
    "@timestamp": "2022-01-28T12:06:20.222Z",
    "host": "ec2-3-110-169-111.ap-south-1.compute.amazonaws.amazonaws.com",
    "tags": ["_grokparsefailure"]
  }
}

These were logs that failed parsing and were actually web traffic hitting the server, not XRootD stash operations. They lacked key transform fields like logical_dirname, filesize, and server.

When the transform encountered enough of these records, grouping keys became null. Combined with malformed or very long field values, the composite aggregation became unstable and hit timeouts.

3. Precision for PetaByte-Scale Data

Not a crash cause, but still important: float is not precise enough for large sums at petabyte scale.

Solution: use double for sums/averages and long for counts.

The Ultimate Solution: Resilience and Precision

The final, robust fix used multiple changes together.

1. Stop and Delete Stale State

Stop the transform and delete the target index to clear bad transform/index state.

POST _plugins/_transform/osdf-summary-2022/_stop
DELETE osdf-summary-2022

2. Recreate Index with Explicit High-Precision Mappings

PUT osdf-summary-2022
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "dirname1": { "type": "keyword" },
      "logical_dirname": { "type": "keyword" },
      "filesize_sum": { "type": "double" },
      "filesize_avg": { "type": "double" },
      "filesize_count": { "type": "long" },
      "doc_count": { "type": "long" }
    }
  }
}

3. Add Intelligent Filtering in data_selection_query

• Exclude _grokparsefailure events.
• Require existence of critical grouping fields.
• Add script guards against empty or oversized keyword values.

"data_selection_query": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": f"{year}-01-01T00:00:00Z",
            "lt": f"{year + 1}-01-01T00:00:00Z"
          }
        }
      }
    ],
    "must_not": [
      { "term": { "tags": "_grokparsefailure" } }
    ],
    "filter": [
      { "exists": { "field": "logical_dirname.keyword" } },
      {
        "script": {
          "script": "doc['logical_dirname.keyword'].size() > 0 && doc['logical_dirname.keyword'].value.length() < 1000"
        }
      }
    ]
  }
}

4. Reduce page_size

Lowering page_size from 1000 to 50 significantly reduced memory pressure per composite aggregation page and helped avoid 502 Bad Gateway failures.

5. Restart the Transform

After recreating the index and updating the transform definition, restart the job.

Conclusion

By combining explicit mappings, stronger filtering, smaller pagination, and a reset of stale transform state, the transform ran reliably and produced accurate summaries without repeated failure loops.

This debugging story reinforced a key lesson: robust pipelines are not just about handling valid data, but actively excluding invalid or malformed records before they poison downstream aggregation logic.

This is a topic about which I've written at length in the past on my blog, and as I read Dr. Reed's latest post (and the Riding the Wave paper that motivated it), I found myself agreeing with a many of his positions but disagreeing with some others.

My own background is in the world at the center of Dr. Reed's writing: traditional HPC for scientific computing at the national scale. However, my outlook has also been colored by the years I spent at Microsoft supporting massive-scale supercomputing infrastructure for training frontier models and the days I now spend at VAST, steeped in the wider enterprise AI market. This undoubtedly results in an unusual lens through which I now view Dr. Reed's position, and I couldn't help but mark up his essay with my own notes as I read through it.

In the event that my perspective--that of an HPC-turned-AI infrastructure practitioner--is of interest to anyone who found Dr. Reed's latest essay as engaging as I did, I've shared them below.

New Maxim Two: Energy and data movement, not floating point operations, are the scarce resources.

This has been true long before exascale in the HPC world. This is not a new maxim. Ironically, it is in the AI world that this maxim is relatively new; as inference overtakes training as the predominant consumer of GPU cycles, we are seeing widespread shortages of DRAM because of the extreme demand for HBM and the memory bandwidth it provides.

New Maxim Three: Benchmarks are mirrors, not levers. Benchmarks rarely drive technical change. Instead, they are snapshots of past and current reality, highlighting progress (or the lack thereof), but they have little power to influence strategic directions.

Benchmarks drive technical change amongst technology providers who act without conviction. The tech industry is full of companies who are blindly chasing consumer demand, and these companies design entire product lines to achieve high benchmark results with the mistaken belief that those benchmarks are a reasonable proxy for actual productivity. And even worse, many buyers (especially in lower-sophistication markets like enterprise) also believe that benchmarks, by virtue of being designed by community organizations who have ostensibly thought deeply about performance, are a good proxy for productivity, make purchasing decisions around these same benchmarks.

The net result is that a bad set of benchmarks can create and sustain an entire economy of buyers and sellers who think they are buying and selling something useful, when in fact they are wasting resources (time, energy, and COGS) because none of them actually understand what really drives productivity within their organizations.

Fortunately, the HPC community is generally savvier than enterprises, and most national computing centers now recognize that HPL is simply not a meaningful yardstick. While it used to be good for convincing politicians and other non-technical funders that good work was being done, the discourse around AI has squarely put R_max in the ground as a meaningful metric. Politicians now understand "hundreds of thousands of GPUs" or "gigawatts," neither of which require a benchmark like HPL to prove.

Also, as an aside, I find it ironic that a paper with Jack Dongarra listed as an author is now saying HPL is a snapshot of the past. I've heard that he is the reason that HPL results achieved using emulated FP64 are not allowed on Top500. Despite achieving the required residuals through more innovative means than simply brute-forcing a problem through FP64 ALUs, using techniques like the Ozaki scheme were deemed incompatible with the purpose of Top500. Which is to say, I think he's the reason why HPL and Top500 has been reduced to a benchmark that reflects outputs (hardware FP64 throughput) rather than outcomes (solving a system of equations using LU decomposition).

New Maxim Four: Winning systems are co-designed end-to-end—workflow first, parts list second.

…

In HPC, we must pivot to funding sustained co-design ecosystems that bet on specific, high-impact scientific workflows

I don't agree with this. Funding sustained co-design is just swimming upstream with more conviction.

The real way forward is to find ways to align scientific discovery with the way the technology landscape is moving. This means truly riding the wave and accepting that scientific discovery may have to turn to completely different techniques that achieve their desired precision and validation through means that may render obsolete the skills and expertise some people have spent their careers developing.

Consider the scaffolding of end-to-end workflow automation; a rich ecosystem of technologies exists in the enterprise and hyperscale worlds that have been used to build extreme-scale, globally distributed, resilient, observable, and high-performance workflows that combine ultra-scalable analytics engines with exascale data warehouses. However, realizing these capabilities in practice requires fundamentally rethinking the software infrastructure on which everything is built. The rigidities of Slurm and the inherent insecurities of relying on ACL- and kernel-based authentication and authorization need to be abandoned, or at least understood to be critically limiting factors that the HPC community chains itself to.

To make this very specific, consider a bulk-synchronous MPI job running across a hundred thousand GPUs; if one node fails, the whole job fails. The "swimming upstream with more conviction" way of solving this problem is to pay a storage company to build a faster file system, pay some researchers to develop a domain-specific checkpoint library that glues the MPI application to platform-specific APIs, and pay SchedMD to automate fast restart based on these two enhancements. Fund all three projects under the same program, and it is arguably a "co-designed end-to-end workflow."

Riding the wave would be something different though: instead of requiring a job requeue and full restart from checkpoint upon job failure, treat the entire job as an end-to-end workflow. If a node fails, the job doesn't stop; it just transitions into a recovery state, where the orchestrator gives it a new node on which the job runtime can rebuild the state of the dead node using distributed parity or domain-specific knowledge. A fast file system is completely unnecessary for failure recovery. But the application developers would have to abandon the model of an application being a single process invocation in favor of the application being a system whose state evolves with the underlying hardware.

Slurm can't do any of this, because Slurm is tied to the MPI model of parallel execution which assumes nothing ever fails. Which is to say, I think co-design should be deferred until a time that the HPC community first recognizes that, so long as they continue to approach end-to-end co-design as an HPC problem to be solved by HPC people using HPC approaches, they will continue to swim upstream regardless of how much co-design they do.

New Maxim Five: Research requires prototyping at scale (and risking failure), otherwise it is procurement. A variant of our 2023 maxim, prototyping – testing new and novel ideas – means accepting the risk of failure, otherwise it is simply incremental development. Implicit in the notion of prototyping is the need to test multiple ideas, then harvest the ones with promise. Remember, a prototype that cannot fail has another name – it’s called a product.

The idea is right, but the title is wrong. Prototyping at scale is the wrong way to think about developing leadership supercomputing capability. The largest commercial AI infrastructure providers do not prototype at scale. Instead, they frame their thinking differently: anything done at scale is production, and if it doesn't work, make it work.

In practice, this means foregoing years-long acceptance test processes and beating up suppliers over hundred-page-long statements of work. Instead, they accept the reality that they share the responsibility of integration with their suppliers, and if things go sideways, they are working with partners who will not walk away when times get tough.

National-scale supercomputing has always been this way in practice, but the HPC community likes to pretend that it isn't. Consider Aurora: if that system wasn't a prototype-at-scale, I don't know what is. That system's deployment and operations was and remains fraught, and it is built on processors and nodes that were cancelled as products before the system even entered production. Yet the theatrics of acceptance testing went on, Intel got paid something, and we all pretend like Aurora just like Frontier or Perlmutter.

AI doesn’t prototype at scale; they just take a risk because the next breakthrough can't wait for every "i" to be dotted and "t" to be crossed. If a hyperscale AI system is a failure, that’s fine. The demand for FLOPS is sufficiently high that it will be utilized by someone for something, even if that use generates low-value results rather than the next frontier model that it was meant to build. The same is true for systems like Aurora; it's not like these systems sit idle, even if they don't live up to their original vision.

And rest assured, AI systems prove to be bad ideas just like HPC systems do. The difference is scale: there are multi-billion-dollar AI supercomputers in existence that were obsolete before they even came online, because the problem they were designed to solve became irrelevant in the years it took to build them. But what was really lost? A bit of money and a little time. The GPUs are still used for day-to-day R&D or inferencing, and the time lost was made up for in lessons learned for the systems that followed.

All the big AI systems are prototypes, because AI workloads themselves are continually evolving prototypes. As a result, the line between prototype and production become blurry, if not meaningless.

All too often, in scientific computing, our gold is buried in disparate, multi-disciplinary datasets. This needs to change; we must build sustainable, multidisciplinary data fusion.

This is so easy to say, but it always feels empty when it is said. What’s stopping this data fusion? I don’t think it’s willpower or resources. It’s just really difficult to figure out what good any of it would be within a standard theory-based modeling framework. Making productive use of fused multimodal data (meshes, particles, and discrete observations, for example) requires multimodal, multiphysics models. And such models are really expensive relative to the insights they deliver.

To me, this means the challenge isn't in getting the world's scientific data to hold hands and sing kumbaya; it's accepting that there's limited value in actually doing this data fusion unless you're willing to also take on more approximations within the models that use them so that the net return--science per dollar--comes out as a net positive over today's physics-based, single-mode scientific models.

The AI community accepts that wholly empirical models are much less interpretable but can much more readily turn multimodal data into results in a meaningfully faster, most resource-efficient way. for example the Aurora model and how it took all sorts of disparate climate datasets to develop an incredibly efficient forecasting tool. In a minute on a single GPU, it produces forecasts of comparable quality to what would take hours across multiple GPUs using a physics-based model. And it achieves this efficiency by having trained on a diverse collection of gridded 3D atmosphere data and tabular data that was fused.

The only problem, of course, is that the model is much less interpretable than a physics-based model. If the Aurora model's forecast is off, forecasters mostly have to shrug and move on with life. But for the purposes of solving the scientific problem at hand (predicting the weather a few days out), that may be good enough.

Governments must now treat advanced computing as a strategic utility, requiring a scale of coordination and investment that rivals the Manhattan Project or the Apollo program.

Manhattan Project and the Apollo mission had distinct goals with a defined "lump of work" required to achieve them. They are not comparable. Computing is a commodity, and it’s a far fairer comparison to liken it to oil or gas reserves. And even then, exactly what good are these computing reserves or capabilities really? Is it one big supercomputer, or many small ones? What are the range of problems that such a strategic utility would be called upon to solve?

In the AI game, advanced computing is certainly a pillar of competitiveness, but it is not necessarily the most limiting one. DeepSeek showed us that ingenuity and massive computing are two orthogonal axes towards developing new capabilities. They showed that, although you can spend a ton of money on GPUs to train a new frontier model, you can also be a lot more clever about how you use much fewer GPUs to do the same thing. And the ratio of people to capital that resulted in DeepSeek-R1 arguably showed that investing in innovation, not just datacenter buildout, has a much higher return on investment.

In the context of the above statement, I think governments would do far better to treat its innovators as a strategic asset and worry less about issuing press releases that lead with how many thousands of GPUs they will deploy. For every thousand GPUs to be deployed on government land in the US this year, how many government researchers, architects, and visionaries have headed out the door and are never coming back?

I am talking to teams where CI is a load-bearing piece of infrastructure. Teams where 20 or 50 or 200 engineers push code daily. Teams where a broken CI pipeline doesn’t mean one person waits a few extra minutes; it means a queue of pull requests backs up, a deploy window gets missed, and product timelines slip. Teams where CI time is measured in engineering-hours-lost-per-week and has a line item on somebody’s OKRs.

It also leans heavily on one of my favorite papers, “Build systems à la carte” by Mokhov et al. From the discussion of that paper:

The real takeaway is not that bash is bad. It’s that the design space of build systems has structure, and that structure has been studied, and that the properties you care about (minimality, correctness, support for dynamic dependencies, cloud caching, early cutoff) correspond to specific architectural choices that live at a level of abstraction bash cannot express. When you write a build pipeline in bash, you are either implementing one of the twelve cells in the Mokhov-Mitchell-Jones matrix (poorly, by hand, with strings and exit codes), or you are living in the busy cell and rebuilding everything every time.

It’s a long read but a good one, go check it out.

“Operations” is not a dirty word, a synonym for toil, or a title for people who can’t write code. May those who shit on ops get the operational outcomes they deserve.

You should absolutely go read the full piece, as well as Charity’s earlier post on the Honeycomb blog: You had one job: Why twenty years of DevOps has failed to do it.

Below find several pull quotes from the post itself, because there were just too many to choose from.

The difference between “dev” and “ops” is not about whether or not you can write code. Dude, it’s 2026: everyone writes software.

The difference between dev and ops is a separation of concerns.

The hardest technical challenges and the long, stubborn tail of intractable problems have always been on the infrastructure side. That’s why we work so hard to try not to have them—to solve them by partnerships, cloud computing, open source, etc. Anything is better than trying to build them again, starting over from scratch. We know the cost of new code in our bones.

As I have said a thousand times: the closer you get to laying bits down on disk, the more conservative (and afraid) you should be.

The difference between dev and ops isn’t about writing code or not. But there are differences. In perspective, priorities, and (often) temperament.

I touched on a number of these in the article I just wrote on feedback loops, so I’m not going to repeat myself here.

The biggest difference I did not mention is that they have different relationships with resources and definitions of success.

Infrastructure is a cost center. You aren’t going to make more money if you give ten laptops to everyone in your company, and you aren’t going to make more money by over-spending on infrastructure, either. Great operations engineers and architects never forget that cost is a first class citizen of their engineering decisions.

Operational rigor and excellence are not, how shall I say this…not yet something you can take for granted in the tech industry. The most striking thing about the 2025 DORA report was that the majority of companies report that AI is just adding more chaos to a system already defined by chaos. In other words, most companies are bad at ops.

Because when the people training these models justify why they’re worth it, they appeal to pretty extreme outcomes. When Dario Amodei wrote his essay Machines of Loving Grace, he wrote that he sees the benefits as being extraordinary: “Reliable prevention and treatment of nearly all natural infectious disease … Elimination of most cancer … Prevention of Alzheimer’s … Improved treatment of most other ailments … Doubling of the human lifespan.” These are the benefits that the CEO of Anthropic uses to justify his belief that LLMs are worth it. If you think that these risks sound fanciful, then I might encourage you to consider what benefits you see LLMs as bringing, and then consider if you think the risks are worth it.

From Carlini’s recent talk/article on Are large language models worth it?

The entire article is well worth reading, but I was struck by this bit near the end. LLM researchers often dismiss (some of) the risks of these models as fanciful. But many of the benefits touted by the labs sound just as fanciful!

When we’re evaluating the worth of this research, it’s a good idea to be consistent about how realistic — or how “galaxy brain” — you want to be, with both risks and benefits.

The trick is to read plainly.

The key word in Artificial General Intelligence is General. That’s the word that makes this AI unlike every other AI: because every other AI was trained for a particular purpose and, & even if it achieved it in spectacular fashion, did not do anything else. Consider landmark models across the decades: the Mark I Perceptron, LeNet, AlexNet, AlphaGo, AlphaFold … these systems were all different, but all alike in this way.

Language models were trained for a purpose, too … but, surprise: the mechanism & scale of that training did something new: opened a wormhole, through which a vast field of action & response could be reached. Towering libraries of human writing, drawn together across time & space, all the dumb reasons for it … that’s rich fuel, if you can hold it all in your head.

It’s important to emphasize that the open-ended capability of these big models was a genuine surprise, even to their custodians. Once understood, the opportunity was quickly grasped … but the magnitude of that initial whoa?! is still ringing the bell of this century.

I’m extreme in this regard: I think 2020’s Language Models are Few-Shot Learners marks the AGI moment. In that paper, OpenAI researchers demonstrated that GPT-3 — at that time, the biggest model of its kind ever trained — performed better on a wide range of linguistic tasks than models trained for those tasks specifically. A more direct title might have been: This Thing Can Do It All?!

“AGI” is such a misused, ill-defined term that I honestly don’t find it too useful… but it’s hard to argue with Sloan’s argument here! Certainly if you showed current LLMs to someone from 20 years ago, or even 10, they’d seem like wild science fiction.

It also reminds me of a quote from Asimov on the definition of “artificial intelligence” and how the goal posts move as new achievements are retrospectively deemed as “not AI”:

[artificial intelligence is] a phrase that we use for any device that does things which, in the past, we have associated only with human intelligence

(via Nicholas Carlini)

So. Do we have AGI? Do we even meaningfully have AI? What would we have to see for the general consensus to agree they had been achieved?

Anyway, they are mostly marketing terms at this point. But it can still be interesting to think about them.

Thoughts from a dog walk listening to the Sloan article using ElevenReader.

Services I run over Tailscale, just for myself, include:

• An RSS feed reader
• A personal git forge
• An IRC bouncer
• A (poorly maintained) wiki
• JupyterLab
• Open WebUI for playing with local LLMs on a GPU workstation
• SSH to a powerful workstation, hosted at home but without complex configs

And probably a few things I’ve forgotten! It’s really just very neat. Sure I could do it all with manual Wireguard configs. But Tailscale just makes the underlying primitive much more ergonomic.

Anyway, back to programming. I have a single suggestion for you, my friend. Whatever you believe about what the Right Thing should be, you can't control it by refusing what is happening right now. Skipping AI is not going to help you or your career. Think about it. Test these new tools, with care, with weeks of work, not in a five minutes test where you can just reinforce your own beliefs. Find a way to multiply yourself, and if it does not work for you, try again every few months.

Yes, maybe you think that you worked so hard to learn coding, and now machines are doing it for you. But what was the fire inside you, when you coded till night to see your project working? It was building. And now you can build more and better, if you find your way to use AI effectively. The fun is still there, untouched

From Don’t fall into the anti-AI hype