HiHAT Meeting Minutes - Revision history

Cnewburn at 18:00, 3 February 2026

2026-02-03T18:00:23Z

← Older revision		Revision as of 18:00, 3 February 2026
Line 28:		Line 28:

	!! Be sure to log in before you save, else changes will be lost !!		!! Be sure to log in before you save, else changes will be lost !!

			= NVIDIA cuTile, Stephen Jones, Feb 3, 2026 =
			* Participants include Stephen Jones, Wilf Pinfold, Michael Wong, Remi Chassagnol, Boeschf, George Bosilca/NV, Hans Johansen, Ian Hendriksen/SNL, Jim Phillips, Jan Ciesko/SNL, Jeongnim Kim/Intel, Joseph Schuchart, Lukas Drescher, Maria Garzaran/Intel, Nitish, PiotrL, Stephen Olivier/SNL, Thiago, Tim Blattner/NIST, Walid Keyrouz/NIST, CJ/NV
			* Concise, high level abstraction, simpler than CUDA. Native types like arrays, no sync. Compilers have more freedom to optimize, more productive
			** Using unreleased AI agent to compare across programming models. See TileGym.
			* Span variations in underlying machine parallelism
			* TileIR is an available compiler target, sits alongside PTX, targeted from Triton, cuTile, existing or new DSLs
			* Family of languages - Rust, C++, others
			* Interops with other CUDA code
			* Compiler tries to figure out best place for memory, does pretty well, e.g. within 10-15% of peak. Customizes to target arch. Can override with hints. Can invoke DMA engine.
			* Hans: Tile size is important? How do you control it? SJ: Autotuner.



	= Template Task Graph, Josh Schuchart, Jan 16 and May 21 and July 16, 2024 =		= Template Task Graph, Josh Schuchart, Jan 16 and May 21 and July 16, 2024 =

Cnewburn: /* Template Task Graph, Josh Schuchart, Jan 16 and May 21 and July 16, 2024 */

2024-07-16T17:04:45Z

Template Task Graph, Josh Schuchart, Jan 16 and May 21 and July 16, 2024

← Older revision		Revision as of 17:04, 16 July 2024
Line 85:		Line 85:
	** Block sparse matrix algos for sparse tensor algos in TiledArray. Not operating on fixed data structures. Tiles flow thru graph without materializing, so no risk of corrupting data. Would otherwise have to put a copy of data where dep could see it, make it globally visible. Instead, runtime can see and access and transparently copy local data.		** Block sparse matrix algos for sparse tensor algos in TiledArray. Not operating on fixed data structures. Tiles flow thru graph without materializing, so no risk of corrupting data. Would otherwise have to put a copy of data where dep could see it, make it globally visible. Instead, runtime can see and access and transparently copy local data.
	** Multi-resolution function analysis in MADNESS		** Multi-resolution function analysis in MADNESS
			* CJ: Let's put a plan together to bring this back for a deeper-dive compare/constrast analysis of perf, resource usage, robustness, debuggability, productivity. Let's bring this back and try to draw in reps from OpenMP, OmpSs, StarPU, HPX to compare/contrast.
			* CJ: How about doing some of that analysis of perf and productivity at p3hpc.org's SC24 workshop? Maybe a panel?
			* Piotr: Maybe a WIP session there.

	= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =		= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =

Cnewburn: /* Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024 */

2024-07-16T17:01:14Z

Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024

← Older revision		Revision as of 17:01, 16 July 2024
Line 29:		Line 29:
	!! Be sure to log in before you save, else changes will be lost !!		!! Be sure to log in before you save, else changes will be lost !!

	= Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024 =		= Template Task Graph, Josh Schuchart, Jan 16 and May 21 and July 16, 2024 =
	* Participants include Wilf, Joseph Schuchart, Alexandre Bardakoff, Andrey Alekseenko, Michael Wong, PiotrL, StephenJ, Szilard Pall, CJ		* 5/21/24 Participants include Wilf, Joseph Schuchart, Alexandre Bardakoff, Andrey Alekseenko, Michael Wong, PiotrL, StephenJ, Szilard Pall, CJ
			* 7/16/24 participants include, Joseph Schuchart, Alexandre Bardakoff, Andrey Alekseenko, Michael Wong, PiotrL, StephenJ, Szilard Pall, Jan Ciesko, Walid Keyrouz, CJ
	* Overview		* Overview
	** Based on graph work at ICL/UTK, including PARSEC		** Based on graph work at ICL/UTK, including PARSEC
Line 41:		Line 42:
	** Task can't execute until all inputs available		** Task can't execute until all inputs available
	** Explicit naming of targets, using keys passed in as a paramter		** Explicit naming of targets, using keys passed in as a paramter
	** CJ: No separation between implementation of a function and the context of that node's use. ~~THis~~ precludes a separation of concerns		** CJ: No separation between implementation of a function and the context of that node's use. This precludes a separation of concerns
	** The intention here is designing the algo that consists of nodes and edges. No anticipation of task reuse.		** The intention here is designing the algo that consists of nodes and edges. No anticipation of task reuse.
	** Szilard: Complex schedule, record a sequence of nodes and edges. Prebaked schedule by a single dev who has most all of the context. If there are extensions, then you might lack reuse.		** Szilard: Complex schedule, record a sequence of nodes and edges. Prebaked schedule by a single dev who has most all of the context. If there are extensions, then you might lack reuse.
Line 64:		Line 65:
	** Results, apps, compare/contrast discussion		** Results, apps, compare/contrast discussion
	** Target next month, June 18		** Target next month, June 18

			7/16/24 continuation
			* Composition granularity
			** Backward looking, based on integration pts [implicit dependencies] - data str (StarPU, Legion), mem loc (OpenMP, OmpSs), futures/promises; or task-wise composition, distr data str, implicit contracts
			** Forward looking [explicit deps], based on pure data flow, task graphs, algos composed thru edges
			** Ex: POTRF
			* Advantages of this approach
			** Treat tasks like functions
			** Free from objects on which deps get inferred
			* Challenges of this approach
			** Can't infer dep graph based on changes; dev has to grok and specify and update the whole graph
			** Lack of object-based dep management introduces potential flow control problems
			** Lots of versions of inputs when producer gets ahead of consumer
			** Back pressure. Add control flow edges as a back channel to send an empty msg to producer task. Experimenting with buffer depth.
			** CJ: Need a global view wrt resources? How does a give programming model approach help or hurt robustness? Consider a 5-stage pipeline where are all resources could get consumed by early stages that prevent progress in later stages.
			** CJ: Is there a separation between creating the graph via implicit inference and explicit construction and the instantiation of the graph with adequate resources?
			** Stephen: But there are more dynamic allocations of resources possible once the graph is known, rather than doing everything statically. There can be sparse data that would drastically overcommit resources in the static case.
			* Target apps
			** Block sparse matrix algos for sparse tensor algos in TiledArray. Not operating on fixed data structures. Tiles flow thru graph without materializing, so no risk of corrupting data. Would otherwise have to put a copy of data where dep could see it, make it globally visible. Instead, runtime can see and access and transparently copy local data.
			** Multi-resolution function analysis in MADNESS

	= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =		= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =

Cnewburn at 18:02, 21 May 2024

2024-05-21T18:02:48Z

← Older revision		Revision as of 18:02, 21 May 2024
Line 26:		Line 26:

	__TOC__		__TOC__

			!! Be sure to log in before you save, else changes will be lost !!

	= Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024 =		= Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024 =

Cnewburn at 18:00, 21 May 2024

2024-05-21T18:00:21Z

← Older revision		Revision as of 18:00, 21 May 2024
Line 26:		Line 26:

	__TOC__		__TOC__

			= Template Task Graph, Josh Schuchart, Jan 16 and May 21, 2024 =
			* Participants include Wilf, Joseph Schuchart, Alexandre Bardakoff, Andrey Alekseenko, Michael Wong, PiotrL, StephenJ, Szilard Pall, CJ
			* Overview
			** Based on graph work at ICL/UTK, including PARSEC
			** distr data flow as abstract teask graph
			** nodes are template tasks, no preemption
			** unrolled during execution - SPMD
			** concurrent execution across task graphs
			* Input and output edges
			** Can send data to some or all outputs
			** Task can't execute until all inputs available
			** Explicit naming of targets, using keys passed in as a paramter
			** CJ: No separation between implementation of a function and the context of that node's use. THis precludes a separation of concerns
			** The intention here is designing the algo that consists of nodes and edges. No anticipation of task reuse.
			** Szilard: Complex schedule, record a sequence of nodes and edges. Prebaked schedule by a single dev who has most all of the context. If there are extensions, then you might lack reuse.
			** THought about splitting out of a task. Hard to send across functors.
			* Data
			** Moved into graph without making a copy
			** Infra for zero copy xfers across nodes
			** Haven't done this in the context of host-side NUMA systems wince it wasn't an issue, but are doing this for GPUs, where automatic load balancing wasn't good enough
			* Memory model - device
			** TTG manages device mem, host is a backup for transparent eviction wrt oversubscription
			** Buffer has logical home on host, currently always materialized there
			** CJ: we as a community are moving to consider accessors vs. materialized data buffers since data sets may not be able to fit in any amount of memory
			** Stephen: Do you shared the or partition data? No, just replicate.
			** Like Kokkos view, can be owning/non-owning. Track last update/access. Always get latest version.
			** Strong consistency, since all deps are explicit.
			* Execution model
			** declare input and scratch data
			** TTG runtime assigns device and execution stream based on inputs and device load
			** co_await select()'s device, co_await kernel() awaits kernel
			** Sending outputs is the last step of a task
			* Next time
			** Results, apps, compare/contrast discussion
			** Target next month, June 18

	= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =		= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =

Cnewburn at 18:00, 17 January 2023

2023-01-17T18:00:04Z

← Older revision		Revision as of 18:00, 17 January 2023
Line 26:		Line 26:

	__TOC__		__TOC__

			= Kokkos Graphs and Conditionals, Jonathan Liflander, Jan 17, '23 =

			Participants: Stephen Jones, Wilf, Jonathan Liflander, SzilardP, CJ

			Simple diamond conditional, pick one saxpy or another.
			* 4 nodes
			* Non-conditional version does a deep copy of the conditional value to CPU, evaluates conditional on CPU and invokes a graph with one side or the other
			Perf
			* More iterations helps - negative for 10 or 100, positive for 1K, 10K
			* Instantiating graph takes about as much time to run the graph once
			** Without conditionals: 216us, seems high
			** With conditions: 2.76ms, seems very high - order of magnitude more
			* Larger problem sizes wash out the impact, as expected
			* If the instantiation with conditionals was similar, conditionals would have been a clear win
			* Warmup didn't matter much. Stephen: been working to reduce sensitivity for that in CUDA 12
			Conclusion
			* Looks promising; some gain now and we'd hope that'd be much more helpful
			* Common motif among several folks at Sandia. Conditionals are a big gating factor. Looking forward to more conversations with Kokkos kernels team.
			* Working on CG now, implementation nearly done, hoping for next week
			* Benefits may be unclear, especially if instantiation costs aren't predictable
			Feedback
			* Stephen working to formalize the mechanism; should reduce overheads
			* Jonathan to share test case with Stephen, would like to have Jonathan chat with the graphs team
			* Not everything you can put in a graph can be put in a while
			Discussion
			* CJ: Are there or will there by Python wrappers for Kokkos? JL: Not that much need. Mostly C++ or Fortran.
			* CJ: Can be difficult to recognize the processing of a list as a loop, which could have implications.
			* Jonathan is also working on Darma, doing work on load balancing in Python, interacts with C++ code. Uses output to guide distribution. This could be relevant in that domain.
			* Szilard: Python used only for high-level drivers, not in GROMACS context. More focused on balancing high- vs. low-level APIs. Some need for interaction with data operated on by inner loop, but the desire is for that interaction to be flexibly done with a Python API.
			* Jonathan: Python used more as a higher level. Data fed from app, in situ.
			* Szilard: Similar - hot loop, capture and export data, e.g. for viz, sometimes modifying. Can imagine some perf requirements. Cases not so well documented, lots in early stages.
			* JL: Can tank perf if not careful -> Python is pure driver code
			* CJ: Expressing control flow in a way that's amenable to Python may be of increasing interest

	= CUDA Graph Updates, Stephen Jones, Oct 18, '22 =		= CUDA Graph Updates, Stephen Jones, Oct 18, '22 =

Cnewburn at 17:02, 18 October 2022

2022-10-18T17:02:12Z

← Older revision		Revision as of 17:02, 18 October 2022
Line 26:		Line 26:

	__TOC__		__TOC__

			= CUDA Graph Updates, Stephen Jones, Oct 18, '22 =

			Participants: Stephen Jones, Wilf, David Fontaine, PiotrL, WalidK, SzilardP, CJ

			* External dependencies via events and memops
			** in/out deps not permitted with CUDA Graphs, have to split them into diff graphs connected with sync
			** Memops: cuStreamWaitValue(), cuStreamWriteValue()
			* Stream-ordered memory alloc: async take a stream argument
			** cudaMalloc/FreeAsync
			** Can allocate in one graph and free in another, virtual addresses are unique across graphs, but phys addresses may be reused
			** Can change topology of graph for edges connecting nodes involving memory management
			** CUDA may add more edges to serialize to avoid OOM. Create a memory pool before beginning to execute a graph instance that satisfies the worst-case paths thru the graph. Wait to start the graph if you can't create that pool. Add extra edges based on the conditions evaluated at the start of that instance's execution.
			** CUDA implicitly inserts postdominators as necessary in order to track when a graph completes and all of its held pool of unallocated memory can be made available for other graphs
			* Dynamic parallelism
			** used to be able to launch kernels from kernels, now you can launch graphs from graphs
			** Encapsulation boundary - GPU stream within a CPU stream
			* Named streams in CUDA 12.0 later this year; CUDA names classes of patterns that you can use; you're not naming streams
			** cudaStreamPerThread - tell CUDA you can optimize around a straightline pattern
			** cudaStreamFireAndForget - device-side kernels issue concurrently vs. sequentially
			** cudaStreamTailLaunch - sequential after parent completes
			** 3x faster with fork/join disabled (22.5 -> 7.2 us), very close to CPU launch (6.5 us). 1.14x in Mandelbrot test
			* Encapsulation for device-side graph launch
			** Whole launching graph, so can't create new dep that induces fork/join within the parent graph
			** Similar named streams for graphs
			** Adding sibling launch for loops. Sibling is launched outside of deps, a level up, creating a nested encapsulation
			** Can make a decision at runtime about launching an appropriately-selected graph by relaunching scheduler after each launch in a loop
			** Launch from device is 2.2x faster than from host 4.5us vs. 9.9us. Lower latency from a shorter control loop.
			** Extra bookkeeping on CPU is avoided, so pretty flat on GPU vs. scales with concurrency on CPU



	= Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022 =		= Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022 =

Pinfold: /* Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022 */

2022-04-19T17:34:40Z

Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022

← Older revision		Revision as of 17:34, 19 April 2022
Line 43:		Line 43:
	Presentation 2:Multi-node Applications - Nitish Shingte		Presentation 2:Multi-node Applications - Nitish Shingte

	[[~~File~~:Presentation HiHat Cluster.pdf\|Presentation HiHat Cluster]]		[[Media:Presentation HiHat Cluster.pdf\|Presentation HiHat Cluster]]

	Goals		Goals

Pinfold: /* Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022 */

2022-04-19T17:34:09Z

Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022

← Older revision		Revision as of 17:34, 19 April 2022
Line 43:		Line 43:
	Presentation 2:Multi-node Applications - Nitish Shingte		Presentation 2:Multi-node Applications - Nitish Shingte

	[[~~Media~~:Presentation HiHat Cluster.pdf\|Presentation HiHat Cluster]]		[[File:Presentation HiHat Cluster.pdf\|Presentation HiHat Cluster]]

	Goals		Goals

Pinfold: /* Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022 */

2022-04-19T17:29:45Z

Hedgehog Static Analysis and Multi-node Applications on Heterogenous Systems April 19, 2022

← Older revision		Revision as of 17:29, 19 April 2022
Line 46:		Line 46:

	Goals		Goals
	*Extend Hdgehog to support ~~Multi-node Applications~~		*Extend Hdgehog to support Multiple Task graphs Multiple Nodes

	[[File:YouTube_icon.png\|20px\|link=https://youtu.be/e2VueSpXsRM]]		[[File:YouTube_icon.png\|20px\|link=https://youtu.be/e2VueSpXsRM]]