HHAT Usage Meeting Minutes - Revision history

imported>Cnewburn at 16:46, 20 July 2021

2021-07-20T16:46:08Z

@@ Line 10: / Line 10: @@
 * Layering, hierarchy, dynamism - especially in a multi-node or NUMA context
 * C++ futures for shared memory distributed system?
 == Characterization of Sample cases

imported>Cnewburn at 17:11, 15 June 2021

2021-06-15T17:11:34Z

@@ Line 10: / Line 10: @@
 * Layering, hierarchy, dynamism - especially in a multi-node or NUMA context
 * C++ futures for shared memory distributed system?
 == Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall ==

imported>Cnewburn: /* Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall */

2021-06-08T20:58:49Z

Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall

@@ Line 18: / Line 18: @@
 Participants for May 18 include
 *David, Piotr, Hans, JohnB, JoseM, Stephen, Szilard, Wael, Waid, Wilf, CJ
@@ Line 125: / Line 92: @@
 ** Szilard: Have plenty of CPU resources, so polling shouldn't be an issue
 * Key ask: Need to work on integrating comms.  Queue this up next month.
 == Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20 and Nov 24, Daisy Hollman ==

imported>Cnewburn: /* Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall */

2021-06-08T20:55:52Z

Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall

imported>Cnewburn: /* Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall */

2021-06-08T20:54:05Z

Heterogeneous task scheduling of molecular dynamics in GROMACS - Szilard Pall

← Older revision		Revision as of 13:54, 8 June 2021
Line 16:		Line 16:
	Participants for Feb 16 include		Participants for Feb 16 include
	*Szilard Pall, Alan Gray; AntonioP, David Bernholdt, HansJ, JiriD, Simon De Conzalo, StephenJ, TimB, Wael, Wilf, CJ		*Szilard Pall, Alan Gray; AntonioP, David Bernholdt, HansJ, JiriD, Simon De Conzalo, StephenJ, TimB, Wael, Wilf, CJ
			Participants for May 18 include
			*David, Piotr, Hans, JohnB, JoseM, Stephen, Szilard, Wael, Waid, Wilf, CJ

			Transition from queue/stream to priorities, critical path acceleration
			○ Thinking of memops and streams, building task graphs vs. just recording them
			○ How should we think about the critical path and expression priorities to the runtime
			○ Internal discussions
			○ GROMACS going through a transition for some back ends that are graph scheduling first. SYCL BE. Has been CUDA (in-order queues) first. SYCL doesn't have support now. CUDA is superior wrt features.
			○ Porting is a big effort for a small team.
			○ Starting with explicit dependencies
			○ After looking at multiple use cases and multiple optimizations, realizing that depending on explicit deps would be a liability for correctness. So thinking more about task-based runtimes, how to build task graphs.
			○ Static problem, know the deps ahead of time. Short iterations. Graph is static over a few iterations.
			○ More of a SW engineering challenge. Mixed codes expression CPU-GPU async as well as CPU only in same code base with multiple code bases.
			○ Don't want breadth first if a small fraction of kernels are on critical path.
			○ Perhaps the runtime could enable measurement of time, so we could derive weights of nodes and/or edges. Have a pretty good guess as a programmer. Already do load balancing on CPU, based on measuring compute without communication.
			○ Priorities in the presence of preemption and a greedy scheduler would not be sufficient, unless there is a mechanism for learning over time.
			○ SP: Would be a step forward to express edge weights as metadata. Averages may not always be representative. Fine-grained changes may not make sense for 10s-100s of usec.
			○ SP: Capturing conditionality would enable building in load-balancing capabilities
			○ SJ: Could gather stats. Could steer control flow.
			○ Example of streams and graphs?
			○ SJ: GPU does its own dynamic load balancing, but you can set priorities. Prioritized speed of decision making over flexibility. Can put a marker at end of stream and query whether it's been reached. Breadth or depth first traversal? Now handled only by HW. Added an editable node property for priority from the beginning.
			○ SP: Want B2B not just priority. Want to avoid latency of draining lower-priority work.
			○ SJ: Long challenge to intentional packing. Often hear case of big rocks first.
			○ SJ: Observatoire de Paris - forks left 90% of time, favor it in priority. Make a decision on CPU side which graph version will be launched. Biasing which you fetch and prebuild.
			○ Hans: use case for CUDA Graphs - iterations where branch taken as a function of # iterations. Creates bottlenecks if you assume lock step, since it eats up occupancy. Common for chem kernels. Have to get something done to get back into lockstep. Can't predict what will provide full utilization. Could be priorities on mem prefetch if that's a bottleneck. Would prefer to swap work in dynamically. Work stealing and stuffing. May use ptr swaps.
			○ CJ: could have side threads packing a slate of work, making a trade-off of packers and workers.
			○ SJ: Can only use all threads; can't sequester a subset of resources for independent graph management on each. There are others asking for this. Can't evict work that's currently on the machine, can only stop dispatching new work.
			○ SP: If initial latency can't be reduced, you at least want to avoid recurring latency from packing.
			○ CJ: Specific use cases to drive an assessment of whether priorities are sufficient.
			○ SP: Next step: plot concrete instances using dot interface, can try recording subgraphs into a stream.
			○ SJ: Interest in control or heuristic influence over execution order.
			○ Piotr: tend to have a well-defined critical path, but can get muddy with very large graphs. Graphs often get latency sensitive since they are sub-usec duration. Small tasks with high priority. No one there has ventured into CUDA Graphs. Mostly inside OpenMP tasks now. Can find out whether a transition from OMP tasks to CUDA Graphs is interest.

imported>Cnewburn at 18:02, 16 February 2021

2021-02-16T18:02:01Z

← Older revision		Revision as of 11:02, 16 February 2021
Line 14:		Line 14:
	Participants for Jan 19 include		Participants for Jan 19 include
	*Szilard Pall, Alan Gray; David Bernholdt, Fabian Cordero, Simon De Conzalo, StephenJ, Wael, Wilf, CJ		*Szilard Pall, Alan Gray; David Bernholdt, Fabian Cordero, Simon De Conzalo, StephenJ, Wael, Wilf, CJ
			Participants for Feb 16 include
			*Szilard Pall, Alan Gray; AntonioP, David Bernholdt, HansJ, JiriD, Simon De Conzalo, StephenJ, TimB, Wael, Wilf, CJ


	Szilard's presentation		Szilard's presentation
			* Context
			** MD, with halo exchange
			** Domain decomp, kernels, reduction, integration
			* Characterization
			** Moderate granularity for tasks. Total time per iterations is <10ms, and 2ms is common for strong scaling, can be 100s of us. Limited by all2all comms.
			** Concurrency across kernels is around 12, could be a total of a couple of dozen
			** Looking at how to improve scheduling with CUDA Graphs, how to increase tput, especially strong scaling
			** Inherent requirement: launch in inner loops and reuse 100s of times, also use on CPU, eventually use internode comms.
	* Caching		* Caching
	** MD runs out of cache, materials are latency sensitive		** MD runs out of cache, materials are latency sensitive
	* Decomposition		* Decomposition
	** Spatial		** Spatial
			** Resource re-allocation and load balancing changes made in outer dashed blue loop, not inner grey.
			** Heuristics are a mix of table lookup and measurement based
			** Published last year - Heterogeneous parallelization and acceleration of molecular dynamics simulations in GROMACS
			J. Chem. Phys. 153, 134110 (2020); https://doi.org/10.1063/5.0018516
	* Parallelization		* Parallelization
	** MPI + OpenMP for CPU, CUDA or OpenCL for GPUs		** MPI + OpenMP for CPU, CUDA or OpenCL for GPUs
	** "A few years ago, would have said anything for perf. Now, with programming model wars, only use models where help implementing is provided."		** "A few years ago, would have said anything for perf. Now, with programming model wars, only use models where help implementing is provided."
			** Subset of ranks dedicated to orange FFT work; faster with a limited # of ranks
			** Haven't found a way to rebalance dynamically. No time to steal tasks or shift data. Quite stable across 100s of iterations.
			** For certain types of data, purple or magenta tasks may go away.
	* Perf challenges		* Perf challenges
	** Making data resident on GPU to avoid PCIe overheads. Migrated MD step to GPU, CPU is helper that enables compatibility with a wider range of features. CPU back-offload does infrequent steps.		** Making data resident on GPU to avoid PCIe overheads. Migrated MD step to GPU, CPU is helper that enables compatibility with a wider range of features. CPU back-offload does infrequent steps.
Line 40:		Line 58:
	** Lower for multi-GPU cases, sometimes slower. Implementation limitation for cudaGraphExecUpdate in a multi-GPU context since may not be recaptured the same way. Projections show would help, but still need threshold to figue out when to apply CUDA Graphs.		** Lower for multi-GPU cases, sometimes slower. Implementation limitation for cudaGraphExecUpdate in a multi-GPU context since may not be recaptured the same way. Projections show would help, but still need threshold to figue out when to apply CUDA Graphs.
	* Trade-offs between one graph per timestep but smaller scope means less benefit, and host nodes which could have perf limitations. Composability with libmp may help? CUDA Graphs now support cuStreamWrite/WaitValue32 natively, but it still involves a kernel flag.		* Trade-offs between one graph per timestep but smaller scope means less benefit, and host nodes which could have perf limitations. Composability with libmp may help? CUDA Graphs now support cuStreamWrite/WaitValue32 natively, but it still involves a kernel flag.
			* Reuse
			** 100s of iterations
			** Update when you can. Likely to capture differently, with different ordering.
			Working on minimizing that effect. Expecting not to be a major problem. Dependencies are stored separately and are determined by app.
			** Reason for re-capture: change in parameters only, update. Now cheap enough to update the whole thing that you don't care, as long as you don't re-instantiate.
			** Recording is cheap, instantiation is expensive
			** SP: Can you not launch a task of size zero?
			** SJ: We have other asks for that and that looks feasible. Could be done by setting a flag to false. Could also use conditional execution - greater generality but higher latency. Sounds like you can know this already at launch time.
			** SP: That could be more elegant, but not necessary except for more irregular apps. Might need 2-3 graphs, because of different steps.

	Discussion		Discussion
Line 48:		Line 74:
	* Should work through timing issues related to dynamic scheduling		* Should work through timing issues related to dynamic scheduling
	* How easily and cleanly was the CUDA Graphs-ness encapsulated?		* How easily and cleanly was the CUDA Graphs-ness encapsulated?
			* HiHAT
			** Truly hierarchical and hetero
			* Affinity
			** Manual management of thread affinity with GPU nodes vs. host nodes
			** Not so sensitive to latency
			** Need reuse of thread pool with affinity
			** Can use two pools, e.g. for blue and magenta tasks, but then you don't get locality. Blue and magenta tasks have very different compute requirements.
			** Focused on sharing in L1, L2. Aggressive in managing affinities if not imposed from outside already.
			** Using TLS within OMP loop scope, so must be in the context of thread with TLS.
			** Stephen: would have to be explicit; In CUDA, register a callback and wait for it. Consider a blocking call to wait for event.
			* CPU nodes: Some code not ported to GPU
			** Can expose as a CPU node (few us), or to reverse offload from a GPU node (few 10s of us).
			** Polling off of memory locations
			** Stephen: need to find ways to burn a thread to spin or suffer interrupt latency
			** Szilard: Have plenty of CPU resources, so polling shouldn't be an issue
			* Key ask: Need to work on integrating comms. Queue this up next month.

	== Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20 and Nov 24, Daisy Hollman ==		== Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20 and Nov 24, Daisy Hollman ==

imported>Cnewburn at 18:00, 19 January 2021

2021-01-19T18:00:35Z

@@ Line 10: / Line 10: @@
 * Layering, hierarchy, dynamism - especially in a multi-node or NUMA context
 * C++ futures for shared memory distributed system?
 == Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20 and Nov 24, Daisy Hollman ==

imported>Cnewburn at 17:57, 24 November 2020

2020-11-24T17:57:14Z

← Older revision		Revision as of 10:57, 24 November 2020
Line 11:		Line 11:
	* C++ futures for shared memory distributed system?		* C++ futures for shared memory distributed system?

	== Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20, Daisy Hollman ==		== Thoughts on Integrating CudaGraph Features into Performance Portability Libraries, Oct 20 and Nov 24, Daisy Hollman ==
	Participants include (19)		Participants for Oct 20 include (19)
	* Daisy Hollman, DavidB, WaelE, StephenO, AlexB, BenoitM, DmitryL, HansJ, JanC, JohnB, JohnS, Leonel, PiotrL, SzilardP, TalB, TimB, Wilf, CJ		* Daisy Hollman, DavidB, WaelE, StephenO, AlexB, BenoitM, DmitryL, HansJ, JanC, JohnB, JohnS, Leonel, PiotrL, SzilardP, TalB, TimB, Wilf, CJ

			Participants for Nov 24 include ()
			* AlexB, AntoninoT, CJ, DaisyH, DavidB, FabianC, Jan Ciesko, JohnB, PiotrL, StephenJ, StephenO, TimB, WaelE, Wilf

	Goals of KokkosGraphs		Goals of KokkosGraphs
Line 23:		Line 26:
	* Could write a higher-level scheduler on top of this		* Could write a higher-level scheduler on top of this
	* Concerns about only being able to change parameters and not attributes and structure. Wants to add/rm nodes.		* Concerns about only being able to change parameters and not attributes and structure. Wants to add/rm nodes.
			* Mapping a restricted programming model onto CUDA Graphs as an execution model

	Questions to ask		Questions to ask
Line 37:		Line 41:
	* Concrete plans in Kokkos for SYCL Graphs, which isn't very mature yet		* Concrete plans in Kokkos for SYCL Graphs, which isn't very mature yet
	* Interesting transitions: Launch-launch, destroy-build		* Interesting transitions: Launch-launch, destroy-build
			* SJ: Trying to expose what's efficient and not? DH: Yes. Good to make it look expensive to reconstruct a new graph.
			* SJ: By removing some arrows, all changes may end up looking equally expensive, which they are not. DH: Trying to recover that, with persistent objects which can be modified while in or out of execution phase. SJ: Makes sense.
			* DH: Really wants conditional execution. SJ: Available via a special experimental header.

	C++ object lifetimes		C++ object lifetimes

imported>Cnewburn at 17:23, 20 October 2020

2020-10-20T17:23:51Z

@@ Line 10: / Line 10: @@
 * Layering, hierarchy, dynamism - especially in a multi-node or NUMA context
 * C++ futures for shared memory distributed system?
 == [[Media:Tasking_in_Accelerators_CudaGraph+OpenACC.pdf|Tasking_in_Accelerators_CudaGraph+OpenACC]], Aug 18, 2020, Leonel Toledo, Antonio Pena ==

imported>Pinfold: /* Tasking_in_Accelerators_CudaGraph+OpenACC, Aug 18, 2020, Leonel Toledo, Antonio Pena */

2020-08-19T03:41:39Z

Tasking_in_Accelerators_CudaGraph+OpenACC, Aug 18, 2020, Leonel Toledo, Antonio Pena

← Older revision		Revision as of 20:41, 18 August 2020
Line 13:		Line 13:
	== [[Media:Tasking_in_Accelerators_CudaGraph+OpenACC.pdf\|Tasking_in_Accelerators_CudaGraph+OpenACC]], Aug 18, 2020, Leonel Toledo, Antonio Pena ==		== [[Media:Tasking_in_Accelerators_CudaGraph+OpenACC.pdf\|Tasking_in_Accelerators_CudaGraph+OpenACC]], Aug 18, 2020, Leonel Toledo, Antonio Pena ==
	Participants included		Participants included
	* Alexandre Bardakoff, CJ, David Bernholdt, James Beyer, Jeff Larkin, Leonel, Stephen Jones, Tim Blattner, Stephen Olivier, Wilfred Pinfold, Wael Elwasif, Fabian Mora Cordero, Piotr Luszczek,		* Alexandre Bardakoff, CJ Newburn, David Bernholdt, James Beyer, Jeff Larkin, Leonel Toledo, Stephen Jones, Tim Blattner, Stephen Olivier, Wilfred Pinfold, Wael Elwasif, Fabian Mora Cordero, Piotr Luszczek

	* Showed speedup from adding CUDA Graphs to OpenACC, compared with OMPSS and other implementations		* Showed speedup from adding CUDA Graphs to OpenACC, compared with OMPSS and other implementations

← Older revision		Revision as of 13:55, 8 June 2021
Line 20:		Line 20:

	Transition from queue/stream to priorities, critical path acceleration		Transition from queue/stream to priorities, critical path acceleration
	○ Thinking of memops and streams, building task graphs vs. just recording them		* Thinking of memops and streams, building task graphs vs. just recording them
	○ How should we think about the critical path and expression priorities to the runtime		* How should we think about the critical path and expression priorities to the runtime
	○ Internal discussions		* Internal discussions
	○ GROMACS going through a transition for some back ends that are graph scheduling first. SYCL BE. Has been CUDA (in-order queues) first. SYCL doesn't have support now. CUDA is superior wrt features.		* GROMACS going through a transition for some back ends that are graph scheduling first. SYCL BE. Has been CUDA (in-order queues) first. SYCL doesn't have support now. CUDA is superior wrt features.
	○ Porting is a big effort for a small team.		* Porting is a big effort for a small team.
	○ Starting with explicit dependencies		* Starting with explicit dependencies
	○ After looking at multiple use cases and multiple optimizations, realizing that depending on explicit deps would be a liability for correctness. So thinking more about task-based runtimes, how to build task graphs.		* After looking at multiple use cases and multiple optimizations, realizing that depending on explicit deps would be a liability for correctness. So thinking more about task-based runtimes, how to build task graphs.
	○ Static problem, know the deps ahead of time. Short iterations. Graph is static over a few iterations.		* Static problem, know the deps ahead of time. Short iterations. Graph is static over a few iterations.
	○ More of a SW engineering challenge. Mixed codes expression CPU-GPU async as well as CPU only in same code base with multiple code bases.		* More of a SW engineering challenge. Mixed codes expression CPU-GPU async as well as CPU only in same code base with multiple code bases.
	○ Don't want breadth first if a small fraction of kernels are on critical path.		* Don't want breadth first if a small fraction of kernels are on critical path.
	○ Perhaps the runtime could enable measurement of time, so we could derive weights of nodes and/or edges. Have a pretty good guess as a programmer. Already do load balancing on CPU, based on measuring compute without communication.		* Perhaps the runtime could enable measurement of time, so we could derive weights of nodes and/or edges. Have a pretty good guess as a programmer. Already do load balancing on CPU, based on measuring compute without communication.
	○ Priorities in the presence of preemption and a greedy scheduler would not be sufficient, unless there is a mechanism for learning over time.		* Priorities in the presence of preemption and a greedy scheduler would not be sufficient, unless there is a mechanism for learning over time.
	○ SP: Would be a step forward to express edge weights as metadata. Averages may not always be representative. Fine-grained changes may not make sense for 10s-100s of usec.		* SP: Would be a step forward to express edge weights as metadata. Averages may not always be representative. Fine-grained changes may not make sense for 10s-100s of usec.
	○ SP: Capturing conditionality would enable building in load-balancing capabilities		* SP: Capturing conditionality would enable building in load-balancing capabilities
	○ SJ: Could gather stats. Could steer control flow.		* SJ: Could gather stats. Could steer control flow.
	○ Example of streams and graphs?		* Example of streams and graphs?
	○ SJ: GPU does its own dynamic load balancing, but you can set priorities. Prioritized speed of decision making over flexibility. Can put a marker at end of stream and query whether it's been reached. Breadth or depth first traversal? Now handled only by HW. Added an editable node property for priority from the beginning.		* SJ: GPU does its own dynamic load balancing, but you can set priorities. Prioritized speed of decision making over flexibility. Can put a marker at end of stream and query whether it's been reached. Breadth or depth first traversal? Now handled only by HW. Added an editable node property for priority from the beginning.
	○ SP: Want B2B not just priority. Want to avoid latency of draining lower-priority work.		* SP: Want B2B not just priority. Want to avoid latency of draining lower-priority work.
	○ SJ: Long challenge to intentional packing. Often hear case of big rocks first.		* SJ: Long challenge to intentional packing. Often hear case of big rocks first.
	○ SJ: Observatoire de Paris - forks left 90% of time, favor it in priority. Make a decision on CPU side which graph version will be launched. Biasing which you fetch and prebuild.		* SJ: Observatoire de Paris - forks left 90% of time, favor it in priority. Make a decision on CPU side which graph version will be launched. Biasing which you fetch and prebuild.
	○ Hans: use case for CUDA Graphs - iterations where branch taken as a function of # iterations. Creates bottlenecks if you assume lock step, since it eats up occupancy. Common for chem kernels. Have to get something done to get back into lockstep. Can't predict what will provide full utilization. Could be priorities on mem prefetch if that's a bottleneck. Would prefer to swap work in dynamically. Work stealing and stuffing. May use ptr swaps.		* Hans: use case for CUDA Graphs - iterations where branch taken as a function of # iterations. Creates bottlenecks if you assume lock step, since it eats up occupancy. Common for chem kernels. Have to get something done to get back into lockstep. Can't predict what will provide full utilization. Could be priorities on mem prefetch if that's a bottleneck. Would prefer to swap work in dynamically. Work stealing and stuffing. May use ptr swaps.
	○ CJ: could have side threads packing a slate of work, making a trade-off of packers and workers.		* CJ: could have side threads packing a slate of work, making a trade-off of packers and workers.
	○ SJ: Can only use all threads; can't sequester a subset of resources for independent graph management on each. There are others asking for this. Can't evict work that's currently on the machine, can only stop dispatching new work.		* SJ: Can only use all threads; can't sequester a subset of resources for independent graph management on each. There are others asking for this. Can't evict work that's currently on the machine, can only stop dispatching new work.
	○ SP: If initial latency can't be reduced, you at least want to avoid recurring latency from packing.		* SP: If initial latency can't be reduced, you at least want to avoid recurring latency from packing.
	○ CJ: Specific use cases to drive an assessment of whether priorities are sufficient.		* CJ: Specific use cases to drive an assessment of whether priorities are sufficient.
	○ SP: Next step: plot concrete instances using dot interface, can try recording subgraphs into a stream.		* SP: Next step: plot concrete instances using dot interface, can try recording subgraphs into a stream.
	○ SJ: Interest in control or heuristic influence over execution order.		* SJ: Interest in control or heuristic influence over execution order.
	○ Piotr: tend to have a well-defined critical path, but can get muddy with very large graphs. Graphs often get latency sensitive since they are sub-usec duration. Small tasks with high priority. No one there has ventured into CUDA Graphs. Mostly inside OpenMP tasks now. Can find out whether a transition from OMP tasks to CUDA Graphs is interest.		* Piotr: tend to have a well-defined critical path, but can get muddy with very large graphs. Graphs often get latency sensitive since they are sub-usec duration. Small tasks with high priority. No one there has ventured into CUDA Graphs. Mostly inside OpenMP tasks now. Can find out whether a transition from OMP tasks to CUDA Graphs is of interest.