Data abstraction interfaces

The purpose of this page is to record input on requirements, offerings, and proposals for data abstraction interfaces, as part of the overall HiHAT community effort. For a WIP picture of memory abstractions, see here.

• Layers
  • User facing
    • Examples: OpenMP, MPI, Sidre, SICM, CHAI, Umpire, ...
  • Retargetable plumbing
    • Examples: HiHAT User and Common Layer APIs
  • User-level implementations
    • Examples: memkind, libpmem, cnmem, tcmalloc, jemalloc, cudaMalloc, SCR
  • Kernel-level support, and interfaces to that support
    • Examples: OS.SICM, mmap, libnuma

• Requirements
  • Kinds of memory objects
    • Regular - simple contiguous arrays, multi-D arrays with stride, block (ref HDF5, MPI data types)
    • Irregular
  • Representation
    • Declarative approach
      • Semantic properties: accessibility (R/W), size
      • Performance properties: affinity to resources, memory kind, layout
      • Mutable state: pinned, cleared, valid, materialized
    • Layout
      • Distinct from logical handle
      • May or may not be changeable without creating a new instance
    • Hierarchy
      • Provide a formulaic prescription for how data could be partitioned, without overdecomposing it a priori
  • Asynchrony
    • Logical handle is available before async operations complete
    • Async operations: virtual alloc, physical materialization, affinitization, pinning, clearing
  • Allocation
    • Pluggable allocators, e.g. unified, per-thread, same-size
  • Operations
    • Conflicts: perfect match, partial match, overlapping ranges without conflicts ({.x} and {.y,.z})
    • Access: slices, projections