parent 8dde327

author Chorazewicz, Igor <[email protected]> 1632834667 +0200
committer Daniel Byrne <[email protected]> 1671070203 -0800

Initial multi-tier support implementation

Extend CompressedPtr to work with multiple tiers

Now it's size is 8 bytes intead of 4.

Original CompressedPtr stored only some offset with a memory Allocator.
For multi-tier implementation, this is not enough. We must also store
tierId and when uncompressing, select a proper allocator.

An alternative could be to just resign from CompressedPtr but they
are leveraged to allow the cache to be mapped to different addresses on shared memory.

Changing CompressedPtr impacted CacheItem size - it increased from 32 to 44 bytes.

Implemented async Item movement between tiers (with corrected
behaivour of tryEvictToNextMemoryTier).

Fix ReaperSkippingSlabTraversalWhileSlabReleasing test

The issue was caused by incorrect behaviour of the
CacheAllocator<CacheTrait>::tryEvictToNextMemoryTier method in case the
evicted item is expired. We cannot simply return a handle to it, but we need
to remove it from the access container and MM container.

Enable workarounds in tests

Add basic multi-tier test

Set correct size for each memory tier

Do not compensate for rounding error when calculating tier sizes (pmem#43)

Compensation results in ratios being different than originially specified.

Fixed total cache size in CacheMemoryStats (pmem#38)

Return a sum of sizes of each tier instead of just 1st tier's size.

Issue75 rebased (pmem#88)

* pmem#75: Use actual tier sizes (rounded down to slab size and decreased by header size) when creating new memory pools

* Added getPoolSize method to calculate combined pool size for all tiers; added pool size validation to tests

* Explicitly specified type for totalCacheSize to avoid overflow

* Minor test change

* Reworked tests

* Minor change

* Deleted redundant tests

* Deleted unused constant

* First set of changes to cache configuration API to enable multi-tier caches (facebook#138)

Summary:
These changes introduce per-tier cache configuration required to implement features discussed here: facebook#102. These specific changes enable single DRAM tier configs only which are compatible with the current version of cachelib. Configuration API will be expanded as multi-tier changes in other parts of the library are introduced.

Pull Request resolved: facebook#138

Reviewed By: therealgymmy

Differential Revision: D36189766

Pulled By: jiayuebao

fbshipit-source-id: 947aa0cd800ea6accffc1b7b6b0c9693aa7fc0a5

Co-authored-by: Victoria McGrath <[email protected]>

Fix eviction flow and removeCb calls

Without this fix removeCb called even in case when Item is moved between
tiers.

Fix issue with "Destorying an unresolved handle"

The issue happened when ReadHandleImpl ctor needs to destroy
waitContext_ because addWaitContextForMovingItem() returns false.
So before destroying waitContext_ we are calling discard method to
notify ~ItemWaitContext() that Item is ready.

Fix slab release code

Get tier id of item before calling any function on allocator
(which needs the tierID).

Author: igchor

cachelib/allocator/CacheAllocator-inl.h

@@ -26,7 +26,6 @@
 #include <string>
 
 #include "cachelib/allocator/Cache.h"
-#include "cachelib/allocator/MemoryTierCacheConfig.h"
 #include "cachelib/allocator/MM2Q.h"
 #include "cachelib/allocator/MemoryMonitor.h"
 #include "cachelib/allocator/MemoryTierCacheConfig.h"
@@ -380,7 +379,6 @@ class CacheAllocatorConfig {
   std::map<std::string, std::string> serialize() const;
 
   // The max number of memory cache tiers
-  // TODO: increase this number when multi-tier configs are enabled
   inline static const size_t kMaxCacheMemoryTiers = 2;
 
   // Cache name for users to indentify their own cache.
@@ -884,11 +882,6 @@ CacheAllocatorConfig<T>& CacheAllocatorConfig<T>::configureMemoryTiers(
 template <typename T>
 const typename CacheAllocatorConfig<T>::MemoryTierConfigs&
 CacheAllocatorConfig<T>::getMemoryTierConfigs() const {
-  for (auto &tier_config: memoryTierConfigs) {
-    if (auto *v = std::get_if<PosixSysVSegmentOpts>(&tier_config.shmOpts)) {
-      const_cast<PosixSysVSegmentOpts*>(v)->usePosix = usePosixShm;
-    }
-  }
   return memoryTierConfigs;
 }
 

cachelib/allocator/CacheAllocator.h

@@ -266,6 +266,21 @@ bool CacheItem<CacheTrait>::isNvmEvicted() const noexcept {
   return ref_.isNvmEvicted();
 }
 
+template <typename CacheTrait>
+void CacheItem<CacheTrait>::markIncomplete() noexcept {
+  ref_.markIncomplete();
+}
+
+template <typename CacheTrait>
+void CacheItem<CacheTrait>::unmarkIncomplete() noexcept {
+  ref_.unmarkIncomplete();
+}
+
+template <typename CacheTrait>
+bool CacheItem<CacheTrait>::isIncomplete() const noexcept {
+  return ref_.isIncomplete();
+}
+
 template <typename CacheTrait>
 void CacheItem<CacheTrait>::markIsChainedItem() noexcept {
   XDCHECK(!hasChainedItem());

cachelib/allocator/CacheAllocatorConfig.h

@@ -141,6 +141,7 @@ class CACHELIB_PACKED_ATTR CacheItem {
    * to be mapped to different addresses on shared memory.
    */
   using CompressedPtr = facebook::cachelib::CompressedPtr;
+  using SingleTierPtrCompressor = MemoryAllocator::SingleTierPtrCompressor<Item>;
   using PtrCompressor = MemoryAllocator::PtrCompressor<Item>;
 
   // Get the required size for a cache item given the size of memory
@@ -241,6 +242,14 @@ class CACHELIB_PACKED_ATTR CacheItem {
   void unmarkNvmEvicted() noexcept;
   bool isNvmEvicted() const noexcept;
 
+  /**
+   * Marks that the item is migrating between memory tiers and
+   * not ready for access now. Accessing thread should wait.
+   */
+  void markIncomplete() noexcept;
+  void unmarkIncomplete() noexcept;
+  bool isIncomplete() const noexcept;
+
   /**
    * Function to set the timestamp for when to expire an item
    *

cachelib/allocator/CacheItem-inl.h

@@ -402,6 +402,12 @@ struct ReadHandleImpl {
       }
     }
 
+   protected:
+    friend class ReadHandleImpl;
+    // Method used only by ReadHandleImpl ctor
+    void discard() {
+      it_.store(nullptr, std::memory_order_relaxed);
+    }
    private:
     // we are waiting on Item* to be set to a value. One of the valid values is
     // nullptr. So choose something that we dont expect to indicate a ptr
@@ -481,7 +487,15 @@ struct ReadHandleImpl {
 
   // Handle which has the item already
   FOLLY_ALWAYS_INLINE ReadHandleImpl(Item* it, CacheT& alloc) noexcept
-      : alloc_(&alloc), it_(it) {}
+      : alloc_(&alloc), it_(it) {
+    if (it_ && it_->isIncomplete()) {
+      waitContext_ = std::make_shared<ItemWaitContext>(alloc);
+      if (!alloc_->addWaitContextForMovingItem(it->getKey(), waitContext_)) {
+        waitContext_->discard();
+        waitContext_.reset();
+      }
+    }
+  }
 
   // handle that has a wait context allocated. Used for async handles
   // In this case, the it_ will be filled in asynchronously and mulitple

cachelib/allocator/CacheItem.h

@@ -71,6 +71,7 @@ class MemoryTierCacheConfig {
 
   const ShmTypeOpts& getShmTypeOpts() const noexcept { return shmOpts; }
 
+private:
   // Ratio is a number of parts of the total cache size to be allocated for this
   // tier. E.g. if X is a total cache size, Yi are ratios specified for memory
   // tiers, and Y is the sum of all Yi, then size of the i-th tier
@@ -81,10 +82,6 @@ class MemoryTierCacheConfig {
   // Options specific to shm type
   ShmTypeOpts shmOpts;
 
- private:
-  // TODO: introduce a container for tier settings when adding support for
-  // file-mapped memory
-
   MemoryTierCacheConfig() = default;
 };
 } // namespace cachelib

cachelib/allocator/Handle.h

@@ -51,6 +51,8 @@ void PoolOptimizer::optimizeRegularPoolSizes() {
 
 void PoolOptimizer::optimizeCompactCacheSizes() {
   try {
+    // TODO: should optimizer look at each tier individually?
+    // If yes, then resizePools should be per-tier
     auto strategy = cache_.getPoolOptimizeStrategy();
     if (!strategy) {
       strategy = strategy_;

cachelib/allocator/MemoryTierCacheConfig.h

@@ -116,6 +116,10 @@ class FOLLY_PACK_ATTR RefcountWithFlags {
     // unevictable in the past.
     kUnevictable_NOOP,
 
+    // Item is accecible but content is not ready yet. Used by eviction
+    // when Item is moved between memory tiers.
+    kIncomplete,
+
     // Unused. This is just to indciate the maximum number of flags
     kFlagMax,
   };
@@ -336,6 +340,14 @@ class FOLLY_PACK_ATTR RefcountWithFlags {
   void unmarkNvmEvicted() noexcept { return unSetFlag<kNvmEvicted>(); }
   bool isNvmEvicted() const noexcept { return isFlagSet<kNvmEvicted>(); }
 
+  /**
+   * Marks that the item is migrating between memory tiers and
+   * not ready for access now. Accessing thread should wait.
+   */
+  void markIncomplete() noexcept { return setFlag<kIncomplete>(); }
+  void unmarkIncomplete() noexcept { return unSetFlag<kIncomplete>(); }
+  bool isIncomplete() const noexcept { return isFlagSet<kIncomplete>(); }
+
   // Whether or not an item is completely drained of access
   // Refcount is 0 and the item is not linked, accessible, nor exclusive
   bool isDrained() const noexcept { return getRefWithAccessAndAdmin() == 0; }

cachelib/allocator/PoolOptimizer.cpp

@@ -50,7 +50,7 @@ AllocationClass::AllocationClass(ClassId classId,
       poolId_(poolId),
       allocationSize_(allocSize),
       slabAlloc_(s),
-      freedAllocations_{slabAlloc_.createPtrCompressor<FreeAlloc>()} {
+      freedAllocations_{slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()} {
   checkState();
 }
 
@@ -102,7 +102,7 @@ AllocationClass::AllocationClass(
       currSlab_(s.getSlabForIdx(*object.currSlabIdx())),
       slabAlloc_(s),
       freedAllocations_(*object.freedAllocationsObject(),
-                        slabAlloc_.createPtrCompressor<FreeAlloc>()),
+                        slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()),
       canAllocate_(*object.canAllocate()) {
   if (!slabAlloc_.isRestorable()) {
     throw std::logic_error("The allocation class cannot be restored.");
@@ -356,9 +356,9 @@ std::pair<bool, std::vector<void*>> AllocationClass::pruneFreeAllocs(
   // allocated slab, release any freed allocations belonging to this slab.
   // Set the bit to true if the corresponding allocation is freed, false
   // otherwise.
-  FreeList freeAllocs{slabAlloc_.createPtrCompressor<FreeAlloc>()};
-  FreeList notInSlab{slabAlloc_.createPtrCompressor<FreeAlloc>()};
-  FreeList inSlab{slabAlloc_.createPtrCompressor<FreeAlloc>()};
+  FreeList freeAllocs{slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()};
+  FreeList notInSlab{slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()};
+  FreeList inSlab{slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()};
 
   lock_->lock_combine([&]() {
     // Take the allocation class free list offline