vllm.core.evictor
BlockMetaData
¶
Data structure for storing key data describe cached block, so that evitor could use to make its decision which one to choose for eviction
Here we use physical block id as the dict key, as there maybe several blocks with the same content hash, but their physical id is unique.
Source code in vllm/core/evictor.py
__init__
¶
EvictionPolicy
¶
Bases: Enum
Enum for eviction policy used by make_evictor to instantiate the correct Evictor subclass.
Source code in vllm/core/evictor.py
Evictor
¶
Bases: ABC
The Evictor subclasses should be used by the BlockAllocator class to handle eviction of freed Blocks.
Source code in vllm/core/evictor.py
__contains__
abstractmethod
¶
__init__
abstractmethod
¶
add
abstractmethod
¶
evict
abstractmethod
¶
Runs the eviction algorithm and returns the evicted block's content hash along with physical block id along with physical block id
update
abstractmethod
¶
LRUEvictor
¶
Bases: Evictor
Evicts in a least-recently-used order using the last_accessed timestamp that's recorded in the Block. If there are multiple blocks with the same last_accessed time, then the one with the largest num_hashed_tokens will be evicted. If two blocks each have the lowest last_accessed time and highest num_hashed_tokens value, then one will be chose arbitrarily
Source code in vllm/core/evictor.py
__contains__
¶
__init__
¶
_cleanup
¶
Source code in vllm/core/evictor.py
_cleanup_if_necessary
¶
add
¶
Source code in vllm/core/evictor.py
evict
¶
Source code in vllm/core/evictor.py
make_evictor
¶
make_evictor(eviction_policy: EvictionPolicy) -> Evictor