Skip to content

vllm.transformers_utils.configs.arctic

Arctic model configuration

ARCTIC_PRETRAINED_CONFIG_ARCHIVE_MAP module-attribute 

ARCTIC_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "arctic": "https://huggingface.co/Snowflake/snowflake-arctic-instruct/tree/main/config.json"
}

logger module-attribute 

logger = get_logger(__name__)

ArcticConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [ArcticModel]. It is used to instantiate an Arctic model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the #TODO(rsamdani): add what model has the default config..

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

Parameters:

Name Type Description Default
vocab_size `int`, *optional*, defaults to 32000

Vocabulary size of the Arctic model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [ArcticModel]

 32000
hidden_size `int`, *optional*, defaults to 4096

Dimension of the hidden representations.

 4096
intermediate_size `int`, *optional*, defaults to 14336

Dimension of the MLP representations.

 14336
num_hidden_layers `int`, *optional*, defaults to 32

Number of hidden layers in the Transformer encoder.

 32
num_attention_heads `int`, *optional*, defaults to 32

Number of attention heads for each attention layer in the Transformer encoder.

 32
num_key_value_heads `int`, *optional*, defaults to 8

This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to8`.

 None
hidden_act `str` or `function`, *optional*, defaults to `"silu"`

The non-linear activation function (function or string) in the decoder.

 'silu'
max_position_embeddings `int`, *optional*, defaults to `4096*32`

The maximum sequence length that this model might ever be used with. Arctic's sliding window attention allows sequence of up to 4096*32 tokens.

 4096
initializer_range `float`, *optional*, defaults to 0.02

The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

 0.02
rms_norm_eps `float`, *optional*, defaults to 1e-05

The epsilon used by the rms normalization layers.

 1e-05
use_cache `bool`, *optional*, defaults to `True`

Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True.

 True
pad_token_id `int`, *optional*

The id of the padding token.

 None
bos_token_id `int`, *optional*, defaults to 1

The id of the "beginning-of-sequence" token.

 1
eos_token_id `int`, *optional*, defaults to 2

The id of the "end-of-sequence" token.

 2
tie_word_embeddings `bool`, *optional*, defaults to `False`

Whether the model's input and output word embeddings should be tied.

 False
rope_theta `float`, *optional*, defaults to 1000000.0

The base period of the RoPE embeddings.

 1000000.0
sliding_window `int`, *optional*

Sliding window attention window size. If not specified, will default to 4096.

 None
attention_dropout `float`, *optional*, defaults to 0.0

The dropout ratio for the attention probabilities.

 0.0
num_experts_per_tok `int`, *optional*, defaults to 2

The number of experts to root per-token, can be also interpreted as the top-p routing parameter

 1
num_local_experts `int`, *optional*, defaults to 8

Number of experts per Sparse MLP layer.

 8
router_aux_loss_coef `float`, *optional*, defaults to 0.001

The aux loss factor for the total loss.

 0.001 
>>> from transformers import ArcticModel, ArcticConfig

>>> # Initializing a Arctic 7B style configuration TODO(rsamdani): verify which model does the default configuration correspond to.
>>> configuration = ArcticConfig()

>>> # Initializing a model from the Arctic 7B style configuration
>>> model = ArcticModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
Source code in vllm/transformers_utils/configs/arctic.py 
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
class ArcticConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`ArcticModel`]. It is used to instantiate an
    Arctic model according to the specified arguments, defining the model architecture. Instantiating a configuration
    with the defaults will yield a similar configuration to that of the #TODO(rsamdani): add what model has the default config..


    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 32000):
            Vocabulary size of the Arctic model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`ArcticModel`]
        hidden_size (`int`, *optional*, defaults to 4096):
            Dimension of the hidden representations.
        intermediate_size (`int`, *optional*, defaults to 14336):
            Dimension of the MLP representations.
        num_hidden_layers (`int`, *optional*, defaults to 32):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 32):
            Number of attention heads for each attention layer in the Transformer encoder.
        num_key_value_heads (`int`, *optional*, defaults to 8):
            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
            by meanpooling all the original heads within that group. For more details checkout [this
            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `8`.
        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
            The non-linear activation function (function or string) in the decoder.
        max_position_embeddings (`int`, *optional*, defaults to `4096*32`):
            The maximum sequence length that this model might ever be used with. Arctic's sliding window attention
            allows sequence of up to 4096*32 tokens.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
            The epsilon used by the rms normalization layers.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models). Only
            relevant if `config.is_decoder=True`.
        pad_token_id (`int`, *optional*):
            The id of the padding token.
        bos_token_id (`int`, *optional*, defaults to 1):
            The id of the "beginning-of-sequence" token.
        eos_token_id (`int`, *optional*, defaults to 2):
            The id of the "end-of-sequence" token.
        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
            Whether the model's input and output word embeddings should be tied.
        rope_theta (`float`, *optional*, defaults to 1000000.0):
            The base period of the RoPE embeddings.
        sliding_window (`int`, *optional*):
            Sliding window attention window size. If not specified, will default to `4096`.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        num_experts_per_tok (`int`, *optional*, defaults to 2):
            The number of experts to root per-token, can be also interpreted as the `top-p` routing
            parameter
        num_local_experts (`int`, *optional*, defaults to 8):
            Number of experts per Sparse MLP layer.
        router_aux_loss_coef (`float`, *optional*, defaults to 0.001):
            The aux loss factor for the total loss.

    ```python
    >>> from transformers import ArcticModel, ArcticConfig

    >>> # Initializing a Arctic 7B style configuration TODO(rsamdani): verify which model does the default configuration correspond to.
    >>> configuration = ArcticConfig()

    >>> # Initializing a model from the Arctic 7B style configuration
    >>> model = ArcticModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```"""

    model_type = "arctic"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=32000,
        hidden_size=4096,
        intermediate_size=14336,
        num_hidden_layers=32,
        num_attention_heads=32,
        num_key_value_heads=None,
        hidden_act="silu",
        max_position_embeddings=4096,
        initializer_range=0.02,
        rms_norm_eps=1e-5,
        use_cache=True,
        pad_token_id=None,
        bos_token_id=1,
        eos_token_id=2,
        tie_word_embeddings=False,
        rope_theta=1e6,
        sliding_window=None,
        attention_dropout=0.0,
        num_experts_per_tok=1,
        num_local_experts=8,
        router_aux_loss_coef=0.001,
        moe_layer_frequency=2,
        parallel_attn_mlp_res=False,
        moe_train_capacity_factor=1,
        moe_eval_capacity_factor=1,
        enable_expert_tensor_parallelism=False,
        moe_min_capacity=0,
        moe_token_dropping=True,
        quantization=None,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.sliding_window = sliding_window

        # for backward compatibility
        if num_key_value_heads is None:
            num_key_value_heads = num_attention_heads

        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.attention_dropout = attention_dropout

        self.num_experts_per_tok = num_experts_per_tok
        self.num_local_experts = num_local_experts
        self.router_aux_loss_coef = router_aux_loss_coef
        self.moe_layer_frequency = moe_layer_frequency
        self.moe_train_capacity_factor = moe_train_capacity_factor
        self.moe_eval_capacity_factor = moe_eval_capacity_factor
        self.enable_expert_tensor_parallelism = enable_expert_tensor_parallelism
        self.moe_min_capacity = moe_min_capacity
        self.moe_token_dropping = moe_token_dropping
        self.parallel_attn_mlp_res = parallel_attn_mlp_res
        if isinstance(quantization, dict):
            self.quantization = ArcticQuantizationConfig(**quantization)
        else:
            self.quantization = quantization

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )

    @classmethod
    def from_dict(cls, config_dict: dict[str, Any], **kwargs) -> "ArcticConfig":
        result = super().from_dict(config_dict, **kwargs)
        config = result[0] if isinstance(result, tuple) else result
        if isinstance(config.quantization, dict):
            config.quantization = ArcticQuantizationConfig(**config.quantization)
        return result

    def to_dict(self) -> dict[str, Any]:
        ret = super().to_dict()
        if isinstance(ret["quantization"], ArcticQuantizationConfig):
            ret["quantization"] = asdict(ret["quantization"])
        return ret

attention_dropout instance-attribute 

attention_dropout = attention_dropout

enable_expert_tensor_parallelism instance-attribute 

enable_expert_tensor_parallelism = (
    enable_expert_tensor_parallelism
)

hidden_act instance-attribute 

hidden_act = hidden_act

hidden_size instance-attribute 

hidden_size = hidden_size

initializer_range instance-attribute 

initializer_range = initializer_range

intermediate_size instance-attribute 

intermediate_size = intermediate_size

keys_to_ignore_at_inference class-attribute instance-attribute 

keys_to_ignore_at_inference = ['past_key_values']

max_position_embeddings instance-attribute 

max_position_embeddings = max_position_embeddings

model_type class-attribute instance-attribute 

model_type = 'arctic'

moe_eval_capacity_factor instance-attribute 

moe_eval_capacity_factor = moe_eval_capacity_factor

moe_layer_frequency instance-attribute 

moe_layer_frequency = moe_layer_frequency

moe_min_capacity instance-attribute 

moe_min_capacity = moe_min_capacity

moe_token_dropping instance-attribute 

moe_token_dropping = moe_token_dropping

moe_train_capacity_factor instance-attribute 

moe_train_capacity_factor = moe_train_capacity_factor

num_attention_heads instance-attribute 

num_attention_heads = num_attention_heads

num_experts_per_tok instance-attribute 

num_experts_per_tok = num_experts_per_tok

num_hidden_layers instance-attribute 

num_hidden_layers = num_hidden_layers

num_key_value_heads instance-attribute 

num_key_value_heads = num_key_value_heads

num_local_experts instance-attribute 

num_local_experts = num_local_experts

parallel_attn_mlp_res instance-attribute 

parallel_attn_mlp_res = parallel_attn_mlp_res

quantization instance-attribute 

quantization = ArcticQuantizationConfig(**quantization)

rms_norm_eps instance-attribute 

rms_norm_eps = rms_norm_eps

rope_theta instance-attribute 

rope_theta = rope_theta

router_aux_loss_coef instance-attribute 

router_aux_loss_coef = router_aux_loss_coef

sliding_window instance-attribute 

sliding_window = sliding_window

use_cache instance-attribute 

use_cache = use_cache

vocab_size instance-attribute 

vocab_size = vocab_size

__init__ 

__init__(
    vocab_size=32000,
    hidden_size=4096,
    intermediate_size=14336,
    num_hidden_layers=32,
    num_attention_heads=32,
    num_key_value_heads=None,
    hidden_act="silu",
    max_position_embeddings=4096,
    initializer_range=0.02,
    rms_norm_eps=1e-05,
    use_cache=True,
    pad_token_id=None,
    bos_token_id=1,
    eos_token_id=2,
    tie_word_embeddings=False,
    rope_theta=1000000.0,
    sliding_window=None,
    attention_dropout=0.0,
    num_experts_per_tok=1,
    num_local_experts=8,
    router_aux_loss_coef=0.001,
    moe_layer_frequency=2,
    parallel_attn_mlp_res=False,
    moe_train_capacity_factor=1,
    moe_eval_capacity_factor=1,
    enable_expert_tensor_parallelism=False,
    moe_min_capacity=0,
    moe_token_dropping=True,
    quantization=None,
    **kwargs,
)
Source code in vllm/transformers_utils/configs/arctic.py 
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
def __init__(
    self,
    vocab_size=32000,
    hidden_size=4096,
    intermediate_size=14336,
    num_hidden_layers=32,
    num_attention_heads=32,
    num_key_value_heads=None,
    hidden_act="silu",
    max_position_embeddings=4096,
    initializer_range=0.02,
    rms_norm_eps=1e-5,
    use_cache=True,
    pad_token_id=None,
    bos_token_id=1,
    eos_token_id=2,
    tie_word_embeddings=False,
    rope_theta=1e6,
    sliding_window=None,
    attention_dropout=0.0,
    num_experts_per_tok=1,
    num_local_experts=8,
    router_aux_loss_coef=0.001,
    moe_layer_frequency=2,
    parallel_attn_mlp_res=False,
    moe_train_capacity_factor=1,
    moe_eval_capacity_factor=1,
    enable_expert_tensor_parallelism=False,
    moe_min_capacity=0,
    moe_token_dropping=True,
    quantization=None,
    **kwargs,
):
    self.vocab_size = vocab_size
    self.max_position_embeddings = max_position_embeddings
    self.hidden_size = hidden_size
    self.intermediate_size = intermediate_size
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.sliding_window = sliding_window

    # for backward compatibility
    if num_key_value_heads is None:
        num_key_value_heads = num_attention_heads

    self.num_key_value_heads = num_key_value_heads
    self.hidden_act = hidden_act
    self.initializer_range = initializer_range
    self.rms_norm_eps = rms_norm_eps
    self.use_cache = use_cache
    self.rope_theta = rope_theta
    self.attention_dropout = attention_dropout

    self.num_experts_per_tok = num_experts_per_tok
    self.num_local_experts = num_local_experts
    self.router_aux_loss_coef = router_aux_loss_coef
    self.moe_layer_frequency = moe_layer_frequency
    self.moe_train_capacity_factor = moe_train_capacity_factor
    self.moe_eval_capacity_factor = moe_eval_capacity_factor
    self.enable_expert_tensor_parallelism = enable_expert_tensor_parallelism
    self.moe_min_capacity = moe_min_capacity
    self.moe_token_dropping = moe_token_dropping
    self.parallel_attn_mlp_res = parallel_attn_mlp_res
    if isinstance(quantization, dict):
        self.quantization = ArcticQuantizationConfig(**quantization)
    else:
        self.quantization = quantization

    super().__init__(
        pad_token_id=pad_token_id,
        bos_token_id=bos_token_id,
        eos_token_id=eos_token_id,
        tie_word_embeddings=tie_word_embeddings,
        **kwargs,
    )

from_dict classmethod 

from_dict(
    config_dict: dict[str, Any], **kwargs
) -> ArcticConfig
Source code in vllm/transformers_utils/configs/arctic.py 
195
196
197
198
199
200
201
@classmethod
def from_dict(cls, config_dict: dict[str, Any], **kwargs) -> "ArcticConfig":
    result = super().from_dict(config_dict, **kwargs)
    config = result[0] if isinstance(result, tuple) else result
    if isinstance(config.quantization, dict):
        config.quantization = ArcticQuantizationConfig(**config.quantization)
    return result

to_dict 

to_dict() -> dict[str, Any]
Source code in vllm/transformers_utils/configs/arctic.py 
203
204
205
206
207
def to_dict(self) -> dict[str, Any]:
    ret = super().to_dict()
    if isinstance(ret["quantization"], ArcticQuantizationConfig):
        ret["quantization"] = asdict(ret["quantization"])
    return ret

ArcticLoRAConfig dataclass

Source code in vllm/transformers_utils/configs/arctic.py 
24
25
26
27
28
@dataclass
class ArcticLoRAConfig:
    lora_r: int = 64
    lora_alpha: float = 16
    shard_base_weights: bool = False

lora_alpha class-attribute instance-attribute 

lora_alpha: float = 16

lora_r class-attribute instance-attribute 

lora_r: int = 64

shard_base_weights class-attribute instance-attribute 

shard_base_weights: bool = False

__init__ 

__init__(
    lora_r: int = 64,
    lora_alpha: float = 16,
    shard_base_weights: bool = False,
) -> None

ArcticQuantizationConfig dataclass

Source code in vllm/transformers_utils/configs/arctic.py 
31
32
33
34
35
36
@dataclass
class ArcticQuantizationConfig:
    q_bits: int = 8
    rounding: str = "nearest"
    mantissa_bits: int = 3
    group_size: int = 128

group_size class-attribute instance-attribute 

group_size: int = 128

mantissa_bits class-attribute instance-attribute 

mantissa_bits: int = 3

q_bits class-attribute instance-attribute 

q_bits: int = 8

rounding class-attribute instance-attribute 

rounding: str = 'nearest'

__init__ 

__init__(
    q_bits: int = 8,
    rounding: str = "nearest",
    mantissa_bits: int = 3,
    group_size: int = 128,
) -> None