NVFP4 (GPTQ, W4A4) build of convaiinnovations/laya

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+multilingual/tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,99 @@
+---
+license: apache-2.0
+base_model: convaiinnovations/laya
+base_model_relation: quantized
+pipeline_tag: text-classification
+tags:
+- laya
+- nvfp4
+- fp4
+- w4a4
+- compressed-tensors
+- llm-compressor
+- classification
+- routing
+- guardrails
+- moderation
+---
+
+# laya, NVFP4 (W4A4)
+
+This repo holds the three [laya](https://huggingface.co/convaiinnovations/laya) checkpoints with both the weights and the input activations of their encoder's linear layers in NVFP4: FP4 (E2M1) values with one FP8 scale per group of 16 and one FP32 scale per tensor. On Blackwell GPUs those layers multiply on FP4 tensor cores. The files are the same size as the [NVFP4A16 build](https://huggingface.co/hoja-solutions/laya-NVFP4A16), 349 MB for the English and typed-decisions checkpoints and 485 MB for the multilingual one, and the model takes about a third of the GPU memory of the original.
+
+The NVFP4A16 build, which keeps activations in bf16, runs at the original's speed, stays closer to the original's answers and gives the same answer for the same question every time. Use this W4A4 build only when GPU memory is what limits you. The [original card](https://huggingface.co/convaiinnovations/laya) explains the model itself.
+
+## Usage
+
+```python
+import sys
+from huggingface_hub import snapshot_download
+
+path = snapshot_download("hoja-solutions/laya-NVFP4")
+sys.path.insert(0, path)
+import laya_nvfp4
+
+agent = laya_nvfp4.load(path, device="cuda")                    # English, at the repo root
+agent_ml = laya_nvfp4.load(path + "/multilingual", device="cuda")
+agent_td = laya_nvfp4.load(path + "/typed-decisions", device="cuda")
+```
+
+`laya_nvfp4.load` returns a regular `laya.Agent`, so `predict` and the rest of laya's API work, and `Router.attach` adds it to laya's router. Each quantized layer quantizes its input on every call. On Blackwell GPUs (sm_100, sm_120) the multiplication runs on FP4 tensor cores through `torch._scaled_mm`, and `torch.compile` fuses the input quantization. On any other device the loader emulates the same arithmetic in fp32, which is correct but slow. Tested with laya 0.4.6 and torch 2.14.
+
+## Quality
+
+The tests are the same as for the NVFP4A16 build: 8,899 questions rebuilt from the laya authors' benchmark scripts (ten application tasks, the typed-decisions test split, MASSIVE intent in 14 languages and XNLI in 15), each checkpoint compared with the original in bf16 on an RTX 5090, accuracy with the shipped temperatures.
+
+| Accuracy, original → this repo | 10 app tasks | typed-decisions | MASSIVE | XNLI |
+|---|---|---|---|---|
+| English | 0.696 → 0.695 | 0.359 → 0.357 | 0.356 → 0.320 | 0.530 → 0.526 |
+| typed-decisions | 0.704 → 0.701 | 0.767 → 0.766 | 0.348 → 0.319 | 0.541 → 0.525 |
+| multilingual | 0.694 → 0.686 | 0.353 → 0.346 | 0.506 → 0.459 | 0.718 → 0.701 |
+
+| Same answer as the original | 10 app tasks | typed-decisions | MASSIVE | XNLI | all questions | when the original is ≥ 90% sure |
+|---|---|---|---|---|---|---|
+| English | 91.9% | 88.8% | 61.7% | 76.3% | 83.8% | 92.5% |
+| typed-decisions | 92.5% | 93.7% | 62.8% | 81.3% | 86.2% | 97.6% |
+| multilingual | 90.6% | 79.5% | 72.4% | 87.0% | 84.6% | 95.4% |
+| NVFP4A16, for comparison | 94.7 to 95.5% | 88.1 to 97.0% | 70.9 to 82.4% | 82.0 to 90.7% | 89.1 to 90.7% | 96.3 to 99.1% |
+| original, run twice | 99.6 to 99.9% | 99.4 to 99.5% | 96.5 to 99.2% | 99.2 to 99.7% | 99.0 to 99.6% | 100% |
+
+Accuracy holds better than agreement because the changed answers go both ways, but this build changes 14 to 16% of the answers, including some the original gives with 90% probability or more.
+
+### The same question can get different answers
+
+Every quantized layer rounds its input to FP4 again. A tiny difference upstream, such as a different floating-point summation order, moves some values across an FP4 rounding boundary, and a full FP4 step is a large error, so the difference grows from layer to layer instead of fading. In practice:
+
+- Batching changes answers. The English checkpoint answered all 8,899 questions twice, with the questions grouped into different batches. 6.1% of its answers differed between the two runs. The original and the NVFP4A16 build change about 1% this way.
+- Hardware changes answers. The emulation path, which repeats the same arithmetic in fp32, agrees with the original about as often as the GPU path (84.0% against 83.8% of all questions), but on different questions: the two runs of this build agree with each other on only 84%.
+
+If you need repeatable decisions, use the NVFP4A16 build.
+
+## Speed
+
+RTX 5090, the original card's latency setup (one support ticket, p50 over 20 calls):
+
+| | 1 question | 10 questions | 50 questions | 4,000 questions, batched | GPU memory for weights |
+|---|---|---|---|---|---|
+| English, original | 10.9 ms | 18.0 ms | 74.9 ms | 775 per second | 1,607 MB |
+| English, this repo | 23.8 ms | 26.6 ms | 72.1 ms | 852 per second | 489 MB |
+| multilingual, original | 8.8 ms | 10.4 ms | 34.3 ms | 1,577 per second | 1,247 MB |
+| multilingual, this repo | 20.4 ms | 21.6 ms | 32.7 ms | 1,507 per second | 868 MB |
+
+An FP4 matrix multiply is about five times faster than bf16 at batch sizes of a few thousand tokens. Matrix multiplies are a small part of laya's run time, though, and every layer pays for quantizing its input. So small requests take about twice as long as with the original, and large batches gain about 10% at best. The memory saving is real: the English checkpoint needs a third of the original's GPU memory, because laya otherwise keeps its weights in fp32.
+
+## How it was made
+
+GPTQ with llm-compressor 0.14.0 (static activation ordering, block size 128, dampening 0.01) and the NVFP4 scheme on the four linear layers of every encoder block. The scales for each group of 16 input values are computed at run time, under a per-tensor scale fixed from the calibration data. Embeddings, norms and the decision head stay in fp16. The calibration set and the NVFP4A16 variants are described on the [NVFP4A16 card](https://huggingface.co/hoja-solutions/laya-NVFP4A16).
+
+Plain rounding instead of GPTQ keeps the same answer on 81.3% (English), 85.0% (typed-decisions) and 82.5% (multilingual) of the questions, so GPTQ helps by 1 to 3 points.
+
+## Limits
+
+- The FP4 tensor-core path needs a Blackwell GPU. Elsewhere the emulation gives the same quality at a fraction of the speed.
+- Tested on an RTX 5090 with torch 2.14 and laya 0.4.6.
+- `laya.load` and `Router` don't read these files by themselves; use `laya_nvfp4.load` and `Router.attach`.
+- Everything on the original card's "Honest Limits" still applies.
+
+## Credits
+
+laya and its three checkpoints are by [Convai Innovations](https://huggingface.co/convaiinnovations) ([GitHub](https://github.com/NandhaKishorM/laya)), released under Apache-2.0. They build on [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) and [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base).

config.json

@@ -0,0 +1,7 @@
+{
+ "model_type": "laya",
+ "architectures": [
+  "LayaTypedDecisions"
+ ],
+ "_comment": "Root config for Hub download tracking. See README for how to load Laya."
+}

encoder/config.json

@@ -0,0 +1,84 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "vocab_size": 50368
+}

laya_nvfp4.py

@@ -0,0 +1,204 @@
+"""Load an NVFP4 laya checkpoint (compressed-tensors format) into a regular `laya.Agent`.
+
+    import laya_nvfp4
+    agent = laya_nvfp4.load("path/to/checkpoint", device="cuda")
+    agent.predict(state, questions)          # every laya API works as usual
+
+The encoder's linear layers are stored as FP4 (E2M1) values with one FP8 (E4M3) scale per 16
+weights and one FP32 scale per tensor. What happens at load time depends on the checkpoint:
+
+* NVFP4A16 (weights only): weights are unpacked to dense once, and the model runs exactly like
+  the original laya, in bf16 on GPU or fp32 on CPU.
+* NVFP4 (weights and activations): each quantized layer also quantizes its input to NVFP4 on the
+  fly (one scale per 16 values, under a fixed per-tensor scale from calibration). On Blackwell
+  GPUs (sm_100, sm_120) the product runs on FP4 tensor cores through `torch._scaled_mm`; on any
+  other device the same arithmetic is emulated in higher precision, so answers match.
+
+Needs only torch and laya (tested with torch 2.14 and laya 0.4.6). The FP4 path also uses
+torch.compile to fuse the activation quantization, and falls back to eager PyTorch without it.
+"""
+
+import json
+import os
+
+import torch
+import torch.nn as nn
+
+E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0)  # FP4 E2M1 magnitudes, indexed by the low 3 bits
+FP8_MAX, FP4_MAX, GROUP = 448.0, 6.0, 16
+
+
+# ----------------------------------------------------------------------------- FP4 packing
+def unpack_fp4(packed: torch.Tensor, dtype=torch.float32) -> torch.Tensor:
+    """uint8 [..., K/2] -> values [..., K]. The low nibble holds the first value of each pair."""
+    codes = torch.stack((packed & 0x0F, packed >> 4), dim=-1).flatten(-2)
+    mag = torch.tensor(E2M1, dtype=dtype, device=packed.device)[(codes & 0x07).long()]
+    return torch.where((codes & 0x08).bool(), -mag, mag)
+
+
+def _fp4_codes(x: torch.Tensor) -> torch.Tensor:
+    """Round to the nearest E2M1 value (x already scaled into [-6, 6]) -> 4-bit codes. Ties go to
+    the even code, like the hardware conversion (cvt.rn.satfinite.e2m1x2)."""
+    a = x.abs()
+    bounds = torch.tensor((0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0), dtype=x.dtype, device=x.device)
+    mag = torch.bucketize(a, bounds)  # 0..7, a tie lands on the lower code
+    mag = mag + ((a == 0.75) | (a == 1.75) | (a == 3.5)).to(mag.dtype)  # those ties belong to the upper, even code
+    return (mag | ((x < 0).to(mag.dtype) << 3)).to(torch.uint8)
+
+
+def dequantize_weight(packed, scale, global_scale, dtype=torch.float32):
+    """W = fp4 * (fp8 group scale / global scale), as compressed-tensors defines it."""
+    w = unpack_fp4(packed, torch.float32)
+    s = (scale.float() / global_scale.float()).repeat_interleave(GROUP, dim=-1)
+    return (w * s).to(dtype)
+
+
+def quantize_activation(x: torch.Tensor, global_scale: torch.Tensor):
+    """Dynamic NVFP4: per group of 16, scale = fp8(global_scale * amax / 6), q = fp4(x / (scale / global_scale)).
+    Returns (packed uint8 [M, K/2], fp8 scales [M, K/16])."""
+    m, k = x.shape
+    g = x.float().view(m, k // GROUP, GROUP)
+    amax = g.abs().amax(-1, keepdim=True)
+    scale = (global_scale * amax / FP4_MAX).clamp(max=FP8_MAX).to(torch.float8_e4m3fn)
+    s = scale.float()
+    s = torch.where(s == 0, torch.ones_like(s), s)
+    codes = _fp4_codes((g * global_scale / s).clamp(-FP4_MAX, FP4_MAX)).view(m, k)
+    return codes[:, 0::2] | (codes[:, 1::2] << 4), scale.view(m, k // GROUP)
+
+
+def to_blocked(scale: torch.Tensor) -> torch.Tensor:
+    """Row-major [rows, cols] block scales -> the 128x4 tiled layout cuBLAS reads for FP4 GEMMs."""
+    rows, cols = scale.shape
+    nr, nc = -(-rows // 128), -(-cols // 4)
+    padded = scale.new_zeros((nr * 128, nc * 4))
+    padded[:rows, :cols] = scale
+    blocks = padded.view(nr, 128, nc, 4).permute(0, 2, 1, 3)
+    return blocks.reshape(-1, 4, 32, 4).transpose(1, 2).reshape(-1, 32, 16).flatten()
+
+
+def fp4_kernels_available(device) -> bool:
+    device = torch.device(device)
+    return (device.type == "cuda" and torch.cuda.get_device_capability(device)[0] in (10, 12)
+            and hasattr(torch, "float4_e2m1fn_x2"))
+
+
+def _quantize_blocked(x, global_scale):
+    q, s = quantize_activation(x, global_scale)
+    return q, to_blocked(s)
+
+
+_compiled = None
+
+
+def quantize_blocked(x, global_scale):
+    """quantize_activation + to_blocked, fused by torch.compile on CUDA (falls back to eager if
+    compilation is unavailable). Eager, these are a dozen small kernels per layer."""
+    global _compiled
+    if _compiled is None:
+        try:
+            _compiled = torch.compile(_quantize_blocked, dynamic=True)
+        except Exception:
+            _compiled = _quantize_blocked
+    try:
+        return _compiled(x, global_scale)
+    except Exception:
+        _compiled = _quantize_blocked
+        return _quantize_blocked(x, global_scale)
+
+
+# ----------------------------------------------------------------------------- the layer
+class NVFP4Linear(nn.Module):
+    """y = x W^T with W stored in NVFP4 and x quantized to NVFP4 on every call (no bias)."""
+
+    def __init__(self, packed, scale, weight_global_scale, input_global_scale, use_kernel: bool):
+        super().__init__()
+        self.out_features, self.in_features = packed.shape[0], packed.shape[1] * 2
+        self.register_buffer("weight_packed", packed)
+        self.register_buffer("weight_scale", scale)
+        self.register_buffer("weight_global_scale", weight_global_scale.float().reshape(()))
+        self.register_buffer("input_global_scale", input_global_scale.float().reshape(()))
+        self.use_kernel = use_kernel
+        self._dense = None   # dequantized weight, for the emulation path
+        self._blocked = None  # weight scales in cuBLAS layout, for the kernel path
+        self._alpha = None    # 1 / (input global scale * weight global scale)
+
+    def extra_repr(self):
+        return "in=%d, out=%d, %s" % (self.in_features, self.out_features, "fp4 kernel" if self.use_kernel else "emulated")
+
+    def _apply(self, fn, *args, **kwargs):  # .to()/.cuda() invalidate the caches
+        self._dense = self._blocked = None
+        return super()._apply(fn, *args, **kwargs)
+
+    def forward(self, x):
+        shape = x.shape
+        x2 = x.reshape(-1, self.in_features)
+        dtype = x.dtype if x.dtype in (torch.bfloat16, torch.float16) else torch.float32
+        if torch.is_autocast_enabled(x.device.type):
+            dtype = torch.get_autocast_dtype(x.device.type)
+        if self.use_kernel:
+            if self._blocked is None:
+                self._blocked = to_blocked(self.weight_scale)
+                self._alpha = (1.0 / (self.input_global_scale * self.weight_global_scale)).float()
+            q, sb = quantize_blocked(x2, self.input_global_scale)
+            y = torch._scaled_mm(q.view(torch.float4_e2m1fn_x2), self.weight_packed.view(torch.float4_e2m1fn_x2).t(),
+                                 sb, self._blocked, out_dtype=torch.bfloat16)
+            y = (y * self._alpha).to(dtype)
+        else:
+            if self._dense is None:
+                self._dense = dequantize_weight(self.weight_packed, self.weight_scale, self.weight_global_scale)
+            q, s = quantize_activation(x2, self.input_global_scale)
+            xq = dequantize_weight(q, s, self.input_global_scale)
+            y = (xq @ self._dense.t()).to(dtype)
+        return y.reshape(*shape[:-1], self.out_features)
+
+
+# ----------------------------------------------------------------------------- loading
+def _read(path):
+    from safetensors.torch import load_file
+    return load_file(os.path.join(path, "model.safetensors"))
+
+
+def dense_state_dict(raw):
+    """Compressed state dict -> the dense state dict laya's DecisionModel expects (fp16)."""
+    out = {}
+    for k, v in raw.items():
+        if k.endswith(".weight_packed"):
+            p = k[: -len(".weight_packed")]
+            out[p + ".weight"] = dequantize_weight(v, raw[p + ".weight_scale"], raw[p + ".weight_global_scale"],
+                                                   torch.float16)
+        elif not k.endswith((".weight_scale", ".weight_global_scale", ".input_global_scale")):
+            out[k] = v
+    return out
+
+
+def load(path, device=None, use_kernel=None, **laya_kwargs):
+    """A `laya.Agent` for an NVFP4 checkpoint directory. `use_kernel=None` picks the FP4 tensor-core
+    path when the GPU has one; False forces the emulation (same answers, any device)."""
+    import laya
+    import safetensors.torch
+
+    raw = _read(path)
+    with open(os.path.join(path, "rl_agent_config.json")) as f:
+        qcfg = json.load(f).get("quantization_config", {})
+    # laya.Agent reads model.safetensors with safetensors.torch.load_file (imported inside its
+    # constructor); hand it the dense weights instead, then swap the quantized layers in below.
+    real_load_file = safetensors.torch.load_file
+    safetensors.torch.load_file = lambda *_args, **_kwargs: dense_state_dict(raw)
+    try:
+        ag = laya.load(path, device=device, backend="eager", **laya_kwargs)
+    finally:
+        safetensors.torch.load_file = real_load_file
+    ag.quantization_config = qcfg
+    act_quant = any(k.endswith(".input_global_scale") for k in raw)
+    if act_quant:
+        kernel = fp4_kernels_available(ag.device) if use_kernel is None else bool(use_kernel)
+        dev = ag.device
+        for name, _ in list(ag.model.named_modules()):
+            if name + ".weight_packed" not in raw:
+                continue
+            parent = ag.model.get_submodule(name.rsplit(".", 1)[0])
+            new = NVFP4Linear(raw[name + ".weight_packed"], raw[name + ".weight_scale"],
+                              raw[name + ".weight_global_scale"], raw[name + ".input_global_scale"], kernel).to(dev)
+            setattr(parent, name.rsplit(".", 1)[1], new)
+    ag.model.eval()
+    return ag

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3566646ec0d40f24342f5f74fbae04fc150909db4b0f5cc75d73d5c0efbac89d
+size 349372970

multilingual/encoder/config.json

@@ -0,0 +1,79 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "vocab_size": 256000
+}

multilingual/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:824e3914dadb20671ff3a4b7383eb4d392b7020d09e160d1f1018c007c18d12b
+size 485312346

multilingual/rl_agent_config.json

@@ -0,0 +1,75 @@
+{
+  "encoder": "jhu-clsp/mmBERT-base",
+  "head_layers": 2,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "amp_dtype": "bf16",
+  "model_name": "rl-agent",
+  "temperature": [
+    1.0,
+    1.0,
+    1.0
+  ],
+  "temperature_by_options": {},
+  "training": {
+    "updates": 15987,
+    "epochs_completed": 4,
+    "hours": 4.97,
+    "world_size": 1,
+    "fine_tuned_from_checkpoint": false
+  },
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "targets": [
+          "re:.*encoder\\.layers\\.\\d+\\.attn\\.(Wqkv|Wo)$",
+          "re:.*encoder\\.layers\\.\\d+\\.mlp\\.(Wi|Wo)$"
+        ],
+        "weights": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": false,
+          "actorder": "static",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {}
+        },
+        "input_activations": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": "local",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "static_minmax",
+          "observer_kwargs": {}
+        },
+        "format": "nvfp4-pack-quantized"
+      }
+    },
+    "quant_method": "compressed-tensors",
+    "format": "nvfp4-pack-quantized",
+    "quantization_status": "compressed",
+    "ignore": [
+      "head.layers.0.linear1",
+      "head.layers.0.linear2",
+      "head.layers.1.linear1",
+      "head.layers.1.linear2",
+      "scorer.1",
+      "scorer.3",
+      "act_head.0",
+      "act_head.2"
+    ]
+  }
+}

multilingual/tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
+size 34363188

multilingual/tokenizer/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": {
+    "extra_0": "<start_of_turn>",
+    "extra_1": "<end_of_turn>"
+  },
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

rl_agent_config.json

@@ -0,0 +1,82 @@
+{
+  "encoder": "answerdotai/ModernBERT-large",
+  "head_layers": 2,
+  "max_len": 512,
+  "head_max_len": 192,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "amp_dtype": "bf16",
+  "model_name": "rl-agent",
+  "temperature": [
+    1.6369030475616455,
+    1.2514300346374512,
+    1.983399510383606
+  ],
+  "temperature_by_options": {
+    "choice:3-5": 1.7601518630981445,
+    "choice:6-10": 1.0000158548355103,
+    "score:3-5": 1.2514300346374512,
+    "noul:2": 1.983399510383606,
+    "choice:11+": 0.10058280825614929,
+    "choice:2": 1.9063563346862793
+  },
+  "training": {
+    "updates": 7313,
+    "epochs_completed": 1,
+    "hours": 1.96,
+    "world_size": 1,
+    "fine_tuned_from_checkpoint": true
+  },
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "targets": [
+          "re:.*encoder\\.layers\\.\\d+\\.attn\\.(Wqkv|Wo)$",
+          "re:.*encoder\\.layers\\.\\d+\\.mlp\\.(Wi|Wo)$"
+        ],
+        "weights": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": false,
+          "actorder": "static",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {}
+        },
+        "input_activations": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": "local",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "static_minmax",
+          "observer_kwargs": {}
+        },
+        "format": "nvfp4-pack-quantized"
+      }
+    },
+    "quant_method": "compressed-tensors",
+    "format": "nvfp4-pack-quantized",
+    "quantization_status": "compressed",
+    "ignore": [
+      "head.layers.0.linear1",
+      "head.layers.0.linear2",
+      "head.layers.1.linear1",
+      "head.layers.1.linear2",
+      "scorer.1",
+      "scorer.3",
+      "act_head.0",
+      "act_head.2"
+    ]
+  }
+}

tokenizer/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}

typed-decisions/encoder/config.json

@@ -0,0 +1,84 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 50368
+}

typed-decisions/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28c7ac1a04b347b19e3fe6883843a19a2af6c0cf8a12820f4382c6ef1c7bdcad
+size 349372970

typed-decisions/rl_agent_config.json

@@ -0,0 +1,85 @@
+{
+  "encoder": "answerdotai/ModernBERT-large",
+  "head_layers": 2,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "amp_dtype": "bf16",
+  "model_name": "laya-typed-decisions",
+  "temperature": [
+    1.0148024559020996,
+    1.0374259948730469,
+    1.0575125217437744
+  ],
+  "temperature_by_options": {
+    "choice:3-5": 1.7601518630981445,
+    "choice:6-10": 1.0000158548355103,
+    "score:3-5": 1.2514300346374512,
+    "noul:2": 1.983399510383606,
+    "choice:11+": 0.10058280825614929,
+    "choice:2": 1.9063563346862793
+  },
+  "training": {
+    "updates": 7313,
+    "epochs_completed": 1,
+    "hours": 1.96,
+    "world_size": 1,
+    "fine_tuned_from_checkpoint": true
+  },
+  "gradient_checkpointing": true,
+  "max_tokens_per_batch": 4096,
+  "fine_tuned": true,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "targets": [
+          "re:.*encoder\\.layers\\.\\d+\\.attn\\.(Wqkv|Wo)$",
+          "re:.*encoder\\.layers\\.\\d+\\.mlp\\.(Wi|Wo)$"
+        ],
+        "weights": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": false,
+          "actorder": "static",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {}
+        },
+        "input_activations": {
+          "num_bits": 4,
+          "type": "float",
+          "symmetric": true,
+          "group_size": 16,
+          "strategy": "tensor_group",
+          "dynamic": "local",
+          "scale_dtype": "torch.float8_e4m3fn",
+          "zp_dtype": null,
+          "observer": "static_minmax",
+          "observer_kwargs": {}
+        },
+        "format": "nvfp4-pack-quantized"
+      }
+    },
+    "quant_method": "compressed-tensors",
+    "format": "nvfp4-pack-quantized",
+    "quantization_status": "compressed",
+    "ignore": [
+      "head.layers.0.linear1",
+      "head.layers.0.linear2",
+      "head.layers.1.linear1",
+      "head.layers.1.linear2",
+      "scorer.1",
+      "scorer.3",
+      "act_head.0",
+      "act_head.2"
+    ]
+  }
+}

typed-decisions/tokenizer/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}