Avey-B Urdu 25M

Urdu masked-language encoder trained from scratch. This is an adaptation of the Avey-B architecture to Urdu. It does not claim the benchmark results reported for Avey-B by Acharya and Hammoud.

Purpose

Use the checkpoint for Urdu classification, token classification, representation extraction, or continued pretraining. It is an encoder, not a chat model or free-text generator.

Upstream attribution

The architecture follows Avey-B (Acharya and Hammoud, ICLR 2026). Upstream code is at github.com/rimads/avey-b. Model code in this repository is included so the checkpoint can be loaded with trust_remote_code=True.

@inproceedings{2026aveyb,
  title={Avey-B},
  author={Acharya, Devang and Hammoud, Mohammad},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026}
}

Data provenance

HPLT packages its dataset under CC0 but does not own the underlying web text. The Apache-2.0 model license does not grant rights to third-party source documents that may be reproduced by the model.

Document filters beyond the quality bins, deduplication, and the tokenizer training corpus are not stated outside MANIFEST.json. This card does not add figures from that file.

Training

Not reported in this card: random seed, number of runs, warmup, weight decay, dropout, and gradient clipping.

Evaluation

The model-index point estimates are accuracy 0.785333 and entity F1 0.812677. The intervals below appear only in this prose table. The number of runs behind each interval is not stated. Repository ids for UrNova, HPLT Urdu BERT, and the XLM-R checkpoint are not stated.

Other figures reported with this checkpoint, without a named device for the latency measurement:

• 1.71 ms encoder latency at batch 1 / length 128, and 148 MiB peak VRAM at batch 16 / length 512.
• 14.29 held-out MLM perplexity, 50.46% top-1, and 70.39% top-5 masked-token accuracy.
• 4.19 characters/token on held-out data.

Limitations

Benchmark coverage in this card is limited to masked-token prediction, movie-review sentiment, and Wikipedia-derived NER. The sentiment limitation stated previously still applies: evaluate on native, domain-specific Urdu before production use. The card describes that sentiment setup as machine-translated movie reviews.

License

Apache-2.0, as declared in the metadata above.

Usage

This model contains a custom Avey architecture, so trust_remote_code=True is required.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "mahwizzzz/avey-b-ur"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True).eval()

text = "پاکستان کا دارالحکومت [MASK] ہے۔"
inputs = tokenizer(text, return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits

mask_index = (inputs.input_ids[0] == tokenizer.mask_token_id).nonzero()[0, 0]
top_ids = logits[0, mask_index].topk(5).indices
print(tokenizer.convert_ids_to_tokens(top_ids.tolist()))

Extract contextual token representations with AutoModel:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mahwizzzz/avey-b-ur", trust_remote_code=True)
encoder = AutoModel.from_pretrained("mahwizzzz/avey-b-ur", trust_remote_code=True)
hidden = encoder(**tokenizer("یہ ایک اردو جملہ ہے۔", return_tensors="pt")).last_hidden_state
print(hidden.shape)  # [batch, tokens, 384]