claude-scientific-skills/scientific-packages/transformers/references/api_reference.md

# Transformers API Reference

This document provides comprehensive API reference for the most commonly used classes and methods in the Transformers library.

## Core Model Classes

### PreTrainedModel

Base class for all models. Handles loading, saving, and common model operations.

**Key Methods:**

```python
from transformers import PreTrainedModel

# Load pretrained model
model = ModelClass.from_pretrained(
    pretrained_model_name_or_path,
    config=None,                   # Custom config
    cache_dir=None,                # Custom cache location
    force_download=False,          # Force re-download
    resume_download=False,         # Resume interrupted download
    proxies=None,                  # HTTP proxies
    local_files_only=False,        # Only use cached files
    token=None,                    # HF auth token
    revision="main",               # Git branch/tag
    trust_remote_code=False,       # Allow custom model code
    device_map=None,               # Device allocation ("auto", "cpu", "cuda:0", etc.)
    torch_dtype=None,              # Model dtype (torch.float16, "auto", etc.)
    low_cpu_mem_usage=False,       # Reduce CPU memory during loading
    **model_kwargs
)

# Save model
model.save_pretrained(
    save_directory,
    save_config=True,              # Save config.json
    state_dict=None,               # Custom state dict
    save_function=torch.save,      # Custom save function
    push_to_hub=False,             # Upload to Hub
    max_shard_size="5GB",          # Max checkpoint size
    safe_serialization=True,       # Use SafeTensors format
    variant=None,                  # Model variant name
)

# Generate text (for generative models)
outputs = model.generate(
    inputs=None,                   # Input token IDs
    max_length=20,                 # Max total length
    max_new_tokens=None,           # Max new tokens to generate
    min_length=0,                  # Minimum length
    do_sample=False,               # Enable sampling
    early_stopping=False,          # Stop when num_beams finish
    num_beams=1,                   # Beam search width
    temperature=1.0,               # Sampling temperature
    top_k=50,                      # Top-k sampling
    top_p=1.0,                     # Nucleus sampling
    repetition_penalty=1.0,        # Penalize repetition
    length_penalty=1.0,            # Beam search length penalty
    no_repeat_ngram_size=0,        # Block repeated n-grams
    num_return_sequences=1,        # Number of sequences to return
    **model_kwargs
)

# Resize token embeddings (after adding tokens)
new_embeddings = model.resize_token_embeddings(
    new_num_tokens,
    pad_to_multiple_of=None
)

# Utility methods
num_params = model.num_parameters(only_trainable=False)
model.gradient_checkpointing_enable()  # Enable gradient checkpointing
model.enable_input_require_grads()     # For PEFT with frozen models
```

### AutoModel Classes

Automatically instantiate the correct model architecture.

**Available Classes:**

- `AutoModel`: Base model (returns hidden states)
- `AutoModelForCausalLM`: Causal language modeling (GPT-style)
- `AutoModelForMaskedLM`: Masked language modeling (BERT-style)
- `AutoModelForSeq2SeqLM`: Sequence-to-sequence (T5, BART)
- `AutoModelForSequenceClassification`: Text classification
- `AutoModelForTokenClassification`: Token classification (NER)
- `AutoModelForQuestionAnswering`: Extractive QA
- `AutoModelForImageClassification`: Image classification
- `AutoModelForObjectDetection`: Object detection
- `AutoModelForSemanticSegmentation`: Semantic segmentation
- `AutoModelForAudioClassification`: Audio classification
- `AutoModelForSpeechSeq2Seq`: Speech-to-text
- `AutoModelForVision2Seq`: Image captioning, VQA

**Usage:**

```python
from transformers import AutoModel, AutoConfig

# Load with default configuration
model = AutoModel.from_pretrained("bert-base-uncased")

# Load with custom configuration
config = AutoConfig.from_pretrained("bert-base-uncased")
config.hidden_dropout_prob = 0.2
model = AutoModel.from_pretrained("bert-base-uncased", config=config)

# Register custom models
from transformers import AutoConfig, AutoModel

AutoConfig.register("my-model", MyModelConfig)
AutoModel.register(MyModelConfig, MyModel)
```

## Tokenizer Classes

### PreTrainedTokenizer / PreTrainedTokenizerFast

Convert text to token IDs and vice versa.

**Key Methods:**

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path,
    use_fast=True,                 # Use fast (Rust) tokenizer if available
    revision="main",
    **kwargs
)

# Encoding (text → token IDs)
encoded = tokenizer(
    text,                          # String or List[str]
    text_pair=None,                # Second sequence for pairs
    add_special_tokens=True,       # Add [CLS], [SEP], etc.
    padding=False,                 # True, False, "longest", "max_length"
    truncation=False,              # True, False, "longest_first", "only_first", "only_second"
    max_length=None,               # Max sequence length
    stride=0,                      # Overlap for split sequences
    return_tensors=None,           # "pt" (PyTorch), "tf" (TensorFlow), "np" (NumPy)
    return_token_type_ids=None,    # Return token type IDs
    return_attention_mask=None,    # Return attention mask
    return_overflowing_tokens=False,  # Return overflowing tokens
    return_special_tokens_mask=False, # Return special token mask
    return_offsets_mapping=False,  # Return char-level offsets (fast only)
    return_length=False,           # Return sequence lengths
    **kwargs
)

# Decoding (token IDs → text)
text = tokenizer.decode(
    token_ids,
    skip_special_tokens=False,     # Remove special tokens
    clean_up_tokenization_spaces=True,  # Clean up spacing
)

# Batch decoding
texts = tokenizer.batch_decode(
    sequences,
    skip_special_tokens=False,
    clean_up_tokenization_spaces=True,
)

# Tokenization (text → tokens)
tokens = tokenizer.tokenize(text, **kwargs)

# Convert tokens to IDs
ids = tokenizer.convert_tokens_to_ids(tokens)

# Convert IDs to tokens
tokens = tokenizer.convert_ids_to_tokens(ids)

# Add new tokens
num_added = tokenizer.add_tokens(["[NEW_TOKEN1]", "[NEW_TOKEN2]"])

# Add special tokens
tokenizer.add_special_tokens({
    "bos_token": "[BOS]",
    "eos_token": "[EOS]",
    "unk_token": "[UNK]",
    "sep_token": "[SEP]",
    "pad_token": "[PAD]",
    "cls_token": "[CLS]",
    "mask_token": "[MASK]",
    "additional_special_tokens": ["[SPECIAL1]", "[SPECIAL2]"],
})

# Chat template formatting
formatted = tokenizer.apply_chat_template(
    conversation,                  # List[Dict[str, str]] with "role" and "content"
    chat_template=None,            # Custom template
    add_generation_prompt=False,   # Add prompt for model to continue
    tokenize=True,                 # Return token IDs
    padding=False,
    truncation=False,
    max_length=None,
    return_tensors=None,
    return_dict=True,
)

# Save tokenizer
tokenizer.save_pretrained(save_directory)

# Get vocab size
vocab_size = len(tokenizer)

# Get special tokens
pad_token = tokenizer.pad_token
pad_token_id = tokenizer.pad_token_id
# Similar for: bos, eos, unk, sep, cls, mask
```

**Special Token Attributes:**

```python
tokenizer.bos_token         # Beginning of sequence
tokenizer.eos_token         # End of sequence
tokenizer.unk_token         # Unknown token
tokenizer.sep_token         # Separator token
tokenizer.pad_token         # Padding token
tokenizer.cls_token         # Classification token
tokenizer.mask_token        # Mask token

# Corresponding IDs
tokenizer.bos_token_id
tokenizer.eos_token_id
# ... etc
```

## Image Processors

### AutoImageProcessor

Preprocess images for vision models.

**Key Methods:**

```python
from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")

# Process images
inputs = processor(
    images,                        # PIL Image, np.array, torch.Tensor, or List
    return_tensors="pt",           # "pt", "tf", "np", None
    do_resize=True,                # Resize to model size
    size=None,                     # Target size dict
    resample=None,                 # Resampling method
    do_rescale=True,               # Rescale pixel values
    do_normalize=True,             # Normalize with mean/std
    image_mean=None,               # Custom mean
    image_std=None,                # Custom std
    do_center_crop=False,          # Center crop
    crop_size=None,                # Crop size
    **kwargs
)

# Returns: BatchFeature with 'pixel_values' key
```

## Training Components

### TrainingArguments

Configuration for the Trainer class.

**Essential Arguments:**

```python
from transformers import TrainingArguments

args = TrainingArguments(
    # ===== Output & Logging =====
    output_dir="./results",              # REQUIRED: Output directory
    overwrite_output_dir=False,          # Overwrite output directory

    # ===== Training Parameters =====
    num_train_epochs=3.0,                # Number of epochs
    max_steps=-1,                        # Max training steps (overrides epochs)
    per_device_train_batch_size=8,       # Train batch size per device
    per_device_eval_batch_size=8,        # Eval batch size per device
    gradient_accumulation_steps=1,       # Accumulation steps

    # ===== Learning Rate & Optimization =====
    learning_rate=5e-5,                  # Initial learning rate
    weight_decay=0.0,                    # Weight decay
    adam_beta1=0.9,                      # Adam beta1
    adam_beta2=0.999,                    # Adam beta2
    adam_epsilon=1e-8,                   # Adam epsilon
    max_grad_norm=1.0,                   # Gradient clipping
    optim="adamw_torch",                 # Optimizer ("adamw_torch", "adafactor", "adamw_8bit")

    # ===== Learning Rate Scheduler =====
    lr_scheduler_type="linear",          # Scheduler type
    warmup_steps=0,                      # Warmup steps
    warmup_ratio=0.0,                    # Warmup ratio (alternative to steps)

    # ===== Evaluation =====
    eval_strategy="no",                  # "no", "steps", "epoch"
    eval_steps=None,                     # Eval every N steps
    eval_delay=0,                        # Delay first eval
    eval_accumulation_steps=None,        # Accumulate eval outputs

    # ===== Checkpointing =====
    save_strategy="steps",               # "no", "steps", "epoch"
    save_steps=500,                      # Save every N steps
    save_total_limit=None,               # Max checkpoints to keep
    save_safetensors=True,               # Save as SafeTensors
    save_on_each_node=False,             # Save on each node (distributed)

    # ===== Best Model Selection =====
    load_best_model_at_end=False,        # Load best checkpoint at end
    metric_for_best_model=None,          # Metric to use
    greater_is_better=None,              # True if higher is better

    # ===== Logging =====
    logging_dir=None,                    # TensorBoard log directory
    logging_strategy="steps",            # "no", "steps", "epoch"
    logging_steps=500,                   # Log every N steps
    logging_first_step=False,            # Log first step
    logging_nan_inf_filter=True,         # Filter NaN/Inf

    # ===== Mixed Precision =====
    fp16=False,                          # Use fp16 training
    fp16_opt_level="O1",                 # Apex AMP optimization level
    fp16_backend="auto",                 # "auto", "apex", "cpu_amp"
    bf16=False,                          # Use bfloat16 training
    bf16_full_eval=False,                # Use bf16 for evaluation
    tf32=None,                           # Use TF32 (Ampere+ GPUs)

    # ===== Memory Optimization =====
    gradient_checkpointing=False,        # Enable gradient checkpointing
    gradient_checkpointing_kwargs=None,  # Kwargs for gradient checkpointing
    torch_empty_cache_steps=None,        # Clear cache every N steps

    # ===== Distributed Training =====
    local_rank=-1,                       # Local rank for distributed
    ddp_backend=None,                    # "nccl", "gloo", "mpi", "ccl"
    ddp_find_unused_parameters=None,     # Find unused parameters
    ddp_bucket_cap_mb=None,              # DDP bucket size
    fsdp="",                             # FSDP configuration
    fsdp_config=None,                    # FSDP config dict
    deepspeed=None,                      # DeepSpeed config

    # ===== Hub Integration =====
    push_to_hub=False,                   # Push to Hugging Face Hub
    hub_model_id=None,                   # Hub model ID
    hub_strategy="every_save",           # "every_save", "checkpoint", "end"
    hub_token=None,                      # Hub authentication token
    hub_private_repo=False,              # Make repo private

    # ===== Data Handling =====
    dataloader_num_workers=0,            # DataLoader workers
    dataloader_pin_memory=True,          # Pin memory
    dataloader_drop_last=False,          # Drop last incomplete batch
    dataloader_prefetch_factor=None,     # Prefetch factor
    remove_unused_columns=True,          # Remove unused dataset columns
    label_names=None,                    # Label column names

    # ===== Other =====
    seed=42,                             # Random seed
    data_seed=None,                      # Data sampling seed
    jit_mode_eval=False,                 # Use PyTorch JIT for eval
    use_ipex=False,                      # Use Intel Extension for PyTorch
    torch_compile=False,                 # Use torch.compile()
    torch_compile_backend=None,          # Compile backend
    torch_compile_mode=None,             # Compile mode
    include_inputs_for_metrics=False,    # Pass inputs to compute_metrics
    skip_memory_metrics=True,            # Skip memory profiling
)
```

### Trainer

Main training class with full training loop.

**Key Methods:**

```python
from transformers import Trainer

trainer = Trainer(
    model=None,                          # Model to train
    args=None,                           # TrainingArguments
    data_collator=None,                  # Data collator
    train_dataset=None,                  # Training dataset
    eval_dataset=None,                   # Evaluation dataset
    tokenizer=None,                      # Tokenizer
    model_init=None,                     # Function to instantiate model
    compute_metrics=None,                # Function to compute metrics
    callbacks=None,                      # List of callbacks
    optimizers=(None, None),             # (optimizer, scheduler) tuple
    preprocess_logits_for_metrics=None,  # Preprocess logits before metrics
)

# Train model
train_result = trainer.train(
    resume_from_checkpoint=None,         # Resume from checkpoint
    trial=None,                          # Optuna/Ray trial
    ignore_keys_for_eval=None,           # Keys to ignore in eval
)

# Evaluate model
eval_result = trainer.evaluate(
    eval_dataset=None,                   # Eval dataset (default: self.eval_dataset)
    ignore_keys=None,                    # Keys to ignore
    metric_key_prefix="eval",            # Prefix for metric names
)

# Make predictions
predictions = trainer.predict(
    test_dataset,                        # Test dataset
    ignore_keys=None,                    # Keys to ignore
    metric_key_prefix="test",            # Metric prefix
)
# Returns: PredictionOutput(predictions, label_ids, metrics)

# Save model
trainer.save_model(output_dir=None)

# Push to Hub
trainer.push_to_hub(
    commit_message="End of training",
    blocking=True,
    **kwargs
)

# Hyperparameter search
best_trial = trainer.hyperparameter_search(
    hp_space=None,                       # Hyperparameter search space
    compute_objective=None,              # Objective function
    n_trials=20,                         # Number of trials
    direction="minimize",                # "minimize" or "maximize"
    backend=None,                        # "optuna", "ray", "sigopt"
    **kwargs
)

# Create optimizer
optimizer = trainer.create_optimizer()

# Create scheduler
scheduler = trainer.create_scheduler(
    num_training_steps,
    optimizer=None
)

# Log metrics
trainer.log_metrics(split, metrics)
trainer.save_metrics(split, metrics)

# Save checkpoint
trainer.save_state()

# Access current step/epoch
current_step = trainer.state.global_step
current_epoch = trainer.state.epoch

# Access training logs
logs = trainer.state.log_history
```

### Seq2SeqTrainer

Specialized trainer for sequence-to-sequence models.

```python
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

# Use Seq2SeqTrainingArguments with additional parameters
training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    predict_with_generate=True,          # Use generate() for evaluation
    generation_max_length=None,          # Max length for generation
    generation_num_beams=None,           # Num beams for generation
    **other_training_arguments
)

# Trainer usage is identical to Trainer
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)
```

## Pipeline Classes

### pipeline()

Unified inference API for all tasks.

```python
from transformers import pipeline

pipe = pipeline(
    task=None,                           # Task name (required)
    model=None,                          # Model name/path or model object
    config=None,                         # Model config
    tokenizer=None,                      # Tokenizer
    feature_extractor=None,              # Feature extractor
    image_processor=None,                # Image processor
    framework=None,                      # "pt" or "tf"
    revision=None,                       # Model revision
    use_fast=True,                       # Use fast tokenizer
    token=None,                          # HF token
    device=None,                         # Device (-1 for CPU, 0+ for GPU)
    device_map=None,                     # Device map for multi-GPU
    torch_dtype=None,                    # Model dtype
    trust_remote_code=False,             # Allow custom code
    model_kwargs=None,                   # Additional model kwargs
    pipeline_class=None,                 # Custom pipeline class
    **kwargs
)

# Use pipeline
results = pipe(
    inputs,                              # Input data
    **task_specific_parameters
)
```

## Data Collators

Batch and pad data for training.

```python
from transformers import (
    DataCollatorWithPadding,             # Dynamic padding for classification
    DataCollatorForTokenClassification,  # Padding for token classification
    DataCollatorForSeq2Seq,              # Padding for seq2seq
    DataCollatorForLanguageModeling,     # MLM/CLM data collation
    default_data_collator,               # Simple collator (no padding)
)

# Text classification
data_collator = DataCollatorWithPadding(
    tokenizer=tokenizer,
    padding=True,
    max_length=None,
    pad_to_multiple_of=None,
)

# Token classification
data_collator = DataCollatorForTokenClassification(
    tokenizer=tokenizer,
    padding=True,
    max_length=None,
    pad_to_multiple_of=None,
    label_pad_token_id=-100,
)

# Seq2Seq
data_collator = DataCollatorForSeq2Seq(
    tokenizer=tokenizer,
    model=None,
    padding=True,
    max_length=None,
    pad_to_multiple_of=None,
    label_pad_token_id=-100,
)

# Language modeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,                            # Masked LM (False for causal LM)
    mlm_probability=0.15,                # Mask probability
    pad_to_multiple_of=None,
)
```

## Optimization & Scheduling

```python
from transformers import (
    AdamW,                               # AdamW optimizer
    Adafactor,                           # Adafactor optimizer
    get_scheduler,                       # Get LR scheduler
    get_linear_schedule_with_warmup,
    get_cosine_schedule_with_warmup,
    get_polynomial_decay_schedule_with_warmup,
)

# Create optimizer
optimizer = AdamW(
    model.parameters(),
    lr=5e-5,
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.01,
)

# Create scheduler
scheduler = get_scheduler(
    name="linear",                       # "linear", "cosine", "polynomial", "constant"
    optimizer=optimizer,
    num_warmup_steps=0,
    num_training_steps=total_steps,
)

# Or use specific schedulers
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=warmup_steps,
    num_training_steps=total_steps,
)

scheduler = get_cosine_schedule_with_warmup(
    optimizer,
    num_warmup_steps=warmup_steps,
    num_training_steps=total_steps,
    num_cycles=0.5,
)
```

## Configuration Classes

```python
from transformers import AutoConfig

# Load configuration
config = AutoConfig.from_pretrained(
    pretrained_model_name_or_path,
    **kwargs
)

# Common configuration attributes
config.vocab_size                        # Vocabulary size
config.hidden_size                       # Hidden layer size
config.num_hidden_layers                 # Number of layers
config.num_attention_heads               # Attention heads
config.intermediate_size                 # FFN intermediate size
config.hidden_dropout_prob               # Dropout probability
config.attention_probs_dropout_prob      # Attention dropout
config.max_position_embeddings           # Max sequence length

# Save configuration
config.save_pretrained(save_directory)

# Create model from config
from transformers import AutoModel
model = AutoModel.from_config(config)
```

## Utility Functions

```python
from transformers import (
    set_seed,                            # Set random seed
    logging,                             # Logging utilities
)

# Set seed for reproducibility
set_seed(42)

# Configure logging
logging.set_verbosity_info()
logging.set_verbosity_warning()
logging.set_verbosity_error()
logging.set_verbosity_debug()

# Get logger
logger = logging.get_logger(__name__)
```

## Model Outputs

All models return model-specific output classes (subclasses of `ModelOutput`):

```python
# Common output attributes
outputs.loss                             # Loss (if labels provided)
outputs.logits                           # Model logits
outputs.hidden_states                    # All hidden states (if output_hidden_states=True)
outputs.attentions                       # Attention weights (if output_attentions=True)

# Seq2Seq specific
outputs.encoder_last_hidden_state
outputs.encoder_hidden_states
outputs.encoder_attentions
outputs.decoder_hidden_states
outputs.decoder_attentions
outputs.cross_attentions

# Access as dict or tuple
logits = outputs.logits
logits = outputs["logits"]
loss, logits = outputs.to_tuple()[:2]
```

This reference covers the most commonly used API components. For complete documentation, refer to https://huggingface.co/docs/transformers.