# arXiv API Reference

## Overview

The arXiv API provides programmatic access to preprint metadata via an Atom XML feed. It supports search queries with field-specific operators, boolean logic, ID-based retrieval, sorting, and pagination. No authentication required.

## Base URL

```
http://export.arxiv.org/api/query
```

## Rate Limiting

- Recommended: **1 request per 3 seconds**
- Aggressive crawling will result in temporary IP bans
- Use `time.sleep(3)` between requests
- Include a descriptive `User-Agent` header

## Query Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `search_query` | Query string with field prefixes and boolean operators | (none) |
| `id_list` | Comma-separated arXiv IDs | (none) |
| `start` | Starting index for pagination (0-based) | `0` |
| `max_results` | Number of results to return (max 300) | `10` |
| `sortBy` | Sort field: `relevance`, `lastUpdatedDate`, `submittedDate` | `relevance` |
| `sortOrder` | Sort direction: `ascending`, `descending` | `descending` |

**Note**: `search_query` and `id_list` can be used together (results are ANDed) or separately.

## Search Query Syntax

### Field Prefixes

| Prefix | Field | Example |
|--------|-------|---------|
| `ti:` | Title | `ti:transformer` |
| `au:` | Author | `au:bengio` |
| `abs:` | Abstract | `abs:attention mechanism` |
| `co:` | Comment | `co:accepted at NeurIPS` |
| `jr:` | Journal Reference | `jr:Nature` |
| `cat:` | Category | `cat:cs.LG` |
| `all:` | All fields | `all:deep learning` |
| `id:` | arXiv ID | `id:2309.10668` |

### Boolean Operators

Operators **must** be uppercase:

```
ti:transformer AND abs:attention           # Both conditions
au:bengio OR au:lecun                      # Either condition
cat:cs.LG ANDNOT cat:cs.CV                # Exclude category
```

### Grouping

Use parentheses for complex queries:

```
(ti:sparse AND ti:autoencoder) AND cat:cs.LG
au:anthropic AND (abs:interpretability OR abs:alignment)
(cat:cs.LG OR cat:cs.CL) AND ti:reinforcement learning
```

### Phrase Search

Quotes for exact phrases:

```
ti:"sparse autoencoder"
au:"Yoshua Bengio"
abs:"reinforcement learning from human feedback"
```

### Wildcards

Not supported by the arXiv API. Use broader terms and filter client-side.

## Example Requests

### Basic keyword search
```
GET http://export.arxiv.org/api/query?search_query=all:sparse+autoencoder&max_results=10
```

### Author + category
```
GET http://export.arxiv.org/api/query?search_query=au:anthropic+AND+cat:cs.LG&max_results=50&sortBy=submittedDate
```

### ID lookup
```
GET http://export.arxiv.org/api/query?id_list=2309.10668,2406.04093
```

### Combined search + ID
```
GET http://export.arxiv.org/api/query?search_query=cat:cs.LG&id_list=2309.10668
```

### Paginated results
```
# Page 1 (results 0-99)
GET ...?search_query=cat:cs.LG&start=0&max_results=100&sortBy=submittedDate

# Page 2 (results 100-199)
GET ...?search_query=cat:cs.LG&start=100&max_results=100&sortBy=submittedDate
```

## Response Format (Atom XML)

The API returns an Atom 1.0 XML feed.

### Feed-level elements

```xml
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:arxiv="http://arxiv.org/schemas/atom">

  <title>ArXiv Query: ...</title>
  <id>http://arxiv.org/api/...</id>
  <updated>2024-01-15T00:00:00-05:00</updated>

  <!-- Total results available (not just returned) -->
  <opensearch:totalResults>1500</opensearch:totalResults>
  <opensearch:startIndex>0</opensearch:startIndex>
  <opensearch:itemsPerPage>50</opensearch:itemsPerPage>

  <entry>...</entry>
  <entry>...</entry>
</feed>
```

### Entry elements

```xml
<entry>
  <!-- Unique identifier (includes version) -->
  <id>http://arxiv.org/abs/2309.10668v2</id>

  <!-- Dates -->
  <published>2023-09-19T17:58:00Z</published>
  <updated>2023-10-04T14:22:00Z</updated>

  <!-- Metadata -->
  <title>Towards Monosemanticity: Decomposing Language Models...</title>
  <summary>We attempt to reverse-engineer a trained neural network...</summary>

  <!-- Authors -->
  <author>
    <name>Trenton Bricken</name>
  </author>
  <author>
    <name>Adly Templeton</name>
  </author>

  <!-- Categories -->
  <arxiv:primary_category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
  <category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
  <category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>

  <!-- Links -->
  <link href="http://arxiv.org/abs/2309.10668v2" rel="alternate" type="text/html"/>
  <link href="http://arxiv.org/pdf/2309.10668v2" rel="related" type="application/pdf" title="pdf"/>

  <!-- Optional -->
  <arxiv:comment>42 pages, 30 figures</arxiv:comment>
  <arxiv:doi>10.48550/arXiv.2309.10668</arxiv:doi>
  <arxiv:journal_ref>...</arxiv:journal_ref>
</entry>
```

### Entry field descriptions

| Field | Description |
|-------|-------------|
| `id` | Canonical arXiv URL with version (e.g., `http://arxiv.org/abs/2309.10668v2`) |
| `published` | First submission date (ISO 8601) |
| `updated` | Last update date (ISO 8601) |
| `title` | Paper title (may contain line breaks in XML) |
| `summary` | Full abstract text |
| `author/name` | Author full name (one per `<author>` element) |
| `arxiv:primary_category` | Primary arXiv category |
| `category` | All categories (multiple elements) |
| `link[@type='text/html']` | Abstract page URL |
| `link[@title='pdf']` | PDF download URL |
| `arxiv:comment` | Author comment (page count, conference, etc.) |
| `arxiv:doi` | Associated DOI (if exists) |
| `arxiv:journal_ref` | Journal publication reference (if published) |

## Complete Category List

### Computer Science (cs.*)

| Category | Name |
|----------|------|
| `cs.AI` | Artificial Intelligence |
| `cs.AR` | Hardware Architecture |
| `cs.CC` | Computational Complexity |
| `cs.CE` | Computational Engineering, Finance, and Science |
| `cs.CG` | Computational Geometry |
| `cs.CL` | Computation and Language |
| `cs.CR` | Cryptography and Security |
| `cs.CV` | Computer Vision and Pattern Recognition |
| `cs.CY` | Computers and Society |
| `cs.DB` | Databases |
| `cs.DC` | Distributed, Parallel, and Cluster Computing |
| `cs.DL` | Digital Libraries |
| `cs.DM` | Discrete Mathematics |
| `cs.DS` | Data Structures and Algorithms |
| `cs.ET` | Emerging Technologies |
| `cs.FL` | Formal Languages and Automata Theory |
| `cs.GL` | General Literature |
| `cs.GR` | Graphics |
| `cs.GT` | Computer Science and Game Theory |
| `cs.HC` | Human-Computer Interaction |
| `cs.IR` | Information Retrieval |
| `cs.IT` | Information Theory |
| `cs.LG` | Machine Learning |
| `cs.LO` | Logic in Computer Science |
| `cs.MA` | Multiagent Systems |
| `cs.MM` | Multimedia |
| `cs.MS` | Mathematical Software |
| `cs.NA` | Numerical Analysis |
| `cs.NE` | Neural and Evolutionary Computing |
| `cs.NI` | Networking and Internet Architecture |
| `cs.OH` | Other Computer Science |
| `cs.OS` | Operating Systems |
| `cs.PF` | Performance |
| `cs.PL` | Programming Languages |
| `cs.RO` | Robotics |
| `cs.SC` | Symbolic Computation |
| `cs.SD` | Sound |
| `cs.SE` | Software Engineering |
| `cs.SI` | Social and Information Networks |
| `cs.SY` | Systems and Control |

### Statistics (stat.*)

| Category | Name |
|----------|------|
| `stat.AP` | Applications |
| `stat.CO` | Computation |
| `stat.ME` | Methodology |
| `stat.ML` | Machine Learning |
| `stat.OT` | Other Statistics |
| `stat.TH` | Statistics Theory |

### Mathematics (math.*)

| Category | Name |
|----------|------|
| `math.AC` | Commutative Algebra |
| `math.AG` | Algebraic Geometry |
| `math.AP` | Analysis of PDEs |
| `math.AT` | Algebraic Topology |
| `math.CA` | Classical Analysis and ODEs |
| `math.CO` | Combinatorics |
| `math.CT` | Category Theory |
| `math.CV` | Complex Variables |
| `math.DG` | Differential Geometry |
| `math.DS` | Dynamical Systems |
| `math.FA` | Functional Analysis |
| `math.GM` | General Mathematics |
| `math.GN` | General Topology |
| `math.GR` | Group Theory |
| `math.GT` | Geometric Topology |
| `math.HO` | History and Overview |
| `math.IT` | Information Theory |
| `math.KT` | K-Theory and Homology |
| `math.LO` | Logic |
| `math.MG` | Metric Geometry |
| `math.MP` | Mathematical Physics |
| `math.NA` | Numerical Analysis |
| `math.NT` | Number Theory |
| `math.OA` | Operator Algebras |
| `math.OC` | Optimization and Control |
| `math.PR` | Probability |
| `math.QA` | Quantum Algebra |
| `math.RA` | Rings and Algebras |
| `math.RT` | Representation Theory |
| `math.SG` | Symplectic Geometry |
| `math.SP` | Spectral Theory |
| `math.ST` | Statistics Theory |

### Physics

| Category | Name |
|----------|------|
| `astro-ph` | Astrophysics (+ subcategories: .CO, .EP, .GA, .HE, .IM, .SR) |
| `cond-mat` | Condensed Matter (+ subcategories) |
| `gr-qc` | General Relativity and Quantum Cosmology |
| `hep-ex` | High Energy Physics - Experiment |
| `hep-lat` | High Energy Physics - Lattice |
| `hep-ph` | High Energy Physics - Phenomenology |
| `hep-th` | High Energy Physics - Theory |
| `math-ph` | Mathematical Physics |
| `nlin` | Nonlinear Sciences (+ subcategories) |
| `nucl-ex` | Nuclear Experiment |
| `nucl-th` | Nuclear Theory |
| `physics` | Physics (+ subcategories: .comp-ph, .data-an, .bio-ph, etc.) |
| `quant-ph` | Quantum Physics |

### Quantitative Biology (q-bio.*)

| Category | Name |
|----------|------|
| `q-bio.BM` | Biomolecules |
| `q-bio.CB` | Cell Behavior |
| `q-bio.GN` | Genomics |
| `q-bio.MN` | Molecular Networks |
| `q-bio.NC` | Neurons and Cognition |
| `q-bio.OT` | Other Quantitative Biology |
| `q-bio.PE` | Populations and Evolution |
| `q-bio.QM` | Quantitative Methods |
| `q-bio.SC` | Subcellular Processes |
| `q-bio.TO` | Tissues and Organs |

### Quantitative Finance (q-fin.*)

| Category | Name |
|----------|------|
| `q-fin.CP` | Computational Finance |
| `q-fin.EC` | Economics |
| `q-fin.GN` | General Finance |
| `q-fin.MF` | Mathematical Finance |
| `q-fin.PM` | Portfolio Management |
| `q-fin.PR` | Pricing of Securities |
| `q-fin.RM` | Risk Management |
| `q-fin.ST` | Statistical Finance |
| `q-fin.TR` | Trading and Market Microstructure |

### Electrical Engineering and Systems Science (eess.*)

| Category | Name |
|----------|------|
| `eess.AS` | Audio and Speech Processing |
| `eess.IV` | Image and Video Processing |
| `eess.SP` | Signal Processing |
| `eess.SY` | Systems and Control |

### Economics (econ.*)

| Category | Name |
|----------|------|
| `econ.EM` | Econometrics |
| `econ.GN` | General Economics |
| `econ.TH` | Theoretical Economics |

## Pagination

The API returns at most 300 results per request. For larger result sets, paginate:

```python
all_results = []
start = 0
batch_size = 100

while True:
    params = {
        "search_query": "cat:cs.LG",
        "start": start,
        "max_results": batch_size,
        "sortBy": "submittedDate",
        "sortOrder": "descending",
    }
    results = fetch(params)  # your fetch function
    if not results:
        break
    all_results.extend(results)
    start += batch_size
    time.sleep(3)  # respect rate limit
```

The total number of results available is in the `opensearch:totalResults` element of the feed.

## Downloading Papers

### PDF
```
http://arxiv.org/pdf/{arxiv_id}
http://arxiv.org/pdf/{arxiv_id}v{version}
```

### Abstract page
```
http://arxiv.org/abs/{arxiv_id}
```

### Source (LaTeX)
```
http://arxiv.org/e-print/{arxiv_id}
```

### HTML (experimental)
```
http://arxiv.org/html/{arxiv_id}
```

## arXiv ID Formats

| Format | Era | Example |
|--------|-----|---------|
| `YYMM.NNNNN` | 2015+ | `2309.10668` |
| `YYMM.NNNN` | 2007-2014 | `0706.0001` |
| `archive/YYMMNNN` | Pre-2007 | `hep-th/9901001` |

All formats are accepted by the API.

## Common Pitfalls

1. **Boolean operators must be UPPERCASE**: `AND`, `OR`, `ANDNOT` (lowercase is treated as search terms)
2. **URL encoding**: Spaces in queries must be encoded as `+` or `%20`
3. **No full-text search**: The API only searches metadata (title, abstract, authors, etc.)
4. **Empty result placeholder**: When no results are found, arXiv may return a single entry with an empty title and the id `http://arxiv.org/api/errors` - filter this out
5. **Version numbering**: `published` date is v1 submission; `updated` is latest version date
6. **Rate limiting**: Exceeding limits can result in 403 errors or temporary bans
7. **Max 300 per request**: Even if `max_results` is set higher, only 300 are returned

## External Resources

- arXiv API documentation: https://info.arxiv.org/help/api/index.html
- arXiv API user manual: https://info.arxiv.org/help/api/user-manual.html
- arXiv bulk data access: https://info.arxiv.org/help/bulk_data.html
- arXiv category taxonomy: https://arxiv.org/category_taxonomy
- OAI-PMH interface (for bulk metadata): http://export.arxiv.org/oai2