mirror of
https://github.com/K-Dense-AI/claude-scientific-skills.git
synced 2026-03-28 07:33:45 +08:00
Improve the arboreto skill
This commit is contained in:
Timothy Kassis
@@ -1,415 +1,250 @@
|
||||
---
|
||||
name: arboreto
|
||||
description: "Gene regulatory network inference with GRNBoost2/GENIE3 algorithms. Infer TF-target relationships from expression data, scalable with Dask, for scRNA-seq and GRN analysis."
|
||||
description: Infer gene regulatory networks (GRNs) from gene expression data using scalable algorithms (GRNBoost2, GENIE3). Use when analyzing transcriptomics data (bulk RNA-seq, single-cell RNA-seq) to identify transcription factor-target gene relationships and regulatory interactions. Supports distributed computation for large-scale datasets.
|
||||
---
|
||||
|
||||
# Arboreto - Gene Regulatory Network Inference
|
||||
# Arboreto
|
||||
|
||||
## Overview
|
||||
|
||||
Arboreto is a Python library for inferring gene regulatory networks (GRNs) from gene expression data using machine learning algorithms. It enables scalable GRN inference from single machines to multi-node clusters using Dask for distributed computing. The skill provides comprehensive support for both GRNBoost2 (fast gradient boosting) and GENIE3 (Random Forest) algorithms.
|
||||
Arboreto is a computational library for inferring gene regulatory networks (GRNs) from gene expression data using parallelized algorithms that scale from single machines to multi-node clusters.
|
||||
|
||||
## When to Use This Skill
|
||||
**Core capability**: Identify which transcription factors (TFs) regulate which target genes based on expression patterns across observations (cells, samples, conditions).
|
||||
|
||||
This skill should be used when:
|
||||
- Inferring regulatory relationships between genes from expression data
|
||||
- Analyzing single-cell or bulk RNA-seq data to identify transcription factor targets
|
||||
- Building the GRN inference component of a pySCENIC pipeline
|
||||
- Comparing GRNBoost2 and GENIE3 algorithm performance
|
||||
- Setting up distributed computing for large-scale genomic analyses
|
||||
- Troubleshooting arboreto installation or runtime issues
|
||||
## Quick Start
|
||||
|
||||
Install arboreto:
|
||||
```bash
|
||||
pip install arboreto
|
||||
```
|
||||
|
||||
Basic GRN inference:
|
||||
```python
|
||||
import pandas as pd
|
||||
from arboreto.algo import grnboost2
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Load expression data (genes as columns)
|
||||
expression_matrix = pd.read_csv('expression_data.tsv', sep='\t')
|
||||
|
||||
# Infer regulatory network
|
||||
network = grnboost2(expression_data=expression_matrix)
|
||||
|
||||
# Save results (TF, target, importance)
|
||||
network.to_csv('network.tsv', sep='\t', index=False, header=False)
|
||||
```
|
||||
|
||||
**Critical**: Always use `if __name__ == '__main__':` guard because Dask spawns new processes.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### 1. Basic GRN Inference
|
||||
|
||||
For standard gene regulatory network inference tasks:
|
||||
For standard GRN inference workflows including:
|
||||
- Input data preparation (Pandas DataFrame or NumPy array)
|
||||
- Running inference with GRNBoost2 or GENIE3
|
||||
- Filtering by transcription factors
|
||||
- Output format and interpretation
|
||||
|
||||
**Key considerations:**
|
||||
- Expression data format: Rows = observations (cells/samples), Columns = genes
|
||||
- If data has genes as rows, transpose it first: `expression_df.T`
|
||||
- Always include `seed` parameter for reproducible results
|
||||
- Transcription factor list is optional but recommended for focused analysis
|
||||
**See**: `references/basic_inference.md`
|
||||
|
||||
**Typical workflow:**
|
||||
```python
|
||||
import pandas as pd
|
||||
from arboreto.algo import grnboost2
|
||||
from arboreto.utils import load_tf_names
|
||||
|
||||
# Load expression data (ensure correct orientation)
|
||||
expression_data = pd.read_csv('expression_data.tsv', sep='\t', index_col=0)
|
||||
|
||||
# Optional: Load TF names
|
||||
tf_names = load_tf_names('transcription_factors.txt')
|
||||
|
||||
# Run inference
|
||||
network = grnboost2(
|
||||
expression_data=expression_data,
|
||||
tf_names=tf_names,
|
||||
seed=42 # For reproducibility
|
||||
)
|
||||
|
||||
# Save results
|
||||
network.to_csv('network_output.tsv', sep='\t', index=False)
|
||||
```
|
||||
|
||||
**Output format:**
|
||||
- DataFrame with columns: `['TF', 'target', 'importance']`
|
||||
- Higher importance scores indicate stronger predicted regulatory relationships
|
||||
- Typically sorted by importance (descending)
|
||||
|
||||
**Multiprocessing requirement:**
|
||||
All arboreto code must include `if __name__ == '__main__':` protection due to Dask's multiprocessing requirements:
|
||||
|
||||
```python
|
||||
if __name__ == '__main__':
|
||||
# Arboreto code goes here
|
||||
network = grnboost2(expression_data=expr_data, seed=42)
|
||||
**Use the ready-to-run script**: `scripts/basic_grn_inference.py` for standard inference tasks:
|
||||
```bash
|
||||
python scripts/basic_grn_inference.py expression_data.tsv output_network.tsv --tf-file tfs.txt --seed 777
|
||||
```
|
||||
|
||||
### 2. Algorithm Selection
|
||||
|
||||
**GRNBoost2 (Recommended for most cases):**
|
||||
- ~10-100x faster than GENIE3
|
||||
- Uses stochastic gradient boosting with early-stopping
|
||||
- Best for: Large datasets (>10k observations), time-sensitive analyses
|
||||
- Function: `arboreto.algo.grnboost2()`
|
||||
Arboreto provides two algorithms:
|
||||
|
||||
**GENIE3:**
|
||||
- Uses Random Forest regression
|
||||
- More established, classical approach
|
||||
- Best for: Small datasets, methodological comparisons, reproducing published results
|
||||
- Function: `arboreto.algo.genie3()`
|
||||
**GRNBoost2 (Recommended)**:
|
||||
- Fast gradient boosting-based inference
|
||||
- Optimized for large datasets (10k+ observations)
|
||||
- Default choice for most analyses
|
||||
|
||||
**When to compare both algorithms:**
|
||||
Use the provided `compare_algorithms.py` script when:
|
||||
- Validating results for critical analyses
|
||||
- Benchmarking performance on new datasets
|
||||
- Publishing research requiring methodological comparisons
|
||||
**GENIE3**:
|
||||
- Random Forest-based inference
|
||||
- Original multiple regression approach
|
||||
- Use for comparison or validation
|
||||
|
||||
Quick comparison:
|
||||
```python
|
||||
from arboreto.algo import grnboost2, genie3
|
||||
|
||||
# Fast, recommended
|
||||
network_grnboost = grnboost2(expression_data=matrix)
|
||||
|
||||
# Classic algorithm
|
||||
network_genie3 = genie3(expression_data=matrix)
|
||||
```
|
||||
|
||||
**For detailed algorithm comparison, parameters, and selection guidance**: `references/algorithms.md`
|
||||
|
||||
### 3. Distributed Computing
|
||||
|
||||
**Local execution (default):**
|
||||
Arboreto automatically creates a local Dask client. No configuration needed:
|
||||
Scale inference from local multi-core to cluster environments:
|
||||
|
||||
**Local (default)** - Uses all available cores automatically:
|
||||
```python
|
||||
network = grnboost2(expression_data=expr_data)
|
||||
network = grnboost2(expression_data=matrix)
|
||||
```
|
||||
|
||||
**Custom local cluster (recommended for better control):**
|
||||
**Custom local client** - Control resources:
|
||||
```python
|
||||
from dask.distributed import Client, LocalCluster
|
||||
from distributed import LocalCluster, Client
|
||||
|
||||
# Configure cluster
|
||||
cluster = LocalCluster(
|
||||
n_workers=4,
|
||||
threads_per_worker=2,
|
||||
memory_limit='4GB',
|
||||
diagnostics_port=8787 # Dashboard at http://localhost:8787
|
||||
)
|
||||
client = Client(cluster)
|
||||
local_cluster = LocalCluster(n_workers=10, memory_limit='8GB')
|
||||
client = Client(local_cluster)
|
||||
|
||||
# Run inference
|
||||
network = grnboost2(
|
||||
expression_data=expr_data,
|
||||
client_or_address=client
|
||||
)
|
||||
network = grnboost2(expression_data=matrix, client_or_address=client)
|
||||
|
||||
# Clean up
|
||||
client.close()
|
||||
cluster.close()
|
||||
local_cluster.close()
|
||||
```
|
||||
|
||||
**Distributed cluster (multi-node):**
|
||||
On scheduler node:
|
||||
```bash
|
||||
dask-scheduler --no-bokeh
|
||||
```
|
||||
|
||||
On worker nodes:
|
||||
```bash
|
||||
dask-worker scheduler-address:8786 --local-dir /tmp
|
||||
```
|
||||
|
||||
In Python:
|
||||
**Cluster computing** - Connect to remote Dask scheduler:
|
||||
```python
|
||||
from dask.distributed import Client
|
||||
from distributed import Client
|
||||
|
||||
client = Client('scheduler-address:8786')
|
||||
network = grnboost2(expression_data=expr_data, client_or_address=client)
|
||||
client = Client('tcp://scheduler:8786')
|
||||
network = grnboost2(expression_data=matrix, client_or_address=client)
|
||||
```
|
||||
|
||||
### 4. Data Preparation
|
||||
|
||||
**Common data format issues:**
|
||||
|
||||
1. **Transposed data** (genes as rows instead of columns):
|
||||
```python
|
||||
# If genes are rows, transpose
|
||||
expression_data = pd.read_csv('data.tsv', sep='\t', index_col=0).T
|
||||
```
|
||||
|
||||
2. **Missing gene names:**
|
||||
```python
|
||||
# Provide gene names if using numpy array
|
||||
network = grnboost2(
|
||||
expression_data=expr_array,
|
||||
gene_names=['Gene1', 'Gene2', 'Gene3', ...],
|
||||
seed=42
|
||||
)
|
||||
```
|
||||
|
||||
3. **Transcription factor specification:**
|
||||
```python
|
||||
# Option 1: Python list
|
||||
tf_names = ['Sox2', 'Oct4', 'Nanog', 'Klf4']
|
||||
|
||||
# Option 2: Load from file (one TF per line)
|
||||
from arboreto.utils import load_tf_names
|
||||
tf_names = load_tf_names('tf_names.txt')
|
||||
```
|
||||
|
||||
### 5. Reproducibility
|
||||
|
||||
Always specify a seed for consistent results:
|
||||
```python
|
||||
network = grnboost2(expression_data=expr_data, seed=42)
|
||||
```
|
||||
|
||||
Without a seed, results will vary between runs due to algorithm randomness.
|
||||
|
||||
### 6. Result Interpretation
|
||||
|
||||
**Understanding the output:**
|
||||
- `TF`: Transcription factor (regulator) gene
|
||||
- `target`: Target gene being regulated
|
||||
- `importance`: Strength of predicted regulatory relationship
|
||||
|
||||
**Typical post-processing:**
|
||||
```python
|
||||
# Filter by importance threshold
|
||||
high_confidence = network[network['importance'] > 10]
|
||||
|
||||
# Get top N predictions
|
||||
top_predictions = network.head(1000)
|
||||
|
||||
# Find all targets of a specific TF
|
||||
sox2_targets = network[network['TF'] == 'Sox2']
|
||||
|
||||
# Count regulations per TF
|
||||
tf_counts = network['TF'].value_counts()
|
||||
```
|
||||
**For cluster setup, performance optimization, and large-scale workflows**: `references/distributed_computing.md`
|
||||
|
||||
## Installation
|
||||
|
||||
**Recommended (via conda):**
|
||||
**Recommended (Conda)**:
|
||||
```bash
|
||||
conda install -c bioconda arboreto
|
||||
```
|
||||
|
||||
**Via pip:**
|
||||
**Alternative (pip)**:
|
||||
```bash
|
||||
pip install arboreto
|
||||
```
|
||||
|
||||
**From source:**
|
||||
```bash
|
||||
git clone https://github.com/tmoerman/arboreto.git
|
||||
cd arboreto
|
||||
pip install .
|
||||
```
|
||||
|
||||
**Dependencies:**
|
||||
- pandas
|
||||
- numpy
|
||||
- scikit-learn
|
||||
- scipy
|
||||
- dask
|
||||
- distributed
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Bokeh error when launching Dask scheduler
|
||||
|
||||
**Error:** `TypeError: got an unexpected keyword argument 'host'`
|
||||
|
||||
**Solutions:**
|
||||
- Use `dask-scheduler --no-bokeh` to disable Bokeh
|
||||
- Upgrade to Dask distributed >= 0.20.0
|
||||
|
||||
### Issue: Workers not connecting to scheduler
|
||||
|
||||
**Symptoms:** Worker processes start but fail to establish connections
|
||||
|
||||
**Solutions:**
|
||||
- Remove `dask-worker-space` directory before restarting workers
|
||||
- Specify adequate `local_dir` when creating cluster:
|
||||
```python
|
||||
cluster = LocalCluster(
|
||||
worker_kwargs={'local_dir': '/tmp'}
|
||||
)
|
||||
```
|
||||
|
||||
### Issue: Memory errors with large datasets
|
||||
|
||||
**Solutions:**
|
||||
- Increase worker memory limits: `memory_limit='8GB'`
|
||||
- Distribute across more nodes
|
||||
- Reduce dataset size through preprocessing (e.g., feature selection)
|
||||
- Ensure expression matrix fits in available RAM
|
||||
|
||||
### Issue: Inconsistent results across runs
|
||||
|
||||
**Solution:** Always specify a `seed` parameter:
|
||||
```python
|
||||
network = grnboost2(expression_data=expr_data, seed=42)
|
||||
```
|
||||
|
||||
### Issue: Import errors or missing dependencies
|
||||
|
||||
**Solution:** Use conda installation to handle numerical library dependencies:
|
||||
**For isolated environment**:
|
||||
```bash
|
||||
conda create --name arboreto-env
|
||||
conda activate arboreto-env
|
||||
conda install -c bioconda arboreto
|
||||
```
|
||||
|
||||
## Provided Scripts
|
||||
**Dependencies**: scipy, scikit-learn, numpy, pandas, dask, distributed
|
||||
|
||||
This skill includes ready-to-use scripts for common workflows:
|
||||
## Common Use Cases
|
||||
|
||||
### scripts/basic_grn_inference.py
|
||||
### Single-Cell RNA-seq Analysis
|
||||
```python
|
||||
import pandas as pd
|
||||
from arboreto.algo import grnboost2
|
||||
|
||||
Command-line tool for standard GRN inference workflow.
|
||||
if __name__ == '__main__':
|
||||
# Load single-cell expression matrix (cells x genes)
|
||||
sc_data = pd.read_csv('scrna_counts.tsv', sep='\t')
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python scripts/basic_grn_inference.py expression_data.tsv \
|
||||
-t tf_names.txt \
|
||||
-o network.tsv \
|
||||
-s 42 \
|
||||
--transpose # if genes are rows
|
||||
# Infer cell-type-specific regulatory network
|
||||
network = grnboost2(expression_data=sc_data, seed=42)
|
||||
|
||||
# Filter high-confidence links
|
||||
high_confidence = network[network['importance'] > 0.5]
|
||||
high_confidence.to_csv('grn_high_confidence.tsv', sep='\t', index=False)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Automatic data loading and validation
|
||||
- Optional TF list specification
|
||||
- Configurable output format
|
||||
- Data transposition support
|
||||
- Summary statistics
|
||||
### Bulk RNA-seq with TF Filtering
|
||||
```python
|
||||
from arboreto.utils import load_tf_names
|
||||
from arboreto.algo import grnboost2
|
||||
|
||||
### scripts/distributed_inference.py
|
||||
if __name__ == '__main__':
|
||||
# Load data
|
||||
expression_data = pd.read_csv('rnaseq_tpm.tsv', sep='\t')
|
||||
tf_names = load_tf_names('human_tfs.txt')
|
||||
|
||||
GRN inference with custom Dask cluster configuration.
|
||||
# Infer with TF restriction
|
||||
network = grnboost2(
|
||||
expression_data=expression_data,
|
||||
tf_names=tf_names,
|
||||
seed=123
|
||||
)
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python scripts/distributed_inference.py expression_data.tsv \
|
||||
-t tf_names.txt \
|
||||
-w 8 \
|
||||
-m 4GB \
|
||||
--threads 2 \
|
||||
--dashboard-port 8787
|
||||
network.to_csv('tf_target_network.tsv', sep='\t', index=False)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Configurable worker count and memory limits
|
||||
- Dask dashboard integration
|
||||
- Thread configuration
|
||||
- Resource monitoring
|
||||
### Comparative Analysis (Multiple Conditions)
|
||||
```python
|
||||
from arboreto.algo import grnboost2
|
||||
|
||||
### scripts/compare_algorithms.py
|
||||
if __name__ == '__main__':
|
||||
# Infer networks for different conditions
|
||||
conditions = ['control', 'treatment_24h', 'treatment_48h']
|
||||
|
||||
Compare GRNBoost2 and GENIE3 side-by-side.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python scripts/compare_algorithms.py expression_data.tsv \
|
||||
-t tf_names.txt \
|
||||
--top-n 100
|
||||
for condition in conditions:
|
||||
data = pd.read_csv(f'{condition}_expression.tsv', sep='\t')
|
||||
network = grnboost2(expression_data=data, seed=42)
|
||||
network.to_csv(f'{condition}_network.tsv', sep='\t', index=False)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Runtime comparison
|
||||
- Network statistics
|
||||
- Prediction overlap analysis
|
||||
- Top prediction comparison
|
||||
## Output Interpretation
|
||||
|
||||
## Reference Documentation
|
||||
Arboreto returns a DataFrame with regulatory links:
|
||||
|
||||
Detailed API documentation is available in [references/api_reference.md](references/api_reference.md), including:
|
||||
- Complete parameter descriptions for all functions
|
||||
- Data format specifications
|
||||
- Distributed computing configuration
|
||||
- Performance optimization tips
|
||||
- Integration with pySCENIC
|
||||
- Comprehensive examples
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| `TF` | Transcription factor (regulator) |
|
||||
| `target` | Target gene |
|
||||
| `importance` | Regulatory importance score (higher = stronger) |
|
||||
|
||||
Load this reference when:
|
||||
- Working with advanced Dask configurations
|
||||
- Troubleshooting complex deployment scenarios
|
||||
- Understanding algorithm internals
|
||||
- Optimizing performance for specific use cases
|
||||
**Filtering strategy**:
|
||||
- Top N links per target gene
|
||||
- Importance threshold (e.g., > 0.5)
|
||||
- Statistical significance testing (permutation tests)
|
||||
|
||||
## Integration with pySCENIC
|
||||
|
||||
Arboreto is the first step in the pySCENIC single-cell analysis pipeline:
|
||||
Arboreto is a core component of the SCENIC pipeline for single-cell regulatory network analysis:
|
||||
|
||||
1. **GRN Inference (arboreto)** ← This skill
|
||||
- Input: Expression matrix
|
||||
- Output: Regulatory network
|
||||
|
||||
2. **Regulon Prediction (pySCENIC)**
|
||||
- Input: Network from arboreto
|
||||
- Output: Refined regulons
|
||||
|
||||
3. **Cell Type Identification (pySCENIC)**
|
||||
- Input: Regulons
|
||||
- Output: Cell type scores
|
||||
|
||||
When working with pySCENIC, use arboreto to generate the initial network, then pass results to the pySCENIC pipeline.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always use seed parameter** for reproducible research
|
||||
2. **Validate data orientation** (rows = observations, columns = genes)
|
||||
3. **Specify TF list** when known to focus inference and improve speed
|
||||
4. **Monitor with Dask dashboard** for distributed computing
|
||||
5. **Save intermediate results** to avoid re-running long computations
|
||||
6. **Filter results** by importance threshold for downstream analysis
|
||||
7. **Use GRNBoost2 by default** unless specifically requiring GENIE3
|
||||
8. **Include multiprocessing guard** (`if __name__ == '__main__':`) in all scripts
|
||||
|
||||
## Quick Reference
|
||||
|
||||
**Basic inference:**
|
||||
```python
|
||||
# Step 1: Use arboreto for GRN inference
|
||||
from arboreto.algo import grnboost2
|
||||
network = grnboost2(expression_data=expr_df, seed=42)
|
||||
network = grnboost2(expression_data=sc_data, tf_names=tf_list)
|
||||
|
||||
# Step 2: Use pySCENIC for regulon identification and activity scoring
|
||||
# (See pySCENIC documentation for downstream analysis)
|
||||
```
|
||||
|
||||
**With TF specification:**
|
||||
## Reproducibility
|
||||
|
||||
Always set a seed for reproducible results:
|
||||
```python
|
||||
network = grnboost2(expression_data=expr_df, tf_names=tf_list, seed=42)
|
||||
network = grnboost2(expression_data=matrix, seed=777)
|
||||
```
|
||||
|
||||
**With custom Dask client:**
|
||||
Run multiple seeds for robustness analysis:
|
||||
```python
|
||||
from dask.distributed import Client, LocalCluster
|
||||
cluster = LocalCluster(n_workers=4)
|
||||
client = Client(cluster)
|
||||
network = grnboost2(expression_data=expr_df, client_or_address=client, seed=42)
|
||||
client.close()
|
||||
cluster.close()
|
||||
from distributed import LocalCluster, Client
|
||||
|
||||
if __name__ == '__main__':
|
||||
client = Client(LocalCluster())
|
||||
|
||||
seeds = [42, 123, 777]
|
||||
networks = []
|
||||
|
||||
for seed in seeds:
|
||||
net = grnboost2(expression_data=matrix, client_or_address=client, seed=seed)
|
||||
networks.append(net)
|
||||
|
||||
# Combine networks and filter consensus links
|
||||
consensus = analyze_consensus(networks)
|
||||
```
|
||||
|
||||
**Load TF names:**
|
||||
```python
|
||||
from arboreto.utils import load_tf_names
|
||||
tf_names = load_tf_names('transcription_factors.txt')
|
||||
```
|
||||
## Troubleshooting
|
||||
|
||||
**Transpose data:**
|
||||
```python
|
||||
expression_df = pd.read_csv('data.tsv', sep='\t', index_col=0).T
|
||||
```
|
||||
**Memory errors**: Reduce dataset size by filtering low-variance genes or use distributed computing
|
||||
|
||||
**Slow performance**: Use GRNBoost2 instead of GENIE3, enable distributed client, filter TF list
|
||||
|
||||
**Dask errors**: Ensure `if __name__ == '__main__':` guard is present in scripts
|
||||
|
||||
**Empty results**: Check data format (genes as columns), verify TF names match gene names
|
||||
|
||||
Reference in New Issue
Block a user