# Bioinformatics and Genomics File Formats Reference

This reference covers file formats used in genomics, transcriptomics, sequence analysis, and related bioinformatics applications.

## Sequence Data Formats

### .fasta / .fa / .fna - FASTA Format
**Description:** Text-based format for nucleotide or protein sequences
**Typical Data:** DNA, RNA, or protein sequences with headers
**Use Cases:** Sequence storage, BLAST searches, alignments
**Python Libraries:**
- `Biopython`: `SeqIO.parse('file.fasta', 'fasta')`
- `pyfaidx`: Fast indexed FASTA access
- `screed`: Fast sequence parsing
**EDA Approach:**
- Sequence count and length distribution
- GC content analysis
- N content (ambiguous bases)
- Sequence ID parsing
- Duplicate detection
- Quality metrics for assemblies (N50, L50)

### .fastq / .fq - FASTQ Format
**Description:** Sequence data with base quality scores
**Typical Data:** Raw sequencing reads with Phred quality scores
**Use Cases:** NGS data, quality control, read mapping
**Python Libraries:**
- `Biopython`: `SeqIO.parse('file.fastq', 'fastq')`
- `pysam`: Fast FASTQ/BAM operations
- `HTSeq`: Sequencing data analysis
**EDA Approach:**
- Read count and length distribution
- Quality score distribution (per-base, per-read)
- GC content and bias
- Duplicate rate estimation
- Adapter contamination detection
- k-mer frequency analysis
- Encoding format validation (Phred33/64)

### .sam - Sequence Alignment/Map
**Description:** Tab-delimited text format for alignments
**Typical Data:** Aligned sequencing reads with mapping quality
**Use Cases:** Read alignment storage, variant calling
**Python Libraries:**
- `pysam`: `pysam.AlignmentFile('file.sam', 'r')`
- `HTSeq`: `HTSeq.SAM_Reader('file.sam')`
**EDA Approach:**
- Mapping rate and quality distribution
- Coverage analysis
- Insert size distribution (paired-end)
- Alignment flags distribution
- CIGAR string patterns
- Mismatch and indel rates
- Duplicate and supplementary alignment counts

### .bam - Binary Alignment/Map
**Description:** Compressed binary version of SAM
**Typical Data:** Aligned reads in compressed format
**Use Cases:** Efficient storage and processing of alignments
**Python Libraries:**
- `pysam`: Full BAM support with indexing
- `bamnostic`: Pure Python BAM reader
**EDA Approach:**
- Same as SAM plus:
- Compression ratio analysis
- Index file (.bai) validation
- Chromosome-wise statistics
- Strand bias detection
- Read group analysis

### .cram - CRAM Format
**Description:** Highly compressed alignment format
**Typical Data:** Reference-compressed aligned reads
**Use Cases:** Long-term storage, space-efficient archives
**Python Libraries:**
- `pysam`: CRAM support (requires reference)
- Reference genome must be accessible
**EDA Approach:**
- Compression efficiency vs BAM
- Reference dependency validation
- Lossy vs lossless compression assessment
- Decompression performance
- Similar alignment metrics as BAM

### .bed - Browser Extensible Data
**Description:** Tab-delimited format for genomic features
**Typical Data:** Genomic intervals (chr, start, end) with annotations
**Use Cases:** Peak calling, variant annotation, genome browsing
**Python Libraries:**
- `pybedtools`: `pybedtools.BedTool('file.bed')`
- `pyranges`: `pyranges.read_bed('file.bed')`
- `pandas`: Simple BED reading
**EDA Approach:**
- Feature count and size distribution
- Chromosome distribution
- Strand bias
- Score distribution (if present)
- Overlap and proximity analysis
- Coverage statistics
- Gap analysis between features

### .bedGraph - BED with Graph Data
**Description:** BED format with per-base signal values
**Typical Data:** Continuous-valued genomic data (coverage, signals)
**Use Cases:** Coverage tracks, ChIP-seq signals, methylation
**Python Libraries:**
- `pyBigWig`: Can convert to bigWig
- `pybedtools`: BedGraph operations
**EDA Approach:**
- Signal distribution statistics
- Genome coverage percentage
- Signal dynamics (peaks, valleys)
- Chromosome-wise signal patterns
- Quantile analysis
- Zero-coverage regions

### .bigWig / .bw - Binary BigWig
**Description:** Indexed binary format for genome-wide signal data
**Typical Data:** Continuous genomic signals (compressed and indexed)
**Use Cases:** Efficient genome browser tracks, large-scale data
**Python Libraries:**
- `pyBigWig`: `pyBigWig.open('file.bw')`
- `pybbi`: BigWig/BigBed interface
**EDA Approach:**
- Signal statistics extraction
- Zoom level analysis
- Regional signal extraction
- Efficient genome-wide summaries
- Compression efficiency
- Index structure analysis

### .bigBed / .bb - Binary BigBed
**Description:** Indexed binary BED format
**Typical Data:** Genomic features (compressed and indexed)
**Use Cases:** Large feature sets, genome browsers
**Python Libraries:**
- `pybbi`: BigBed reading
- `pybigtools`: Modern BigBed interface
**EDA Approach:**
- Feature density analysis
- Efficient interval queries
- Zoom level validation
- Index performance metrics
- Feature size statistics

### .gff / .gff3 - General Feature Format
**Description:** Tab-delimited format for genomic annotations
**Typical Data:** Gene models, transcripts, exons, regulatory elements
**Use Cases:** Genome annotation, gene prediction
**Python Libraries:**
- `BCBio.GFF`: Biopython GFF module
- `gffutils`: `gffutils.create_db('file.gff3')`
- `pyranges`: GFF support
**EDA Approach:**
- Feature type distribution (gene, exon, CDS, etc.)
- Gene structure validation
- Strand balance
- Hierarchical relationship validation
- Phase validation for CDS
- Attribute completeness
- Gene model statistics (introns, exons per gene)

### .gtf - Gene Transfer Format
**Description:** GFF2-based format for gene annotations
**Typical Data:** Gene and transcript annotations
**Use Cases:** RNA-seq analysis, gene quantification
**Python Libraries:**
- `pyranges`: `pyranges.read_gtf('file.gtf')`
- `gffutils`: GTF database creation
- `HTSeq`: GTF reading for counts
**EDA Approach:**
- Transcript isoform analysis
- Gene structure completeness
- Exon number distribution
- Transcript length distribution
- TSS and TES analysis
- Biotype distribution
- Overlapping gene detection

### .vcf - Variant Call Format
**Description:** Text format for genetic variants
**Typical Data:** SNPs, indels, structural variants with annotations
**Use Cases:** Variant calling, population genetics, GWAS
**Python Libraries:**
- `pysam`: `pysam.VariantFile('file.vcf')`
- `cyvcf2`: Fast VCF parsing
- `PyVCF`: Older but comprehensive
**EDA Approach:**
- Variant count by type (SNP, indel, SV)
- Quality score distribution
- Allele frequency spectrum
- Transition/transversion ratio
- Heterozygosity rates
- Missing genotype analysis
- Hardy-Weinberg equilibrium
- Annotation completeness (if annotated)

### .bcf - Binary VCF
**Description:** Compressed binary variant format
**Typical Data:** Same as VCF but binary
**Use Cases:** Efficient variant storage and processing
**Python Libraries:**
- `pysam`: Full BCF support
- `cyvcf2`: Optimized BCF reading
**EDA Approach:**
- Same as VCF plus:
- Compression efficiency
- Indexing validation
- Read performance metrics

### .gvcf - Genomic VCF
**Description:** VCF with reference confidence blocks
**Typical Data:** All positions (variant and non-variant)
**Use Cases:** Joint genotyping workflows, GATK
**Python Libraries:**
- `pysam`: GVCF support
- Standard VCF parsers
**EDA Approach:**
- Reference block analysis
- Coverage uniformity
- Variant density
- Genotype quality across genome
- Reference confidence distribution

## RNA-Seq and Expression Data

### .counts - Gene Count Matrix
**Description:** Tab-delimited gene expression counts
**Typical Data:** Gene IDs with read counts per sample
**Use Cases:** RNA-seq quantification, differential expression
**Python Libraries:**
- `pandas`: `pd.read_csv('file.counts', sep='\t')`
- `scanpy` (for single-cell): `sc.read_csv()`
**EDA Approach:**
- Library size distribution
- Detection rate (genes per sample)
- Zero-inflation analysis
- Count distribution (log scale)
- Outlier sample detection
- Correlation between replicates
- PCA for sample relationships

### .tpm / .fpkm - Normalized Expression
**Description:** Normalized gene expression values
**Typical Data:** TPM (transcripts per million) or FPKM values
**Use Cases:** Cross-sample comparison, visualization
**Python Libraries:**
- `pandas`: Standard CSV reading
- `anndata`: For integrated analysis
**EDA Approach:**
- Expression distribution
- Highly expressed gene identification
- Sample clustering
- Batch effect detection
- Coefficient of variation analysis
- Dynamic range assessment

### .mtx - Matrix Market Format
**Description:** Sparse matrix format (common in single-cell)
**Typical Data:** Sparse count matrices (cells × genes)
**Use Cases:** Single-cell RNA-seq, large sparse matrices
**Python Libraries:**
- `scipy.io`: `scipy.io.mmread('file.mtx')`
- `scanpy`: `sc.read_mtx('file.mtx')`
**EDA Approach:**
- Sparsity analysis
- Cell and gene filtering thresholds
- Doublet detection metrics
- Mitochondrial fraction
- UMI count distribution
- Gene detection per cell

### .h5ad - Anndata Format
**Description:** HDF5-based annotated data matrix
**Typical Data:** Expression matrix with metadata (cells, genes)
**Use Cases:** Single-cell RNA-seq analysis with Scanpy
**Python Libraries:**
- `scanpy`: `sc.read_h5ad('file.h5ad')`
- `anndata`: Direct AnnData manipulation
**EDA Approach:**
- Cell and gene counts
- Metadata completeness
- Layer availability (raw, normalized)
- Embedding presence (PCA, UMAP)
- QC metrics distribution
- Batch information
- Cell type annotation coverage

### .loom - Loom Format
**Description:** HDF5-based format for omics data
**Typical Data:** Expression matrices with metadata
**Use Cases:** Single-cell data, RNA velocity analysis
**Python Libraries:**
- `loompy`: `loompy.connect('file.loom')`
- `scanpy`: Can import loom files
**EDA Approach:**
- Layer analysis (spliced, unspliced)
- Row and column attribute exploration
- Graph connectivity analysis
- Cluster assignments
- Velocity-specific metrics

### .rds - R Data Serialization
**Description:** R object storage (often Seurat objects)
**Typical Data:** R analysis results, especially single-cell
**Use Cases:** R-Python data exchange
**Python Libraries:**
- `pyreadr`: `pyreadr.read_r('file.rds')`
- `rpy2`: For full R integration
- Conversion tools to AnnData
**EDA Approach:**
- Object type identification
- Data structure exploration
- Metadata extraction
- Conversion validation

## Alignment and Assembly Formats

### .maf - Multiple Alignment Format
**Description:** Text format for multiple sequence alignments
**Typical Data:** Genome-wide or local multiple alignments
**Use Cases:** Comparative genomics, conservation analysis
**Python Libraries:**
- `Biopython`: `AlignIO.parse('file.maf', 'maf')`
- `bx-python`: MAF-specific tools
**EDA Approach:**
- Alignment block statistics
- Species coverage
- Gap analysis
- Conservation scoring
- Alignment quality metrics
- Block length distribution

### .axt - Pairwise Alignment Format
**Description:** Pairwise alignment format (UCSC)
**Typical Data:** Pairwise genomic alignments
**Use Cases:** Genome comparison, synteny analysis
**Python Libraries:**
- Custom parsers (simple format)
- `bx-python`: AXT support
**EDA Approach:**
- Alignment score distribution
- Identity percentage
- Syntenic block identification
- Gap size analysis
- Coverage statistics

### .chain - Chain Alignment Format
**Description:** Genome coordinate mapping chains
**Typical Data:** Coordinate transformations between genome builds
**Use Cases:** Liftover, coordinate conversion
**Python Libraries:**
- `pyliftover`: Chain file usage
- Custom parsers for chain format
**EDA Approach:**
- Chain score distribution
- Coverage of source genome
- Gap analysis
- Inversion detection
- Mapping quality assessment

### .psl - Pattern Space Layout
**Description:** BLAT/BLAST alignment format
**Typical Data:** Alignment results from BLAT
**Use Cases:** Transcript mapping, similarity searches
**Python Libraries:**
- Custom parsers (tab-delimited)
- `pybedtools`: Can handle PSL
**EDA Approach:**
- Match percentage distribution
- Gap statistics
- Query coverage
- Multiple mapping analysis
- Alignment quality metrics

## Genome Assembly and Annotation

### .agp - Assembly Golden Path
**Description:** Assembly structure description
**Typical Data:** Scaffold composition, gap information
**Use Cases:** Genome assembly representation
**Python Libraries:**
- Custom parsers (simple tab-delimited)
- Assembly analysis tools
**EDA Approach:**
- Scaffold statistics (N50, L50)
- Gap type and size distribution
- Component length analysis
- Assembly contiguity metrics
- Unplaced contig analysis

### .scaffolds / .contigs - Assembly Sequences
**Description:** Assembled sequences (usually FASTA)
**Typical Data:** Assembled genomic sequences
**Use Cases:** Genome assembly output
**Python Libraries:**
- Same as FASTA format
- Assembly-specific tools (QUAST)
**EDA Approach:**
- Assembly statistics (N50, N90, etc.)
- Length distribution
- Coverage analysis
- Gap (N) content
- Duplication assessment
- BUSCO completeness (if annotations available)

### .2bit - Compressed Genome Format
**Description:** UCSC compact genome format
**Typical Data:** Reference genomes (highly compressed)
**Use Cases:** Efficient genome storage and access
**Python Libraries:**
- `py2bit`: `py2bit.open('file.2bit')`
- `twobitreader`: Alternative reader
**EDA Approach:**
- Compression efficiency
- Random access performance
- Sequence extraction validation
- Masked region analysis
- N content and distribution

### .sizes - Chromosome Sizes
**Description:** Simple format with chromosome lengths
**Typical Data:** Tab-delimited chromosome names and sizes
**Use Cases:** Genome browsers, coordinate validation
**Python Libraries:**
- Simple file reading with pandas
- Built into many genomic tools
**EDA Approach:**
- Genome size calculation
- Chromosome count
- Size distribution
- Karyotype validation
- Completeness check against reference

## Phylogenetics and Evolution

### .nwk / .newick - Newick Tree Format
**Description:** Parenthetical tree representation
**Typical Data:** Phylogenetic trees with branch lengths
**Use Cases:** Evolutionary analysis, tree visualization
**Python Libraries:**
- `Biopython`: `Phylo.read('file.nwk', 'newick')`
- `ete3`: `ete3.Tree('file.nwk')`
- `dendropy`: Phylogenetic computing
**EDA Approach:**
- Tree structure analysis (tips, internal nodes)
- Branch length distribution
- Tree balance metrics
- Ultrametricity check
- Bootstrap support analysis
- Topology validation

### .nexus - Nexus Format
**Description:** Rich format for phylogenetic data
**Typical Data:** Alignments, trees, character matrices
**Use Cases:** Phylogenetic software interchange
**Python Libraries:**
- `Biopython`: Nexus support
- `dendropy`: Comprehensive Nexus handling
**EDA Approach:**
- Data block analysis
- Character type distribution
- Tree block validation
- Taxa consistency
- Command block parsing
- Format compliance checking

### .phylip - PHYLIP Format
**Description:** Sequence alignment format (strict/relaxed)
**Typical Data:** Multiple sequence alignments
**Use Cases:** Phylogenetic analysis input
**Python Libraries:**
- `Biopython`: `AlignIO.read('file.phy', 'phylip')`
- `dendropy`: PHYLIP support
**EDA Approach:**
- Alignment dimensions
- Sequence length uniformity
- Gap position analysis
- Informative site calculation
- Format variant detection (strict vs relaxed)

### .paml - PAML Output
**Description:** Output from PAML phylogenetic software
**Typical Data:** Evolutionary model results, dN/dS ratios
**Use Cases:** Molecular evolution analysis
**Python Libraries:**
- Custom parsers for specific PAML programs
- `Biopython`: Basic PAML parsing
**EDA Approach:**
- Model parameter extraction
- Likelihood values
- dN/dS ratio distribution
- Branch-specific results
- Convergence assessment

## Protein and Structure Data

### .embl - EMBL Format
**Description:** Rich sequence annotation format
**Typical Data:** Sequences with extensive annotations
**Use Cases:** Sequence databases, genome records
**Python Libraries:**
- `Biopython`: `SeqIO.read('file.embl', 'embl')`
**EDA Approach:**
- Feature annotation completeness
- Sequence length and type
- Reference information
- Cross-reference validation
- Feature overlap analysis

### .genbank / .gb / .gbk - GenBank Format
**Description:** NCBI's sequence annotation format
**Typical Data:** Annotated sequences with features
**Use Cases:** Sequence databases, annotation transfer
**Python Libraries:**
- `Biopython`: `SeqIO.parse('file.gb', 'genbank')`
**EDA Approach:**
- Feature type distribution
- CDS analysis (start codons, stops)
- Translation validation
- Annotation completeness
- Source organism extraction
- Reference and publication info
- Locus tag consistency

### .sff - Standard Flowgram Format
**Description:** 454/Roche sequencing data format
**Typical Data:** Raw pyrosequencing flowgrams
**Use Cases:** Legacy 454 sequencing data
**Python Libraries:**
- `Biopython`: `SeqIO.parse('file.sff', 'sff')`
- Platform-specific tools
**EDA Approach:**
- Read count and length
- Flowgram signal quality
- Key sequence detection
- Adapter trimming validation
- Quality score distribution

### .hdf5 (Genomics Specific)
**Description:** HDF5 for genomics (10X, Hi-C, etc.)
**Typical Data:** High-throughput genomics data
**Use Cases:** 10X Genomics, spatial transcriptomics
**Python Libraries:**
- `h5py`: Low-level access
- `scanpy`: For 10X data
- `cooler`: For Hi-C data
**EDA Approach:**
- Dataset structure exploration
- Barcode statistics
- UMI counting
- Feature-barcode matrix analysis
- Spatial coordinates (if applicable)

### .cool / .mcool - Cooler Format
**Description:** HDF5-based Hi-C contact matrices
**Typical Data:** Chromatin interaction matrices
**Use Cases:** 3D genome analysis, Hi-C data
**Python Libraries:**
- `cooler`: `cooler.Cooler('file.cool')`
- `hicstraw`: For .hic format
**EDA Approach:**
- Resolution analysis
- Contact matrix statistics
- Distance decay curves
- Compartment analysis
- TAD boundary detection
- Balance factor validation

### .hic - Hi-C Binary Format
**Description:** Juicer binary Hi-C format
**Typical Data:** Multi-resolution Hi-C matrices
**Use Cases:** Hi-C analysis with Juicer tools
**Python Libraries:**
- `hicstraw`: `hicstraw.HiCFile('file.hic')`
- `straw`: C++ library with Python bindings
**EDA Approach:**
- Available resolutions
- Normalization methods
- Contact statistics
- Chromosomal interactions
- Quality metrics

### .bw (ChIP-seq / ATAC-seq specific)
**Description:** BigWig files for epigenomics
**Typical Data:** Coverage or enrichment signals
**Use Cases:** ChIP-seq, ATAC-seq, DNase-seq
**Python Libraries:**
- `pyBigWig`: Standard bigWig access
**EDA Approach:**
- Peak enrichment patterns
- Background signal analysis
- Sample correlation
- Signal-to-noise ratio
- Library complexity metrics

### .narrowPeak / .broadPeak - ENCODE Peak Formats
**Description:** BED-based formats for peaks
**Typical Data:** Peak calls with scores and p-values
**Use Cases:** ChIP-seq peak calling output
**Python Libraries:**
- `pybedtools`: BED-compatible
- Custom parsers for peak-specific fields
**EDA Approach:**
- Peak count and width distribution
- Signal value distribution
- Q-value and p-value analysis
- Peak summit analysis
- Overlap with known features
- Motif enrichment preparation

### .wig - Wiggle Format
**Description:** Dense continuous genomic data
**Typical Data:** Coverage or signal tracks
**Use Cases:** Genome browser visualization
**Python Libraries:**
- `pyBigWig`: Can convert to bigWig
- Custom parsers for wiggle format
**EDA Approach:**
- Signal statistics
- Coverage metrics
- Format variant (fixedStep vs variableStep)
- Span parameter analysis
- Conversion efficiency to bigWig

### .ab1 - Sanger Sequencing Trace
**Description:** Binary chromatogram format
**Typical Data:** Sanger sequencing traces
**Use Cases:** Capillary sequencing validation
**Python Libraries:**
- `Biopython`: `SeqIO.read('file.ab1', 'abi')`
- `tracy` tools: For quality assessment
**EDA Approach:**
- Base calling quality
- Trace quality scores
- Mixed base detection
- Primer and vector detection
- Read length and quality region
- Heterozygosity detection

### .scf - Standard Chromatogram Format
**Description:** Sanger sequencing chromatogram
**Typical Data:** Base calls and confidence values
**Use Cases:** Sequencing trace analysis
**Python Libraries:**
- `Biopython`: SCF format support
**EDA Approach:**
- Similar to AB1 format
- Quality score profiles
- Peak height ratios
- Signal-to-noise metrics

### .idx - Index Files (Generic)
**Description:** Index files for various formats
**Typical Data:** Fast random access indices
**Use Cases:** Efficient data access (BAM, VCF, etc.)
**Python Libraries:**
- Format-specific libraries handle indices
- `pysam`: Auto-handles BAI, CSI indices
**EDA Approach:**
- Index completeness validation
- Binning strategy analysis
- Access performance metrics
- Index size vs data size ratio