Commit Graph

322 Commits

Author SHA1 Message Date
Timothy Kassis
29c869326e Enhance README by adding a description of K-Dense Web, clarifying its role as a hosted platform built on the scientific skills collection. This addition aims to promote the platform's features and benefits. 2026-02-28 07:23:48 -08:00
Timothy Kassis
a472690126 Revise README to streamline content and promote K-Dense Web, highlighting its features and benefits. Remove outdated sections and enhance calls to action for cloud-based execution and publication-ready outputs. 2026-02-27 13:42:12 -08:00
Timothy Kassis
c58a309012 Update README to reflect the increase in scientific skills from 148 to 250+ databases, and enhance descriptions for Python package and integration skills. Adjust badge and feature tables accordingly. 2026-02-27 13:36:31 -08:00
Timothy Kassis
6040d02c8c Update README and documentation to reflect the addition of the pyzotero skill and increment skill count from 147 to 148. Bump version to 2.24.0 in marketplace.json. v2.24.0 2026-02-27 09:38:51 -08:00
Timothy Kassis
8dc5701063 Use Nano Banana 2 2026-02-26 08:59:40 -08:00
Timothy Kassis
d4177ce3a5 Merge pull request #61 from connerlambden/add-bgpt-skill
Add BGPT paper search skill
2026-02-24 14:23:19 -08:00
connerlambden
f54b1bb174 Add BGPT paper search skill 2026-02-24 14:42:31 -07:00
Timothy Kassis
29ae12d2c0 Merge pull request #62 from leipzig/feature/add-tiledb-vcf-skill
Feature/add tiledb-vcf skill
2026-02-24 12:43:57 -08:00
Timothy Kassis
77883baba2 Merge pull request #60 from renato-umeton/main
Fix architecture diagram
2026-02-24 12:41:32 -08:00
Jeremy Leipzig
791fd2361c Update skill counts and add TileDB-VCF to repository documentation
- Update skill count badges and descriptions from 146 to 147 skills
- Add TileDB-VCF to genomic tools list in bioinformatics section
- Add variant database management use case for TileDB-VCF
- Add comprehensive TileDB-VCF entry to docs/scientific-skills.md
2026-02-24 12:07:21 -07:00
Jeremy Leipzig
730531e0d7 Remove all reference documentation files and clean up references
- Delete references/population_genomics.md
- Remove all references to deleted documentation files
- Clean up References section since no reference files remain
- Simplify skill to standalone main file only
2026-02-24 11:30:31 -07:00
Jeremy Leipzig
55811bdbbe Remove references/querying.md
- Delete detailed querying reference documentation
- Update main skill to remove references to querying.md
- Leave only population_genomics.md reference file
2026-02-24 11:29:01 -07:00
Jeremy Leipzig
c576d2e66a Remove references/export.md and references/ingestion.md
- Delete detailed export and ingestion reference documentation
- Update main skill to remove references to deleted files
- Simplify skill to focus on core querying and population genomics
- Keep querying.md and population_genomics.md reference files
2026-02-24 11:27:59 -07:00
Jeremy Leipzig
ba2afda31c Fix documentation URLs to point to correct TileDB Academy
- All documentation is at https://cloud.tiledb.com/academy/
- Remove incorrect service URLs (docs.tiledb.com, support portal, etc.)
- Consolidate to academy and main platform URLs only
- Update contact information to sales@tiledb.com
2026-02-24 11:22:34 -07:00
Jeremy Leipzig
e3a7a85122 Remove multiple advanced export sections
- Remove VEP annotation preparation section
- Remove Cloud Export (S3) section
- Remove Export Validation section
- Remove Efficient Export Strategies section
- Simplify export guide to focus on core export functionality
- Maintain essential VCF/BCF and TSV export examples
2026-02-24 11:17:41 -07:00
Jeremy Leipzig
518261c4f2 Remove Streaming Export for Large Datasets section
- Remove chunked export functionality
- Remove parallel export functionality
- Simplify export guide to focus on basic export operations
2026-02-24 11:13:01 -07:00
Jeremy Leipzig
70a34bd652 Remove Custom Field Selection and Population-Specific Exports sections
- Remove detailed custom TSV field configuration section
- Remove population-based export workflow section
- Simplify export guide to focus on core functionality
2026-02-24 11:11:53 -07:00
Jeremy Leipzig
b4b8572244 Fix CLI subcommands to match actual TileDB-VCF interface
- Replace incorrect subcommands (create-dataset, ingest, list-datasets)
- Use correct subcommands: create, store, export, list, stat, utils, version
- Update examples to match actual CLI usage patterns
- Add comprehensive list of all available subcommands with descriptions
2026-02-24 11:08:13 -07:00
Jeremy Leipzig
3f76537f75 Add critical VCF ingestion requirements
- VCFs must be single-sample (multi-sample not supported)
- Index files (.csi or .tbi) are required for all VCF/BCF files
- Add indexing examples with bcftools and tabix
- Document requirements prominently in both main skill and ingestion guide
2026-02-24 11:07:20 -07:00
Jeremy Leipzig
07e8e0e284 Fix TileDB-Cloud VCF query API syntax
- Correct method: tiledb.cloud.vcf.read() not query_variants()
- Fix parameter: attrs not attributes
- Add namespace parameter for billing account
- Add .to_pandas() conversion step
- Use realistic example with TileDB-Inc dataset URI
2026-02-24 11:00:51 -07:00
Jeremy Leipzig
3feaa90860 Reorganize TileDB-VCF skill structure and update examples
- Remove Java references (focus on Python and CLI)
- Move all TileDB-Cloud content to bottom of document
- Update export example to show VCF format with .export() method
- Simplify 'When to Use' section focusing on open source capabilities
- Better document organization with cloud scaling at the end
2026-02-24 10:59:39 -07:00
Jeremy Leipzig
6fcc786915 Update TileDB-VCF installation with preferred conda/mamba method
- Add preferred conda environment setup with Python <3.10
- Include M1 Mac specific configuration (CONDA_SUBDIR=osx-64)
- Install tiledbvcf-py via mamba from tiledb channel
- Restore normal Python examples (not Docker-only)
- Keep Docker as alternative installation method
2026-02-24 10:21:14 -07:00
Jeremy Leipzig
18ecbc3b30 Fix TileDB-VCF installation instructions
- Correct installation method: Docker images, not pip packages
- Update examples to show Docker container usage
- Based on actual TileDB-VCF repository documentation
2026-02-24 10:02:34 -07:00
Jeremy Leipzig
3c98f0cada Add TileDB-VCF skill for genomic variant analysis
- Add comprehensive TileDB-VCF skill by Jeremy Leipzig
- Covers open source TileDB-VCF for learning and moderate-scale work
- Emphasizes TileDB-Cloud for large-scale production genomics (1000+ samples)
- Includes detailed reference documentation:
  * ingestion.md - Dataset creation and VCF ingestion
  * querying.md - Efficient variant queries
  * export.md - Data export and format conversion
  * population_genomics.md - GWAS and population analysis workflows
- Features accurate TileDB-Cloud API patterns from official repository
- Highlights scale transition: open source → TileDB-Cloud for enterprise
2026-02-24 09:31:48 -07:00
renato-umeton
fa3a20ca4d Fix architecture diagram in markdown 2026-02-23 22:52:10 -05:00
Timothy Kassis
9bc98cabe8 Merge pull request #58 from K-Dense-AI/fix-yaml-frontmatter
Fix allowed-tools YAML frontmatter format across all skills
2026-02-23 13:45:04 -08:00
Timothy Kassis
a33b572e44 Support Alpha Advantage for more financial data v2.23.0 2026-02-23 13:43:11 -08:00
Timothy Kassis
ea9e0b60e7 Fix version number v2.22.0 2026-02-23 13:33:58 -08:00
Timothy Kassis
5490490294 Support for Hedge Fund Monitor from the Office of Financial Research 2026-02-23 13:32:09 -08:00
Timothy Kassis
86b5d1d30b Add support for FiscalData.treasury.gov 2026-02-23 13:20:34 -08:00
Timothy Kassis
0ffa12a0e2 Support EdgarTools to access and analyze SEC Edgar filings, XBRL financial statements, 10-K, 10-Q, and 8-K reports v2.21.0 2026-02-23 13:02:15 -08:00
Vinayak Agarwal
f6f3023d3d Update allowed-tools formatting in SKILL.md files across multiple scientific skills to improve consistency and readability. 2026-02-23 12:36:05 -08:00
Timothy Kassis
f8da4bf9a7 Forecasting examples 2026-02-23 10:50:24 -08:00
Timothy Kassis
8bbf1fc840 Merge pull request #53 from borealBytes/feat/timesfm-forecasting-skill
feat(ml): add timesfm-forecasting skill for local time series forecasting
2026-02-23 09:43:01 -08:00
Timothy Kassis
6df504f03c Merge pull request #57 from renato-umeton/claude/implement-issue-56-tdd-SOjek
Add open-notebook skill with comprehensive API documentation
2026-02-23 09:42:19 -08:00
Clayton Young
df58339850 feat(timesfm): complete all three examples with quality docs
- anomaly-detection: full two-phase rewrite (context Z-score + forecast PI),
  2-panel viz, Sep 2023 correctly flagged CRITICAL (z=+3.03)
- covariates-forecasting: v3 rewrite with variable-shadowing bug fixed,
  2x2 shared-axis viz showing actionable covariate decomposition,
  108-row CSV with distinct per-store price arrays
- global-temperature: output/ subfolder reorganization (all 6 output files
  moved, 5 scripts + shell script paths updated)
- SKILL.md: added Examples table, Quality Checklist, Common Mistakes (8 items),
  Validation & Verification with regression assertions
- .gitattributes already at repo root covering all binary types
2026-02-23 07:43:04 -05:00
Clayton Young
509190118f fix(examples): correct quantile indices, variable shadowing, and test design in anomaly + covariates examples
Anomaly detection fixes:
- Fix critical quantile index bug: index 0 is mean not q10; correct indices are q10=1, q20=2, q80=8, q90=9
- Redesign test: use all 36 months as context, inject 3 synthetic anomalies into future
- Result: 3 CRITICAL detected (was 11/12 — caused by test-set leakage + wrong indices)
- Update severity labels: CRITICAL = outside 80% PI, WARNING = outside 60% PI

Covariates fixes:
- Fix variable-shadowing bug: inner dict comprehension overwrote outer loop store_id
  causing all stores to get identical covariate arrays (store_A's price for everyone)
- Give each store a distinct price baseline (premium $12, standard $10, discount $7.50)
- Trim CONTEXT_LEN from 48 → 24 weeks; CSV now 108 rows (was 180)
- Add NOTE ON REAL DATA comment: temp file pattern for large external datasets

Both scripts regenerated with clean outputs.
2026-02-23 07:43:04 -05:00
Clayton Young
0d98fa353c feat(examples): add anomaly detection and covariates examples
Anomaly Detection Example:
- Uses quantile forecasts as prediction intervals
- Flags values outside 80%/90% CI as warnings/critical anomalies
- Includes visualization with deviation plot

Covariates (XReg) Example:
- Demonstrates forecast_with_covariates() API
- Shows dynamic numerical/categorical covariates
- Shows static categorical covariates
- Includes synthetic retail sales data with price, promotion, holiday

SKILL.md Updates:
- Added anomaly detection section with code example
- Expanded covariates section with covariate types table
- Added XReg modes explanation
- Updated 'When not to use' section to note anomaly detection workaround
2026-02-23 07:43:04 -05:00
Clayton Young
1a65439ebf fix(html): embed animation data for CORS-safe local file access
- Created generate_html.py to embed JSON data directly in HTML
- No external fetch() needed - works when opened directly in browser
- File size: 149.5 KB (self-contained)
- Shows forecast horizon (12-36 months) in stats
2026-02-23 07:43:04 -05:00
Clayton Young
96372cee99 feat(animation): extend forecasts to final date with dynamic horizon
- Each forecast now extends to 2025-12 regardless of historical data length
- Step 1 (12 points): forecasts 36 months ahead to 2025-12
- Step 25 (36 points): forecasts 12 months ahead to 2025-12
- GIF shows full forecast horizon at every animation step
2026-02-23 07:43:04 -05:00
Clayton Young
7b7110eebb fix(animation): use fixed axes showing full observed data in background
- X-axis fixed to 2022-01 to 2025-12 (full data range)
- Y-axis fixed to 0.72°C to 1.52°C (full value range)
- Background shows all observed data (faded gray) + final forecast reference (faded red dashed)
- Foreground shows current step data (bright blue) + current forecast (bright red)
- GIF size reduced from 918KB to 659KB
2026-02-23 07:43:04 -05:00
Clayton Young
1506a60993 feat(example): add interactive forecast animation with slider
Create an all-out demonstration showing how TimesFM forecasts evolve
as more historical data is added:

- generate_animation_data.py: Runs 25 incremental forecasts (12→36 points)
- interactive_forecast.html: Single-file HTML with Chart.js slider
  - Play/Pause animation control
  - Shows historical data, forecast, 80%/90% CIs, and actual future data
  - Live stats: forecast mean, max, min, CI width
- generate_gif.py: Creates animated GIF for embedding in markdown
- forecast_animation.gif: 25-frame animation (896 KB)

Interactive features:
- Slider to manually step through forecast evolution
- Auto-play with 500ms per frame
- Shows how each additional data point changes the forecast
- Confidence intervals narrow as more data is added
2026-02-23 07:43:04 -05:00
Clayton Young
910bcfdc8b fix(example): update visualization title to clarify demo purpose
- Change title from 'Above 1951-1980 Baseline' to clearer example description
- New title: 'TimesFM Zero-Shot Forecast Example / 36-month Temperature Anomaly → 12-month Forecast'
- Makes it clear this is a demonstration with limited input data
2026-02-23 07:43:04 -05:00
Clayton Young
dcde063723 chore: remove markdown-mermaid-writing skill from this branch
This branch was originally created from feat/markdown-mermaid-writing-skill
for development purposes, but the timesfm-forecasting skill should be
independent of PR #50.

- Remove scientific-skills/markdown-mermaid-writing/ directory
- Remove reference to markdown-mermaid-writing from SKILL.md integration section
- This PR now stands alone and does not require PR #50 to be merged first
2026-02-23 07:43:04 -05:00
Clayton Young
88300014e2 docs(skill): add note that model weights are not stored in repo
Model weights (~800 MB) download on-demand from HuggingFace when skill
is first used. Preflight checker ensures sufficient resources before
any download begins.
2026-02-23 07:43:04 -05:00
Clayton Young
c7c5bc21ff feat(example): add working TimesFM forecast example with global temperature data
- Add NOAA GISTEMP global temperature anomaly dataset (36 months, 2022-2024)
- Run TimesFM 1.0 PyTorch forecast for 2025 (12-month horizon)
- Generate fan chart visualization with 80%/90% confidence intervals
- Create comprehensive markdown report with findings and API notes

API Discovery:
- TimesFM 2.5 PyTorch checkpoint has file format issue (model.safetensors
  vs expected torch_model.ckpt)
- Working API uses TimesFmHparams + TimesFmCheckpoint + TimesFm() constructor
- Documented API in GitHub README differs from actual pip package

Includes:
- temperature_anomaly.csv (input data)
- forecast_output.csv (point forecast + quantiles)
- forecast_output.json (machine-readable output)
- forecast_visualization.png (LFS-tracked)
- run_forecast.py (reusable script)
- visualize_forecast.py (chart generation)
- run_example.sh (one-click runner)
- README.md (full report with findings)
2026-02-23 07:43:04 -05:00
Clayton Young
98670bcf47 feat(skill): add timesfm-forecasting skill for time series forecasting
Add comprehensive TimesFM forecasting skill with mandatory system
preflight checks (RAM/GPU/disk), end-to-end CSV forecasting script,
full API reference, data preparation guide, and hardware requirements
documentation. Supports TimesFM 2.5 (200M), 2.0 (500M), and legacy
v1.0 with automatic batch size recommendations based on hardware.
2026-02-23 07:43:04 -05:00
Clayton Young
a0f81aeaa3 chore: remove docs/project files from PR per review feedback
Per borealBytes review comment, removing the docs/project directory
from this PR since only the skill content should be included.

The docs/project content remains in my local fork for reference.

Refs: PR #50
2026-02-23 07:43:04 -05:00
Clayton Young
79e03ea0f6 docs(skill): add common pitfalls section with radar-beta syntax guide
Added '## ⚠️ Common pitfalls' section covering:
- Radar chart syntax (radar-beta vs radar, axis vs x-axis, curve syntax)
- XY Chart vs Radar syntax comparison table
- Accessibility notes for diagrams that don't support accTitle/accDescr

Prevents the x-axis → radar-beta confusion that occurred in the example
research report.
2026-02-23 07:43:04 -05:00
Clayton Young
21bbff2c4e fix(example): correct radar chart syntax from x-axis to radar-beta
Changed from invalid 'radar' with 'x-axis' syntax to proper 'radar-beta'
syntax with axis/curve keywords as per references/diagrams/radar.md.

Also removed accTitle/accDescr (radar-beta doesn't support them) and
added italic description above the code block per accessibility requirements.
2026-02-23 07:43:04 -05:00