Commit Graph

305 Commits

Author SHA1 Message Date
Jeremy Leipzig
70a34bd652 Remove Custom Field Selection and Population-Specific Exports sections
- Remove detailed custom TSV field configuration section
- Remove population-based export workflow section
- Simplify export guide to focus on core functionality
2026-02-24 11:11:53 -07:00
Jeremy Leipzig
b4b8572244 Fix CLI subcommands to match actual TileDB-VCF interface
- Replace incorrect subcommands (create-dataset, ingest, list-datasets)
- Use correct subcommands: create, store, export, list, stat, utils, version
- Update examples to match actual CLI usage patterns
- Add comprehensive list of all available subcommands with descriptions
2026-02-24 11:08:13 -07:00
Jeremy Leipzig
3f76537f75 Add critical VCF ingestion requirements
- VCFs must be single-sample (multi-sample not supported)
- Index files (.csi or .tbi) are required for all VCF/BCF files
- Add indexing examples with bcftools and tabix
- Document requirements prominently in both main skill and ingestion guide
2026-02-24 11:07:20 -07:00
Jeremy Leipzig
07e8e0e284 Fix TileDB-Cloud VCF query API syntax
- Correct method: tiledb.cloud.vcf.read() not query_variants()
- Fix parameter: attrs not attributes
- Add namespace parameter for billing account
- Add .to_pandas() conversion step
- Use realistic example with TileDB-Inc dataset URI
2026-02-24 11:00:51 -07:00
Jeremy Leipzig
3feaa90860 Reorganize TileDB-VCF skill structure and update examples
- Remove Java references (focus on Python and CLI)
- Move all TileDB-Cloud content to bottom of document
- Update export example to show VCF format with .export() method
- Simplify 'When to Use' section focusing on open source capabilities
- Better document organization with cloud scaling at the end
2026-02-24 10:59:39 -07:00
Jeremy Leipzig
6fcc786915 Update TileDB-VCF installation with preferred conda/mamba method
- Add preferred conda environment setup with Python <3.10
- Include M1 Mac specific configuration (CONDA_SUBDIR=osx-64)
- Install tiledbvcf-py via mamba from tiledb channel
- Restore normal Python examples (not Docker-only)
- Keep Docker as alternative installation method
2026-02-24 10:21:14 -07:00
Jeremy Leipzig
18ecbc3b30 Fix TileDB-VCF installation instructions
- Correct installation method: Docker images, not pip packages
- Update examples to show Docker container usage
- Based on actual TileDB-VCF repository documentation
2026-02-24 10:02:34 -07:00
Jeremy Leipzig
3c98f0cada Add TileDB-VCF skill for genomic variant analysis
- Add comprehensive TileDB-VCF skill by Jeremy Leipzig
- Covers open source TileDB-VCF for learning and moderate-scale work
- Emphasizes TileDB-Cloud for large-scale production genomics (1000+ samples)
- Includes detailed reference documentation:
  * ingestion.md - Dataset creation and VCF ingestion
  * querying.md - Efficient variant queries
  * export.md - Data export and format conversion
  * population_genomics.md - GWAS and population analysis workflows
- Features accurate TileDB-Cloud API patterns from official repository
- Highlights scale transition: open source → TileDB-Cloud for enterprise
2026-02-24 09:31:48 -07:00
Timothy Kassis
9bc98cabe8 Merge pull request #58 from K-Dense-AI/fix-yaml-frontmatter
Fix allowed-tools YAML frontmatter format across all skills
2026-02-23 13:45:04 -08:00
Timothy Kassis
a33b572e44 Support Alpha Advantage for more financial data v2.23.0 2026-02-23 13:43:11 -08:00
Timothy Kassis
ea9e0b60e7 Fix version number v2.22.0 2026-02-23 13:33:58 -08:00
Timothy Kassis
5490490294 Support for Hedge Fund Monitor from the Office of Financial Research 2026-02-23 13:32:09 -08:00
Timothy Kassis
86b5d1d30b Add support for FiscalData.treasury.gov 2026-02-23 13:20:34 -08:00
Timothy Kassis
0ffa12a0e2 Support EdgarTools to access and analyze SEC Edgar filings, XBRL financial statements, 10-K, 10-Q, and 8-K reports v2.21.0 2026-02-23 13:02:15 -08:00
Vinayak Agarwal
f6f3023d3d Update allowed-tools formatting in SKILL.md files across multiple scientific skills to improve consistency and readability. 2026-02-23 12:36:05 -08:00
Timothy Kassis
f8da4bf9a7 Forecasting examples 2026-02-23 10:50:24 -08:00
Timothy Kassis
8bbf1fc840 Merge pull request #53 from borealBytes/feat/timesfm-forecasting-skill
feat(ml): add timesfm-forecasting skill for local time series forecasting
2026-02-23 09:43:01 -08:00
Timothy Kassis
6df504f03c Merge pull request #57 from renato-umeton/claude/implement-issue-56-tdd-SOjek
Add open-notebook skill with comprehensive API documentation
2026-02-23 09:42:19 -08:00
Clayton Young
df58339850 feat(timesfm): complete all three examples with quality docs
- anomaly-detection: full two-phase rewrite (context Z-score + forecast PI),
  2-panel viz, Sep 2023 correctly flagged CRITICAL (z=+3.03)
- covariates-forecasting: v3 rewrite with variable-shadowing bug fixed,
  2x2 shared-axis viz showing actionable covariate decomposition,
  108-row CSV with distinct per-store price arrays
- global-temperature: output/ subfolder reorganization (all 6 output files
  moved, 5 scripts + shell script paths updated)
- SKILL.md: added Examples table, Quality Checklist, Common Mistakes (8 items),
  Validation & Verification with regression assertions
- .gitattributes already at repo root covering all binary types
2026-02-23 07:43:04 -05:00
Clayton Young
509190118f fix(examples): correct quantile indices, variable shadowing, and test design in anomaly + covariates examples
Anomaly detection fixes:
- Fix critical quantile index bug: index 0 is mean not q10; correct indices are q10=1, q20=2, q80=8, q90=9
- Redesign test: use all 36 months as context, inject 3 synthetic anomalies into future
- Result: 3 CRITICAL detected (was 11/12 — caused by test-set leakage + wrong indices)
- Update severity labels: CRITICAL = outside 80% PI, WARNING = outside 60% PI

Covariates fixes:
- Fix variable-shadowing bug: inner dict comprehension overwrote outer loop store_id
  causing all stores to get identical covariate arrays (store_A's price for everyone)
- Give each store a distinct price baseline (premium $12, standard $10, discount $7.50)
- Trim CONTEXT_LEN from 48 → 24 weeks; CSV now 108 rows (was 180)
- Add NOTE ON REAL DATA comment: temp file pattern for large external datasets

Both scripts regenerated with clean outputs.
2026-02-23 07:43:04 -05:00
Clayton Young
0d98fa353c feat(examples): add anomaly detection and covariates examples
Anomaly Detection Example:
- Uses quantile forecasts as prediction intervals
- Flags values outside 80%/90% CI as warnings/critical anomalies
- Includes visualization with deviation plot

Covariates (XReg) Example:
- Demonstrates forecast_with_covariates() API
- Shows dynamic numerical/categorical covariates
- Shows static categorical covariates
- Includes synthetic retail sales data with price, promotion, holiday

SKILL.md Updates:
- Added anomaly detection section with code example
- Expanded covariates section with covariate types table
- Added XReg modes explanation
- Updated 'When not to use' section to note anomaly detection workaround
2026-02-23 07:43:04 -05:00
Clayton Young
1a65439ebf fix(html): embed animation data for CORS-safe local file access
- Created generate_html.py to embed JSON data directly in HTML
- No external fetch() needed - works when opened directly in browser
- File size: 149.5 KB (self-contained)
- Shows forecast horizon (12-36 months) in stats
2026-02-23 07:43:04 -05:00
Clayton Young
96372cee99 feat(animation): extend forecasts to final date with dynamic horizon
- Each forecast now extends to 2025-12 regardless of historical data length
- Step 1 (12 points): forecasts 36 months ahead to 2025-12
- Step 25 (36 points): forecasts 12 months ahead to 2025-12
- GIF shows full forecast horizon at every animation step
2026-02-23 07:43:04 -05:00
Clayton Young
7b7110eebb fix(animation): use fixed axes showing full observed data in background
- X-axis fixed to 2022-01 to 2025-12 (full data range)
- Y-axis fixed to 0.72°C to 1.52°C (full value range)
- Background shows all observed data (faded gray) + final forecast reference (faded red dashed)
- Foreground shows current step data (bright blue) + current forecast (bright red)
- GIF size reduced from 918KB to 659KB
2026-02-23 07:43:04 -05:00
Clayton Young
1506a60993 feat(example): add interactive forecast animation with slider
Create an all-out demonstration showing how TimesFM forecasts evolve
as more historical data is added:

- generate_animation_data.py: Runs 25 incremental forecasts (12→36 points)
- interactive_forecast.html: Single-file HTML with Chart.js slider
  - Play/Pause animation control
  - Shows historical data, forecast, 80%/90% CIs, and actual future data
  - Live stats: forecast mean, max, min, CI width
- generate_gif.py: Creates animated GIF for embedding in markdown
- forecast_animation.gif: 25-frame animation (896 KB)

Interactive features:
- Slider to manually step through forecast evolution
- Auto-play with 500ms per frame
- Shows how each additional data point changes the forecast
- Confidence intervals narrow as more data is added
2026-02-23 07:43:04 -05:00
Clayton Young
910bcfdc8b fix(example): update visualization title to clarify demo purpose
- Change title from 'Above 1951-1980 Baseline' to clearer example description
- New title: 'TimesFM Zero-Shot Forecast Example / 36-month Temperature Anomaly → 12-month Forecast'
- Makes it clear this is a demonstration with limited input data
2026-02-23 07:43:04 -05:00
Clayton Young
dcde063723 chore: remove markdown-mermaid-writing skill from this branch
This branch was originally created from feat/markdown-mermaid-writing-skill
for development purposes, but the timesfm-forecasting skill should be
independent of PR #50.

- Remove scientific-skills/markdown-mermaid-writing/ directory
- Remove reference to markdown-mermaid-writing from SKILL.md integration section
- This PR now stands alone and does not require PR #50 to be merged first
2026-02-23 07:43:04 -05:00
Clayton Young
88300014e2 docs(skill): add note that model weights are not stored in repo
Model weights (~800 MB) download on-demand from HuggingFace when skill
is first used. Preflight checker ensures sufficient resources before
any download begins.
2026-02-23 07:43:04 -05:00
Clayton Young
c7c5bc21ff feat(example): add working TimesFM forecast example with global temperature data
- Add NOAA GISTEMP global temperature anomaly dataset (36 months, 2022-2024)
- Run TimesFM 1.0 PyTorch forecast for 2025 (12-month horizon)
- Generate fan chart visualization with 80%/90% confidence intervals
- Create comprehensive markdown report with findings and API notes

API Discovery:
- TimesFM 2.5 PyTorch checkpoint has file format issue (model.safetensors
  vs expected torch_model.ckpt)
- Working API uses TimesFmHparams + TimesFmCheckpoint + TimesFm() constructor
- Documented API in GitHub README differs from actual pip package

Includes:
- temperature_anomaly.csv (input data)
- forecast_output.csv (point forecast + quantiles)
- forecast_output.json (machine-readable output)
- forecast_visualization.png (LFS-tracked)
- run_forecast.py (reusable script)
- visualize_forecast.py (chart generation)
- run_example.sh (one-click runner)
- README.md (full report with findings)
2026-02-23 07:43:04 -05:00
Clayton Young
98670bcf47 feat(skill): add timesfm-forecasting skill for time series forecasting
Add comprehensive TimesFM forecasting skill with mandatory system
preflight checks (RAM/GPU/disk), end-to-end CSV forecasting script,
full API reference, data preparation guide, and hardware requirements
documentation. Supports TimesFM 2.5 (200M), 2.0 (500M), and legacy
v1.0 with automatic batch size recommendations based on hardware.
2026-02-23 07:43:04 -05:00
Clayton Young
a0f81aeaa3 chore: remove docs/project files from PR per review feedback
Per borealBytes review comment, removing the docs/project directory
from this PR since only the skill content should be included.

The docs/project content remains in my local fork for reference.

Refs: PR #50
2026-02-23 07:43:04 -05:00
Clayton Young
79e03ea0f6 docs(skill): add common pitfalls section with radar-beta syntax guide
Added '## ⚠️ Common pitfalls' section covering:
- Radar chart syntax (radar-beta vs radar, axis vs x-axis, curve syntax)
- XY Chart vs Radar syntax comparison table
- Accessibility notes for diagrams that don't support accTitle/accDescr

Prevents the x-axis → radar-beta confusion that occurred in the example
research report.
2026-02-23 07:43:04 -05:00
Clayton Young
21bbff2c4e fix(example): correct radar chart syntax from x-axis to radar-beta
Changed from invalid 'radar' with 'x-axis' syntax to proper 'radar-beta'
syntax with axis/curve keywords as per references/diagrams/radar.md.

Also removed accTitle/accDescr (radar-beta doesn't support them) and
added italic description above the code block per accessibility requirements.
2026-02-23 07:43:04 -05:00
Clayton Young
313ba28adf fix(attribution): standardize Boreal Bytes → borealBytes (GitHub username)
All instances of 'Boreal Bytes' updated to 'borealBytes' (as @borealBytes
in narrative context) across issue, PR, kanban, and SKILL.md.

Files: issue-00000050, pr-00000050, SKILL.md
2026-02-23 07:43:04 -05:00
Clayton Young
672a49bb6a docs(pr): add author LinkedIn and email to PR-00000050 2026-02-23 07:43:04 -05:00
Clayton Young
2198b84be2 docs(kanban): update board to reflect PR #50 open and in review
PR https://github.com/K-Dense-AI/claude-scientific-skills/pull/50 is live.
Moved 'Push branch + open PR' task to Done column. Status updated to
In Review. Pie chart updated to match.
2026-02-23 07:43:04 -05:00
Clayton Young
f8d0f97660 docs(project): renumber to #50, add kanban board, link all tracking files
GitHub issue/PR counter on K-Dense-AI/claude-scientific-skills is at 49.
Renumbered all tracking docs to #50 (next available):

- issue-00000050-markdown-mermaid-skill.md (renamed from 00000001)
- pr-00000050-markdown-mermaid-skill.md (renamed from 00000001)
- kanban/feat-00000050-markdown-mermaid-skill.md (new)

Cross-references updated throughout. PR record now links to kanban board.
Upstream PR URL set to https://github.com/K-Dense-AI/claude-scientific-skills/pull/50
2026-02-23 07:43:04 -05:00
Clayton Young
54a592d7f1 fix(mermaid): replace \n with <br/> in all node labels
Mermaid renders literal \n as text on GitHub — line breaks inside
node labels require <br/> syntax. Fixed 12 occurrences across 4 files:

- SKILL.md: three-phase workflow (Phase 1/2/3 nodes)
- issue-00000001: three-phase workflow nodes
- pr-00000001: skill name node
- example-research-report.md: Stage 1-5 nodes in experimental workflow
2026-02-23 07:43:04 -05:00
Clayton Young
ea5a287cf9 fix(attribution): correct source repo URL to SuperiorByteWorks-LLC/agent-project
All 40 references to borealBytes/opencode updated to the correct source:
https://github.com/SuperiorByteWorks-LLC/agent-project

Affected files: SKILL.md, all 24 diagram guides, 9 templates, issue and PR
docs, plus assets/examples/example-research-report.md (new file).

The example report demonstrates full skill usage: flowchart, sequence,
timeline, xychart, radar diagrams — all with accTitle/accDescr and
classDef colors, no %%{init}. Covers HEK293T CRISPR editing efficiency
as a realistic scientific context.
2026-02-23 07:43:04 -05:00
Clayton Young
97d7901870 feat(skill): add markdown-mermaid-writing skill with source format philosophy
New skill establishing markdown + Mermaid diagrams as the default and
canonical documentation format for all scientific skill outputs.

Core principle (from K-Dense Discord, 2026-02-19): Mermaid in markdown
is the source of truth — text-based, version-controlled, token-efficient,
universally renderable. Python/AI images are downstream conversions only.

SKILL.md includes:
- Full 'source format' philosophy with three-phase workflow diagram
- 24-entry diagram type selection table with links to each guide
- 9-entry document template index
- Per-skill integration guides (scientific-schematics, scientific-writing,
  literature-review, and any other output-producing skill)
- Quality checklist for finalizing documents from any skill
- Full attribution for ported Apache-2.0 content

Originated from conversation between Clayton Young (Boreal Bytes) and the
K-Dense team regarding documentation standards for shared scientific skills.
2026-02-23 07:43:04 -05:00
Clayton Young
39bb842a21 docs(references): port style guides, 24 diagram guides, and 9 templates from opencode
All content ported from borealBytes/opencode under Apache-2.0 license with
attribution headers prepended to each file.

- references/markdown_style_guide.md (~733 lines): full markdown formatting,
  citation, collapsible sections, emoji, Mermaid integration, and template
  selection guide
- references/mermaid_style_guide.md (~458 lines): full Mermaid standards —
  emoji set, classDef color palette, accessibility (accTitle/accDescr),
  theme neutrality (no %%{init}), and diagram type selection table
- references/diagrams/ (24 files): per-type exemplars, tips, and templates
  for all Mermaid diagram types
- templates/ (9 files): PR, issue, kanban, ADR, presentation, how-to,
  status report, research paper, project docs

Source: https://github.com/borealBytes/opencode
2026-02-23 07:43:04 -05:00
Clayton Young
21f8536cef docs(pr): add PR record for markdown-mermaid writing skill
Skeleton PR document following the Everything is Code convention.
Contains full changes inventory, impact classification, testing steps,
architecture diagram showing integration with existing skills, and
design decision notes.

Files touched: docs/project/pr/pr-00000001-markdown-mermaid-skill.md
2026-02-23 07:43:04 -05:00
Clayton Young
0607ad9cf8 docs(issues): add feature request for markdown-mermaid writing skill
Documents the feature request to add a skill establishing markdown+Mermaid
as the canonical source format for scientific documentation. Includes the
originating K-Dense Discord conversation, three-phase workflow diagram
(Mermaid → Python → AI images), acceptance criteria, and technical spec
for the skill directory structure.

Files touched: docs/project/issues/issue-00000001-markdown-mermaid-skill.md
2026-02-23 07:43:04 -05:00
Claude
259e01f7fd Add open-notebook skill: self-hosted NotebookLM alternative (issue #56)
Implements the open-notebook skill as a comprehensive integration for the
open-source, self-hosted alternative to Google NotebookLM. Addresses the
gap created by Google not providing a public NotebookLM API.

Developed using TDD with 44 tests covering skill structure, SKILL.md
frontmatter/content, reference documentation, example scripts, API
endpoint coverage, and marketplace.json registration.

Includes:
- SKILL.md with full documentation, code examples, and provider matrix
- references/api_reference.md covering all 20+ REST API endpoint groups
- references/examples.md with complete research workflow examples
- references/configuration.md with Docker, env vars, and security setup
- references/architecture.md with system design and data flow diagrams
- scripts/ with 3 example scripts (notebook, source, chat) + test suite
- marketplace.json updated to register the new skill

Closes #56

https://claude.ai/code/session_015CqcNWNYmDF9sqxKxziXcz
2026-02-23 00:18:19 +00:00
Timothy Kassis
f7585b7624 Merge pull request #50 from borealBytes/feat/markdown-mermaid-writing-skill
feat(scientific-communication): add markdown-mermaid-writing skill
2026-02-21 20:49:19 -08:00
borealBytes
1c8470a7c5 chore: remove docs/project files from PR per review feedback
Per borealBytes review comment, removing the docs/project directory
from this PR since only the skill content should be included.

The docs/project content remains in my local fork for reference.

Refs: PR #50
2026-02-20 19:27:17 -05:00
borealBytes
b955648f14 docs(skill): add common pitfalls section with radar-beta syntax guide
Added '## ⚠️ Common pitfalls' section covering:
- Radar chart syntax (radar-beta vs radar, axis vs x-axis, curve syntax)
- XY Chart vs Radar syntax comparison table
- Accessibility notes for diagrams that don't support accTitle/accDescr

Prevents the x-axis → radar-beta confusion that occurred in the example
research report.
2026-02-19 22:09:55 -05:00
borealBytes
dc250634e4 fix(example): correct radar chart syntax from x-axis to radar-beta
Changed from invalid 'radar' with 'x-axis' syntax to proper 'radar-beta'
syntax with axis/curve keywords as per references/diagrams/radar.md.

Also removed accTitle/accDescr (radar-beta doesn't support them) and
added italic description above the code block per accessibility requirements.
2026-02-19 22:03:33 -05:00
borealBytes
1f59444cec fix(attribution): standardize Boreal Bytes → borealBytes (GitHub username)
All instances of 'Boreal Bytes' updated to 'borealBytes' (as @borealBytes
in narrative context) across issue, PR, kanban, and SKILL.md.

Files: issue-00000050, pr-00000050, SKILL.md
2026-02-19 18:48:23 -05:00
borealBytes
99f23be117 docs(pr): add author LinkedIn and email to PR-00000050 2026-02-19 18:47:50 -05:00