Testing Checklist

Every provider contribution should include tests that prove the provider's own API shape. Do not only copy Anthropic-shaped fixtures.

Definition Tests

• ProviderDefinitionSchema validates the new definition.
• PROVIDER_DEFINITIONS contains the provider.
• vendorKey, adapterKey, protocol, auth metadata, default endpoint, and default model are correct.
• Local providers mark auth.required: false; remote providers declare their env var and vault namespace.
• Built-in provider leaves do not hand-author wellKnownProviderId; generated catalogs derive stable ids from vendorKey.
• CLI-style providers declare the correct executionCapabilityProfile on the provider definition.
• API-key providers declare auth.header with the correct header name and raw or bearer scheme.
• Providers that support dynamic model discovery declare modelListEndpoint, modelListFormat, and capabilities.modelListing.
• modelListFormat matches the response envelope shape the endpoint actually returns; openai-models and anthropic-models are shape names, not vendor-only labels.

Codegen Tests

• Generated catalogs are fresh.
• Public exports include the certified provider leaf.
• The provider leaf has the required generator files.

Adapter Tests

• formatRequest(...) maps system prompt, context, tool definitions, and model requirements correctly.
• parseResponse(...) handles text output, tool calls, malformed output, and provider-specific response shapes.
• parseResponse(...) returns a fallback instead of throwing.
• Streaming and tool-use behavior are covered when the adapter declares those capabilities.
• CLI-style adapters cover request/response behavior appropriate to the execution profile declared by the provider definition.

Provider Implementation Tests

• Incoming input is validated through TextModelInputSchema or the correct provider-specific schema.
• Base endpoint handling avoids double slashes and full-path misconfiguration.
• Credentials are sent through the provider's required auth shape.
• 401, 403, 429, unavailable, timeout, and abort behavior map to the expected Nous errors.
• Stream events are parsed and released correctly when streaming is supported.

Runtime Tests

• resolveAdapterKeyFromConfig(...) resolves through provider definitions and falls back safely.
• ProviderRegistry can construct the provider when credentials are present.
• Missing credentials skip or fail according to current runtime behavior.
• Dynamic model discovery is covered for each declared modelListFormat, including successful parse, HTTP failure, parse failure, and network failure fallback behavior.
• Key-test behavior uses healthCheckEndpoint when present and falls back to modelListEndpoint otherwise.
• Remote discovery cache behavior and local-provider cache bypass behavior are covered when the provider participates in discovery.
• Tests prove fetched model IDs are returned as reported by the provider; do not add model-family or prefix filtering expectations.
• Agent-CLI providers cover runtime CLI session manager behavior when chat-bound sessions are supported: fixture-aware session keys, wrapper reuse, scoped teardown, restart failure surfacing, and one-shot preservation for transient/batch invocations.
• Role assignment tests prove incompatible CLI providers are disabled/rejected for Cortex persistent-chat roles and remain selectable for compatible agent roles.
• Provider registry tests prove both derived built-in ids and legacy persisted ids resolve to the same built-in provider definition where compatibility aliases are required.

Suggested Commands

pnpm --filter @nous/subcortex-providers run check:generated
pnpm --filter @nous/subcortex-providers run typecheck
pnpm --filter @nous/subcortex-providers exec vitest run src/__tests__/provider-codegen.test.ts src/__tests__/public-exports.test.ts src/__tests__/provider-definitions src/__tests__/adapter-resolver.test.ts src/__tests__/provider-pipeline-integration.test.ts --config vitest.config.ts

For docs-only work:

pnpm --filter docs lint
pnpm --filter docs build