Tokenizer, Model, and Telemetry

The Effect AI SDK ships several small supporting modules that the higher-level LanguageModel builds on. They are useful on their own when you need to count tokens, inspect which provider is active, generate stable identifiers, or attach OpenTelemetry GenAI attributes to your spans.

All of these modules live under effect/unstable/ai:

import {
  IdGenerator,
  Model,
  ResponseIdTracker,
  Telemetry,
  Tokenizer
} from "effect/unstable/ai"

Unstable

These modules are part of the unstable AI surface. Import paths and APIs may change between minor releases.

Tokenizer

A Tokenizer converts a prompt into provider-specific token ids and truncates a prompt to fit within a token budget. It is a service because tokenization depends on the target model, provider, and encoding rules — not on Effect itself. Provider packages usually supply a Tokenizer, but you can build your own with Tokenizer.make.

The common cases are counting tokens before choosing a model or request size, and dropping older conversation messages before a chat completion call.

import { Effect } from "effect"
import { Tokenizer } from "effect/unstable/ai"

// A simple word-based tokenizer for tests / unsupported providers.
// `prompt.content` is a normalized array of messages.
const wordTokenizer = Tokenizer.make({
  tokenize: (prompt) =>
    Effect.succeed(
      prompt.content
        .flatMap((msg) =>
          typeof msg.content === "string"
            ? msg.content.split(" ")
            : msg.content.flatMap((part) =>
                part.type === "text" ? part.text.split(" ") : []
              )
        )
        .map((_, index) => index)
    )
})

const program = Effect.gen(function* () {
  const tokenizer = yield* Tokenizer.Tokenizer

  // Count tokens in a raw string prompt
  const tokens = yield* tokenizer.tokenize("Count the tokens in this prompt")
  yield* Effect.log(`token count: ${tokens.length}`) // => token count: 6

  // Keep the prompt under a 100-token budget
  const trimmed = yield* tokenizer.truncate(
    "A potentially very long conversation...",
    100
  )
  return trimmed
}).pipe(Effect.provideService(Tokenizer.Tokenizer, wordTokenizer))

Note

truncate keeps complete messages from the end of the prompt until the next message would exceed the budget; it never splits a message to fit the remaining space. Token ids are model-specific, so counts from one tokenizer will not match another model’s tokenizer.

Reference

Tokenizer (service)

Context.Service tag (id "effect/ai/Tokenizer") used to access or provide a tokenization service. Programs that read it require a tokenizer to be provided.

import { Effect } from "effect"
import { Tokenizer } from "effect/unstable/ai"

const useTokenizer = Effect.gen(function* () {
  const tokenizer = yield* Tokenizer.Tokenizer
  const tokens = yield* tokenizer.tokenize("Hello, world!")
  return tokens.length // => number
})

Service

The service interface. Has two methods, both accepting a Prompt.RawInput (a string, a message array, or a Prompt):

• tokenize(input) → Effect<Array<number>, AiError> — token ids for the input.
• truncate(input, tokens) → Effect<Prompt, AiError> — a prompt trimmed to at most tokens tokens.

import { Effect } from "effect"
import { Prompt } from "effect/unstable/ai"
import type { Tokenizer } from "effect/unstable/ai"

const custom: Tokenizer.Service = {
  tokenize: (input) =>
    Effect.succeed(input.toString().split(" ").map((_, i) => i)),
  truncate: (input, maxTokens) =>
    Effect.succeed(Prompt.make(input.toString().slice(0, maxTokens * 5)))
}

make

Builds a Service from a single tokenize function that operates on a normalized Prompt. The truncate implementation is derived for you (it repeatedly tokenizes trailing messages and stops when the budget is reached).

import { Effect } from "effect"
import { Tokenizer } from "effect/unstable/ai"

const tokenizer = Tokenizer.make({
  // Pretend each message is one token
  tokenize: (prompt) => Effect.succeed(prompt.content.map((_, i) => i))
})
// tokenizer.tokenize / tokenizer.truncate are now available

Model

A Model wraps a provider-specific language model Layer together with the metadata Effect AI needs at runtime. A Model is a Layer, but providing it also installs two metadata services — Model.ProviderName and Model.ModelName — so any program can inspect which provider/model is in use (useful for telemetry, routing, or provider-agnostic services).

Provider packages (OpenAI, Anthropic, Bedrock, etc.) construct Models for you; Model.make is the building block they use.

import { Effect, Layer } from "effect"
import { LanguageModel, Model } from "effect/unstable/ai"

declare const languageModelLayer: Layer.Layer<LanguageModel.LanguageModel>

// Combine a provider id, a model id, and the layer that provides AI services.
const model = Model.make("openai", "gpt-5", languageModelLayer)

const program = Effect.gen(function* () {
  // ProviderName and ModelName are provided automatically by the Model
  const provider = yield* Model.ProviderName // => "openai"
  const modelName = yield* Model.ModelName // => "gpt-5"

  const response = yield* LanguageModel.generateText({
    prompt: `Hello from ${provider}/${modelName}!`
  })
  return response.text
}).pipe(Effect.provide(model))

Note

The provider and model names are metadata only — the actual implementation still comes from the layer passed to Model.make. If that layer has requirements, they must be satisfied before the model can be provided.

Reference

Model<Provider, Provides, Requires>

An interface extending Layer.Layer<Provides | ProviderName | ModelName, never, Requires>. Beyond the Layer shape it carries:

• provider — the provider identifier (e.g. "openai", "anthropic", "amazon-bedrock").
• captureRequirements — an Effect that captures the current context into a concrete Layer whose requirements are already satisfied. Useful when you need a Model from inside an Effect service and want to “lift” its dependencies into the parent Effect.

import { Effect, Layer } from "effect"
import { LanguageModel, Model } from "effect/unstable/ai"

declare const layer: Layer.Layer<LanguageModel.LanguageModel>
const model = Model.make("anthropic", "claude-3-5-haiku", layer)

model.provider // => "anthropic"

// Lift the Model's requirements into the surrounding Effect
const lifted = Effect.gen(function* () {
  const ready: Layer.Layer<unknown> = yield* model.captureRequirements
  return ready
})

ProviderName (service)

Context.Service<ProviderName, string> (id "effect/unstable/ai/Model/ProviderName"). Automatically provided by a Model; read it to discover the active provider name.

import { Effect } from "effect"
import { Model } from "effect/unstable/ai"

const readProvider = Effect.gen(function* () {
  const provider = yield* Model.ProviderName
  return provider // => the provider string, e.g. "openai"
})

ModelName (service)

Context.Service<ModelName, string> (id "effect/unstable/ai/Model/ModelName"). Automatically provided by a Model; read it to discover the active model name.

import { Effect } from "effect"
import { Model } from "effect/unstable/ai"

const readModel = Effect.gen(function* () {
  const modelName = yield* Model.ModelName
  return modelName // => the model string, e.g. "gpt-5"
})

make

Creates a Model from a provider name, a model name, and a Layer of AI services. The result merges the provided layer with the ProviderName / ModelName metadata services.

import { Layer } from "effect"
import { LanguageModel, Model } from "effect/unstable/ai"

declare const bedrockLayer: Layer.Layer<LanguageModel.LanguageModel>

const model = Model.make("amazon-bedrock", "claude-3-5-haiku", bedrockLayer)
// => Model<"amazon-bedrock", LanguageModel, never>

IdGenerator

IdGenerator is a pluggable service for generating unique identifiers for tool calls, assistant messages, and other generated items. It supports custom alphabets, prefixes, separators, and sizes — useful when you want deterministic IDs in tests or provider-specific ID formats in production.

import { Effect } from "effect"
import { IdGenerator } from "effect/unstable/ai"

// A custom generator for tool-call IDs
const toolCallIdLayer = IdGenerator.layer({
  alphabet: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ",
  prefix: "tool_call",
  separator: "_",
  size: 12
})

const program = Effect.gen(function* () {
  const idGen = yield* IdGenerator.IdGenerator
  const id = yield* idGen.generateId()
  return id // => "tool_call_A7XK9MP2QR5T"
}).pipe(Effect.provide(toolCallIdLayer))

If you do not provide your own, use IdGenerator.defaultIdGenerator, which emits id_XXXXXXXXXXXXXXXX-style identifiers.

Note

The random portion is drawn from the Effect Random service, so seeding it (for example with Effect.withRandom) makes generated IDs deterministic in tests.

Reference

IdGenerator (service)

Context.Service<IdGenerator, Service> (id "@effect/ai/IdGenerator"). Access or provide ID generation throughout your application.

import { Effect } from "effect"
import { IdGenerator } from "effect/unstable/ai"

const useIdGenerator = Effect.gen(function* () {
  const idGenerator = yield* IdGenerator.IdGenerator
  return yield* idGenerator.generateId() // => string
})

Service

The service interface: a single generateId(): Effect<string> method. Any implementation that satisfies this can be provided as the service.

import { Effect } from "effect"
import type { IdGenerator } from "effect/unstable/ai"

let nextId = 0
const customService: IdGenerator.Service = {
  generateId: () => Effect.sync(() => `custom_${++nextId}`)
}
// customService.generateId() => Effect that yields "custom_1", "custom_2", ...

MakeOptions

Configuration for custom generators:

• alphabet — character set for the random portion (required).
• prefix — optional string prepended to the id.
• separator — character between prefix and random portion (must not be part of the alphabet).
• size — length of the random portion.

import type { IdGenerator } from "effect/unstable/ai"

const options: IdGenerator.MakeOptions = {
  alphabet: "0123456789ABCDEF",
  prefix: "tool",
  separator: "_",
  size: 8
}
// Generates IDs like: "tool_A1B2C3D4"

defaultIdGenerator

A ready-made Service using the "id" prefix and a 16-character alphanumeric random portion.

import { Effect } from "effect"
import { IdGenerator } from "effect/unstable/ai"

const program = Effect.gen(function* () {
  const id = yield* IdGenerator.defaultIdGenerator.generateId()
  return id // => "id_A7xK9mP2qR5tY8uV"
})

make

An Effect that builds a custom Service from MakeOptions. It fails with Cause.IllegalArgumentError if the separator appears in the alphabet (which would make IDs ambiguous to parse).

import { Effect } from "effect"
import { IdGenerator } from "effect/unstable/ai"

const program = Effect.gen(function* () {
  const gen = yield* IdGenerator.make({
    alphabet: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ",
    prefix: "msg",
    separator: "-",
    size: 10
  })
  return yield* gen.generateId() // => "msg-A7X9K2M5P8"
})

layer

The recommended way to provide an IdGenerator. Wraps make in a Layer; the layer fails during construction if the options are invalid.

import { Effect } from "effect"
import { IdGenerator } from "effect/unstable/ai"

const idLayer = IdGenerator.layer({
  alphabet: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ",
  prefix: "tool_call",
  separator: "_",
  size: 12
})

const program = Effect.gen(function* () {
  const idGen = yield* IdGenerator.IdGenerator
  return yield* idGen.generateId() // => "tool_call_A7XK9MP2QR5T"
}).pipe(Effect.provide(idLayer))

ResponseIdTracker

Some providers can continue from a prior response by accepting a previousResponseId plus only the messages added since that response. ResponseIdTracker is a small mutable cache that remembers which exact prompt message objects were included in each provider response, then prepares a shorter prompt when a later call extends the same conversation.

It is an optimization cache used by language model implementations — not conversation storage. You typically just provide it and let the provider use it.

import { Effect, Option } from "effect"
import { Prompt, ResponseIdTracker } from "effect/unstable/ai"

const program = Effect.gen(function* () {
  const tracker = yield* ResponseIdTracker.make

  // After a provider responds, associate the messages that were sent
  // with the provider's response id.
  const firstPrompt = Prompt.make("Hello")
  tracker.markParts(firstPrompt.content, "resp_123")

  // On a follow-up turn, prepare a shorter prompt. If the prefix is fully
  // tracked, you get back the previous response id plus only the new messages.
  const followUp = Prompt.fromMessages([
    ...firstPrompt.content,
    { role: "assistant", content: "Hi!" },
    { role: "user", content: "How are you?" }
  ] as any)

  const prepared = tracker.prepareUnsafe(followUp)
  if (Option.isSome(prepared)) {
    prepared.value.previousResponseId // => "resp_123"
    prepared.value.prompt // => prompt with only the messages after the last assistant turn
  }
})

Caution

Tracking is based on object identity — equivalent message values are only recognized if they are the same objects. The service intentionally exposes Unsafe methods because callers coordinate it inside provider request/response code. A prefix mismatch clears all tracked state so a response id is never reused for the wrong prompt.

Reference

ResponseIdTracker (service)

Context.Service<ResponseIdTracker, Service> (id "effect/ai/ResponseIdTracker"). Provide it so a language model can send only new prompt messages alongside the provider’s prior response id.

import { Effect } from "effect"
import { ResponseIdTracker } from "effect/unstable/ai"

const useTracker = Effect.gen(function* () {
  const tracker = yield* ResponseIdTracker.ResponseIdTracker
  return tracker // => ResponseIdTracker.Service
})

Service

The mutable service interface:

• markParts(parts, responseId) — record the message objects that produced a response.
• prepareUnsafe(prompt) → Option<PrepareResult> — returns the previous response id and the untracked suffix when the prefix is fully recognized; otherwise clears state and returns Option.none().
• clearUnsafe() — drop all tracked state.

import { Option } from "effect"
import type { ResponseIdTracker } from "effect/unstable/ai"

declare const tracker: ResponseIdTracker.Service
tracker.clearUnsafe() // => void, forgets everything tracked so far

PrepareResult

The shape returned (inside an Option) by prepareUnsafe:

• previousResponseId: string — pass this to the provider as previousResponseId.
• prompt: Prompt — only the new messages after the latest assistant turn.

import type { ResponseIdTracker } from "effect/unstable/ai"

declare const result: ResponseIdTracker.PrepareResult
result.previousResponseId // => "resp_123"
result.prompt // => Prompt containing only the new messages

make

An Effect<Service> that constructs a fresh in-memory tracker.

import { Effect } from "effect"
import { ResponseIdTracker } from "effect/unstable/ai"

const program = Effect.gen(function* () {
  const tracker = yield* ResponseIdTracker.make // => ResponseIdTracker.Service
  return tracker
})

Telemetry

The Telemetry module models the OpenTelemetry GenAI semantic-convention attributes (gen_ai.*) and exposes small helpers for writing them onto Effect tracing spans. Provider implementations use it to annotate spans with request, response, and token-usage metadata; applications can use it to add consistent GenAI metadata around their own model calls.

These helpers annotate spans that already exist — they do not create or scope spans. Attribute writers mutate the provided span, emit only non-nullish values, and convert camelCase field names to snake_case keys.

import { Effect } from "effect"
import { LanguageModel, Telemetry } from "effect/unstable/ai"

// Wrap a generateText call with GenAI annotations on the current span.
const generateWithTelemetry = Effect.gen(function* () {
  const response = yield* LanguageModel.generateText({
    prompt: "Explain Effect in one sentence"
  })

  const span = yield* Effect.currentSpan
  Telemetry.addGenAIAnnotations(span, {
    system: "openai",
    operation: { name: "chat" },
    request: { model: "gpt-5", temperature: 0.7 },
    response: {
      finishReasons: [response.finishReason]
    },
    usage: {
      inputTokens: response.usage.inputTokens.total,
      outputTokens: response.usage.outputTokens.total
    }
  })

  return response.text
}).pipe(Effect.withSpan("generate-text"))
// Span now has gen_ai.system, gen_ai.operation.name, gen_ai.request.model,
// gen_ai.request.temperature, gen_ai.response.finish_reasons,
// gen_ai.usage.input_tokens, gen_ai.usage.output_tokens

Reference

addGenAIAnnotations

Applies standard GenAI attributes to a span. Dual-signature: call it with a span and options directly, or pre-apply options to get (span) => void. Only non-nullish values are written. Mutates the span in-place.

import { Effect } from "effect"
import { Telemetry } from "effect/unstable/ai"

const annotate = Effect.gen(function* () {
  const span = yield* Effect.currentSpan
  Telemetry.addGenAIAnnotations(span, {
    system: "anthropic",
    usage: { inputTokens: 100, outputTokens: 50 }
  })
  // => sets gen_ai.system, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens

  // Curried form
  const apply = Telemetry.addGenAIAnnotations({ request: { model: "claude-3-5" } })
  apply(span) // => sets gen_ai.request.model
})

addSpanAttributes

Creates a reusable attribute writer for an arbitrary key prefix and key transformer. The returned function writes each non-nullish attribute as ${prefix}.${transformKey(key)}.

import { String } from "effect"
import type { Tracer } from "effect"
import { Telemetry } from "effect/unstable/ai"

const addCustom = Telemetry.addSpanAttributes("custom.ai", String.camelToSnake)

declare const span: Tracer.Span
addCustom(span, { modelName: "gpt-5", maxTokens: 1000 })
// => sets custom.ai.model_name and custom.ai.max_tokens

GenAITelemetryAttributeOptions

The options object accepted by addGenAIAnnotations. It is BaseAttributes (so system lives at the top level) plus optional grouped namespaces: operation, request, response, token, and usage.

import type { Telemetry } from "effect/unstable/ai"

const options: Telemetry.GenAITelemetryAttributeOptions = {
  system: "openai",
  operation: { name: "chat" },
  request: { model: "gpt-4-turbo", temperature: 0.7, maxTokens: 2000 },
  response: { id: "chatcmpl-123", model: "gpt-4-turbo", finishReasons: ["stop"] },
  usage: { inputTokens: 50, outputTokens: 25 }
}

Attribute interfaces

The grouped attribute interfaces map directly to gen_ai.* namespaces. All fields are optional and nullable; nullish values are skipped when written.

• BaseAttributes — gen_ai.*. Field: system (a WellKnownSystem or custom string).
• OperationAttributes — gen_ai.operation.*. Field: name (a WellKnownOperationName or custom string).
• TokenAttributes — gen_ai.token.*. Field: type.
• UsageAttributes — gen_ai.usage.*. Fields: inputTokens, outputTokens.
• RequestAttributes — gen_ai.request.*. Fields: model, temperature, topK, topP, maxTokens, encodingFormats, stopSequences, frequencyPenalty, presencePenalty, seed.
• ResponseAttributes — gen_ai.response.*. Fields: id, model, finishReasons.

import type { Telemetry } from "effect/unstable/ai"

const request: Telemetry.RequestAttributes = {
  model: "gpt-5",
  temperature: 0.7,
  maxTokens: 1024,
  stopSequences: ["\n\n"]
}
const usage: Telemetry.UsageAttributes = { inputTokens: 50, outputTokens: 25 }

There are also two aggregate types: AllAttributes (the intersection of all six interfaces) and GenAITelemetryAttributes (the fully prefixed, snake-cased shape produced on the span).

WellKnownOperationName

The well-known values for gen_ai.operation.name: "chat", "embeddings", "text_completion". A custom string may be used if none apply.

import type { Telemetry } from "effect/unstable/ai"

const op: Telemetry.WellKnownOperationName = "chat" // => "chat"

WellKnownSystem

The well-known values for gen_ai.system, including "anthropic", "aws.bedrock", "az.ai.inference", "az.ai.openai", "cohere", "deepseek", "gemini", "groq", "ibm.watsonx.ai", "mistral_ai", "openai", "perplexity", "vertex_ai", "xai". A custom string may be used if none apply.

import type { Telemetry } from "effect/unstable/ai"

const system: Telemetry.WellKnownSystem = "openai" // => "openai"

SpanTransformer

A function that receives the provider request options plus the array of response parts and mutates the span. Use it to derive custom attributes from the actual response content.

import type { Telemetry } from "effect/unstable/ai"

const transformer: Telemetry.SpanTransformer = ({ response, span }) => {
  const totalTextLength = response.reduce(
    (sum, part) => sum + (part.type === "text" ? part.text.length : 0),
    0
  )
  span.attribute("total_text_length", totalTextLength) // => sets total_text_length
}

CurrentSpanTransformer (service)

Context.Service<CurrentSpanTransformer, SpanTransformer> (id "effect/ai/Telemetry/CurrentSpanTransformer"). Provide a SpanTransformer through context so a language model implementation can annotate the span after it has seen the provider response.

import { Effect } from "effect"
import { Telemetry } from "effect/unstable/ai"

const transformer: Telemetry.SpanTransformer = ({ span }) => {
  span.attribute("annotated_by", "custom-transformer")
}

const program = Effect.gen(function* () {
  // language model operations within this scope will run the transformer
  return yield* Effect.void
}).pipe(Effect.provideService(Telemetry.CurrentSpanTransformer, transformer))

Utility types

• AttributesWithPrefix<Attributes, Prefix> — prefixes each key with Prefix. and snake-cases it (e.g. { maxTokens: number } with prefix "gen_ai.request" → { "gen_ai.request.max_tokens": number }).
• FormatAttributeName<T> — converts a camelCase literal to snake_case (e.g. "modelName" → "model_name").

import type { Telemetry } from "effect/unstable/ai"

type Formatted = Telemetry.FormatAttributeName<"maxTokens"> // => "max_tokens"