diff --git a/.github/workflows/ruby-llm-docs.yml b/.github/workflows/ruby-llm-docs.yml index cbf99d490..4a6a81c3f 100644 --- a/.github/workflows/ruby-llm-docs.yml +++ b/.github/workflows/ruby-llm-docs.yml @@ -6,6 +6,7 @@ on: paths: - src/pages/docs/integrations/frameworks/ruby-llm.mdx - scripts/ruby-llm-*.test.* + - examples/frameworks/ruby-llm/** - .github/workflows/ruby-llm-docs.yml permissions: @@ -42,3 +43,28 @@ jobs: sudo --preserve-env=GEM_HOME,GEM_PATH env PATH="$PATH" \ RUBY_LLM_VERSION="${{ matrix.ruby_llm }}" \ unshare --net sh -c 'ip link set lo up; ruby scripts/ruby-llm-native.test.rb' + + legacy-privacy: + name: Legacy privacy (Ruby ${{ matrix.ruby }}, RubyLLM ${{ matrix.ruby_llm }}) + runs-on: ubuntu-latest + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + ruby: ['3.2', '3.4'] + ruby_llm: ['1.15.0', '1.16.0'] + env: + RUBY_LLM_VERSION: ${{ matrix.ruby_llm }} + BUNDLE_GEMFILE: ${{ github.workspace }}/examples/frameworks/ruby-llm/Gemfile + steps: + - uses: actions/checkout@v4 + - uses: ruby/setup-ruby@v1 + with: + ruby-version: ${{ matrix.ruby }} + - name: Install legacy fixture dependencies + run: bundle install + - name: Run privacy and wire regressions without external networking + working-directory: examples/frameworks/ruby-llm + run: | + sudo --preserve-env=GEM_HOME,GEM_PATH,BUNDLE_GEMFILE,RUBY_LLM_VERSION env PATH="$PATH" \ + unshare --net sh -c 'ip link set lo up; bundle exec rake test' diff --git a/examples/frameworks/ruby-llm/Gemfile b/examples/frameworks/ruby-llm/Gemfile index d9546d9d9..9ff07f3f9 100644 --- a/examples/frameworks/ruby-llm/Gemfile +++ b/examples/frameworks/ruby-llm/Gemfile @@ -7,8 +7,8 @@ ruby ">= 3.2" # Override with RUBY_LLM_VERSION=1.15.0 to run the older leg, or 2.1.1 to reproduce the failure. gem "ruby_llm", ENV.fetch("RUBY_LLM_VERSION", "1.16.0") gem "opentelemetry-instrumentation-ruby_llm", "0.7.1" -gem "opentelemetry-sdk" -gem "opentelemetry-exporter-otlp" +gem "opentelemetry-sdk", "1.10.0" +gem "opentelemetry-exporter-otlp", "0.32.0" # Compatible with the Ruby 3.2 floor. group :test do gem "minitest", "~> 5.25" diff --git a/examples/frameworks/ruby-llm/README.md b/examples/frameworks/ruby-llm/README.md index 7e8b8fc9e..a07e3933d 100644 --- a/examples/frameworks/ruby-llm/README.md +++ b/examples/frameworks/ruby-llm/README.md @@ -1,19 +1,18 @@ -# Ruby LLM with Future AGI +# RubyLLM legacy regression fixture -Runnable companion to the page `docs/integrations/frameworks/ruby-llm`. +For application setup, use the [native RubyLLM integration guide](https://docs.futureagi.com/docs/integrations/frameworks/ruby-llm). This directory retains the old 1.x implementation for regression tests; it is not a maintained, installable integration package or a replacement for the native setup. -- `futureagi_tracing.rb` is the whole integration: OpenTelemetry SDK + OTLP/HTTP exporter (both auth headers, `project_name` resource attribute, no gzip), the `opentelemetry-instrumentation-ruby_llm` instrumentation, and a `SanitizingExporter` that keeps spans metadata-only unless `FI_CAPTURE_CONTENT=1`. -- `app.rb` is a demo with six scenarios: `chat`, `stream`, `tool`, `agent`, `session`, `error`. -- `test/` runs with a WebMock stand-in for the model (canned OpenAI-shaped replies, NOT a real model) and a WebMock stand-in for Future AGI (the real OTLP exporter posts to it). No network needed. -- `run_e2e.sh` installs the gems, runs the tests, then runs the scenarios against a real model and a real Future AGI, once metadata only and once with the opt-in. +The earlier helper trusted entire `gen_ai.request.*`, `gen_ai.response.*` and `gen_ai.usage.*` namespaces, allowing custom attributes to export content without opt-in. The retained fixture now allows only the exact span-attribute names in `METADATA_ATTRIBUTES`. It still drops events and, in metadata-only mode, error status descriptions. This is not a secret scanner: values of allowed attributes, span names, resource attributes and other instrumentation remain application-owned. Content opt-in keeps arbitrary span attributes. -Tested: `ruby_llm` 1.16.0 and 1.15.0 with `opentelemetry-instrumentation-ruby_llm` 0.7.1, Ruby 3.2.2 and 3.4.11. `ruby_llm` 2.x is not supported by that instrumentation (it raises `NoMethodError` on `model_id`); `RUBY_LLM_VERSION=2.1.1 bundle install` reproduces it. +Do not remove the sanitizer from an exporting legacy installation. Follow the guide's instructions to disable legacy tracing until you can migrate to native tracing or use a maintained privacy-filtering package. This fixture does not supply that package. + +## Run the regression tests + +Use Ruby 3.2 or newer. The fixture covers RubyLLM 1.15.0 and 1.16.0 with third-party instrumentation 0.7.1; that instrumentation does not support RubyLLM 2.x. ```bash -export FI_API_KEY=... FI_SECRET_KEY=... OPENAI_API_KEY=... -# self-hosted Future AGI / OpenAI-compatible endpoint: -# export FI_BASE_URL=http://localhost:4318 OPENAI_API_BASE=http://127.0.0.1:8317/v1 MODEL_ID=my-model -./run_e2e.sh +bundle install +bundle exec rake test ``` -Needs Ruby 3.2 or newer on PATH. Knobs: `RUBY_LLM_VERSION` (default 1.16.0), `SCENARIOS`, `SKIP_TESTS=1`, `FI_PROJECT_NAME`, `MARKER`. +Set `RUBY_LLM_VERSION=1.15.0` for the older compatibility leg. The tests use synthetic data and WebMock for both model requests and OTLP export. Native 2.1 tests live in `scripts/ruby-llm-native.test.rb` at the repository root. The historical `app.rb` and `run_e2e.sh` are not the current customer setup. diff --git a/examples/frameworks/ruby-llm/Rakefile b/examples/frameworks/ruby-llm/Rakefile index 8f715b8fa..f6918df69 100644 --- a/examples/frameworks/ruby-llm/Rakefile +++ b/examples/frameworks/ruby-llm/Rakefile @@ -1,5 +1,5 @@ -# Three files, three processes: the SDK and the instrumentation are configured once per process. -TEST_FILES = %w[test/test_recipe.rb test/test_optin.rb test/test_wire.rb].freeze +# Separate processes: SDK/patch state must not leak into the paused integration test. +TEST_FILES = %w[test/test_recipe.rb test/test_optin.rb test/test_wire.rb test/test_paused.rb].freeze desc "Run every test file in its own process" task :test do diff --git a/examples/frameworks/ruby-llm/futureagi_tracing.rb b/examples/frameworks/ruby-llm/futureagi_tracing.rb index 051b82eb6..b7d23b1e1 100644 --- a/examples/frameworks/ruby-llm/futureagi_tracing.rb +++ b/examples/frameworks/ruby-llm/futureagi_tracing.rb @@ -1,9 +1,9 @@ # frozen_string_literal: true -# Future AGI tracing for ruby_llm. This file is the whole integration: it configures the -# OpenTelemetry Ruby SDK, enables opentelemetry-instrumentation-ruby_llm, and (by default) -# keeps only metadata on every span, because that instrumentation exports tool arguments and -# results even when capture_content is off. +# Retained legacy regression fixture, not a maintained installation path. See README.md. +# Configures the OpenTelemetry Ruby SDK and opentelemetry-instrumentation-ruby_llm. +# By default, restricts span attributes to an exact allowlist and removes events because +# the instrumentation exports tool arguments and results even when capture_content is off. # # require_relative "futureagi_tracing" # FutureAGITracing.setup # once, at boot @@ -21,15 +21,23 @@ require "opentelemetry-instrumentation-ruby_llm" module FutureAGITracing - # Attributes that describe a call without containing what was said or done. Anything not - # listed is dropped unless content is opted in, so an attribute added by a later release of - # the instrumentation (or one your code sets with with_otel_attributes) stays private until - # you add it here. opentelemetry-instrumentation-ruby_llm 0.7.1 writes tool arguments, - # tool results and tool descriptions onto execute_tool spans even with capture_content off - # (patches/chat.rb, execute_tool), so its own switch is not enough. + # Exact span-attribute contract for the retained legacy regression fixture, not an + # installable integration. Never trust entire gen_ai namespaces: custom attributes + # can put prompts or credentials under otherwise plausible request/response keys. + # Values of allowed attributes, span names and resource attributes remain caller-owned; + # this is not a secret scanner. The instrumentation's capture switch alone leaves + # tool arguments, results, descriptions and exception events enabled. METADATA_ATTRIBUTES = %w[ gen_ai.operation.name gen_ai.provider.name + gen_ai.request.model + gen_ai.request.stream + gen_ai.request.temperature + gen_ai.response.model + gen_ai.usage.input_tokens + gen_ai.usage.output_tokens + gen_ai.usage.cache_creation.input_tokens + gen_ai.usage.cache_read.input_tokens gen_ai.tool.name gen_ai.tool.call.id gen_ai.tool.type @@ -39,7 +47,6 @@ module FutureAGITracing session.id user.id ].freeze - METADATA_PREFIXES = %w[gen_ai.request. gen_ai.response. gen_ai.usage.].freeze # Wraps an exporter. Always drops span events (the instrumentation records exception # messages and stack traces there, which contain local paths and raw provider text). Unless @@ -62,7 +69,7 @@ def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout) private def metadata?(key) - METADATA_ATTRIBUTES.include?(key) || METADATA_PREFIXES.any? { |prefix| key.start_with?(prefix) } + METADATA_ATTRIBUTES.include?(key) end def sanitize(span_data) diff --git a/examples/frameworks/ruby-llm/test/test_paused.rb b/examples/frameworks/ruby-llm/test/test_paused.rb new file mode 100644 index 000000000..a5d4a83b2 --- /dev/null +++ b/examples/frameworks/ruby-llm/test/test_paused.rb @@ -0,0 +1,56 @@ +# frozen_string_literal: true + +# Separate process: a restart must remove the old patch, not reinstall it. +require "bundler/setup" +require "minitest/autorun" +require "webmock/minitest" +require "json" +require "ruby_llm" +require "opentelemetry/sdk" + +WebMock.disable_net_connect! + +class PausedLegacyTest < Minitest::Test + def test_documented_session_runs_without_the_instrumentation_or_exports + page = File.read(File.expand_path("../../../../src/pages/docs/integrations/frameworks/ruby-llm.mdx", __dir__)) + snippet = page[/```ruby title="legacy-paused.rb"\n(.*?)```/m, 1] + refute_nil snippet, "The pause procedure needs a runnable replacement for the old session chain" + + # A shared provider can remain active for unrelated integrations. + exporter = OpenTelemetry::SDK::Trace::Export::InMemorySpanExporter.new + OpenTelemetry::SDK.configure do |config| + config.add_span_processor(OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(exporter)) + end + ENV["OPENAI_API_KEY"] = "synthetic-openai-key" + requests = [] + stub_request(:post, "https://api.openai.com/v1/chat/completions").to_return do |request| + requests << JSON.parse(request.body) + { + status: 200, headers: { "Content-Type" => "application/json" }, + body: JSON.generate( + id: "chatcmpl-paused", object: "chat.completion", created: 1, model: "gpt-4o-mini", + choices: [{ index: 0, message: { role: "assistant", content: "Hello" }, finish_reason: "stop" }], + usage: { prompt_tokens: 3, completion_tokens: 1, total_tokens: 4 } + ) + } + end + + refute RubyLLM::Chat.method_defined?(:with_otel_attributes) + eval(snippet, TOPLEVEL_BINDING, "legacy-paused.rb") + assert_equal 2, requests.size + assert_equal ["user"], requests.first.fetch("messages").map { |message| message.fetch("role") } + assert_equal ["user", "assistant", "user"], requests.last.fetch("messages").map { |message| message.fetch("role") } + OpenTelemetry.tracer_provider.force_flush + assert_empty exporter.finished_spans, "Paused RubyLLM must not emit spans to the shared provider" + refute defined?(FutureAGITracing), "The legacy helper must not be loaded" + refute RubyLLM::Chat.method_defined?(:with_otel_attributes), "Do not restore the instrumentation patch" + refute $LOADED_FEATURES.any? { |path| path.include?("opentelemetry-instrumentation-ruby_llm") } + + OpenTelemetry.tracer_provider.tracer("other-integration").in_span("unrelated") {} + OpenTelemetry.tracer_provider.force_flush + assert_equal ["unrelated"], exporter.finished_spans.map(&:name) + ensure + OpenTelemetry.tracer_provider.shutdown if exporter + ENV.delete("OPENAI_API_KEY") + end +end diff --git a/examples/frameworks/ruby-llm/test/test_recipe.rb b/examples/frameworks/ruby-llm/test/test_recipe.rb index 4797e6e4f..3e882a759 100644 --- a/examples/frameworks/ruby-llm/test/test_recipe.rb +++ b/examples/frameworks/ruby-llm/test/test_recipe.rb @@ -102,6 +102,22 @@ def test_unlisted_attributes_are_dropped_by_default refute attrs.key?("custom.note") end + def test_metadata_namespaces_do_not_allow_arbitrary_content + FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "ok")) + private_attributes = { + "gen_ai.request.parameters" => JSON.generate(api_key: "SYNTHETIC_PRIVATE_KEY", prompt: "SYNTHETIC_PRIVATE_PROMPT"), + "gen_ai.response.content" => "SYNTHETIC_PRIVATE_RESPONSE", + "gen_ai.usage.details" => "SYNTHETIC_PRIVATE_USAGE", + "gen_ai.request.model.extra" => "SYNTHETIC_PRIVATE_SUFFIX" + } + new_chat.with_otel_attributes(private_attributes).ask("hi") + spans = finished + private_attributes.each_key { |key| refute spans.first.attributes.key?(key), "exported #{key}" } + refute_includes all_text(spans), "SYNTHETIC_PRIVATE" + assert_equal "fake-model", spans.first.attributes["gen_ai.request.model"] + assert_equal 11, spans.first.attributes["gen_ai.usage.input_tokens"] + end + def test_no_credentials_or_paths_on_any_span FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "ok")) new_chat.ask("hi") diff --git a/examples/frameworks/ruby-llm/test/test_wire.rb b/examples/frameworks/ruby-llm/test/test_wire.rb index d32804d1e..45704d85f 100644 --- a/examples/frameworks/ruby-llm/test/test_wire.rb +++ b/examples/frameworks/ruby-llm/test/test_wire.rb @@ -14,7 +14,11 @@ class WireTest < Minitest::Test def test_export_request_shape stub = WebMock.stub_request(:post, "https://fi.example.test/tracer/v1/traces").to_return(status: 200, body: "") FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "WIREREPLY-MARKER")) - new_chat.ask("WIREPROMPT-MARKER") + new_chat.with_otel_attributes( + "gen_ai.request.parameters" => "SYNTHETIC_PRIVATE_KEY", + "gen_ai.response.content" => "SYNTHETIC_PRIVATE_RESPONSE", + "gen_ai.usage.details" => "SYNTHETIC_PRIVATE_USAGE" + ).ask("WIREPROMPT-MARKER") FutureAGITracing.shutdown FutureAGITracing.shutdown # second call is a no-op assert_requested stub, times: 1 @@ -32,5 +36,6 @@ def test_export_request_shape assert_equal "chat fake-model", span.name refute_includes req.body, "WIREPROMPT-MARKER" refute_includes req.body, "WIREREPLY-MARKER" + refute_includes req.body, "SYNTHETIC_PRIVATE" end end diff --git a/scripts/ruby-llm-native.test.rb b/scripts/ruby-llm-native.test.rb index 718c83b07..d836c8896 100644 --- a/scripts/ruby-llm-native.test.rb +++ b/scripts/ruby-llm-native.test.rb @@ -40,6 +40,7 @@ def setup end @server.mount_proc('/v1') { |req, res| provider_response(req, res) } @thread = Thread.new { @server.start } + Thread.pass until @server.status == :Running @origin = "http://127.0.0.1:#{@server.config[:Port]}" end @@ -193,6 +194,132 @@ def test_stream_and_error_metadata_remain_private assert_empty error.events end + # Characterize released behavior rather than silently treating UNSET as success. + # The page-contract test requires the caveat until upstream changes this behavior. + def test_outer_timeout_finishes_span_without_error_metadata + items = run_app(<<~RUBY) + require_relative 'tracing' + require 'timeout' + chat = RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions) + started = false + begin + Timeout.timeout(1) do + chat.ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk| + started = true + sleep 10 + end + end + raise 'timeout did not fire' + rescue Timeout::Error + raise 'timed out before stream callback' unless started + end + RUBY + assert_interrupted_span(items) + end + + def test_interrupt_finishes_span_without_error_metadata + items = run_app(<<~RUBY) + require_relative 'tracing' + begin + RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions).ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk| + raise Interrupt, 'SYNTHETIC_PRIVATE_INTERRUPTED' + end + raise 'interrupt did not fire' + rescue Interrupt + end + RUBY + assert_interrupted_span(items) + end + + def test_killed_worker_finishes_span_without_error_metadata + items = run_app(<<~RUBY) + require_relative 'tracing' + ready = Queue.new + worker = Thread.new do + RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions).ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk| + ready << true + sleep 10 + end + end + ready.pop + worker.kill.join + RUBY + assert_interrupted_span(items) + end + + def assert_interrupted_span(items) + assert_envelope(items, 'none') + assert_equal 1, spans(items).size + span = spans(items).first + assert_equal :STATUS_CODE_UNSET, span.status.code + refute attributes(span.attributes).key?('error.type') + assert_empty span.events + assert_empty span.status.message + end + + def test_cooperative_stream_cancellation_records_error_without_content + items = run_app(<<~RUBY) + require_relative 'tracing' + chat = RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions) + begin + chat.ask('SYNTHETIC_PRIVATE_STREAM') { |_chunk| chat.cancel } + raise 'cancellation not raised' + rescue RubyLLM::CancelledError + end + RUBY + assert_envelope(items, 'none') + assert_equal 1, spans(items).size + span = spans(items).first + assert_equal :STATUS_CODE_ERROR, span.status.code + assert_equal 'RubyLLM::CancelledError', attributes(span.attributes)['error.type'] + assert_empty span.events + assert_empty span.status.message + end + + def test_http_timeout_records_error_without_content + timeout_config = snippet('timeouts.rb').sub('config.request_timeout = 30', 'config.request_timeout = 0.05') + items = run_app(<<~RUBY) + require_relative 'tracing' + #{timeout_config} + begin + RubyLLM.chat(model: 'gpt-4o-mini').ask('SYNTHETIC_PRIVATE_HTTP_TIMEOUT') + raise 'HTTP timeout not raised' + rescue Faraday::TimeoutError + end + RUBY + assert_envelope(items, 'none') + assert_equal 1, spans(items).size + span = spans(items).first + assert_equal :STATUS_CODE_ERROR, span.status.code + assert_equal 'Faraday::TimeoutError', attributes(span.attributes)['error.type'] + assert_empty span.events + assert_empty span.status.message + end + + def test_tool_exception_records_error_without_content + items = run_app(<<~RUBY) + require_relative 'tracing' + class Lookup < RubyLLM::Tool + def execute(value:) + raise ArgumentError, 'SYNTHETIC_PRIVATE_TOOL_ERROR' + end + end + begin + RubyLLM.chat(model: 'gpt-4o-mini').with_tools(Lookup).ask('SYNTHETIC_PRIVATE_PROMPT') + raise 'tool exception not raised' + rescue ArgumentError + end + RUBY + assert_envelope(items, 'none') + assert_equal 2, spans(items).size + tool = spans(items).find { |span| span.name == 'execute_tool lookup' } + refute_nil tool + assert_equal :STATUS_CODE_ERROR, tool.status.code + assert_equal 'ArgumentError', attributes(tool.attributes)['error.type'] + assert_empty tool.events + assert_empty tool.status.message + end + def test_exported_payload_round_trips_json_and_protobuf_plain_and_gzip # Ruby exports protobuf only. JSON is a wire-codec replay, not a claimed Ruby exporter mode. original = run_app.first[:request] @@ -219,6 +346,7 @@ def test_exported_payload_round_trips_json_and_protobuf_plain_and_gzip def provider_response(req, res) data = JSON.parse(req.body) @provider_requests << { path: req.path, data: data } + sleep 0.2 if req.body.include?('SYNTHETIC_PRIVATE_HTTP_TIMEOUT') res['Content-Type'] = 'application/json' if req.body.include?('SYNTHETIC_PRIVATE_ERROR') res.status = 400 diff --git a/scripts/ruby-llm-page.test.mjs b/scripts/ruby-llm-page.test.mjs index 7447640d5..64e14bd96 100644 --- a/scripts/ruby-llm-page.test.mjs +++ b/scripts/ruby-llm-page.test.mjs @@ -26,13 +26,41 @@ test('native setup does not promise legacy content or session parity', () => { assert.match(native, /Do not enable.*third-party.*native/s); }); -test('existing 1.x users retain their bounded compatibility and privacy reference', () => { - assert.match(legacy, /gem "ruby_llm", "1\.16\.0"/); - assert.match(legacy, /gem "opentelemetry-instrumentation-ruby_llm", "0\.7\.1"/); - assert.match(legacy, /class SanitizingExporter/); - assert.match(legacy, /copy.events = nil/); - assert.match(legacy, /capture_content: true/); +test('native error coverage distinguishes cooperative failures from forced interruption', () => { + assert.match(native, /StandardError/); + for (const mechanism of ['Timeout.timeout', 'Interrupt', 'Thread#kill']) { + assert.ok(native.includes(mechanism), `Missing ${mechanism} caveat`); + } + assert.match(native, /UNSET.*error\.type/s); + assert.match(native, /UNSET.*not.*success/s); + assert.match(native, /config\.request_timeout = 30/); + assert.match(native, /config\.max_retries = 0/); + assert.match(native, /not.*wall-clock deadline/s); + assert.match(native, /chat\.cancel/); + assert.match(native, /RubyLLM::CancelledError/); + assert.match(native, /cooperative.*checkpoint/s); +}); + +test('legacy pause removes instrumentation-only application calls before restarting', () => { + const pause = legacy.split('Keep your application')[0]; + assert.match(pause, /Remove.*with_otel_attributes.*before restarting/s); + assert.match(pause, /NoMethodError/); + assert.match(pause, /```ruby title="legacy-paused.rb"/); + assert.match(pause, /session.*grouping.*unavailable/s); + const snippet = pause.match(/```ruby title="legacy-paused.rb"\n([\s\S]*?)```/)?.[1]; + assert.ok(snippet); + assert.doesNotMatch(snippet, /with_otel_attributes|FutureAGITracing|futureagi_tracing|OpenTelemetry/); +}); + +test('legacy export is held without telling users to remove only its safeguard', () => { + assert.match(legacy, /Stop exporting.*1\.x/s); + assert.match(legacy, /gen_ai\.request\.parameters/); + assert.match(legacy, /capture_content: false/); + assert.match(legacy, /Do not remove.*SanitizingExporter.*leaving.*instrumentation/s); + assert.match(legacy, /stop loading.*futureagi_tracing\.rb/s); + assert.match(legacy, /restart/i); + assert.match(legacy, /maintained.*package/s); assert.match(legacy, /with_otel_attributes/); assert.match(legacy, /invoke_agent/); - assert.match(legacy, /token counts.*inflated/s); + assert.doesNotMatch(page, /class SanitizingExporter|METADATA_PREFIXES|title="futureagi_tracing\.rb"/); }); diff --git a/src/pages/docs/integrations/frameworks/ruby-llm.mdx b/src/pages/docs/integrations/frameworks/ruby-llm.mdx index 89fa55b33..8a5ad3721 100644 --- a/src/pages/docs/integrations/frameworks/ruby-llm.mdx +++ b/src/pages/docs/integrations/frameworks/ruby-llm.mdx @@ -105,262 +105,70 @@ The exit hook flushes pending spans on a normal process exit. For a managed serv - **Chat and streaming:** a `chat ` span for each model request, with model/provider metadata and token usage when reported by the provider. Streaming produces one span for the request, not one per chunk. - **Tools:** `execute_tool ` spans with the tool name and call ID, without arguments, results or descriptions. Follow-up model requests are separate chat spans. -- **Errors:** an `ERROR` status and `error.type`, without the provider error message or exception stack trace. +- **Errors:** exceptions derived from `StandardError` that escape an instrumented operation set `ERROR` and `error.type`, without the error message or exception stack trace. This includes provider errors and ordinary tool exceptions; forced interruption is different, as described below. These privacy defaults apply to RubyLLM's native spans, not to attributes or events added by your application or other instrumentation. Use non-sensitive tool and workflow names because names are exported. Native RubyLLM 2.1.1 does not add an `invoke_agent` root or session and user attributes, and does not expose the third-party `with_otel_attributes` API. Do not expect a 1.x application's content capture or session grouping to carry over automatically. Follow the [RubyLLM upgrade guide](https://rubyllm.com/upgrading/) for application API changes as well. -## Troubleshooting - -- **No traces:** confirm both Future AGI keys, the origin in `FI_BASE_URL`, and the `project_name` resource attribute. Check exporter logs for HTTP failures. -- **`NoMethodError` mentioning `model_id`:** the old third-party patch is still active with RubyLLM 2.x. Remove it and restart the process before using native tracing. -- **No prompt or response text:** expected for native tracing in 2.1.1. Setting `FI_CAPTURE_CONTENT` or the third-party capture environment variable does not enable native content capture. -- **Missing spans at exit:** allow the shutdown hook to complete. Forced termination such as `kill -9` cannot flush queued spans. - -## Existing RubyLLM 1.x installations - -Keep an existing 1.x setup on its pinned dependencies until you can migrate its application APIs and tracing behavior together. The legacy route below retains opt-in content capture, `with_otel_attributes`, and `invoke_agent` spans; the native 2.1.1 route is not a drop-in replacement for those features. Its exporter safeguard is still required: the legacy instrumentation's `capture_content: false` does not suppress tool data or exception events by itself. - -
-Legacy reference for existing 1.x applications - -This reference is for applications already using `FutureAGITracing`. For new applications, use the native setup above. Do not combine the two configurations. - -### Legacy dependencies - -```ruby title="Gemfile" -source "https://rubygems.org" - -gem "ruby_llm", "1.16.0" -gem "opentelemetry-instrumentation-ruby_llm", "0.7.1" -gem "opentelemetry-sdk" -gem "opentelemetry-exporter-otlp" -``` - -Keep the OpenTelemetry version pins already working in your 1.x application. The instrumentation gem depends only on `opentelemetry-api` and `opentelemetry-instrumentation-base`; the SDK and exporter remain separate dependencies. Keep `ruby_llm` below 2.0 (`"~> 1.16"`). - -### Legacy environment variables - -```bash -export FI_API_KEY="YOUR_API_KEY" -export FI_SECRET_KEY="YOUR_SECRET_KEY" -export FI_PROJECT_NAME="my-chatbot" # the Observe project the traces land in -export OPENAI_API_KEY="YOUR_OPENAI_KEY" # model provider key -# Set this to your self-hosted Future AGI origin. -export FI_BASE_URL="http://localhost:4318" # replace with your instance -# export FI_CAPTURE_CONTENT=1 # opt in to prompts, answers and tool input/output -``` - -Set `FI_BASE_URL` to the origin only. The code adds `/tracer/v1/traces`. - -### Keep the legacy exporter safeguard +## Timeouts and cancellation -The instrumentation gem sets no endpoint, headers or project. The OpenTelemetry SDK you configure does, and Future AGI needs all three: the OTLP/HTTP endpoint, both `X-Api-Key` and `X-Secret-Key`, and the resource attribute `project_name` (a span attribute of that name is not a substitute). Keep the following safeguard in your existing `futureagi_tracing.rb`: - -```ruby title="futureagi_tracing.rb" -# frozen_string_literal: true + +In RubyLLM 2.1.0–2.1.1, an outer `Timeout.timeout`, `Interrupt`, or `Thread#kill` can finish a native request span with status `UNSET` and no `error.type`, even though the request did not complete. `UNSET` is not evidence of success. Do not use native span status alone to count successful requests when your application uses forced interruption. + -# Future AGI tracing for ruby_llm. This file is the whole integration: it configures the -# OpenTelemetry Ruby SDK, enables opentelemetry-instrumentation-ruby_llm, and (by default) -# keeps only metadata on every span, because that instrumentation exports tool arguments and -# results even when capture_content is off. -# -# require_relative "futureagi_tracing" -# FutureAGITracing.setup # once, at boot -# ... -# FutureAGITracing.shutdown # before the process exits (also registered with at_exit) -# -# Environment: -# FI_API_KEY, FI_SECRET_KEY required -# FI_PROJECT_NAME Observe project the traces land in (default "ruby-llm") -# FI_BASE_URL self-hosted origin only; default https://api.futureagi.com -# FI_CAPTURE_CONTENT=1 opt in to prompts, answers, tool arguments and tool results +For HTTP timeouts, configure RubyLLM before creating the chat instead of wrapping the call in `Timeout.timeout`: -require "opentelemetry/sdk" -require "opentelemetry/exporter/otlp" -require "opentelemetry-instrumentation-ruby_llm" - -module FutureAGITracing - # Attributes that describe a call without containing what was said or done. Anything not - # listed is dropped unless content is opted in, so an attribute added by a later release of - # the instrumentation (or one your code sets with with_otel_attributes) stays private until - # you add it here. opentelemetry-instrumentation-ruby_llm 0.7.1 writes tool arguments, - # tool results and tool descriptions onto execute_tool spans even with capture_content off - # (patches/chat.rb, execute_tool), so its own switch is not enough. - METADATA_ATTRIBUTES = %w[ - gen_ai.operation.name - gen_ai.provider.name - gen_ai.tool.name - gen_ai.tool.call.id - gen_ai.tool.type - gen_ai.agent.name - gen_ai.conversation.id - error.type - session.id - user.id - ].freeze - METADATA_PREFIXES = %w[gen_ai.request. gen_ai.response. gen_ai.usage.].freeze - - # Wraps an exporter. Always drops span events (the instrumentation records exception - # messages and stack traces there, which contain local paths and raw provider text). Unless - # content is opted in, also keeps only METADATA_ATTRIBUTES, and keeps the ERROR status - # without its text. - class SanitizingExporter - def initialize(inner, keep_content:) - @inner = inner - @keep_content = keep_content - end - - def export(span_datas, timeout: nil) - @inner.export(span_datas.map { |sd| sanitize(sd) }, timeout: timeout) - end - - def force_flush(timeout: nil) = @inner.force_flush(timeout: timeout) - - def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout) - - private - - def metadata?(key) - METADATA_ATTRIBUTES.include?(key) || METADATA_PREFIXES.any? { |prefix| key.start_with?(prefix) } - end - - def sanitize(span_data) - copy = span_data.dup - copy.events = nil - copy.total_recorded_events = 0 - unless @keep_content - copy.attributes = (span_data.attributes || {}).select { |key, _| metadata?(key) } - status = span_data.status - copy.status = OpenTelemetry::Trace::Status.error if status && status.code == OpenTelemetry::Trace::Status::ERROR - end - copy - end - end - - class << self - # Configure the SDK. Pass exporter: to replace the OTLP exporter (tests do). - def setup(project_name: ENV.fetch("FI_PROJECT_NAME", "ruby-llm"), - capture_content: ENV["FI_CAPTURE_CONTENT"] == "1", - exporter: nil, batch: true, flush_at_exit: true) - exporter ||= otlp_exporter - exporter = SanitizingExporter.new(exporter, keep_content: capture_content) - processor = if batch - OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(exporter) - else - OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(exporter) - end - - OpenTelemetry::SDK.configure do |c| - c.service_name = ENV.fetch("OTEL_SERVICE_NAME", project_name) - c.resource = OpenTelemetry::SDK::Resources::Resource.create( - "project_name" => project_name, - "project_type" => "observe" - ) - c.add_span_processor(processor) - c.use "OpenTelemetry::Instrumentation::RubyLLM", capture_content: capture_content - end - at_exit { shutdown } if flush_at_exit - self - end - - # Flush and stop. Safe to call twice. - def shutdown - return if @shut_down - - @shut_down = true - OpenTelemetry.tracer_provider.shutdown - end - - def endpoint - "#{ENV.fetch('FI_BASE_URL', 'https://api.futureagi.com').chomp('/')}/tracer/v1/traces" - end - - private - - def otlp_exporter - OpenTelemetry::Exporter::OTLP::Exporter.new( - endpoint: endpoint, - headers: { "X-Api-Key" => ENV.fetch("FI_API_KEY"), "X-Secret-Key" => ENV.fetch("FI_SECRET_KEY") }, - compression: "none" # the exporter gzips by default; do not - ) - end - end +```ruby title="timeouts.rb" +RubyLLM.configure do |config| + config.request_timeout = 30 + config.max_retries = 0 end ``` -Two details in it matter. `compression: "none"` is there because the Ruby OTLP exporter gzips by default and self-hosted releases before v1.56.0 answered gzip with HTTP 400. Future AGI cloud and self-hosted v1.56.0 and later accept gzip, so the line is optional there. The `SanitizingExporter` is there because of what the instrumentation exports, described under privacy below. +An HTTP timeout that raises `Faraday::TimeoutError` inside the native request span is recorded as an error. The HTTP timeout is not a wall-clock deadline for an entire conversation, tool loop, or continuously active stream. Adjust retries deliberately because they can extend the total duration. -### Legacy application setup +To cancel an active stream, call `chat.cancel` from your application's cancellation handler or stream callback, and handle `RubyLLM::CancelledError` around `chat.ask`. Cancellation is cooperative: it takes effect at a cancellation checkpoint, not immediately while a network read or tool is blocked. When that exception leaves the active request span, native tracing records `ERROR` and `error.type`. Let the operation unwind before shutting down tracing; do not kill its worker thread to cancel it. If you must enforce a hard deadline or terminate a worker, track that outcome in your application's own request lifecycle rather than infer it from a native span's status. -Existing applications call `setup` once at boot, before any `RubyLLM` call: +## Troubleshooting -```ruby -require_relative "futureagi_tracing" -require "ruby_llm" +- **No traces:** confirm both Future AGI keys, the origin in `FI_BASE_URL`, and the `project_name` resource attribute. Check exporter logs for HTTP failures. +- **`NoMethodError` mentioning `model_id`:** the old third-party patch is still active with RubyLLM 2.x. Remove it and restart the process before using native tracing. +- **No prompt or response text:** expected for native tracing in 2.1.1. Setting `FI_CAPTURE_CONTENT` or the third-party capture environment variable does not enable native content capture. +- **Missing spans at exit:** allow the shutdown hook to complete. Forced termination such as `kill -9` cannot flush queued spans. -FutureAGITracing.setup +## Existing RubyLLM 1.x installations -RubyLLM.configure { |c| c.openai_api_key = ENV.fetch("OPENAI_API_KEY") } + +Stop exporting traces from the old RubyLLM 1.x helper when you require metadata-only capture. Its prefix filter accepts arbitrary `gen_ai.request.*`, `gen_ai.response.*`, and `gen_ai.usage.*` attributes: for example, `with_otel_attributes` can put prompts or credentials in `gen_ai.request.parameters`. Neither `capture_content: false` nor leaving `FI_CAPTURE_CONTENT` unset prevents this bypass. The instrumentation's capture switch also leaves tool data and exception events enabled. + -chat = RubyLLM.chat(model: "gpt-4o-mini") -chat.ask("What is OpenTelemetry?") +Do not remove `SanitizingExporter` while leaving the third-party instrumentation exporting: that exposes additional data. To pause the old integration: -FutureAGITracing.shutdown # flush before the process exits -``` +1. In startup configuration, stop loading `futureagi_tracing.rb` and calling `FutureAGITracing.setup`. Remove any separate `config.use "OpenTelemetry::Instrumentation::RubyLLM"` call and disable automatic loading of that instrumentation. If tracing shares a provider with other instrumentation, keep that provider and its exporter for the other integrations rather than shutting it down globally. +2. Remove instrumentation-only application calls such as `with_otel_attributes` before restarting, including calls that attach session, conversation or user attributes. That method comes from the third-party patch, not RubyLLM 1.x; leaving it in a chat chain raises `NoMethodError` once the patch is disabled. Keep the chat object and ordinary RubyLLM calls, but omit the telemetry-only chain as shown below. Future AGI session/user grouping is unavailable while this tracing is paused; the chat's conversation history still works. +3. Restart the application with both changes. Changes to startup configuration do not remove an already installed patch. Do not reinstall the patch just to retain its attribute methods. -Run `ruby chat.rb`, then open the project in Observe to inspect the trace and its spans. +For example, replace a session chain such as `RubyLLM.chat(model: "gpt-4o-mini").with_otel_attributes(...)` with an ordinary chat, reusing it for each turn: -`setup` also registers `at_exit { shutdown }`, so a normal exit flushes. A short script that is killed (`kill -9`) loses the last batch. In Rails, call `FutureAGITracing.setup` from an initializer and `FutureAGITracing.shutdown` from your server's stop hook if it does not exit normally. +```ruby title="legacy-paused.rb" +require "ruby_llm" -To tie several calls to one session and one user, set the attributes on the chat: +RubyLLM.configure { |config| config.openai_api_key = ENV.fetch("OPENAI_API_KEY") } -```ruby chat = RubyLLM.chat(model: "gpt-4o-mini") - .with_otel_attributes("session.id" => conversation.id.to_s, "user.id" => current_user.id.to_s) -chat.ask("Hi") -chat.ask("And now?") +chat.ask("Say hi") +chat.ask("Say bye") ``` -The legacy spans include the same `session.id` and `user.id` on both calls for session grouping. RubyLLM spans carry no session or user unless you set them. The instrumentation sets `gen_ai.conversation.id` only for persisted `acts_as_chat` records or when you set it yourself; `gen_ai.conversation.id` on its own is outside this guide’s coverage as a session key. - -### Legacy captured data - -| Case | What you get | -| --- | --- | -| `chat.ask` | One `llm` span `chat ` with the request and response model, provider (`openai`), and input and output token counts. | -| Streaming (`chat.ask` with a block) | One span for the whole stream, not one per chunk, with token counts (the provider reports usage on the last chunk) and `gen_ai.request.stream=true`. | -| Tool call | One `execute_tool ` span of type `tool` (tool name and call id), a child of the chat span that ran it. The follow-up completion is a second `chat` span under it. Tool arguments, results and the tool description are not exported by default. | -| `RubyLLM::Agent` | One `invoke_agent ` span as trace root, with the chat and tool spans below it. Needs `ruby_llm` 1.12.1 or newer. | -| Provider error | The chat span has status `ERROR` and `error.type` (for example `RubyLLM::BadRequestError` for an HTTP 400, and the base `RubyLLM::Error` for OpenAI's 404 on an unknown model id). The error text and stack trace are not exported. | -| Session and user | `session.id` and `user.id` when you set them as above. | - -Everything RubyLLM does not report is absent, not zero. - -### Legacy privacy settings - -With no opt-in, Future AGI receives model, provider, token counts, span names and timing, tool names, agent names, status and `error.type`, and the `session.id`, `user.id` and `gen_ai.conversation.id` you set. It does not receive prompts, answers, system instructions, tool arguments, tool results, tool descriptions, error messages, stack traces, local paths, host names, or any other attribute you set with `with_otel_attributes`. - -The instrumentation's `capture_content` switch alone does not suppress tool arguments, results, descriptions or exception events. Keep `SanitizingExporter` in place to remove them before export when content capture is off. - -To opt in, set `FI_CAPTURE_CONTENT=1` or call `FutureAGITracing.setup(capture_content: true)`. That turns on `gen_ai.input.messages`, `gen_ai.output.messages` and `gen_ai.system_instructions` (prompts and answers), keeps the tool arguments, results and descriptions, and keeps every other attribute you set. Span events, and so exception messages and stack traces, are never exported, in either mode. Content arrives as span attributes and may not appear in the trace's Input and Output panels. - -### Legacy limitations +This example does not load the old helper or enable RubyLLM tracing. -- The legacy instrumentation does not support `ruby_llm` 2.x. Use the native route above when migrating to 2.1. -- In a tool loop the outer `chat` span reports the token counts of the final response, the same numbers as the follow-up `chat` span under it, so summed token counts for the trace are inflated. This comes from the instrumentation: it reads the last message after the nested completion returns. -- `invoke_agent` spans show as type `unknown` in Future AGI: the instrumentation sets no `fi.span.kind`. -- The instrumentation exports `gen_ai.usage.cache_creation.input_tokens=0` when the provider reports none. Future AGI tokens are unaffected. +Keep your application on its pinned RubyLLM 1.x dependencies while you prepare the upgrade, but leave its legacy tracing disabled. Resume RubyLLM tracing with the native setup above after migrating to 2.1.1, or with a maintained privacy-filtering package that explicitly supports your 1.x version. This guide does not provide an installable legacy replacement; do not replace the old helper with an unfiltered exporter. -### Legacy troubleshooting +Before switching to native tracing, account for these differences: -- **`NoMethodError ... 'model_id'` on `chat.ask`.** You are on `ruby_llm` 2.x. Pin `ruby_llm` to 1.x. -- **No traces for the project.** `project_name` must be a resource attribute, which `setup` sets. A missing `FI_PROJECT_NAME` falls back to `ruby-llm`: look there. -- **`KeyError: key not found: "FI_API_KEY"`.** Both `FI_API_KEY` and `FI_SECRET_KEY` are required. -- **HTTP errors from the exporter in the log.** Check the endpoint (`FI_BASE_URL` is the origin, no path). If gzip is back on: releases before v1.56.0 answered HTTP 400; current releases accept gzip. -- **Short script, zero spans.** Call `FutureAGITracing.shutdown` before exit, and do not `kill -9` the process. -- **Content shows up when you did not opt in.** Check that `FI_CAPTURE_CONTENT` is unset and that you are using `SanitizingExporter` through `setup`. `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` turns the instrumentation's content on, but the exporter still drops it unless you opted in here. -- **`RubyLLM::ModelNotFoundError: Unknown model`**. Pass `provider: :openai, assume_model_exists: true` for a model RubyLLM does not list, such as a model behind an OpenAI-compatible endpoint. +- Native tracing does not provide content capture, `with_otel_attributes`, automatic session/user attributes, or `invoke_agent` roots. Migrate application calls that rely on the old API; simply updating the gem version is not enough. +- The third-party instrumentation 0.7.1 does not support RubyLLM 2.x. Remove that gem and its configuration, then restart before enabling native tracing. Do not run both integrations together. +- Use only non-sensitive model, tool and workflow names. Native privacy defaults do not sanitize attributes, events, resource values or names supplied by your own application or other instrumentation. -
+Follow the [RubyLLM upgrade guide](https://rubyllm.com/upgrading/) for the application migration.