Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/ruby-llm-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ on:
paths:
- src/pages/docs/integrations/frameworks/ruby-llm.mdx
- scripts/ruby-llm-*.test.*
- examples/frameworks/ruby-llm/**
- .github/workflows/ruby-llm-docs.yml

permissions:
Expand Down Expand Up @@ -42,3 +43,28 @@ jobs:
sudo --preserve-env=GEM_HOME,GEM_PATH env PATH="$PATH" \
RUBY_LLM_VERSION="${{ matrix.ruby_llm }}" \
unshare --net sh -c 'ip link set lo up; ruby scripts/ruby-llm-native.test.rb'

legacy-privacy:
name: Legacy privacy (Ruby ${{ matrix.ruby }}, RubyLLM ${{ matrix.ruby_llm }})
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
ruby: ['3.2', '3.4']
ruby_llm: ['1.15.0', '1.16.0']
env:
RUBY_LLM_VERSION: ${{ matrix.ruby_llm }}
BUNDLE_GEMFILE: ${{ github.workspace }}/examples/frameworks/ruby-llm/Gemfile
steps:
- uses: actions/checkout@v4
- uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby }}
- name: Install legacy fixture dependencies
run: bundle install
- name: Run privacy and wire regressions without external networking
working-directory: examples/frameworks/ruby-llm
run: |
sudo --preserve-env=GEM_HOME,GEM_PATH,BUNDLE_GEMFILE,RUBY_LLM_VERSION env PATH="$PATH" \
unshare --net sh -c 'ip link set lo up; bundle exec rake test'
4 changes: 2 additions & 2 deletions examples/frameworks/ruby-llm/Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ ruby ">= 3.2"
# Override with RUBY_LLM_VERSION=1.15.0 to run the older leg, or 2.1.1 to reproduce the failure.
gem "ruby_llm", ENV.fetch("RUBY_LLM_VERSION", "1.16.0")
gem "opentelemetry-instrumentation-ruby_llm", "0.7.1"
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-sdk", "1.10.0"
gem "opentelemetry-exporter-otlp", "0.32.0" # Compatible with the Ruby 3.2 floor.

group :test do
gem "minitest", "~> 5.25"
Expand Down
23 changes: 11 additions & 12 deletions examples/frameworks/ruby-llm/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
# Ruby LLM with Future AGI
# RubyLLM legacy regression fixture

Runnable companion to the page `docs/integrations/frameworks/ruby-llm`.
For application setup, use the [native RubyLLM integration guide](https://docs.futureagi.com/docs/integrations/frameworks/ruby-llm). This directory retains the old 1.x implementation for regression tests; it is not a maintained, installable integration package or a replacement for the native setup.

- `futureagi_tracing.rb` is the whole integration: OpenTelemetry SDK + OTLP/HTTP exporter (both auth headers, `project_name` resource attribute, no gzip), the `opentelemetry-instrumentation-ruby_llm` instrumentation, and a `SanitizingExporter` that keeps spans metadata-only unless `FI_CAPTURE_CONTENT=1`.
- `app.rb` is a demo with six scenarios: `chat`, `stream`, `tool`, `agent`, `session`, `error`.
- `test/` runs with a WebMock stand-in for the model (canned OpenAI-shaped replies, NOT a real model) and a WebMock stand-in for Future AGI (the real OTLP exporter posts to it). No network needed.
- `run_e2e.sh` installs the gems, runs the tests, then runs the scenarios against a real model and a real Future AGI, once metadata only and once with the opt-in.
The earlier helper trusted entire `gen_ai.request.*`, `gen_ai.response.*` and `gen_ai.usage.*` namespaces, allowing custom attributes to export content without opt-in. The retained fixture now allows only the exact span-attribute names in `METADATA_ATTRIBUTES`. It still drops events and, in metadata-only mode, error status descriptions. This is not a secret scanner: values of allowed attributes, span names, resource attributes and other instrumentation remain application-owned. Content opt-in keeps arbitrary span attributes.

Tested: `ruby_llm` 1.16.0 and 1.15.0 with `opentelemetry-instrumentation-ruby_llm` 0.7.1, Ruby 3.2.2 and 3.4.11. `ruby_llm` 2.x is not supported by that instrumentation (it raises `NoMethodError` on `model_id`); `RUBY_LLM_VERSION=2.1.1 bundle install` reproduces it.
Do not remove the sanitizer from an exporting legacy installation. Follow the guide's instructions to disable legacy tracing until you can migrate to native tracing or use a maintained privacy-filtering package. This fixture does not supply that package.

## Run the regression tests

Use Ruby 3.2 or newer. The fixture covers RubyLLM 1.15.0 and 1.16.0 with third-party instrumentation 0.7.1; that instrumentation does not support RubyLLM 2.x.

```bash
export FI_API_KEY=... FI_SECRET_KEY=... OPENAI_API_KEY=...
# self-hosted Future AGI / OpenAI-compatible endpoint:
# export FI_BASE_URL=http://localhost:4318 OPENAI_API_BASE=http://127.0.0.1:8317/v1 MODEL_ID=my-model
./run_e2e.sh
bundle install
bundle exec rake test
```

Needs Ruby 3.2 or newer on PATH. Knobs: `RUBY_LLM_VERSION` (default 1.16.0), `SCENARIOS`, `SKIP_TESTS=1`, `FI_PROJECT_NAME`, `MARKER`.
Set `RUBY_LLM_VERSION=1.15.0` for the older compatibility leg. The tests use synthetic data and WebMock for both model requests and OTLP export. Native 2.1 tests live in `scripts/ruby-llm-native.test.rb` at the repository root. The historical `app.rb` and `run_e2e.sh` are not the current customer setup.
4 changes: 2 additions & 2 deletions examples/frameworks/ruby-llm/Rakefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Three files, three processes: the SDK and the instrumentation are configured once per process.
TEST_FILES = %w[test/test_recipe.rb test/test_optin.rb test/test_wire.rb].freeze
# Separate processes: SDK/patch state must not leak into the paused integration test.
TEST_FILES = %w[test/test_recipe.rb test/test_optin.rb test/test_wire.rb test/test_paused.rb].freeze

desc "Run every test file in its own process"
task :test do
Expand Down
31 changes: 19 additions & 12 deletions examples/frameworks/ruby-llm/futureagi_tracing.rb
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# frozen_string_literal: true

# Future AGI tracing for ruby_llm. This file is the whole integration: it configures the
# OpenTelemetry Ruby SDK, enables opentelemetry-instrumentation-ruby_llm, and (by default)
# keeps only metadata on every span, because that instrumentation exports tool arguments and
# results even when capture_content is off.
# Retained legacy regression fixture, not a maintained installation path. See README.md.
# Configures the OpenTelemetry Ruby SDK and opentelemetry-instrumentation-ruby_llm.
# By default, restricts span attributes to an exact allowlist and removes events because
# the instrumentation exports tool arguments and results even when capture_content is off.
#
# require_relative "futureagi_tracing"
# FutureAGITracing.setup # once, at boot
Expand All @@ -21,15 +21,23 @@
require "opentelemetry-instrumentation-ruby_llm"

module FutureAGITracing
# Attributes that describe a call without containing what was said or done. Anything not
# listed is dropped unless content is opted in, so an attribute added by a later release of
# the instrumentation (or one your code sets with with_otel_attributes) stays private until
# you add it here. opentelemetry-instrumentation-ruby_llm 0.7.1 writes tool arguments,
# tool results and tool descriptions onto execute_tool spans even with capture_content off
# (patches/chat.rb, execute_tool), so its own switch is not enough.
# Exact span-attribute contract for the retained legacy regression fixture, not an
# installable integration. Never trust entire gen_ai namespaces: custom attributes
# can put prompts or credentials under otherwise plausible request/response keys.
# Values of allowed attributes, span names and resource attributes remain caller-owned;
# this is not a secret scanner. The instrumentation's capture switch alone leaves
# tool arguments, results, descriptions and exception events enabled.
METADATA_ATTRIBUTES = %w[
gen_ai.operation.name
gen_ai.provider.name
gen_ai.request.model
gen_ai.request.stream
gen_ai.request.temperature
gen_ai.response.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cache_creation.input_tokens
gen_ai.usage.cache_read.input_tokens
gen_ai.tool.name
gen_ai.tool.call.id
gen_ai.tool.type
Expand All @@ -39,7 +47,6 @@ module FutureAGITracing
session.id
user.id
].freeze
METADATA_PREFIXES = %w[gen_ai.request. gen_ai.response. gen_ai.usage.].freeze

# Wraps an exporter. Always drops span events (the instrumentation records exception
# messages and stack traces there, which contain local paths and raw provider text). Unless
Expand All @@ -62,7 +69,7 @@ def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout)
private

def metadata?(key)
METADATA_ATTRIBUTES.include?(key) || METADATA_PREFIXES.any? { |prefix| key.start_with?(prefix) }
METADATA_ATTRIBUTES.include?(key)
end

def sanitize(span_data)
Expand Down
56 changes: 56 additions & 0 deletions examples/frameworks/ruby-llm/test/test_paused.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# frozen_string_literal: true

# Separate process: a restart must remove the old patch, not reinstall it.
require "bundler/setup"
require "minitest/autorun"
require "webmock/minitest"
require "json"
require "ruby_llm"
require "opentelemetry/sdk"

WebMock.disable_net_connect!

class PausedLegacyTest < Minitest::Test
def test_documented_session_runs_without_the_instrumentation_or_exports
page = File.read(File.expand_path("../../../../src/pages/docs/integrations/frameworks/ruby-llm.mdx", __dir__))
snippet = page[/```ruby title="legacy-paused.rb"\n(.*?)```/m, 1]
refute_nil snippet, "The pause procedure needs a runnable replacement for the old session chain"

# A shared provider can remain active for unrelated integrations.
exporter = OpenTelemetry::SDK::Trace::Export::InMemorySpanExporter.new
OpenTelemetry::SDK.configure do |config|
config.add_span_processor(OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(exporter))
end
ENV["OPENAI_API_KEY"] = "synthetic-openai-key"
requests = []
stub_request(:post, "https://api.openai.com/v1/chat/completions").to_return do |request|
requests << JSON.parse(request.body)
{
status: 200, headers: { "Content-Type" => "application/json" },
body: JSON.generate(
id: "chatcmpl-paused", object: "chat.completion", created: 1, model: "gpt-4o-mini",
choices: [{ index: 0, message: { role: "assistant", content: "Hello" }, finish_reason: "stop" }],
usage: { prompt_tokens: 3, completion_tokens: 1, total_tokens: 4 }
)
}
end

refute RubyLLM::Chat.method_defined?(:with_otel_attributes)
eval(snippet, TOPLEVEL_BINDING, "legacy-paused.rb")
assert_equal 2, requests.size
assert_equal ["user"], requests.first.fetch("messages").map { |message| message.fetch("role") }
assert_equal ["user", "assistant", "user"], requests.last.fetch("messages").map { |message| message.fetch("role") }
OpenTelemetry.tracer_provider.force_flush
assert_empty exporter.finished_spans, "Paused RubyLLM must not emit spans to the shared provider"
refute defined?(FutureAGITracing), "The legacy helper must not be loaded"
refute RubyLLM::Chat.method_defined?(:with_otel_attributes), "Do not restore the instrumentation patch"
refute $LOADED_FEATURES.any? { |path| path.include?("opentelemetry-instrumentation-ruby_llm") }

OpenTelemetry.tracer_provider.tracer("other-integration").in_span("unrelated") {}
OpenTelemetry.tracer_provider.force_flush
assert_equal ["unrelated"], exporter.finished_spans.map(&:name)
ensure
OpenTelemetry.tracer_provider.shutdown if exporter
ENV.delete("OPENAI_API_KEY")
end
end
16 changes: 16 additions & 0 deletions examples/frameworks/ruby-llm/test/test_recipe.rb
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,22 @@ def test_unlisted_attributes_are_dropped_by_default
refute attrs.key?("custom.note")
end

def test_metadata_namespaces_do_not_allow_arbitrary_content
FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "ok"))
private_attributes = {
"gen_ai.request.parameters" => JSON.generate(api_key: "SYNTHETIC_PRIVATE_KEY", prompt: "SYNTHETIC_PRIVATE_PROMPT"),
"gen_ai.response.content" => "SYNTHETIC_PRIVATE_RESPONSE",
"gen_ai.usage.details" => "SYNTHETIC_PRIVATE_USAGE",
"gen_ai.request.model.extra" => "SYNTHETIC_PRIVATE_SUFFIX"
}
new_chat.with_otel_attributes(private_attributes).ask("hi")
spans = finished
private_attributes.each_key { |key| refute spans.first.attributes.key?(key), "exported #{key}" }
refute_includes all_text(spans), "SYNTHETIC_PRIVATE"
assert_equal "fake-model", spans.first.attributes["gen_ai.request.model"]
assert_equal 11, spans.first.attributes["gen_ai.usage.input_tokens"]
end

def test_no_credentials_or_paths_on_any_span
FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "ok"))
new_chat.ask("hi")
Expand Down
7 changes: 6 additions & 1 deletion examples/frameworks/ruby-llm/test/test_wire.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@ class WireTest < Minitest::Test
def test_export_request_shape
stub = WebMock.stub_request(:post, "https://fi.example.test/tracer/v1/traces").to_return(status: 200, body: "")
FakeOpenAI.stub_chat(FakeOpenAI.completion(content: "WIREREPLY-MARKER"))
new_chat.ask("WIREPROMPT-MARKER")
new_chat.with_otel_attributes(
"gen_ai.request.parameters" => "SYNTHETIC_PRIVATE_KEY",
"gen_ai.response.content" => "SYNTHETIC_PRIVATE_RESPONSE",
"gen_ai.usage.details" => "SYNTHETIC_PRIVATE_USAGE"
).ask("WIREPROMPT-MARKER")
FutureAGITracing.shutdown
FutureAGITracing.shutdown # second call is a no-op
assert_requested stub, times: 1
Expand All @@ -32,5 +36,6 @@ def test_export_request_shape
assert_equal "chat fake-model", span.name
refute_includes req.body, "WIREPROMPT-MARKER"
refute_includes req.body, "WIREREPLY-MARKER"
refute_includes req.body, "SYNTHETIC_PRIVATE"
end
end
128 changes: 128 additions & 0 deletions scripts/ruby-llm-native.test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ def setup
end
@server.mount_proc('/v1') { |req, res| provider_response(req, res) }
@thread = Thread.new { @server.start }
Thread.pass until @server.status == :Running
@origin = "http://127.0.0.1:#{@server.config[:Port]}"
end

Expand Down Expand Up @@ -193,6 +194,132 @@ def test_stream_and_error_metadata_remain_private
assert_empty error.events
end

# Characterize released behavior rather than silently treating UNSET as success.
# The page-contract test requires the caveat until upstream changes this behavior.
def test_outer_timeout_finishes_span_without_error_metadata
items = run_app(<<~RUBY)
require_relative 'tracing'
require 'timeout'
chat = RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions)
started = false
begin
Timeout.timeout(1) do
chat.ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk|
started = true
sleep 10
end
end
raise 'timeout did not fire'
rescue Timeout::Error
raise 'timed out before stream callback' unless started
end
RUBY
assert_interrupted_span(items)
end

def test_interrupt_finishes_span_without_error_metadata
items = run_app(<<~RUBY)
require_relative 'tracing'
begin
RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions).ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk|
raise Interrupt, 'SYNTHETIC_PRIVATE_INTERRUPTED'
end
raise 'interrupt did not fire'
rescue Interrupt
end
RUBY
assert_interrupted_span(items)
end

def test_killed_worker_finishes_span_without_error_metadata
items = run_app(<<~RUBY)
require_relative 'tracing'
ready = Queue.new
worker = Thread.new do
RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions).ask('SYNTHETIC_PRIVATE_STREAM') do |_chunk|
ready << true
sleep 10
end
end
ready.pop
worker.kill.join
RUBY
assert_interrupted_span(items)
end

def assert_interrupted_span(items)
assert_envelope(items, 'none')
assert_equal 1, spans(items).size
span = spans(items).first
assert_equal :STATUS_CODE_UNSET, span.status.code
refute attributes(span.attributes).key?('error.type')
assert_empty span.events
assert_empty span.status.message
end

def test_cooperative_stream_cancellation_records_error_without_content
items = run_app(<<~RUBY)
require_relative 'tracing'
chat = RubyLLM.chat(model: 'gpt-4o-mini', protocol: :chat_completions)
begin
chat.ask('SYNTHETIC_PRIVATE_STREAM') { |_chunk| chat.cancel }
raise 'cancellation not raised'
rescue RubyLLM::CancelledError
end
RUBY
assert_envelope(items, 'none')
assert_equal 1, spans(items).size
span = spans(items).first
assert_equal :STATUS_CODE_ERROR, span.status.code
assert_equal 'RubyLLM::CancelledError', attributes(span.attributes)['error.type']
assert_empty span.events
assert_empty span.status.message
end

def test_http_timeout_records_error_without_content
timeout_config = snippet('timeouts.rb').sub('config.request_timeout = 30', 'config.request_timeout = 0.05')
items = run_app(<<~RUBY)
require_relative 'tracing'
#{timeout_config}
begin
RubyLLM.chat(model: 'gpt-4o-mini').ask('SYNTHETIC_PRIVATE_HTTP_TIMEOUT')
raise 'HTTP timeout not raised'
rescue Faraday::TimeoutError
end
RUBY
assert_envelope(items, 'none')
assert_equal 1, spans(items).size
span = spans(items).first
assert_equal :STATUS_CODE_ERROR, span.status.code
assert_equal 'Faraday::TimeoutError', attributes(span.attributes)['error.type']
assert_empty span.events
assert_empty span.status.message
end

def test_tool_exception_records_error_without_content
items = run_app(<<~RUBY)
require_relative 'tracing'
class Lookup < RubyLLM::Tool
def execute(value:)
raise ArgumentError, 'SYNTHETIC_PRIVATE_TOOL_ERROR'
end
end
begin
RubyLLM.chat(model: 'gpt-4o-mini').with_tools(Lookup).ask('SYNTHETIC_PRIVATE_PROMPT')
raise 'tool exception not raised'
rescue ArgumentError
end
RUBY
assert_envelope(items, 'none')
assert_equal 2, spans(items).size
tool = spans(items).find { |span| span.name == 'execute_tool lookup' }
refute_nil tool
assert_equal :STATUS_CODE_ERROR, tool.status.code
assert_equal 'ArgumentError', attributes(tool.attributes)['error.type']
assert_empty tool.events
assert_empty tool.status.message
end

def test_exported_payload_round_trips_json_and_protobuf_plain_and_gzip
# Ruby exports protobuf only. JSON is a wire-codec replay, not a claimed Ruby exporter mode.
original = run_app.first[:request]
Expand All @@ -219,6 +346,7 @@ def test_exported_payload_round_trips_json_and_protobuf_plain_and_gzip
def provider_response(req, res)
data = JSON.parse(req.body)
@provider_requests << { path: req.path, data: data }
sleep 0.2 if req.body.include?('SYNTHETIC_PRIVATE_HTTP_TIMEOUT')
res['Content-Type'] = 'application/json'
if req.body.include?('SYNTHETIC_PRIVATE_ERROR')
res.status = 400
Expand Down
Loading
Loading