Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 4 additions & 13 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Roadmap & Next Steps

Status: **v0.1.0 — alpha, release-ready for local/self-hosted use.**
Status: **v0.3.1 — alpha.** CI live at [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Public VPS+gVisor path: [`docs/DEPLOY_GVISOR.md`](docs/DEPLOY_GVISOR.md).

## 🟢 Ready to use right now

Expand All @@ -22,21 +22,12 @@ Status: **v0.1.0 — alpha, release-ready for local/self-hosted use.**
```
Or Docker: `docker compose up -d`.

2. **Enable GitHub Actions CI**
The CI workflow is staged at [`.github-pending/ci.yml`](.github-pending/ci.yml)
because the local `gh` token is missing the `workflow` scope. To enable:
```bash
gh auth refresh -s workflow
mkdir -p .github/workflows
mv .github-pending/ci.yml .github/workflows/ci.yml
rmdir .github-pending
git add .github && git commit -m "ci: enable GitHub Actions"
git push
```
2. **GitHub Actions CI**
CI is live at [`.github/workflows/ci.yml`](.github/workflows/ci.yml).

3. **Publish to PyPI**
- Reserve the name: <https://pypi.org/project/openfindata/>
- Create a release: `git tag v0.1.0 && git push --tags`
- Create a release: `git tag v0.3.1 && git push --tags`
- Add a `release.yml` workflow that runs on tags and publishes via
[trusted publishing](https://docs.pypi.org/trusted-publishers/).

Expand Down
67 changes: 67 additions & 0 deletions deploy/docker-compose.gvisor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# gVisor-hardened openfindata deploy.
# Never attach this service to hermes/wealthuman networks.
# Code mode must stay off (do not set FINDATA_MCP_CODE_MODE).
# Requires Docker runtime runsc (gVisor).
services:
openfindata:
build:
context: ..
dockerfile: Dockerfile
image: openfindata:latest
container_name: openfindata
runtime: runsc
restart: unless-stopped
ports:
- "127.0.0.1:8000:8000"
# gVisor does not reliably use Docker's 127.0.0.11 stub DNS.
# Mount a resolv.conf that points straight at public resolvers.
dns:
- 1.1.1.1
- 8.8.8.8
volumes:
- ./resolv.gvisor.conf:/etc/resolv.conf:ro
read_only: true
tmpfs:
- /tmp:mode=1777
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
mem_limit: 512m
cpus: 1.0
user: "65534:65534"
pids_limit: 256
environment:
FINDATA_RATE_LIMIT_ENABLED: "true"
FINDATA_RATE_LIMIT_DEFAULT: "60/minute;1000/day"
# Explicitly off — do not override via .env on public VPS.
FINDATA_MCP_CODE_MODE: "0"
healthcheck:
test:
- "CMD"
- "python"
- "-c"
- "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/health', timeout=2).status==200 else 1)"
interval: 30s
timeout: 3s
retries: 3
start_period: 5s
logging:
driver: json-file
options:
max-size: "10m"
max-file: "5"
networks:
- openfindata_net
labels:
- traefik.enable=true
- traefik.docker.network=deploy_openfindata_net
# Production: export OPENFINDATA_HOST=api.seudominio.com before up.
- traefik.http.routers.openfindata.rule=Host(`${OPENFINDATA_HOST:-findata.localhost}`)
- traefik.http.routers.openfindata.entrypoints=websecure
- traefik.http.routers.openfindata.tls.certresolver=letsencrypt
- traefik.http.services.openfindata.loadbalancer.server.port=8000

networks:
openfindata_net:
driver: bridge
5 changes: 5 additions & 0 deletions deploy/resolv.gvisor.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Used by deploy/docker-compose.gvisor.yml — gVisor does not reliably
# follow Docker's 127.0.0.11 stub resolver.
nameserver 1.1.1.1
nameserver 8.8.8.8
options edns0
115 changes: 115 additions & 0 deletions docs/DEPLOY_GVISOR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Deploy público com gVisor (VPS)

Guia prático para subir o **Dados Financeiros Abertos** em VPS com runtime
**runsc (gVisor)**, Traefik em host mode e rede isolada.

> Esta VPS **não tem KVM aninhado**. O gVisor em modo **systrap** é a camada de
> sandbox do processo do container, **não** uma segunda VM.

## Pré-requisitos

- Docker Engine com runtime **runsc** instalado
- Traefik já em host mode na monvanti-vps (entrypoints `websecure`, certresolver
`letsencrypt`)
- Domínio apontando para a VPS

### Instalar gVisor / runsc (snippet)

```bash
# Exemplo baseado em release oficial do gVisor (ajuste a arquitetura se preciso)
ARCH=$(uname -m)
URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
curl -fsSL "${URL}/runsc" -o /tmp/runsc
curl -fsSL "${URL}/runsc.sha512" -o /tmp/runsc.sha512
(cd /tmp && sha512sum -c runsc.sha512)
sudo mv /tmp/runsc /usr/local/bin/runsc
sudo chmod 755 /usr/local/bin/runsc

# Registrar o runtime sem sobrescrever o daemon.json existente
sudo /usr/local/bin/runsc install
sudo systemctl reload docker || sudo systemctl restart docker
```

Verifique:

```bash
docker info | grep -i runsc
docker run --rm --runtime=runsc hello-world
```

## Clone / update em `/opt/openfindata`

```bash
sudo mkdir -p /opt
sudo git clone https://github.com/robertoecf/openfindata.git /opt/openfindata
# ou, se já existir:
cd /opt/openfindata && sudo git pull --ff-only
cd /opt/openfindata
```

## Variável de host

```bash
export OPENFINDATA_HOST=seu.dominio
# opcional: persistir em deploy/.env ao lado do compose
```

## Subir o serviço

```bash
cd /opt/openfindata
docker compose -f deploy/docker-compose.gvisor.yml up -d --build
```

O compose publica só em `127.0.0.1:8000` e usa labels Traefik. **Não** anexe esta
rede a stacks hermes/wealthuman. **Não** habilite code mode
(`FINDATA_MCP_CODE_MODE` deve permanecer ausente).

## Smoke checks

```bash
curl -sS http://127.0.0.1:8000/health
curl -sS http://127.0.0.1:8000/stats
curl -sS 'http://127.0.0.1:8000/bcb/series/name/selic?n=3'
# MCP HTTP transport:
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/mcp
```

Pelo domínio (via Traefik):

```bash
curl -sS "https://${OPENFINDATA_HOST}/health"
curl -sS "https://${OPENFINDATA_HOST}/stats"
curl -sS "https://${OPENFINDATA_HOST}/bcb/series/name/selic?n=3"
```

## Checklist de segurança

- [ ] `runtime: runsc` ativo no container
- [ ] publish apenas em loopback (`127.0.0.1:8000`)
- [ ] sem mount de `docker.sock`
- [ ] sem `network_mode: host`
- [ ] code mode desligado (sem `FINDATA_MCP_CODE_MODE`)
- [ ] rede isolada `openfindata_net` (não compartilhada com hermes/wealthuman)
- [ ] limite de memória (`mem_limit: 512m`) e CPU (`cpus: 1.0`)
- [ ] `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true`
- [ ] DNS via `deploy/resolv.gvisor.conf` (gVisor + `127.0.0.11` falha)

## Troubleshooting

```bash
# runtime registrado?
docker info | grep -i runsc

# runtime executa?
docker run --rm --runtime=runsc hello-world

# container e health
docker compose -f deploy/docker-compose.gvisor.yml ps
docker inspect --format '{{.HostConfig.Runtime}}' openfindata
curl -v http://127.0.0.1:8000/health
```

Se o Traefik não rotear, confira `OPENFINDATA_HOST`, se o Traefik enxerga a rede
do container e se o entrypoint `websecure` + `letsencrypt` já estão válidos na
monvanti-vps.
4 changes: 3 additions & 1 deletion docs/DEPLOY_PUBLIC.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Deploy público do Dados Financeiros Abertos no seu PC (WSL + Cloudflare Tunnel)

> **VPS + gVisor:** para deploy em VPS com runtime runsc, veja [`docs/DEPLOY_GVISOR.md`](DEPLOY_GVISOR.md).

> **Meta:** expor o **Dados Financeiros Abertos** como **servidor MCP público** — acessível via
> HTTPS, com TLS, rate limit, e _sem_ abrir porta no roteador nem pagar nada.
>
Expand Down Expand Up @@ -71,7 +73,7 @@ Pronto. Em ~30s:

```bash
curl https://findata.seudominio.com.br/health
# {"status":"ok","version":"0.1.0"}
# {"status":"ok","version":"0.3.1"}

curl https://findata.seudominio.com.br/stats
# { ... uptime, cache, rate limits ... }
Expand Down
5 changes: 3 additions & 2 deletions docs/MCP_SURFACE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MCP surface: curated tools over the REST API

> Status: prototype / design proposal (alpha 0.3.x). Non-breaking: the REST API
> Status: implemented (alpha curated catalog). Non-breaking: the REST API
> is untouched. Implemented in [`src/findata/api/mcp_app.py`](../src/findata/api/mcp_app.py).

## Problem
Expand Down Expand Up @@ -65,7 +65,7 @@ b3_quote b3_cotahist b3_index (B3: 9 → 3)
tesouro_bonds tesouro_siconfi (Tesouro: 6 → 2)
ibge_indicator ibge_ipca_breakdown (IBGE: 4 → 2)
ipea_series ipea_search (IPEA: 4 → 2)
anbima (ANBIMA: 3 → 1)
anbima (ANBIMA: ima|ettj|debentures|tpf)
openfinance_directory (Open Finance: 15 → 1)
basedosdados_search basedosdados_sql (BdD: 7 → 2)
receita_arrecadacao aneel_leiloes susep_empresas
Expand All @@ -85,6 +85,7 @@ findata_run_code (code mode, opt-in)
| `b3_index` | index portfolio + monthly + list | `dataset`, omit `symbol` to list |
| `tesouro_bonds` | bonds list/search/history | `dataset` |
| `tesouro_siconfi` | `rreo`, `rgf`, `entes` | `report` |
| `anbima` | ima, ettj, debentures, tpf | `dataset=ima\|ettj\|debentures\|tpf` |
| `openfinance_directory` | participants/endpoints/resources/roles | `dataset` |

## Tradeoffs
Expand Down
4 changes: 3 additions & 1 deletion docs/SOURCES_AND_ENDPOINTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Para testar interativamente, rode `findata serve` e abra `/api/docs` ou `/redoc`
| Open Finance Brasil | Diretório público, participantes, recursos, JWKS e Portal de Dados | `/openfinance/resources`, `/openfinance/participants`, `/openfinance/endpoints`, `/openfinance/directory/api-resources`, `/openfinance/portal/datasets` | Não para dados públicos |
| B3 | Cotações, COTAHIST oficial, composição teórica e evolução mensal de índices | `/b3/quote/{ticker}`, `/b3/history/{ticker}`, `/b3/quotes`, `/b3/cotahist/year/{year}`, `/b3/indices`, `/b3/indices/{symbol}`, `/b3/indices/{symbol}/monthly` | Não |
| Yahoo Finance | Endpoint experimental de gráfico de preços | `/yahoo/chart/{symbol}` | Não; fonte não oficial |
| ANBIMA | IMA, ETTJ e debêntures via arquivos públicos | `/anbima/ima`, `/anbima/ettj`, `/anbima/debentures` | Não para os arquivos usados |
| ANBIMA | IMA, ETTJ, debêntures e TPF via arquivos públicos | `/anbima/ima`, `/anbima/ettj`, `/anbima/debentures`, `/anbima/tpf` | Não para os arquivos usados |
| Receita Federal | Arrecadação por período, UF e tributo | `/receita/arrecadacao`, `/receita/tributos` | Não |
| ANEEL | Leilões de geração e transmissão | `/aneel/leiloes/geracao`, `/aneel/leiloes/transmissao` | Não |
| SUSEP | Entidades supervisionadas | `/susep/empresas`, `/susep/empresas/search` | Não |
Expand Down Expand Up @@ -101,6 +101,8 @@ projeto.

### ANBIMA

No MCP, a tool `anbima` inclui o seletor `dataset=tpf` além de `ima`, `ettj` e `debentures`.

O módulo atual usa arquivos públicos em `www.anbima.com.br/informacoes/*`
(XLS/CSV/TXT), não a API comercial autenticada Sensedia. Produtos autenticados
futuros devem seguir o padrão de `docs/SOURCES_WITH_AUTH.md` e nunca embutir
Expand Down
42 changes: 42 additions & 0 deletions docs/snapshots/obm_day0_robots_sitemap.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"base_url": "https://obm.com.br",
"robots": {
"status": 200,
"disallow": [
"/api/",
"/ops"
],
"body_preview": "# As a condition of accessing this website, you agree to abide by the following\n# content signals:\n\n# (a) If a Content-Signal = yes, you may collect content for the corresponding\n# use.\n# (b) If a Content-Signal = no, you may not collect content for the\n# corresponding use.\n# (c) If the website operator does not include a Content-Signal for a\n# corresponding use, the website operator neither grants nor restricts\n# permission via Content-Signal with respect to the corresponding use.\n\n# The content signals and their meanings are:\n\n# search: building a search index and providing search results (e.g., returning\n# hyperlinks and short excerpts from your website's contents). Search does not\n# include providing AI-generated search summaries.\n# ai-input"
},
"sitemap": {
"url": "https://obm.com.br/sitemap.xml",
"status": 200,
"url_count": 21,
"by_section": {
"/sitemap": 21
},
"sample": [
"https://obm.com.br/sitemap/static.xml",
"https://obm.com.br/sitemap/funds.xml",
"https://obm.com.br/sitemap/equities.xml",
"https://obm.com.br/sitemap/bdrs.xml",
"https://obm.com.br/sitemap/etfs.xml",
"https://obm.com.br/sitemap/fiis.xml",
"https://obm.com.br/sitemap/treasuries.xml",
"https://obm.com.br/sitemap/debentures.xml",
"https://obm.com.br/sitemap/letras-financeiras.xml",
"https://obm.com.br/sitemap/indices.xml",
"https://obm.com.br/sitemap/crypto.xml",
"https://obm.com.br/sitemap/blog.xml",
"https://obm.com.br/sitemap/glossario.xml",
"https://obm.com.br/sitemap/fidc.xml",
"https://obm.com.br/sitemap/fip.xml",
"https://obm.com.br/sitemap/fiagro.xml",
"https://obm.com.br/sitemap/fi-infra.xml",
"https://obm.com.br/sitemap/cri.xml",
"https://obm.com.br/sitemap/cra.xml",
"https://obm.com.br/sitemap/empresas.xml",
"https://obm.com.br/sitemap/entidades.xml"
]
}
}
Loading
Loading