VigiCall backend

FastAPI backend for receiving ESP32 PCM audio, running real-time speech and fraud-analysis pipelines, persisting results, and broadcasting them to React Native clients.

See the asynchronous architecture for the complete pipeline.

Prerequisites

• Python 3.11
• The fine-tuned local model artifacts listed below
• Node.js 18+ for the React Native client

Backend setup and run

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
Copy-Item .env.example .env

If the Python launcher has not refreshed after installation, create the environment with the full executable path:

& "$env:LOCALAPPDATA\Programs\Python\Python311\python.exe" -m venv .venv

Before starting, place your trained artifacts at the configured .env paths:

models/
  muril_scam_classifier/  # Hugging Face config, tokenizer, weights; id2label contains Scam
  wav2vec2_ser/           # Wav2Vec2 config, feature extractor, weights; labels: Neutral, Calm, Happy, Angry, Fear
  user_voice_profile.pt   # Enrolled ECAPA voiceprint used by final speaker identification

For final speaker analysis, set HF_TOKEN in .env (or provide it through the backend process environment). The token is never logged and must be authorized for pyannote/speaker-diarization-community-1.

Start FastAPI:

.\.venv\Scripts\python.exe -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Open http://127.0.0.1:8000/docs for the API documentation. Use your computer's LAN IP, rather than localhost, for the ESP32 and physical mobile device.

Mobile app

cd mobile
npm.cmd install
npx expo start

Set the FastAPI LAN URL in the application’s Settings screen, for example http://192.168.1.100:8000.

ESP32 streamer

The currently used live-call sketch is esp32/VigiCall_ESP32/VigiCall_ESP32/VigiCall_ESP32.ino. It streams 16 kHz mono PCM to 192.168.0.100:8000 and accepts runtime serial commands: id <UUID>, start, stop, end, status, and help.

firmware/esp32_audio_streamer/ is a separate configuration-header/PlatformIO implementation. It uses a compile-time session UUID and does not provide those runtime serial session commands; retain it only if that alternate workflow is intentional.

Call flow

1. Create a session: POST /api/v1/sessions.
2. Connect React Native to /ws/clients/{session_id}.
3. Configure the same UUID in the ESP32 and let it connect to /ws/audio/{session_id}.
4. The backend emits transcript, rule evaluation, MuRIL, emotion, combined risk, and final speaker-analysis events.
5. End the session: POST /api/v1/sessions/{session_id}/end.

With default settings, the ESP32’s 16 kHz mono PCM stream is grouped into five-second windows with a one-second overlap. Faster-Whisper, MuRIL, and Wav2Vec2 SER load once at application startup and only run inference. Emotion analysis is independent and never affects scam risk. After a call ends, PyAnnote diarization and ECAPA speaker identification run against the finalized WAV and persist speaker segments.