Proposal: Add FastAPI + Ollama (Local AI/LLM Inference) Compose sample

[VimalN2005]

Description

This Pull Request adds a new application sample: FastAPI + Ollama (Local AI/LLM Inference) under fastapi-ollama/.

Architecture & Features

• Ollama service: Official Ollama container with model volume persistence (ollama_data:/root/.ollama) and optional NVIDIA GPU acceleration config.
• FastAPI backend: Python 3.11 service with connection pooling via lifespan context manager, exposing /generate (streaming & non-streaming), /chat, and /models endpoints.
• Robust Error Handling: Status code validation before streaming chunks and graceful network error handling during generation.
• Documentation: Step-by-step README.md covering deployment, pulling models (llama3.2:1b), testing endpoints, and enabling GPU acceleration.
• Root README: Updated main sample catalog.

Signed-off-by: Vimal Sahani vimalsahani2005@gmail.com

[stewartmbofana]

Thanks for contributing this sample! Having an Ollama + FastAPI sample in awesome-compose is a great addition for developers looking to run local LLM inference with Docker Compose.

Before this can be merged, please address the following changes:

1. Remove UTF-8 BOM (Byte Order Mark) from files

Several files have been saved with a UTF-8 BOM (\xef\xbb\xbf):

• fastapi-ollama/Dockerfile
• fastapi-ollama/compose.yaml
• fastapi-ollama/requirements.txt
• fastapi-ollama/app/main.py
• fastapi-ollama/README.md

In particular, the BOM at the beginning of fastapi-ollama/Dockerfile causes Docker BuildKit to fail to recognize the # syntax=docker/dockerfile:1.4 directive, because parser directives must strictly start at byte 0. Please re-save all files as standard UTF-8 (without BOM).

2. Remove --no-cache-dir when using pip cache mount in Dockerfile

In fastapi-ollama/Dockerfile:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

Using --mount=type=cache,target=/root/.cache/pip alongside pip install --no-cache-dir is contradictory because --no-cache-dir instructs pip to bypass the cache directory entirely. Please remove --no-cache-dir so pip can utilize the cache mount:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

3. Streaming error handling & client lifecycle in app/main.py

• Streaming exception handling: In /generate and /chat, the try...except httpx.RequestError block does not catch connection or network failures during streaming because stream_generator() is consumed asynchronously by Starlette after the route returns.
• Status code check on stream: If Ollama returns a non-200 status code (e.g. 404 when a model has not yet been pulled), client.stream does not raise an exception, and the error response body is streamed under an HTTP 200 response. Consider verifying response.status_code == 200 before yielding chunks.
• Client lifecycle: Rather than creating a new httpx.AsyncClient inside each request, consider managing a shared client via FastAPI's lifespan context manager (@asynccontextmanager async def lifespan(app: FastAPI): ...) to enable HTTP connection pooling and proper cleanup.

4. GPU Acceleration Note

Most users running Ollama in Docker will be interested in GPU pass-through. It would be very helpful to add a note or commented-out configuration in compose.yaml and fastapi-ollama/README.md showing how to enable Nvidia GPU support (e.g., via deploy.resources.reservations.devices).

5. PR Title & Description

Please update the PR title and description to reflect that this is an implemented Pull Request ready for review rather than an issue proposal.

[VimalN2005]

Hi @stewartmbofana, thank you for the helpful review!

I have addressed all the requested changes in the latest commit:

1. Removed UTF-8 BOM: Re-saved all files as standard UTF-8 without BOM so Docker BuildKit recognizes # syntax=docker/dockerfile:1.4 properly.
2. Removed --no-cache-dir: Updated the Dockerfile to allow pip to utilize the cache mount.
3. Streaming & Client Lifecycle in app/main.py:
   • Implemented FastAPI lifespan context manager to manage a shared httpx.AsyncClient with connection pooling.
   • Added pre-stream status code check to raise proper HTTPException if Ollama returns a non-200 response.
   • Wrapped the streaming generator in a try...except httpx.RequestError block with finally: await response.aclose().
4. GPU Acceleration: Added a commented deploy.resources.reservations.devices configuration in compose.yaml and documented the setup in fastapi-ollama/README.md.
5. PR Title & Description: Updated the title and description to reflect the ready-for-review implementation.

Please let me know if any further adjustments are needed!

[VimalN2005]

Hi @stewartmbofana, friendly ping! All requested changes (BOM removal, streaming lifecycle, GPU notes) were pushed and CI is clean. Could you please take a re-look when you have a moment? Thank you!