encoder-layer — Transformer Encoder Block (Pure C)

Part 10 of: Transformer from Scratch in Pure C

Goal: Implement the full "Attention is All You Need" paper in pure C, building up from primitives to a complete machine translation pipeline.

encoder-layer marks a major architectural milestone. It introduces almost zero new mathematics. Instead, it assembles previously validated sub-layers into the canonical Encoder Block.

---

Architecture Context

This project implements the exact Encoder Block architecture described in the original 2017 paper:

                 Input
                   │
                   ▼
        Multi-Head Attention
                   │
             + Residual
                   │
                   ▼
             LayerNorm
                   │
                   ▼
         Feed Forward Network
                   │
             + Residual
                   │
                   ▼
             LayerNorm
                   │
                   ▼
                 Output

---

Highlights

• Pure C (C99), builds directly upon six previous projects.
• Composition over Reimplementation — Reuses MultiHeadAttention, LayerNorm, and FeedForward exactly as implemented in earlier projects.
• Post-LayerNorm — Faithfully follows the original 2017 Transformer architecture pipeline.
• Residual Connections — Wraps attention and feed-forward sub-layers with additive identity bypasses.
• 11-test suite validating end-to-end memory safety, component reuse, and normalization statistics.
• Built on mini-tensor, multi-head-attention, layernorm, & feed-forward ← | Foundation for encoder-stack →

---

The Composition Philosophy

Every module in this repository was built from scratch in previous projects. Project 10 proves that these standalone components compose perfectly.

Primitive	Source Project
Matrix library	01
Linear layer	02
Multi-Head Attention	07
LayerNorm	08
Feed Forward	09
Residual / Encoder Layer	10 (This Project)

---

What encoder-layer provides

A single public header (include/encoder_layer.h) exposes:

Encoder Layer

typedef struct {
    MultiHeadAttention *mha;   /* Project 07 */
    LayerNorm          *ln1;   /* Project 08 */
    FeedForward        *ffn;   /* Project 09 */
    LayerNorm          *ln2;   /* Project 08 */

    size_t d_model;
    size_t num_heads;
    size_t d_ff;
} EncoderLayer;

EncoderLayer* encoder_layer_create(size_t d_model, size_t num_heads, size_t d_ff);
Matrix*       encoder_layer_forward(EncoderLayer *encoder, const Matrix *input);
size_t        encoder_layer_num_parameters(const EncoderLayer *encoder);
void          encoder_layer_free(EncoderLayer *encoder);

---

Example usage

#include "encoder_layer.h"

int main(void) {
    size_t d_model = 512;
    size_t heads = 8;
    size_t d_ff = 2048; 
    size_t seq_len = 10;

    /* Create the full Encoder Layer */
    EncoderLayer *enc = encoder_layer_create(d_model, heads, d_ff);

    /* Embeddings with positional encodings */
    Matrix *input = matrix_create(seq_len, d_model);

    /* Run the complete self-attention + FFN pipeline */
    Matrix *output = encoder_layer_forward(enc, input);

    matrix_free(output);
    matrix_free(input);
    encoder_layer_free(enc);
    
    return 0;
}

---

Directory structure

encoder-layer/
├── include/
│   └── encoder_layer.h      # Public API
├── src/
│   ├── encoder_layer.c      # Pure composition orchestrator
│   └── demo.c               # Visual pipeline demo
├── tests/
│   └── test_encoder_layer.c # 11-test suite
├── Makefile                 # Links 6 previous projects
├── README.md
└── Analysis.md

---

Build & Run

Requirements

• GCC (C99 compatible)
• GNU Make
• libm (math library)
• Six previous projects: mini-tensor, neural-net, attention, multi-head-attention, layernorm, and feed-forward must be adjacent in the directory tree.

Build

make

Run tests

make test

Run visual demo

make demo

---

Pipeline Demonstration Output

=========================================
  Encoder Layer — Architecture Pipeline
=========================================

--- Model Details ---
d_model : 8
heads   : 2
d_ff    : 32

Multi-Head Attention : 256
LayerNorm 1          : 16
Feed-Forward         : 552
LayerNorm 2          : 16
--------------------------------
Total Parameters     : 840

--- Tracing Token 0 Through the Pipeline ---

Input (3x8)          : [  -5.00  -4.58  -4.17  -3.75  -3.33  -2.92  -2.50  -2.08 ]
After MHA (3x8)      : [   0.31  -2.94   3.92   0.74   2.51  -3.24  -6.77   0.85 ]
Residual #1 (3x8)    : [  -4.69  -7.52  -0.25  -3.01  -0.83  -6.16  -9.27  -1.23 ]
LayerNorm #1 (3x8)   : [  -0.18  -1.09   1.24   0.36   1.05  -0.65  -1.65   0.92 ]
  -> LN1 Token 0 Stats : μ ≈   0.00, σ² ≈   1.00
Feed Forward (3x8)   : [   0.08   0.52   0.17   0.60   0.32   1.01  -0.99  -0.15 ]
Residual #2 (3x8)    : [  -0.11  -0.57   1.41   0.95   1.37   0.36  -2.64   0.77 ]
LayerNorm #2 (3x8)   : [  -0.24  -0.61   0.97   0.61   0.94   0.13  -2.27   0.46 ]
  -> LN2 Token 0 Stats : μ ≈   0.00, σ² ≈   1.00

Encoder Block output successfully normalized and ready for stacking!

---

Test suite

=== Encoder Layer Test Suite ===
Running test_encoder_create...
  Pass.
Running test_forward_shape...
  Pass.
Running test_residual_add...
  Pass.
Running test_residual_identity...
  Pass.
Running test_layernorm_after_attention...
  Pass.
Running test_parameter_count...
  Pass.
Running test_invalid_configuration...
  (Expect error messages below)
[encoder-layer] ERROR: dimensions must be > 0.
[encoder-layer] ERROR: dimensions must be > 0.
[encoder-layer] ERROR: dimensions must be > 0.
[encoder-layer] ERROR: d_model (9) must be divisible by num_heads (2).
  Pass.
Running test_output_not_null...
  Pass.
Running test_sequence_length_preserved...
  Pass.
Running test_token_dimension_preserved...
  Pass.
Running test_component_reuse...
  Pass.
All 11 tests passed!

---

What's explicitly NOT in this project

Missing feature	Why	Where it belongs
Encoder Stack	This repository strictly implements a single block	Part 11 — Encoder Stack
Embeddings	Input is assumed to already be embedded	Part 04 / Part 05
Decoder	Focus is solely on the Encoder half of the architecture	Part 12 — Decoder Layer

---

Engineering Design © 2026 Shahid Ul Islam.
Built with passion for Mathematical Rigor and Technical Excellence.