Email Generation Assistant - AI Engineer Candidate Assessment

A production-ready email generation system that produces professional, contextually-appropriate emails using advanced LLM prompt engineering. Includes comprehensive evaluation metrics for measuring email quality.

📋 Project Overview

Goals

1. Email Generation: Build an assistant that generates professional emails from Intent, Facts, and Tone
2. Custom Metrics: Implement 3 custom evaluation metrics tailored to email quality assessment
3. Model Comparison: Compare two prompting strategies (Advanced vs. Baseline) using the same metrics

Deliverables

• ✅ Working email generation assistant with advanced prompt engineering
• ✅ 3 custom evaluation metrics with LLM-as-Judge evaluation
• ✅ 10 test scenarios with reference emails
• ✅ Structured evaluation results (JSON, CSV)
• ✅ Comparative analysis report
• ✅ Complete code repository with documentation

🏗️ Project Structure

email-generation-assistant/
├── src/
│   ├── email_generator.py      # Email generation with 2 strategies
│   ├── evaluator.py            # Main evaluation orchestrator
│   └── config.py               # Configuration management
├── data/
│   └── test_scenarios.py        # 10 test scenarios with reference emails
├── metrics/
│   └── evaluation_metrics.py    # 3 custom metrics implementation
├── output/
│   ├── evaluation_results.json  # Raw evaluation data
│   ├── evaluation_summary.csv   # Metric scores in CSV format
│   ├── analysis_report.md       # Comparative analysis
│   └── METRICS_DOCUMENTATION.md # Metric definitions
├── requirements.txt
├── .env.example
└── README.md

🚀 Quick Start

1. Setup

# Clone repository
git clone <repo-url>
cd email-generation-assistant

# Install dependencies
pip install -r requirements.txt

# Configure API keys
cp .env.example .env
# Edit .env with your API keys:
# OPENAI_API_KEY=your_key_here
# GEMINI_API_KEY=your_key_here (optional)

2. Run Evaluation

# Full evaluation with all 10 scenarios
python3 src/evaluator.py --api openai

# Using Gemini API
python3 src/evaluator.py --api gemini

3. View Results

Results are automatically generated in output/:

• evaluation_results.json - Complete raw scores
• evaluation_summary.csv - Summary table
• analysis_report.md - Model comparison analysis
• METRICS_DOCUMENTATION.md - Metric definitions

📊 The 3 Custom Metrics

Metric 1: Fact Incorporation & Specificity (0-100)

Purpose: Measure how well required facts are included and naturally integrated

Sub-components:

1. Fact Coverage: Are all key facts present? (0-100)
2. Natural Integration: Do facts flow naturally or are they forced? (0-100)
3. Specificity: Is the detail level appropriate? (0-100)

Evaluation: LLM-as-Judge assesses each component, returns average score

---

Metric 2: Tone Consistency & Appropriateness (0-100)

Purpose: Evaluate if the email matches the specified tone

Sub-components:

1. Tone Match: Does overall tone match specified style? (0-100)
2. Vocabulary Appropriateness: Are word choices appropriate? (0-100)
3. Emotional Resonance: Does it evoke the right emotional response? (0-100)

Evaluation: LLM-as-Judge calibrated against reference emails

---

Metric 3: Clarity & Effectiveness (0-100)

Purpose: Measure clarity, structure, and likelihood to achieve goal

Sub-components:

1. Clarity & Structure: Is the email well-organized? (0-100)
2. Actionability: Is there a clear call-to-action? (0-100)
3. Professional Quality: Free of errors and professional? (0-100)

Evaluation: LLM-as-Judge assesses business communication effectiveness

---

Overall Score: (Metric1 + Metric2 + Metric3) / 3 = 0-100

💡 Email Generation Strategies

Strategy A: Advanced Prompting

Combines three advanced techniques:

1. Role-Playing: Position generator as expert professional
2. Few-Shot Examples: Show 2 example input-output pairs
3. Chain-of-Thought: Request step-by-step reasoning

Result: Higher quality, more consistent outputs

Strategy B: Simple Prompting

Basic prompt without advanced techniques - serves as baseline for comparison

Result: Faster generation but lower quality

---

📝 Test Data: 10 Email Scenarios

Each scenario includes Intent, Facts, Tone, and Reference Email:

1. Follow up after client consultation (professional & warm)
2. Request urgent action on stalled project (urgent & direct)
3. Congratulate colleague on promotion (genuine & encouraging)
4. Request proposal and pricing (formal & business-like)
5. Apologize for missed deadline (apologetic & solution-focused)
6. Introduce new team member (informative & welcoming)
7. Request budget approval for tools (persuasive & data-driven)
8. Follow up after job interview (professional & enthusiastic)
9. Decline partnership offer (respectful & diplomatic)
10. Request project status update (concerned & analytical)

---

📊 Output Files

1. evaluation_results.json

Complete evaluation data for all 20 results (10 scenarios × 2 strategies)

[
  {
    "scenario_id": 1,
    "strategy": "A",
    "intent": "Follow up after...",
    "metric_1_fact_incorporation": 87.5,
    "metric_2_tone_consistency": 91.2,
    "metric_3_clarity_effectiveness": 89.3,
    "average_score": 89.33
  }
]

2. evaluation_summary.csv

Tabular format for easy analysis

scenario_id,strategy,metric_1,metric_2,metric_3,average_score
1,A,87.5,91.2,89.3,89.33
1,B,72.1,75.8,71.2,73.03
...

3. analysis_report.md

Comparative analysis with key findings and recommendations

4. METRICS_DOCUMENTATION.md

Complete metric definitions and evaluation methodology

---

🔧 API Configuration

OpenAI

OPENAI_API_KEY=sk-...

• Model: gpt-3.5-turbo
• Used for both generation and evaluation

Gemini

GEMINI_API_KEY=AIza...

• Model: gemini-pro
• Requires API key with Generative AI enabled

---

📈 Model Comparison

Run evaluation to get comparison:

python3 src/evaluator.py --api openai

Generates analysis showing:

• ✓ Strategy A vs Strategy B performance
• ✓ Metric-by-metric breakdown
• ✓ Failure mode analysis
• ✓ Production recommendation

---

🎯 Key Findings

The evaluation framework shows:

• Strategy A (Advanced): Consistently higher scores across all metrics
• Strategy B (Simple): Baseline comparison, useful for cost/speed tradeoff
• Best Metric: Varies by language model and scenario
• Production Recommendation: Strategy A for quality, Strategy B for speed

---

📝 Prompt Template

system_prompt = f"""
You are an expert professional email ghostwriter with 15+ years of experience.

[Few-Shot Examples showing 2 examples]

INSTRUCTIONS:
Think step-by-step following Chain-of-Thought:
1. Identify core message and desired action
2. Determine best email structure
3. Select vocabulary matching tone
4. Integrate all facts seamlessly
5. Review for clarity and consistency

Generate email for:
Intent: {intent}
Facts: {facts}
Tone: {tone}
"""

---

🛠️ Technical Stack

• Python 3.9+
• LLM APIs: OpenAI (GPT-3.5-turbo), Google Gemini
• Libraries:
  • openai - OpenAI API client
  • google-generativeai - Gemini API client
  • python-dotenv - Environment configuration
  • pandas - Data analysis
  • textblob - Text analysis

---

📜 Dependencies

See requirements.txt for complete list:

openai==1.3.0
google-generativeai>=0.4.0
python-dotenv==1.0.0
pandas==2.0.3
textblob==0.17.1
nltk==3.8.1
requests==2.31.0

---

🚦 Running the Full Pipeline

# 1. Setup
pip install -r requirements.txt
cp .env.example .env  # Add API keys

# 2. Run evaluation
python3 src/evaluator.py --api openai

# 3. Check results
cat output/analysis_report.md
cat output/evaluation_summary.csv

# 4. Review detailed metrics
cat output/METRICS_DOCUMENTATION.md

---

📄 Output Format

All outputs are in structured, production-ready formats:

• JSON: For programmatic analysis and integration
• CSV: For spreadsheet analysis and visualization
• Markdown: For human-readable reports and documentation

---

✅ Assessment Criteria Met

• Working email generation assistant
• 3 custom metrics with clear definitions
• 10 test scenarios with reference emails
• Structured evaluation results
• Model comparison analysis
• Advanced prompt engineering (Role-Playing + Few-Shot + CoT)
• LLM-as-Judge evaluation methodology
• Production-ready code
• Complete documentation

---

📚 Additional Resources

• output/METRICS_DOCUMENTATION.md - Detailed metric explanations
• output/analysis_report.md - Comparative analysis results
• src/email_generator.py - Generation implementation
• metrics/evaluation_metrics.py - Metric implementation
• data/test_scenarios.py - Test data

---

🔍 File Reference

Core Implementation:

• src/email_generator.py - Email generation with Strategy A & B
• src/evaluator.py - Evaluation orchestrator
• metrics/evaluation_metrics.py - Custom metrics (all 3)
• data/test_scenarios.py - 10 test scenarios

Configuration & Data:

• requirements.txt - Python dependencies
• .env.example - API key template
• output/ - Evaluation results

---

📞 Support

For issues or questions:

1. Check output/METRICS_DOCUMENTATION.md for metric details
2. Review output/analysis_report.md for results interpretation
3. See src/evaluator.py for execution details

---

Project Status: ✅ Complete and Production-Ready ├── .env.example # Template for environment variables └── README.md # This file

## Setup Instructions

### 1. Install Dependencies

```bash
cd "/Users/apple/Desktop/chat assistant"
pip install -r requirements.txt

2. Configure API Key

# Copy the example file
cp .env.example .env

# Edit .env and add your OpenAI API key
# OPENAI_API_KEY=sk-...

3. Run the Evaluation

python src/evaluator.py

This will:

1. Generate emails for all 10 scenarios using Strategy A (advanced prompting)
2. Generate emails for all 10 scenarios using Strategy B (simple prompting)
3. Evaluate each email against the three custom metrics
4. Output results to CSV, JSON, and markdown report files

Test Scenarios (10 Total)

1. Follow up after client consultation
2. Request urgent action on stalled project
3. Congratulate colleague on promotion
4. Request proposal and pricing information
5. Apologize for missed deadline with remediation plan
6. Introduce new team member to stakeholders
7. Request budget approval for software tools
8. Follow up on job interview
9. Professionally decline partnership offer
10. Request detailed project status and risk assessment

Output Files

After running the evaluation, you'll find:

• evaluation_results_YYYYMMDD_HHMMSS.json - Full results with generated emails
• evaluation_summary_YYYYMMDD_HHMMSS.csv - Metrics summary (easy for analysis)
• analysis_report_YYYYMMDD_HHMMSS.md - Comparative analysis and recommendations
• METRICS_DOCUMENTATION.md - Detailed explanation of the three metrics

Key Features

Advanced Prompt Engineering

The email generator implements three advanced techniques:

1. Role-Playing

"You are an expert professional email ghostwriter with 15+ years of experience..."

2. Few-Shot Examples Shows the model example inputs and outputs to guide generation

3. Chain-of-Thought

"Think through this step-by-step:
1. Identify the core message
2. Determine the best structure
3. Select appropriate vocabulary
4. Ensure all facts are included
5. Review for clarity and tone"

Custom Evaluation Metrics

Each metric uses LLM-as-a-Judge to provide nuanced evaluation:

• Not just simple keyword matching
• Considers natural integration and contextual appropriateness
• Compares against human-written reference emails
• Evaluates sub-components to provide detailed feedback

Customization

Adding New Test Scenarios

Edit data/test_scenarios.py and add new scenarios to the TEST_SCENARIOS list.

Changing the Model

Edit src/email_generator.py and update the model parameter:

generator = EmailGenerator(model="gpt-4")  # Use different model

Adjusting Metrics

Modify the criteria in metrics/evaluation_metrics.py to focus on different aspects of email quality.

Example Usage

from src.email_generator import generate_email

# Strategy A: Advanced prompting
email = generate_email(
    intent="Follow up after design review meeting",
    facts=[
        "Discussed new dashboard mockups",
        "Team agreed on implementation timeline of 3 weeks",
        "Budget approved for contractors"
    ],
    tone="professional and collaborative",
    strategy="A"
)

print(email)

Architecture Decisions

Why These Three Metrics?

• Fact Incorporation: Ensures the assistant remembers and includes all requirements
• Tone Consistency: Validates that email matches the intended communication style
• Clarity & Effectiveness: Confirms emails will achieve their business goals

Why Two Strategies?

• Strategy A (Advanced): Shows the power of well-engineered prompts
• Strategy B (Simple): Provides a baseline to demonstrate improvements

Why LLM-as-Judge?

• More nuanced than keyword matching or simple heuristics
• Can evaluate semantic appropriateness, not just surface features
• Handles the subjective nature of email quality assessment

Evaluation Methodology

1. Generate: Create emails for each scenario with both strategies
2. Evaluate: Score each email against three metrics
3. Compare: Analyze differences between strategies
4. Recommend: Identify best approach for production use

Metric Scoring

• Each metric scored 0-100
• Sub-components averaged together
• Overall score = (Metric1 + Metric2 + Metric3) / 3
• Higher scores indicate better performance

Results Interpretation

Expected Outcomes

• Strategy A typically outperforms Strategy B by 10-20 points
• Best improvements in Metric 2 (Tone Consistency) and Metric 3 (Clarity)
• Strategy A more consistent across different scenarios

Analyzing Your Results

1. Check analysis_report_*.md for summary findings
2. Review evaluation_summary_*.csv for scenario-level performance
3. Study generated emails in evaluation_results_*.json for qualitative feedback
4. Focus on low-scoring scenarios to understand failure modes

Troubleshooting

API Key Issues

Error: OpenAI API key not found
Solution: Check .env file has OPENAI_API_KEY set correctly

Rate Limiting

Error: Rate limit exceeded
Solution: Add delays between requests or use a higher API tier

Model Not Available

Error: Model 'gpt-4' not found
Solution: Ensure you have access to that model or use 'gpt-3.5-turbo'

System Requirements

• Python 3.8+
• OpenAI API key (with available credits)
• ~5-10 minutes runtime for full evaluation
• ~100 API calls total (generates + evaluates 20 emails)

License

This project is provided as-is for educational and demonstration purposes.

Questions or Issues?

Review the comments in the source code for detailed explanations of implementation decisions.