Repository Guide | English | 한국어
Last Updated: 2026-11-19
Introduction to Algorithms, Cormen, Leiserson, Rivest, and Stein, Ch 2, 3, 4
Learning Objectives:
- State the formal properties of an algorithm and trace through the Euclidean GCD algorithm
- Count basic operations in iterative code and express the result as T(n)
- Distinguish between best-case, worst-case, and average-case analysis
- Explain why asymptotic notation is preferred over exact operation counts, and define Big-O, Big-Omega, and Big-Theta notation and prove membership using the formal definition
- Solve simple recurrence relations using repeated substitution and the guess-and-verify method
- Apply the Master Theorem to determine the time complexity of divide-and-conquer recurrences
- 1. Algorithm Basics
- 2. Time Complexity
- 3. Asymptotic Notation
- 4. Recurrence Relations
- Summary
- Self-Check Questions
- An algorithm is a step-by-step procedure or method for solving a problem.
- The problems covered in this course are solved using computers.
- An algorithm takes input and produces output (a solution).
flowchart LR
I["Input"] --> A["Algorithm"]
A --> O["Output"]
Figure 1. An algorithm taking input and producing output
| Property | Description |
|---|---|
| Correctness | Must produce the correct output for all valid inputs (same input → same output) |
| Executability | Each step must be executable on a computer |
| Finiteness | Must terminate within a finite amount of time |
| Efficiency | The more efficient in time and space, the more valuable |
Table 1. Four properties of algorithms
Key Point: Finiteness is especially important. A program stuck in an infinite loop is not an algorithm. For example,
while True: passis not an algorithm. In contrast, the Euclidean GCD always terminates because b decreases with each step and must eventually reach 0.
It is one of the oldest known algorithms, described around 300 BCE.
- The Greatest Common Divisor (GCD) is the largest number that divides both natural numbers.
- The key insight is GCD(A, B) = GCD(B, A mod B).
Mathematical Basis:
Given
Here,
Repeatedly applying this reduces
Proof: Since
Pseudocode:
Euclid(a, b)
Input: integers a, b (a >= b >= 0)
Output: GCD(a, b)
if b == 0
return a
return Euclid(b, a mod b)
Trace: GCD(24, 14)
| Call | a | b | a mod b |
|---|---|---|---|
| 1 | 24 | 14 | 10 |
| 2 | 14 | 10 | 4 |
| 3 | 10 | 4 | 2 |
| 4 | 4 | 2 | 0 |
| 5 | 2 | 0 | (return 2) |
Table 2. Call-by-call trace of GCD(24, 14)
Iterative Version:
def gcd_iter(a, b):
while b != 0:
a, b = b, a % b
return aRecursive Version:
def gcd_rec(a, b):
if b == 0:
return a
return gcd_rec(b, a % b)Both versions produce the same result. Recursion uses the call stack, where each call pushes a new frame. Iteration avoids stack overhead.
[Data Structures] Recursion is when a function calls itself, and every recursive function consists of a base case, which stops the recursion, and a recursive step, which reduces the problem to a smaller size and calls the function itself. In the GCD function above,
b == 0is the base case andgcd_rec(b, a % b)is the recursive step, and since the second argument shrinks with every call, the base case is always reached and the recursion ends.
Recursive Call Stack Trace:
gcd_rec(24, 14)
└─► gcd_rec(14, 10)
└─► gcd_rec(10, 4)
└─► gcd_rec(4, 2)
└─► gcd_rec(2, 0) → returns 2
returns 2
returns 2
returns 2
returns 2
Figure 2. Recursive calls and returns of gcd_rec(24, 14)
Each recursive call pushes a frame onto the stack. When b == 0, the value is popped back up.
[Data Structures] A stack is a LIFO (Last In, First Out) data structure. Program function calls are also managed using a stack structure: a function call pushes, and a return pops. If recursion goes too deep, a Stack Overflow can occur. Python's default recursion depth limit is 1000.
An algorithm is a step-by-step procedure, similar to a cooking recipe. There are three common ways to represent one.
| Method | Description |
|---|---|
| Natural Language | Describe each step in words |
| Pseudocode | A structured notation similar to programming languages (most common) |
| Flowchart | A visual diagram with shapes and arrows |
Table 3. Three ways to represent an algorithm
Example: Finding the Maximum
Problem: Find the largest number among
Figure 3. Comparing and arranging cards in hand
Natural Language:
- Read the first card's number and remember it.
- Read the next card and compare it with the remembered number.
- Keep the larger number in memory.
- If cards remain, go to step 2.
- The remembered number is the maximum.
Pseudocode:
FindMax(A[], n)
max = A[0]
for i = 1 to n-1
if A[i] > max
max = A[i]
return max
Pseudocode is not a specific programming language but a conventional notation for clearly expressing the logic of an algorithm. The CLRS textbook writes algorithms in its own pseudocode style.
Flowchart:
flowchart TD
S(["Start"]) --> I["max = A[0], i = 1"]
I --> C1{"i < n ?"}
C1 -->|"No"| R["return max"]
C1 -->|"Yes"| C2{"A[i] > max ?"}
C2 -->|"Yes"| U["max = A[i]"]
C2 -->|"No"| N["i = i + 1"]
U --> N
N --> C1
Figure 4. Flowchart for finding the maximum
Flowcharts are intuitive for simple algorithms but become complex for large programs, so pseudocode is predominantly used in modern textbooks.
The classification below gives an overview of algorithm design strategies; divide and conquer is covered in detail in Week 4, greedy algorithms in Week 5, dynamic programming in Week 6, and approximation algorithms in Weeks 13-14.
By problem-solving strategy, algorithms fall into the following six major types.
| Type | Core Idea |
|---|---|
| Divide and Conquer | Split into subproblems → solve each → combine results. E.g., Merge Sort splits an array in half, sorts each half, then merges the sorted halves. |
| Greedy | Make the locally optimal choice at each step. E.g., Dijkstra's Algorithm always extends the shortest known path to reach all vertices in a graph. |
| Dynamic Programming (DP) | Solve subproblems and store results for reuse. E.g., the Knapsack Problem: given items with weights and values, find the most valuable combination that fits in a bag. |
| Approximation | Near-optimal solutions for NP-hard problems. E.g., for the Traveling Salesman Problem (TSP), find a near-shortest route visiting all cities when the exact solution is too costly to compute. |
| Backtracking | Explore all possibilities, pruning infeasible branches. E.g., N-Queens places N queens on an N x N chessboard so that no two queens attack each other. |
| Branch and Bound | Similar to backtracking, but uses bounds for more aggressive pruning. E.g., for Knapsack (exact), find the optimal item selection by pruning branches that cannot beat the current best. |
Table 4. Six types of algorithms by problem-solving strategy
Key Point: These 6 categories form the core framework running through the entire course. Among them, Divide and Conquer, Greedy, and Dynamic Programming are the three most heavily covered strategies.
Divide and Conquer:
Divide a large problem into smaller subproblems, solve each recursively, then combine the results. Advantage: Effective when the problem structure is naturally recursive. Examples: Merge Sort, Quick Sort, Binary Search.
Figure 5. Merge sort on the sequence 5, 2, 4, 7, 1, 3, 2, 6
Divide and conquer always consists of the following 3 stages, covered in detail in Week 4.
- Divide: Split the problem into smaller subproblems
- Conquer: Solve each subproblem recursively (solve directly if small enough)
- Combine: Merge the solutions of subproblems to form the solution to the original problem
In merge sort, Divide splits the array in half, Conquer recursively sorts each half, and Combine merges the two sorted halves.
Greedy Algorithms:
Make the choice that looks best right now at each step. Does not always guarantee a globally optimal solution, but does under specific conditions. Examples: Minimum Spanning Tree (Kruskal, Prim), Dijkstra's Shortest Path, Huffman Coding.
For a greedy algorithm to guarantee an optimal solution, two conditions are required.
- Greedy Choice Property: A locally optimal choice leads to a globally optimal solution
- Optimal Substructure: The optimal solution of the problem contains optimal solutions of subproblems
If these conditions are not met, the greedy approach can yield suboptimal solutions. This will be covered rigorously in Weeks 5-6.
Dynamic Programming (DP):
Divide the problem into subproblems (like divide and conquer), but store and reuse the results of overlapping subproblems.
Key conditions:
- Overlapping Subproblems
- Optimal Substructure
Examples: Fibonacci sequence, Longest Common Subsequence (LCS), Knapsack Problem.
DP vs Divide and Conquer: In divide and conquer, subproblems are independent; in DP, subproblems overlap.
Fibonacci illustrates the difference as follows.
-
Divide and Conquer: The calls expand as fib(5) → fib(4) + fib(3), fib(4) → fib(3) + fib(2), and fib(3) is computed redundantly. The time complexity is
$O(2^n)$ . - DP: Compute sequentially from fib(1), fib(2), storing results in a table for reuse. No redundant computation, yielding O(n).
Approximation, Backtracking, Branch and Bound: Of the strategies in Table 4, approximation algorithms trade optimality for speed in NP-hard problems where exact solutions are too costly, as in TSP approximation and Vertex Cover. Backtracking is used for N-Queens and maze search, and branch and bound for exact solutions of TSP and the knapsack problem.
Note: An NP-hard problem is one to which every problem in NP reduces in polynomial time, and no polynomial-time algorithm for it is known. For example, for the Traveling Salesman Problem (TSP), checking all cases becomes practically impossible as n grows. This will be covered in detail in Week 13.
The key difference between backtracking and branch and bound is as follows.
- Backtracking: Checks whether a solution can exist in this direction (feasibility check).
- Branch and Bound: Checks whether this direction can improve upon the current best solution (optimality check).
Branch and bound computes upper or lower bounds and prunes entire branches that have no chance of improving the current best solution.
Other Classifications:
- By problem domain: Sorting, Graphs, Computational Geometry
- By computing environment: Parallel, Distributed, Quantum
- Other: AI and Machine Learning algorithms
Time complexity is the number of basic operations as a function of input size
This counting assumes the RAM (Random Access Machine) model, where each basic operation (arithmetic, comparison, memory access) takes exactly one unit of time.
Example: Finding the maximum among
- Sequential scan: compare each card with the current maximum.
- Number of comparisons:
$n - 1$ - Time complexity:
$T(n) = n - 1$
Key Point: The key to computing time complexity is counting how many basic operations are performed for input size n. It is an abstract count of operations, not actual execution time (seconds). This allows comparing the fundamental efficiency of algorithms regardless of computer performance.
Example 1: O(1)
sample1(A[], n)
k = n / 2
return A[k]
It performs only 1 division and 1 array access, so it takes constant time regardless of
Example 2: O(n)
sample2(A[], n)
sum = 0
for i = 1 to n
sum = sum + A[i]
return sum
The loop executes
Example 3: O(n²)
sample3(A[], n)
sum = 0
for i = 1 to n
for j = 1 to n
sum = sum + A[i] * A[j]
return sum
The nested loops each execute
The basic principle for determining nested loop time complexity is as follows. If the inner loop's iteration count does not depend on the outer loop, the total is inner loop iterations × outer loop iterations. Here it is n × n = n². A triple-nested loop would be O(n³).
Example 4: O(n²) (Triangular Loop)
sample4(A[], n)
sum = 0
for i = 1 to n-1
for j = i+1 to n
sum = sum + A[i] * A[j]
return sum
The inner loop executes
Example 5: O(n) (Recursion)
factorial(n)
if n == 1 return 1
return n * factorial(n - 1)
The recursion depth is
The time complexity of a recursive function is determined by how many recursive calls occur in total. factorial(n) is called exactly n times, so it is O(n). Since the work done in each call (one multiplication) is O(1), the total time complexity is O(n) × O(1) = O(n).
| Type | Meaning |
|---|---|
| Worst-case | Upper bound; guarantees that no input takes longer than this |
| Average-case | Expected time under a probability distribution (usually uniform) |
| Best-case | Fastest execution; used to find the optimal algorithm |
| Amortized | Average cost per operation over a sequence of operations |
Table 5. Four types of complexity analysis
Amortized analysis is different from average-case analysis. It considers the worst-case total cost of a sequence of operations, then divides by the number of operations. For example, a dynamic array occasionally needs to resize (expensive), but most insertions are cheap, so the amortized cost per insertion is O(1).
In practice, worst-case analysis is the most commonly used.
Key Point: The reason worst-case is primarily used is that it provides a guarantee. A statement such as "this algorithm will never exceed n² time for any input" is a solid promise. Average-case analysis, on the other hand, is an expected value under an assumed input distribution, so it gives no guarantee for an individual input. For example, quick sort averages O(n log n) but has a worst case of O(n²).
Commute Time Analogy:
Figure 6. Seoul Metro
Scenario: Home → Station (6 min) → Subway (20 min) → Classroom (10 min)
| Case | Time | Description |
|---|---|---|
| Best | 36 min | The train arrives immediately |
| Worst | 40 min | The train just left, so a 4 min wait |
| Average | 38 min | Average wait of about 2 min |
Table 6. Best, worst, and average cases of commute time
Algorithm analysis follows the same way of thinking: the worst case provides a guarantee.
Section 2 computed exact operation counts like
Figure 7. Graphical meaning of Θ, O, and Ω notation
- Time complexity is a function of
$n$ (usually a polynomial with multiple terms). - As
$n$ grows, only the dominant term matters. - Asymptotic notation simplifies the expression by focusing on the growth rate.
| Notation | Meaning | Intuitive Interpretation |
|---|---|---|
|
|
Upper bound | Grows at most this fast |
|
|
Lower bound | Grows at least this fast |
|
|
Tight bound | Exact growth rate |
Table 7. Meaning of the three asymptotic notations
Key Point: Asymptotic means behavior as n approaches infinity. For example, in
$3n^2 + 100n + 5000$ , when n is 1,000,000,$3n^2 = 3 \times 10^{12}$ and$100n = 10^8$ , so$100n$ is negligibly small compared to$3n^2$ . That is why we keep only the highest-order term and ignore the rest.
Formal Definition:
-
$g(n)$ is the asymptotic upper bound of$f(n)$ . - Convention: Technically
$f(n) \in O(g(n))$ , but we write$f(n) = O(g(n))$ .
Reading the definition: "There exist constants
Intuitive Meaning:
In plain terms, for sufficiently large n (n ≥ n₀), f(n) is always at most c × g(n). Here, c and n₀ are constants that need to be found, and to prove that a function belongs to O(g(n)), it suffices to exhibit specific values of c and n₀ that satisfy this condition.
Proof Example:
We need to find constants
Choose
Rearranging gives
The expansion is as follows. Subtracting
Therefore
Note: Choosing c. For a polynomial
$a_k n^k + a_{k-1} n^{k-1} + \cdots + a_0$ with a positive leading coefficient, taking the sum of absolute values of all coefficients$|a_k| + |a_{k-1}| + \cdots + |a_0|$ as c always works with$n_0 = 1$ . This is because each term satisfies$a_i n^i \leq |a_i| n^k$ for$n \geq 1$ . Example: for$2n^2 - 8n + 3$ ,$c = 2 + 8 + 3 = 13$ ,$n_0 = 1$ .
Functions contained in
-
$3n^2 + 2n$ yes,$7n^2 - 100n$ yes,$n \log n + 5n$ yes,$3n$ yes
Functions NOT in
-
$n^3$ no,$3^n$ no
Best Practice: Write as tightly as possible. Write
The statements "this algorithm is O(n²)" and "this algorithm is O(n log n)" can both be true, but O(n log n) is more useful information, just as "this apple costs at most 50 won" is more useful than "this apple costs at most 100 won."
More Big-O Examples:
| Expression | Big-O | Reason |
|---|---|---|
|
|
||
|
|
||
|
|
||
|
|
Table 8. Examples of Big-O bounds and their justification
Formal Definition:
-
$g(n)$ is the asymptotic lower bound of$f(n)$ . - It is symmetric to Big-O.
Intuitive Meaning:
Examples:
| Expression | Big-Omega | Reason |
|---|---|---|
|
|
||
|
|
||
|
|
||
|
|
Table 9. Examples of Big-Omega bounds and their justification
Best Practice: Write as tightly as possible. Write
Proof Example:
Choose
Formal Definition:
Intuitive Meaning:
Key Point: Θ is the most precise notation.
- O(n²): This algorithm is at most at the n² level (it could be faster).
- Ω(n²): This algorithm is at least at the n² level (it could be slower).
- Θ(n²): This algorithm is exactly at the n² level.
Examples:
| Expression | Theta |
|---|---|
|
|
|
Table 10. Examples of Theta bounds
Counterexample:
Proof Example:
Upper bound:
| Notation | Name | Example |
|---|---|---|
| Constant | Array index access | |
| Logarithmic | Binary search | |
| Linear | Linear search | |
| Log-linear | Merge sort | |
| Quadratic | Bubble sort | |
| Cubic | Matrix multiplication (naive) | |
| Exponential | Subset exhaustive search |
Table 11. Common complexity classes and examples
Hierarchy:
Key Point: Practical intuition is as follows.
- O(1): A hash table lookup (on average); it finishes instantly.
- O(log n): Even for n=1 billion, about 30 steps; very fast.
- O(n): For n=1 billion, 1 billion steps; typically a few seconds.
- O(n log n): Most sorting falls here; the practical limit.
- O(n²): For n=100,000, 10 billion steps; starts getting slow.
$O(2^n)$ : For n=40, about 1 trillion steps; practically impossible.
Growth Rate Comparison:
| 10 | 3.3 | 10 | 33 | 100 | 1,000 | 1,024 |
| 100 | 6.6 | 100 | 664 | 10,000 | ||
| 1,000 | 10 | 1,000 | 10,000 | |||
| 20 | - |
Table 12. Growth of function values by input size
The gap between polynomial and exponential is enormous.
Note: In the table above,
$\log n$ means$\log_2 n$ . In CS, log without specifying a base usually means base 2. In Big-O, the base does not matter, because$\log_a n = \frac{\log_b n}{\log_b a}$ , so changing bases is just a constant factor, and constants are ignored in Big-O.
Sorting 1 billion numbers:
| Algorithm |
|
|
|
|---|---|---|---|
|
|
< 1 sec | about 3 hours | about 300 years |
|
|
< 1 sec | about 1 sec | about 12 days |
|
|
< 1 sec | < 1 sec | about 5 min |
|
|
< 1 sec | < 1 sec | < 1 sec |
Table 13. Sorting time by algorithm and hardware
The table assumes a PC performs about
Key Point: An efficient algorithm is worth more than a supercomputer.
Investing in a better algorithm is far more cost-effective than investing in better hardware.
For large enough inputs, running O(n log n) on a regular PC is faster than running O(n²) on a supercomputer; at n = 1 billion it takes about 5 minutes versus about 12 days, a gap of about 3,300×. Hardware can only provide constant-factor improvement, but an algorithm improvement changes the order itself.
When we analyze recursive algorithms, we cannot simply count loop iterations as in Section 2. Instead, the running time
A recurrence relation expresses a function in terms of its values on smaller inputs.
Examples:
$a_n = a_{n-1} + 2$ -
$f(n) = n \cdot f(n-1)$ : Factorial -
$f(n) = f(n-1) + f(n-2)$ : Fibonacci -
$f(n) = f(n/2) + n$ : Binary reduction
Many recursive algorithms naturally produce recurrence relations for their running time.
A recurrence relation can be thought of as a recursively defined function. The running time of a recursive algorithm is difficult to express directly in closed form, so instead we express T(n) in terms of T(n/2), T(n-1), and so on. The goal is to solve the recurrence and obtain a closed form.
Merge Sort Recurrence:
-
$2T(n/2)$ : Two recursive calls on halves -
$n$ : Merge overhead (linear scan) $T(1) = 1$
Merge sort works as follows, and Weeks 3 and 4 cover it in detail.
- Split array [5, 2, 4, 1, 3] in half: [5, 2] / [4, 1, 3]
- Recursively sort each half: [2, 5] / [1, 3, 4]
- Merge the two sorted arrays: [1, 2, 3, 4, 5]
The merge process compares elements from the front of both arrays, taking the smaller one, so it is O(n). The
Three methods for solving recurrence relations:
| Method | Idea |
|---|---|
| Repeated Substitution | Expand the recurrence step by step to find a pattern |
| Guess and Verify | Guess the form, then prove it by mathematical induction |
| Master Theorem | Apply a formula directly to recurrences of a specific form |
Table 14. Three methods for solving recurrences
Example:
This is the recurrence for factorial, with linear time.
The key strategy of repeated substitution is to repeatedly substitute the recurrence into T(n) to find a pattern, then generalize it. In the example above, the pattern is that after k substitutions we get T(n-k) + kc. Substituting k = n-1 gives T(1) + (n-1)c.
Applied to Merge Sort:
Assume
The expansion pattern is as follows.
- 1 substitution:
$2T(n/2) + n$ → coefficient 2, problem size n/2 - 2 substitutions:
$4T(n/4) + 2n$ → coefficient 4, problem size n/4 - k substitutions:
$2^k T(n/2^k) + kn$
If
Recurrence:
Guess:
[Discrete Mathematics] The 3 steps of mathematical induction are as follows.
- Base Case: Show the statement is true for the smallest case
- Inductive Hypothesis: Assume the statement is true for n = k
- Inductive Step: Show it is also true for n = k+1
Completing these three steps proves the statement for all natural numbers. The form used in the proof below, which assumes the statement for inputs smaller than n (such as n/2), is strong induction and rests on the same principle, and the guess-and-verify method proves a guessed bound in this way.
Proof by Mathematical Induction:
Base case:
Inductive hypothesis: Assume
Inductive step:
Therefore
Common Pitfall:
Caution: The constant
$c$ in the hypothesis must be the same$c$ in the conclusion.
As an incorrect proof attempt, guess
This is
Fix: strengthen the guess to
A common mistake: The most common mistake in induction proofs is getting cn + (constant) and concluding that it is O(n) and therefore enough. In induction, you must show exactly the same form (
A systematic method for determining the constant
Consider recurrences of the following form.
Let
| Case | Condition | Result |
|---|---|---|
| Case 1 |
|
|
| Case 2 | ||
| Case 3 |
|
Table 15. The three cases of the Master Theorem
Polynomially smaller or larger means the following. The requirement that
Key Point: The Master Theorem is a powerful tool that solves divide-and-conquer recurrences in one step. The core is simple.
$a$ : Number of recursive calls$b$ : Factor by which the problem shrinks$f(n)$ : Cost of dividing and combining$n^{\log_b a}$ : Cost at the leaf level of the recursion treeThe 3 Cases determine whether the leaves dominate (Case 1), the root dominates (Case 3), or it is a tie (Case 2).
Intuition:
Each level of the recursion tree does work. Which part dominates?
Figure 8. Recursion tree of the merge sort recurrence with per-level costs
| Scenario | Winner | Result |
|---|---|---|
| Leaves dominate | Leaf work wins | |
| Root dominates | Root work wins | |
| Balanced | Equal per level |
Table 16. Result by the dominating part of the recursion tree
The tree height is
[Data Structures] A recursion tree is an application of trees. The root is the original problem, each internal node is a recursive call, and leaves are base cases. The tree height is
$\log_b n$ because the problem size shrinks by a factor of$1/b$ at each level (n → n/b → n/b² → … → 1).
Examples:
Example 1:
-
$a = 2$ ,$b = 3$ ,$h(n) = n^{\log_3 2} \approx n^{0.63}$ ,$f(n) = c = O(1)$ -
$f(n)$ is polynomially smaller → Case 1 $T(n) = \Theta(n^{\log_3 2})$
Example 2:
-
$a = 2$ ,$b = 4$ ,$h(n) = n^{\log_4 2} = n^{0.5} = \sqrt{n}$ ,$f(n) = n$ -
$f(n)$ is polynomially larger → Case 3 $T(n) = \Theta(n)$
Example 3:
-
$a = 2$ ,$b = 2$ ,$h(n) = n^{\log_2 2} = n$ ,$f(n) = n$ -
$f(n) = \Theta(h(n))$ → Case 2 $T(n) = \Theta(n \log n)$
The Master Theorem is applied in the following order.
- Identify a, b, and f(n)
- Compute
$n^{\log_b a}$ - Compare f(n) with
$n^{\log_b a}$ - Apply the corresponding Case
Example 3 is precisely the recurrence for merge sort. Using the Master Theorem, you get Θ(n log n) immediately without repeated substitution or induction.
The regularity condition ensures that
Checking Case 3: Case 3 is the hardest to apply because the regularity condition (
| Concept | Key Summary |
|---|---|
| Algorithm | It is a step-by-step procedure for solving problems, and correctness, executability, finiteness, and efficiency are its essential properties. |
| Euclidean GCD | It is one of the oldest known algorithms (around 300 BCE) and repeatedly applies GCD(A, B) = GCD(B, A mod B). |
| Iteration vs Recursion | Iteration uses loops and recursion uses self-calls; the results are the same, but recursion uses the call stack. |
| Algorithm Representation | Algorithms are represented in natural language, pseudocode (most common), or flowcharts. |
| Algorithm Classification | Algorithms divide into divide and conquer, greedy, DP, approximation, backtracking, and branch and bound, and the three core strategies are divide and conquer, greedy, and DP. |
| Time Complexity | It is the number of basic operations as a function of input size |
| Complexity Analysis Types | They are worst-case (most common, providing a guarantee), average-case, best-case, and amortized. |
| Big-O ( |
It is an asymptotic upper bound, meaning constants |
| Big-Omega ( |
It is an asymptotic lower bound, meaning constants |
| Theta ( |
It is a tight bound, |
| Write tightly |
|
| Complexity Hierarchy | The hierarchy is |
| Algorithm > Hardware | For large enough inputs, running O(n log n) on a PC is faster than running O(n²) on a supercomputer. |
| Recurrence Relations | A recurrence expresses a function in terms of its values on smaller inputs and is used to analyze the time complexity of recursive algorithms. |
| Repeated Substitution | It repeatedly expands the recurrence to find and generalize a pattern. |
| Guess and Verify | It guesses the form and proves it by mathematical induction, and you must beware the pitfall that the constant c must be identical. |
| Master Theorem | It applies to |
-
Euclidean GCD: Trace GCD(48, 18) step by step. How many iterations does it take?
Answer
Apply
GCD(A, B) = GCD(B, A mod B)repeatedly:GCD(48, 18) → GCD(18, 12) → GCD(12, 6) → GCD(6, 0) = 6. The recursion stops whenb == 0, returning the other argument as the GCD. It takes 3 iterations (or 4 calls if you count the terminating call). The algorithm is fast becausebshrinks quickly; by Lamé's theorem, the number of steps isO(log min(a, b)). -
Operation Counting: Given the code
for i in range(n): for j in range(i, n): count += 1, how many times doescount += 1execute? Express as a function of n.Answer
This is the triangular double loop pattern. The inner loop runs
n - itimes for eachifrom0ton-1, so the total isn + (n-1) + (n-2) + ... + 1 = n(n+1)/2. Asymptotically this is Θ(n²), keeping only the highest-order term and dropping the constant factor1/2. The triangular structure does not save an order of growth, only a constant factor. -
Big-O Proof: Prove that 5n³ + 2n² + 7 = O(n³) by finding appropriate c and n₀.
Answer
We need
c > 0andn₀ ≥ 0such that5n³ + 2n² + 7 ≤ c · n³for alln ≥ n₀. Using the sum-of-absolute-coefficients trick, choosec = 5 + 2 + 7 = 14andn₀ = 1. Forn ≥ 1:5n³ + 2n² + 7 ≤ 5n³ + 2n³ + 7n³ = 14n³. Therefore5n³ + 2n² + 7 = O(n³)withc = 14,n₀ = 1. (Tighter values likec = 6,n₀ = 3also work.) -
Big-Omega vs Big-O: Is it true that n² = O(n³)? Is it true that n² = Ω(n³)? Explain why or why not.
Answer
n² = O(n³)is true.n²does not grow faster thann³, sincen² ≤ 1 · n³for alln ≥ 1(c = 1,n₀ = 1).n² = Ω(n³)is false. Big-Omega requiresn² ≥ c · n³for somec > 0and largen, butn²/n³ = 1/n → 0, so no such constant exists. The lesson: Big-O gives an upper bound that can be loose, while Big-Omega demands a genuine lower bound. -
Theta: If f(n) = Θ(g(n)), what does this tell you about the relationship between f and g? Can f(n) = Θ(n²) and f(n) = O(n³) both be true simultaneously?
Answer
f(n) = Θ(g(n))meansfandggrow at exactly the same rate up to constant factors; formally,f = O(g)andf = Ω(g). Yes,f(n) = Θ(n²)andf(n) = O(n³)can both be true: a Θ-bound implies the matching O-bound, and any function that isΘ(n²)is also (loosely)O(n³). Θ is the tightest statement; the corresponding O statement is always weaker but not contradictory. -
Master Theorem: Apply the Master Theorem to T(n) = 4T(n/2) + n. Identify a, b, f(n), compute
$n^{\log_b a}$ , determine the case, and state T(n).Answer
Here
a = 4,b = 2,f(n) = n, so$n^{\log_b a} = n^{\log_2 4} = n^2$ . Comparef(n) = nwithn²:nis polynomially smaller thann²(use$\varepsilon = 1$ , since$n = O(n^{2-1})$ ). This is Case 1, so$T(n) = \Theta(n^{\log_b a}) = \Theta(n^2)$ . Intuitively, the leaves of the recursion tree dominate, because there aren²of them and each does constant work. -
Guess and Verify: For T(n) = T(n/2) + 1, guess that T(n) = O(log n) and verify using induction. The base case must also be checked.
Answer
Guess:
T(n) ≤ c · log nfor somec > 0. Base case (n = 2):T(2) = T(1) + 1. AssumingT(1) = 1,T(2) = 2, and we need2 ≤ c · log₂ 2 = c, soc ≥ 2. Inductive step: AssumeT(n/2) ≤ c · log(n/2). ThenT(n) = T(n/2) + 1 ≤ c · log(n/2) + 1 = c · log n - c · log 2 + 1 ≤ c · log nwheneverc · log 2 ≥ 1, i.e.,c ≥ 1/log 2. With base 2,log 2 = 1, soc ≥ 1(with the natural log it would bec ≥ 1/ln 2 ≈ 1.44). Takingc = 2satisfies both, soT(n) = O(log n). -
Why does it matter? Comparing a PC running an O(n log n) algorithm with a supercomputer running an O(n²) algorithm, at what input size does the PC win?
Answer
Assume the PC does
10⁸ops/sec and the supercomputer does10¹²ops/sec, a 10,000× hardware gap. The PC wins whenn log n / 10⁸ < n² / 10¹², i.e., whenlog n / n < 10⁻⁴, roughlyn > 200,000. Beyond a few hundred thousand inputs, the PC's better algorithmic order beats the supercomputer's raw speed. This illustrates the central message that a better algorithm beats a better machine: hardware gives a constant-factor improvement, while algorithmic order changes the growth class itself.




