Active Learning
目录
人工标注的成本，来自标注者投入的时间、专业知识，以及组织和审核标注的过程。面对大量未标注数据，一个自然的问题是：如果只能获得少量标签，应该先标注哪些样本？
Active Learning（主动学习）让模型参与这个选择过程。模型先评估未标注样本，再把其中一些交给人类标注，用新增标签更新自己。
diff --git a/blog/behavior-cloning-plid/index.html b/blog/behavior-cloning-plid/index.html index 884ac81..bd9de97 100644 --- a/blog/behavior-cloning-plid/index.html +++ b/blog/behavior-cloning-plid/index.html @@ -21,7 +21,13 @@Behavior Cloning 与 PLID
目录
Behavior Cloning 把专家示范当作监督学习数据，但模型部署后会访问训练时没有覆盖的状态。本文从模仿学习的基本形式化出发，说明 covariate shift 与 compounding errors，并推导 PLID / DAgger 如何通过当前策略 rollout、再由专家补充标签，逐步修正状态分布偏移。
前置知识：概率、条件概率、监督学习损失，以及基本的 Markov Decision Process（MDP）概念。KL 散度的定义见 Active Learning。
diff --git a/blog/distribution-matching-gail/index.html b/blog/distribution-matching-gail/index.html index aeea5c9..9b34dd3 100644 --- a/blog/distribution-matching-gail/index.html +++ b/blog/distribution-matching-gail/index.html @@ -21,7 +21,13 @@Distribution Matching 与 GAIL
目录
Behavior Cloning 直接拟合专家动作，PLID 则让专家在模型实际访问的状态上补充标签。若不能持续询问专家，能否仍让模型产生与专家相似的轨迹或状态—动作分布？本文从轨迹概率、Forward KL 与状态—动作分布出发，说明 GAIL 如何用判别器间接完成分布匹配。
系列上一篇：Behavior Cloning 与 PLID。前置知识还包括 MDP、条件概率，以及 Active Learning 中的 KL divergence。
diff --git a/blog/index.html b/blog/index.html index e70d792..6d2fd82 100644 --- a/blog/index.html +++ b/blog/index.html @@ -5,25 +5,31 @@ - + -文章
IRL、RLHF 与 DPO
从逆强化学习出发，说明如何从专家行为与人类偏好学习目标，并连接到 RLHF 与 DPO。
阅读全文Distribution Matching 与 GAIL
从轨迹与状态—动作分布匹配出发，说明 Forward KL 为何可能退化为 BC，以及 GAIL 如何用判别器完成分布匹配。
阅读全文Behavior Cloning 与 PLID
说明 Behavior Cloning 的状态分布偏移，并推导 PLID / DAgger 如何用交互式专家标签修正策略。
阅读全文Active Learning
介绍主动学习的基本流程，比较不确定性采样、QBC 和 BALD 三种选样方法。
阅读全文Blog
系列Imitation Learning
其他文章
IRL、RLHF 与 DPO
目录
Behavior Cloning、PLID 和 GAIL 主要学习专家如何行动。若希望模型在新环境中仍能做出合理决策，就需要进一步理解专家行为背后的目标。本文从 Inverse Reinforcement Learning 出发，说明如何从专家行为推断奖励，并连接到 RLHF 与 DPO。
系列前文：Behavior Cloning 与 PLID、Distribution Matching 与 GAIL。
diff --git a/content/news.json b/content/news.json new file mode 100644 index 0000000..941b9f6 --- /dev/null +++ b/content/news.json @@ -0,0 +1,17 @@ +[ + { + "date": "2026-11-03", + "text": "发布 Imitation Learning 系列第三篇：IRL、RLHF 与 DPO。", + "href": "/blog/irl-rlhf-dpo/" + }, + { + "date": "2026-11-02", + "text": "更新 Distribution Matching 与 GAIL，并整理仿作学习系列。", + "href": "/blog/distribution-matching-gail/" + }, + { + "date": "2026-10-20", + "text": "发布 Behavior Cloning 与 PLID。", + "href": "/blog/behavior-cloning-plid/" + } +] diff --git a/content/pages/about.md b/content/pages/about.md new file mode 100644 index 0000000..2e461ca --- /dev/null +++ b/content/pages/about.md @@ -0,0 +1,8 @@ +--- +title: About me +summary: 关于 Musicaldg。 +--- + +机器学习笔记与个人主页。更多内容稍后补充。 + +目前主要整理模仿学习、主动学习等相关笔记，源码与页面托管在 [GitHub](https://github.com/Musicaldg)。 diff --git a/content/pages/cv.md b/content/pages/cv.md new file mode 100644 index 0000000..7b524e4 --- /dev/null +++ b/content/pages/cv.md @@ -0,0 +1,8 @@ +--- +title: CV +summary: Curriculum Vitae。 +--- + +CV 页面已预留，正文与可下载版本稍后补充。 + +也可先通过 [GitHub](https://github.com/Musicaldg) 查看公开项目与提交记录。 diff --git a/cv/index.html b/cv/index.html new file mode 100644 index 0000000..012c7a1 --- /dev/null +++ b/cv/index.html @@ -0,0 +1,37 @@ + + + + + + + + + + + +
+
+
+
+News
+
+
+ Latest Writing
+ All posts +系列Imitation Learning
Active Learning
介绍主动学习的基本流程，比较不确定性采样、QBC 和 BALD 三种选样方法。
{esc(p["title"])}
{esc(p["summary"])}
阅读全文 ' for p in selected) or EMPTY + return ''.join( + f'' + f'' + f'{esc(p["title"])}
{esc(p["summary"])}
' + f'阅读全文 ' + for p in selected + ) or EMPTY def compact_card(post, index=None): index_html = f'{index:02d}' if index is not None else '' return ( - f'' + f'' f'{index_html}' f''
f'
{esc(post["title"])}
' @@ -222,14 +305,32 @@ def compact_card(post, index=None): f'' ) - def series_folder(series, open_by_default=False): + def roomy_card(post, index=None): + index_html = f'{index:02d}' if index is not None else '' + return ( + f'' + f'{index_html}' + f''
+ f''
+ f'
'
+ )
+
+ def series_folder(series, *, card='compact', open_by_default=False, variant=''):
span = series['start'] if series['start'] == series['end'] else f'{series["start"]} — {series["end"]}'
- cards = ''.join(compact_card(post, index=i) for i, post in enumerate(series['posts'], 1))
+ maker = roomy_card if card == 'roomy' else compact_card
+ cards = ''.join(maker(post, index=i) for i, post in enumerate(series['posts'], 1))
open_attr = ' open' if open_by_default else ''
+ class_name = 'series-folder' + (f' {variant}' if variant else '')
return (
- f'{esc(post["title"])}
' + f'{esc(post["summary"])}
' + f'阅读全文 ' + f''
+ f''
+ f'{text} '
+ )
+ return '' + blog_rows(posts) + ' ', 'blog')
- page('404.html', '页面未找到 · ' + config['name'], '页面未找到', '')
+ page(
+ 'blog/index.html',
+ 'Blog · ' + config['name'],
+ '机器学习笔记。',
+ ''
+ '' + blog_feed() + ' ',
+ 'blog',
+ extra_assets='',
+ )
+ page(
+ '404.html',
+ '页面未找到 · ' + config['name'],
+ '页面未找到',
+ '',
+ )
+ for item in pages:
+ body = (
+ f''
+ f''
+ )
+ page(
+ f'{item["slug"]}/index.html',
+ item['title'] + ' · ' + config['name'],
+ item['summary'],
+ body,
+ item['slug'],
+ page_class='site-page',
+ )
for series in series_list:
- cards = ''.join(compact_card(post, index=i) for i, post in enumerate(series['posts'], 1))
+ cards = ''.join(roomy_card(post, index=i) for i, post in enumerate(series['posts'], 1))
body = (
- f'
'
f''
- f''
+ f''
f'系列'
f'{esc(series["title"])}'
f''
@@ -238,55 +339,155 @@ def series_folder(series, open_by_default=False):
f'
'
)
+ def news_list(items, limit=3):
+ selected = items[:limit]
+ if not selected:
+ return ''
- f''
+ f''
f'系列'
f'{esc(series["title"])}'
f''
@@ -238,55 +339,155 @@ def series_folder(series, open_by_default=False):
f'
No news yet.
' + rows = [] + for item in selected: + text = esc(item['text']) + if item['href']: + text = f'{text}' + rows.append( + f'- ' + ''.join(rows) + '
No posts yet.
' - home = Template((ROOT / 'templates/home.html').read_text(encoding='utf-8')).substitute(feed=home_feed()) + def blog_feed(): + parts = [series_folder(series, card='roomy', variant='series-folder-roomy') for series in series_list] + if standalone: + if series_list: + parts.append('其他文章
') + parts.append('' + blog_rows(standalone) + '
')
+ elif not series_list:
+ parts.append('' + EMPTY + '
')
+ return ''.join(parts)
+
+ home = Template((ROOT / 'templates/home.html').read_text(encoding='utf-8')).substitute(
+ news=news_list(news_items, 3),
+ feed=home_feed(),
+ )
page('index.html', config['name'], config['description'], home, 'home', extra_assets='')
- page('blog/index.html', '文章 · ' + config['name'], '机器学习笔记。', '文章
Blog
{esc(item["title"])}
' + f'{esc(item["summary"])}
{item["body"]}
← 首页'
+ f''
f'{cards} '
)
- page(f'series/{series["id"]}/index.html', series['title'] + ' · ' + config['name'], series['summary'], body, 'home')
+ page(
+ f'series/{series["id"]}/index.html',
+ series['title'] + ' · ' + config['name'],
+ series['summary'],
+ body,
+ 'blog',
+ )
for post in posts:
toc = ''
- body = f'← 全部文章 '
- page(f'blog/{post["slug"]}/index.html', post['title'] + ' · ' + config['name'], post['summary'], body, 'blog', 'article')
+ body = (
+ f''
+ f'← 全部文章'
+ f''
+ f' '
+ f''
+ )
+ page(
+ f'blog/{post["slug"]}/index.html',
+ post['title'] + ' · ' + config['name'],
+ post['summary'],
+ body,
+ 'blog',
+ 'article',
+ )
feed = ET.Element('rss', version='2.0')
channel = ET.SubElement(feed, 'channel')
- for key, value in [('title', config['name']), ('link', config['url']), ('description', config['description']), ('language', 'zh-CN')]:
+ for key, value in [
+ ('title', config['name']),
+ ('link', config['url']),
+ ('description', config['description']),
+ ('language', 'zh-CN'),
+ ]:
ET.SubElement(channel, key).text = value
for post in posts:
item = ET.SubElement(channel, 'item')
- for key, value in [('title', post['title']), ('link', config['url'] + '/blog/' + post['slug'] + '/'), ('guid', config['url'] + '/blog/' + post['slug'] + '/'), ('description', post['summary']), ('pubDate', dt.datetime.strptime(post['date'], '%Y-%m-%d').strftime('%a, %d %b %Y 00:00:00 GMT'))]:
+ for key, value in [
+ ('title', post['title']),
+ ('link', config['url'] + '/blog/' + post['slug'] + '/'),
+ ('guid', config['url'] + '/blog/' + post['slug'] + '/'),
+ ('description', post['summary']),
+ ('pubDate', dt.datetime.strptime(post['date'], '%Y-%m-%d').strftime('%a, %d %b %Y 00:00:00 GMT')),
+ ]:
ET.SubElement(item, key).text = value
write('feed.xml', ET.tostring(feed, encoding='utf-8', xml_declaration=True).decode('utf-8'))
sitemap = ET.Element('urlset', xmlns='http://www.sitemaps.org/schemas/sitemap/0.9')
for path in generated:
if path.endswith('index.html'):
entry = ET.SubElement(sitemap, 'url')
- ET.SubElement(entry, 'loc').text = config['url'] + ('/' if path == 'index.html' else '/' + path.removesuffix('index.html'))
+ ET.SubElement(entry, 'loc').text = config['url'] + (
+ '/' if path == 'index.html' else '/' + path.removesuffix('index.html')
+ )
write('sitemap.xml', ET.tostring(sitemap, encoding='utf-8', xml_declaration=True).decode('utf-8'))
write('robots.txt', 'User-agent: *\nAllow: /\nSitemap: ' + config['url'] + '/sitemap.xml')
+ page_slugs = {p['slug'] for p in pages}
for obsolete in set(previous) - set(generated):
- if obsolete.endswith('/index.html') and (obsolete.startswith('blog/') or obsolete.startswith('series/')):
+ if not obsolete.endswith('/index.html') or obsolete == 'index.html':
+ continue
+ top = obsolete.split('/', 1)[0]
+ if (
+ obsolete.startswith('blog/')
+ or obsolete.startswith('series/')
+ or top in page_slugs
+ ):
target = ROOT / obsolete
if target.exists():
target.unlink()
write('generated-files.json', json.dumps(generated, indent=2))
- print(f'构建完成：{len(posts)} 篇已发布文章。')
+ print(f'构建完成：{len(posts)} 篇已发布文章，{len(pages)} 个页面。')
if __name__ == '__main__':
diff --git a/series/imitation-learning/index.html b/series/imitation-learning/index.html
index 9e4c101..530621a 100644
--- a/series/imitation-learning/index.html
+++ b/series/imitation-learning/index.html
@@ -21,9 +21,15 @@
Musicaldg
-
+
- 010203 010203
{esc(post["title"])}
目录
{toc}{post["body"]}
{esc(post["title"])}
' + f'' + f'目录
{toc}{post["body"]}
Behavior Cloning 与 PLID
说明 Behavior Cloning 的状态分布偏移，并推导 PLID / DAgger 如何用交互式专家标签修正策略。
Distribution Matching 与 GAIL
从轨迹与状态—动作分布匹配出发，说明 Forward KL 为何可能退化为 BC，以及 GAIL 如何用判别器完成分布匹配。
IRL、RLHF 与 DPO
从逆强化学习出发，说明如何从专家行为与人类偏好学习目标，并连接到 RLHF 与 DPO。
Behavior Cloning 与 PLID
说明 Behavior Cloning 的状态分布偏移，并推导 PLID / DAgger 如何用交互式专家标签修正策略。
阅读全文Distribution Matching 与 GAIL
从轨迹与状态—动作分布匹配出发，说明 Forward KL 为何可能退化为 BC，以及 GAIL 如何用判别器完成分布匹配。
阅读全文IRL、RLHF 与 DPO
从逆强化学习出发，说明如何从专家行为与人类偏好学习目标，并连接到 RLHF 与 DPO。
阅读全文