​（比如LLM的词向量、多模态的特征emb），没法直接塞进新模型的Lookup Table里初始化。

不能把所有的emb全dump到样本里面吧，这存储不得爆炸了？

因此需要使用一些骚操作来处理，

​把这些冻结的emb当成“标签”​

，然后训练一个新的Lookup Table，让它学得跟冻结emb一模一样。

“如何让一个emb收敛至另外一个emb？”

 最直接的想法当然是构建一个简单的训练流程：

 我们通过两个emb的余弦相似度来衡量二者，从图中可以看出，大约训练了30轮之后，cos相似度≈1，确实可以基本完全收敛。

但是仔细想想，这个任务很简单，真的有必要训练这么多轮？就算加大学习率，按照目前的配置最少也要个十几轮。

因此，现在的问题就变成了：“能不能让一个emb快速收敛至另外一个emb？最好是一步”

想要一步收敛，复杂的优化器（如Adam）可能有些“杀鸡用牛刀”，反而引入不必要的动量或自适应项。让我们回归最原始、最纯粹的优化算法：​

这个是标准的MSE公式，如果我的emb维度是128，那对于每条样本来说，loss就是：

如果考虑momentum=0.0的简单情况，那么梯度就是（其中LR是学习率）：

这时候发现，如果学习率LR=64，恰好可以一步收敛：

所以，如果我的emb维度是128，我的学习率配置成64就可以了？

以上情况是考虑单个样本，也就是batchsize=1的情况。但是训练的时候，batchsize往往比较大，以batchsize=2048为例。此时的MSE公式根据batch_loss的不同处理方式变成了不同的样子：

的loss处理方式，因为从公式可以看出，这样LR的配置可以和batchsize解耦。因为在真实训练的时候，最后一个batch往往不能塞满，如果此时还按照 batchsize配置LR，会导致最后一批embedding都学偏。比如下图这里，embedding都变大了3倍。

LR=13w？！是不是还挺反直觉的？Sum Loss与Mean Loss在SGD中的表现差异，本质是梯度幅度与学习率的博弈。原理其实并不难，甚至可以当作校招面试题，只是平时太多玄学调参了，忘了一些本质的数学原理。

另外，如果非要让LR=1.0一步收敛怎么办？那就去魔改一下loss呗~

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# 设置中文字体
try:
    plt.rcParams['font.sans-serif'] = ['Microsoft YaHei']
    plt.rcParams['axes.unicode_minus'] = False
except:
    try:
        plt.rcParams['font.sans-serif'] = ['SimHei']  # 黑体
    except:
        print("警告: 无法加载中文字体，图表中的中文可能无法正常显示")


# 设置随机种子以保证结果可复现
tf.random.set_seed(42)
np.random.seed(42)

# 使用提供的3个标签示例
label_examples = {
    "0": [0.1650,0.1043,-0.2367,0.2781,-0.1600,0.1100,-0.1014,0.1454,0.0150,-0.0368,-0.0786,-0.0097,0.0147,0.0352,-0.0038,-0.0682,-0.0377,-0.0941,0.0537,-0.0282,-0.0856,0.0397,0.0237,-0.0667,-0.0816,0.0299,-0.1135,0.0215,-0.0430,-0.1201,-0.0704,0.0507,-0.0725,0.1234,0.1061,0.0279,0.0778,-0.0225,0.0678,-0.0330,0.0798,-0.0138,0.0442,-0.0075,0.0652,0.0400,0.0043,-0.0473,-0.0644,0.1271,-0.0314,-0.0075,0.0601,0.0940,-0.0457,-0.0633,0.0455,-0.0643,-0.0671,-0.0847,0.1479,0.0065,0.0130,-0.0102,-0.0366,-0.0292,0.0589,0.1359,-0.0297,0.1030,-0.0288,-0.1139,0.0303,0.0577,0.0631,0.0473,0.0016,0.0085,-0.0017,0.0843,-0.0529,-0.0150,0.0113,-0.1452,-0.0160,-0.0700,-0.0454,-0.0147,0.0898,-0.0661,-0.0072,-0.0654,0.0899,0.0392,-0.1007,0.1151,-0.1294,-0.0225,0.0015,-0.0613,-0.0455,-0.0364,-0.0312,-0.0353,-0.1115,-0.0618,0.0001,0.0053,-0.0383,-0.0474,-0.0183,0.0426,0.0578,-0.1057,-0.0200,0.0271,0.0341,-0.0332,-0.0358,-0.0059,-0.0318,-0.0060,0.0038,-0.0312,0.0109,-0.0292,-0.0500,-0.0195],
    "1": [0.0077,0.0944,-0.0428,0.1155,-0.3050,-0.2412,-0.0830,0.2189,0.1693,0.0103,0.0748,-0.0186,0.0466,0.0762,0.0796,0.1353,-0.0357,-0.1261,0.0572,-0.0780,-0.0211,0.0839,0.0269,-0.0529,-0.0222,-0.0199,-0.1045,-0.0080,0.0040,-0.1030,-0.0947,0.0282,0.0155,0.2036,0.1035,0.0545,0.1193,-0.0350,0.1458,-0.0669,0.0975,-0.0189,0.0569,-0.0561,0.0466,0.0647,-0.0680,-0.0612,-0.0009,0.1773,-0.0656,-0.0345,0.0701,0.1478,-0.0325,-0.0864,0.0290,-0.1142,-0.1041,-0.0886,0.1599,-0.0373,-0.0338,-0.0773,-0.0230,-0.0584,0.0717,0.1360,-0.0685,0.1513,-0.0297,-0.1376,0.0077,0.0912,0.1246,0.0486,-0.0121,0.0099,-0.0133,0.1053,-0.0813,-0.0688,0.0225,-0.1743,0.0153,-0.0717,-0.0471,-0.0253,0.1406,-0.0746,-0.0062,-0.0977,0.0663,-0.0074,-0.0987,0.1128,-0.1203,-0.0290,0.0867,-0.0547,-0.0110,-0.0722,-0.0671,-0.0109,-0.1558,-0.0811,0.0521,0.0325,-0.0161,-0.0850,-0.0438,0.0462,0.0874,-0.1192,-0.0534,0.0599,0.0581,-0.0545,0.0121,-0.0575,-0.0161,-0.0778,0.0860,-0.0636,0.0036,-0.0070,-0.0555,-0.0186],
    "2": [-0.2175,-0.1049,-0.0487,0.0613,-0.1672,0.0619,-0.0058,0.1916,0.0644,-0.0907,0.0876,0.0609,0.0257,-0.0962,0.0678,0.0333,-0.0904,-0.0900,0.1516,-0.1562,-0.0654,0.0606,0.0988,-0.0080,-0.0468,0.0064,-0.0473,0.0118,-0.0886,-0.1223,0.0281,0.0119,-0.0618,0.1628,0.0990,0.0564,0.1271,-0.0351,0.1439,-0.0791,0.0923,-0.0680,0.0051,0.0117,0.0566,0.0936,0.0036,-0.0484,-0.0556,0.1759,-0.1002,-0.0082,0.1414,0.0753,-0.0106,-0.0485,0.0459,-0.0255,-0.0688,-0.0757,0.1886,0.0325,0.0145,-0.0385,-0.0313,0.0075,0.1179,0.1594,0.0187,0.0887,-0.0438,-0.1319,0.0354,0.0379,0.1274,0.0455,-0.0032,-0.0034,0.0084,0.1091,-0.0209,-0.0611,-0.0070,-0.1151,-0.0321,-0.0479,-0.0577,-0.0086,0.0716,-0.0832,0.0096,-0.0801,0.1189,0.0320,-0.0938,0.1504,-0.1370,-0.0095,-0.0285,-0.0275,-0.0289,-0.0367,-0.0024,-0.0490,-0.1220,-0.0897,0.0005,-0.0194,0.0028,-0.0507,-0.0333,0.0679,0.0598,-0.1042,-0.0116,0.0608,0.0281,-0.0411,-0.0040,0.0079,-0.0264,-0.0126,0.0041,-0.0508,0.0474,-0.0756,-0.0407,-0.0243]
}

# 将字符串ID转换为整数ID (0, 1, 2)
input_ids = np.array([0, 1, 2])  # 对应于3个示例的索引
embedding_dim = 128  # 嵌入层的输出维度

# 将标签转换为numpy数组
labels = np.array(list(label_examples.values()))

# 创建TensorFlow数据集
dataset = tf.data.Dataset.from_tensor_slices((input_ids, labels))
dataset = dataset.batch(3)  

# 定义模型
class SimpleEmbeddingModel(tf.keras.Model):
    def __init__(self, vocab_size, embedding_dim):
        super(SimpleEmbeddingModel, self).__init__()
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        
    def call(self, inputs):
        return self.embedding(inputs)

# 实例化模型
vocab_size = 3  # 只有3个ID (0, 1, 2)
model = SimpleEmbeddingModel(vocab_size, embedding_dim)

# 编译模型
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='mse')

# 用于存储训练过程中的点积值
dot_products_history = []

# 每个epoch后计算点积值
class DotProductCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch % 1 == 0:  # 每5个epoch记录一次
            predictions = self.model.predict(input_ids, verbose=0)
            dot_products = []
            for i, pred in enumerate(predictions):
                # 计算预测向量和真实标签向量的点积
                dot_product = np.sum(pred * labels[i])
                # 计算向量的范数
                pred_norm = np.sqrt(np.sum(pred**2))
                label_norm = np.sqrt(np.sum(labels[i]**2))
                # 计算余弦相似度
                cosine_similarity = dot_product / (pred_norm * label_norm)
                dot_products.append((dot_product, cosine_similarity))
            dot_products_history.append(dot_products)

dot_product_callback = DotProductCallback()

# 训练模型
history = model.fit(dataset, epochs=100, verbose=1, callbacks=[dot_product_callback])

# 绘制损失曲线
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.plot(history.history['loss'])
plt.title('模型训练损失')
plt.ylabel('损失')
plt.xlabel('Epoch')
plt.grid(True)
plt.yscale('log')  # 使用对数刻度更好地观察损失变化

# 绘制点积值曲线
plt.subplot(1, 2, 2)
epochs = range(0, 100, 1)  # 每5个epoch记录一次
dot_products = np.array(dot_products_history)

# 绘制余弦相似度
plt.plot(epochs, [dp[0][1] for dp in dot_products], '--', label='ID 0 余弦相似度')

plt.title('余弦相似度变化')
plt.ylabel('值')
plt.xlabel('Epoch')
plt.grid(True)
plt.legend(loc='lower right')

plt.tight_layout()
plt.show()

# 测试模型
test_ids = np.array([0, 1, 2])
predictions = model.predict(test_ids)
print(f"预测结果形状: {predictions.shape}")
print(f"预测结果与实际标签的MSE:")
for i, pred in enumerate(predictions):
    mse = np.mean((pred - labels[i])**2)
    print(f"ID {i}: {mse:.6f}")
    
    # 计算并打印点积和余弦相似度
    dot_product = np.sum(pred * labels[i])
    pred_norm = np.sqrt(np.sum(pred**2))
    label_norm = np.sqrt(np.sum(labels[i]**2))
    cosine_similarity = dot_product / (pred_norm * label_norm)
    print(f"ID {i} 点积: {dot_product:.4f}, 余弦相似度: {cosine_similarity:.4f}")
    print(pred)

如何让一个emb一步收敛至另外一个emb

当业务框架限制无法直接导入冻结Embedding时，常需训练一个Lookup Table进行逼近。常规MSE+Adam优化需30轮收敛，效率低。使用 SGD并精准设置学习率（LR），可一步收敛至目标Embedding。关键在于MSE损失函数的梯度缩放方式。

高级算法工程师

人工智能

文章探讨如何让预训练Embedding快速收敛至目标Embedding，先介绍无脑训练法，后提出SGD一步收敛法，分析不同Batchsize下MSE公式及学习率配置，还给出验证代码，最后得出结论。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云AI代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

功能1上新10个字符

功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符。

功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符。

功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符

功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符

功能4上新

文章&问答评论现已支持表情

全新交互，全新视觉，新增快捷键、悬浮工具栏、高亮块等功能并同时优化现有功能，全面提升创作效率和体验

社区富文本编辑器全新改版！诚邀体验～ 

精选全网热门MCP server，让你的AI更好用 🚀

💥开发者 MCP广场重磅上线！

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

如何让一个emb一步收敛至另外一个emb

如何让一个emb一步收敛至另外一个emb

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐