10:59 [动手写神经网络] pytorch 高维张量 Tensor 维度操作与处理,einops 23:03 [动手写 Transformer] 手动实现 Transformer Decoder(交叉注意力,encoder-decoder cross attentio) 14:43 [动手写神经网络] kSparse AutoEncoder 稀疏性激活的显示实现(SAE on LLM
This paper proposes a novel k-sparse denoising autoencoder (KDAE) with a softmax classifier for HSI classification. Based on the stack-type autoencoder, KDAE adopts k-sparsity and random noise, employs the dropout method at the hidden layers, and finally classi...
In order to accelerate training, K-means clustering optimizing deep stacked sparse autoencoder (K-means sparse SAE) is presented in this paper. First, the input features are divided into K small subsets by K-means clustering, then each subset is input into corresponding autoencoder model for ...
现在我们来跑AE模型(Auto-encoder),简单说说AE模型,主要步骤很简单,有三层,输入-隐含-输出,把数据input进去,encode然后再decode,cost_function就是output与input之间的“差值”(有公式),差值越小,目标函数值越优。简单地说,就是你输入n维的数据,输出的还是n维的数据,有人可能会问,这有什么用呢,其实也没什么用,...
现在我们来跑AE模型(Auto-encoder),简单说说AE模型,主要步骤很简单,有三层,输入-隐含-输出,把数据input进去,encode然后再decode,cost_function就是output与input之间的“差值”(有公式),差值越小,目标函数值越优。简单地说,就是你输入n维的数据,输出的还是n维的数据,有人可能会问,这有什么用呢,其实也没什么用,...
现在我们来跑AE模型(Auto-encoder),简单说说AE模型,主要步骤很简单,有三层,输入-隐含-输出,把数据input进去,encode然后再decode,cost_function就是output与input之间的“差值”(有公式),差值越小,目标函数值越优。简单地说,就是你输入n维的数据,输出的还是n维的数据,有人可能会问,这有什么用呢,其实也没什么用,...
详细见Algorithm 2。a倍的放大连接过程是重要的过程。当a=0的时候,梯度后向传播时不会传播经过这些失败的神经元,模型就会变成常规的k-sparse自编码器。当a>2/k时(为什么是2/k,而不是直接是a>0,还没想清楚),后向梯度会经过这些失败神经元传播。从经验上来说,放大的过程帮助提升KATE模型。
当然了,本文依旧是参考论文An Analysis of Single-Layer Networks in Unsupervised Feature Learning,Adam Coates, Honglak Lee, and Andrew Y. Ng. InAISTATS 14, 2011.只是重点在分析4个算法中的kemans算法(因为作者只提供关于kmeans的demo,呵呵,当然了另一个原因是sparse autoencoder在前面的博文中也介绍很多了)...
[27] adopt stacked sparse autoencoder to learn important features and experiments validate the effectiveness, but it increases the difficulties of training network. Chang et al. [28] present a deep clustering model that adds global and local constraints of pattern relationships. However, it performs...
Interpretability: Mechanistic interpretability techniques like Sparse Autoencoders (SAEs) made remarkable progress to provide insights about the inner workings of LLMs. This has also been applied with techniques such as abliteration, which allow you to modify the behavior of models without training. Te...