cmake 3.27.9 pypi_0 pypi comm 0.1.2 py38h06a4308_0 crypto Collaborator Author Author Atrix21commentedDec 19, 2023 Atrix21commentedDec 20, 2023 Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment...
DNABERT-2 的作者在论文中提到,他们采用 SentencePiece 的框架实现了 BPE Tokenization 的过程,但我并未找到具体的代码和细节。通过查看 Hugging Face Repo,他们所采用的框架应该是 Hugging Face 的 tokenizers。由于文章篇幅和格式所限,我把Notebook发布在了专栏的GitHub Repo中,里面除了 tokenizers 的实现,还有 Sente...
Hi @Zhihan1996, You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at zhihan1996/DNABERT-2-117M and are newly initialized: ['classifier.bias','bert.pooler.dense.bias','classifier.weight','bert.pooler.dense.weight'] You should probably TRAIN this model on a down-stream ...
zhaoweiyu-github commented Aug 9, 2023 I've encountered a similar issue. I believe the recently released flash-attention version 2 significantly improves the execution time for DNA-BERT2. By the way, I greatly appreciate your work; it has been very helpful to me. Collaborator Zhihan1996 ...