*/ #pragma once #include "encoding_utils.h" #include <string> #include <unordered_map> #include <vector> class IResourceReader { public: virtual std::vector<std::string> readLines() = 0; }; class EmbeddedResourceReader : public IResourceReader { public: EmbeddedResourceReader(const std::...
Cpp-Tiktoken This is a C++ implementation of a tiktoken tokenizer library for C++. It was heavily inspired byhttps://github.com/dmitry-brazhenko/SharpToken To use, first somewhere have a lines in your project that reads something like: ...
gh-markt/cpp-tiktokenPublic NotificationsYou must be signed in to change notification settings Fork7 Star16 Code Issues Pull requests Actions Projects Security Insights Additional navigation options Commit Browse filesBrowse the repository at this point in the history ...
Breadcrumbs torchchat /tokenizer / tiktoken.cppTop File metadata and controls Code Blame 390 lines (339 loc) · 11.6 KB Raw // @lint-ignore-every LICENSELINT /*** Copyright (c) 2023 sewenew Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file ex...
gh-markt / cpp-tiktoken Public Notifications Fork 6 Star 14 Code Issues Pull requests Actions Projects Security Insights New issue Add O200K_BASE #9 Merged gh-markt merged 3 commits into gh-markt:master from ydot13:master Jun 18, 2024 ...
master qwen_cpp tests third_party tiktoken_cpp .gitignore .gitmodules CMakeLists.txt LICENSE MANIFEST.in NOTICE README.md base64.h main.cpp pyproject.toml qwen.cpp qwen.h qwen_pybind.cpp qwen_test.cpp setup.py tiktoken.h unordered_dense.hBreadcrumbs...
GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.
All activity All usersAll time Showing most recent first ensure lammma3 tokenizer is copied over gh-marktpushed 1 commit to master • 994693b…50a281f • on Oct 29, 2024 Merge pull request #13 from nguyenhoangthuan99/master Pull request merge gh-marktpushed 6 commits to master • ...
qwen2 and llama3 cpp implementation. Contribute to yvonwin/qwen2.cpp development by creating an account on GitHub.
std::shared_ptr<GptEncoding> GptEncoding::get_encoding(LanguageModel model) std::shared_ptr<GptEncoding> GptEncoding::get_encoding(IModelGetter& model_getter) { ModelParams model_params = ModelParamsGenerator::get_model_params(model); ModelParams model_params = model_getter.get();...