py3-tiktoken-0.13.0
fast BPE tokenizer for OpenAI models
Back to search · Project homepage
Description
tiktoken is a fast BPE (byte pair encoding) tokenizer for use with
OpenAI's models.
Package information
- Ports path
- textproc/py-tiktoken
- Package architecture
- amd64
- Maintainer
- The OpenBSD ports mailing-list <ports@openbsd.org>
- Categories
- textproc, lang/python, lang/rust
- Available flavors
- None listed
- Only for architectures
- aarch64, amd64, riscv64, sparc64
These are ports metadata. Binary availability depends on the release, architecture and mirror. Build and test dependencies are not an installation checklist.
Direct dependencies
Runtime
Build
- lang/python/3
- devel/py-build
- devel/py-installer
- devel/py-setuptools py3-setuptools->=79v0
- devel/py-wheel
- devel/py-setuptools-rust
- devel/cargo-generate-vendor STEM->=1.1
- lang/rust
Test
Used by (4 dependency relationships)
Includes library, runtime, build and test relationships. Results load 100 at a time.