Skip to contents

Authors

  • David Zimmermann-Kollenda. Author, maintainer.

  • Roger Zurawicki. Author.
    tiktoken-rs Rust library

  • Authors of the dependent Rust crates. Author.
    see AUTHORS file

Citation

Source: DESCRIPTION

Zimmermann-Kollenda D, Roger Zurawicki, Authors of the dependent Rust crates (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.0.6, https://github.com/DavZim/rtiktoken/, https://davzim.github.io/rtiktoken/.

@Manual{,
  title = {rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models},
  author = {David Zimmermann-Kollenda and {Roger Zurawicki} and {Authors of the dependent Rust crates}},
  year = {2024},
  note = {R package version 0.0.6, https://github.com/DavZim/rtiktoken/},
  url = {https://davzim.github.io/rtiktoken/},
}

Additional details

Authors of vendored cargo crates

Additional details

 - aho-corasick 1.1.3: Andrew Gallant

Additional details

 - anyhow 1.0.89: David Tolnay

Additional details

 - autocfg 1.4.0: Josh Stone

Additional details

 - base64 0.21.7: Alice Maz, Marshall Pierce

Additional details

 - bit-set 0.5.3: Alexis Beingessner

Additional details

 - bit-vec 0.6.3: Alexis Beingessner

Additional details

 - bitflags 2.6.0: The Rust Project Developers

Additional details

 - bstr 1.10.0: Andrew Gallant

Additional details

 - cfg-if 1.0.0: Alex Crichton

Additional details

 - extendr-api 0.7.1: andy-thomason, Thomas Down, Mossa Merhi Reimert, Claus O. Wilke, Hiroaki Yutani, Ilia A. Kosenkov, Michael Milton

Additional details

 - extendr-macros 0.7.1: andy-thomason, Thomas Down, Mossa Merhi Reimert, Claus O. Wilke, Hiroaki Yutani, Ilia A. Kosenkov, Michael Milton

Additional details

 - fancy-regex 0.12.0: Raph Levien, Robin Stocker

Additional details

 - lazy_static 1.5.0: Marvin Löbel

Additional details

 - libR-sys 0.7.1: andy-thomason, Thomas Down, Mossa Merhi Reimert, Claus O. Wilke, Ilia A. Kosenkov, Hiroaki Yutani

Additional details

 - libc 0.2.159: The Rust Project Developers

Additional details

 - lock_api 0.4.12: Amanieu d'Antras

Additional details

 - memchr 2.7.4: Andrew Gallant, bluss

Additional details

 - once_cell 1.20.2: Aleksey Kladov

Additional details

 - parking_lot 0.12.3: Amanieu d'Antras

Additional details

 - parking_lot_core 0.9.10: Amanieu d'Antras

Additional details

 - paste 1.0.15: David Tolnay

Additional details

 - proc-macro2 1.0.87: David Tolnay, Alex Crichton

Additional details

 - quote 1.0.37: David Tolnay

Additional details

 - redox_syscall 0.5.7: Jeremy Soller

Additional details

 - regex 1.11.0: The Rust Project Developers, Andrew Gallant

Additional details

 - regex-automata 0.4.8: The Rust Project Developers, Andrew Gallant

Additional details

 - regex-syntax 0.8.5: The Rust Project Developers, Andrew Gallant

Additional details

 - rustc-hash 1.1.0: The Rust Project Developers

Additional details

 - scopeguard 1.2.0: bluss

Additional details

 - serde 1.0.210: Erick Tryzelaar, David Tolnay

Additional details

 - serde_derive 1.0.210: Erick Tryzelaar, David Tolnay

Additional details

 - smallvec 1.13.2: The Servo Project Developers

Additional details

 - syn 2.0.79: David Tolnay

Additional details

 - tiktoken-rs 0.5.9: Roger Zurawicki

Additional details

 - unicode-ident 1.0.13: David Tolnay

Additional details

 - windows-targets 0.52.6: Microsoft

Additional details

 - windows_aarch64_gnullvm 0.52.6: Microsoft

Additional details

 - windows_aarch64_msvc 0.52.6: Microsoft

Additional details

 - windows_i686_gnu 0.52.6: Microsoft

Additional details

 - windows_i686_gnullvm 0.52.6: Microsoft

Additional details

 - windows_i686_msvc 0.52.6: Microsoft

Additional details

 - windows_x86_64_gnu 0.52.6: Microsoft

Additional details

 - windows_x86_64_gnullvm 0.52.6: Microsoft

Additional details

 - windows_x86_64_msvc 0.52.6: Microsoft

Additional details


    

Additional details

(This file was auto-generated from 'cargo metadata' on 2024-10-11)