GGUF Models Now Run Directly in Hugging Face Transformers with ggml Metal Kernels

Key Info

Hugging Face integrates ggml's Metal kernels into the transformers ecosystem, allowing GGUF models (commonly used with llama.cpp) to run directly in transformers, enhancing compatibility and local inference performance on Mac.

Highlights

• Millions of GGUF downloads later, the same llama.cpp checkpoints can now be used in Hugging Face transformers.
• Integration of ggml kernels brings fast local inference on Mac powered by Metal.
• A blog post and kernel repository link are provided for more details.