EVA-Qwen2.5-32B-v0.1-GGUF
| Property | Value |
|---|---|
| Parameter Count | 32.8B |
| License | Apache 2.0 |
| Author | bartowski |
| Base Model | EVA-UNIT-01/EVA-Qwen2.5-32B-v0.1 |
What is EVA-Qwen2.5-32B-v0.1-GGUF?
EVA-Qwen2.5-32B-v0.1-GGUF is a comprehensive collection of quantized versions of the EVA-Qwen2.5-32B model, optimized for different deployment scenarios and hardware configurations. This model has been trained on 10 diverse datasets and offers various quantization options ranging from full 16-bit precision to highly compressed versions.
Implementation Details
The model uses the llama.cpp framework for quantization and implements an innovative imatrix approach for optimization. It supports a specific prompt format using im_start and im_end tokens for system and user interactions.
- Multiple quantization options from F16 (65.54GB) to IQ2_XXS (9.03GB)
- Specialized versions for ARM inference with different optimization levels
- Implementations using both K-quants and I-quants for different use cases
Core Capabilities
- Text generation with high-quality outputs across multiple domains
- Flexible deployment options for different hardware configurations
- Optimized performance on various platforms (CPU, GPU, ARM)
- Memory-efficient operation with minimal quality loss in recommended versions
Frequently Asked Questions
Q: What makes this model unique?
The model stands out for its comprehensive range of quantization options and careful optimization using imatrix calibration, allowing users to choose the perfect balance between model size and performance for their specific needs.
Q: What are the recommended use cases?
For most users, the Q4_K_M quantization (19.85GB) offers a good balance of quality and size. Users with more RAM might prefer Q6_K_L (27.26GB) for near-perfect quality, while those with limited resources can use IQ3_XS (13.71GB) for decent performance at a smaller size.
