Quantization

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Reno Lu

Oct 2

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization

4 min read

Jon Handler

Oct 2

1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch

#vectorsearch #airetrieval #opensearch #quantization

8 min read

Nainik Mehta

Sep 23

Hot Take: Deploy Small Language Models to the Edge

#edge #machinelearning #mlops #quantization

4 min read

Syed Anzar

Sep 18

What Model Quantization Actually Does: From Float16 to 4-Bit Weights

#ai #llm #quantization #machinelearning

6 min read

Qasim Parray

Sep 30

Testing the 9x Smaller Local LLM Claim on a GPU-less VPS

#localllm #quantization #selfhosting #gguf

6 min read

Anaz S. Aji for Codecora Dev

Sep 2

A Small Transformer Trained in 1.5 Hours Beat Many LLMs on ARC

#machinelearning #vectorsearch #quantization #benchmark

5 min read

Seth Wheeler

Aug 25

A Better FP4 Gradient Quantizer That Training Couldn't Notice

#llm #measurement #quantization #training

7 min read

Eryk Kubiak

Sep 25

Why Does Your Local Model Crash at 32k Tokens?

#llm #gpu #quantization #machinelearning

12 min read

Seth Wheeler

Aug 21

Error Feedback, Gradient Compression, and Why Adam Breaks It

#llm #measurement #quantization #training

8 min read

Aomi Qaza

Aug 15

KV Cache INT4 Quantization for 1M+ Token Context Windows

#aiengineering #quantization

3 min read

Mr Say Nothing

Sep 13

GGUF Quantization: Which Level Should You Use?

#ai #gguf #quantization #localllm

5 min read

Seth Wheeler

Aug 22

Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors

#llm #quantization #training #measurement

7 min read

xbill for Google Developer Experts

Aug 19

Deploying a QAT Checkpoint Your Serving Stack Can't Load: Gemma 4 E2B in Pure JAX on One TPU

#tpu #jax #llm #quantization

10 min read

Pneumetron

Jul 17

Unsloth Releases Qwen3.6-27B-NVFP4: Enhanced Throughput and Agentic Coding for Developers

#aiml #largelanguagemodels #quantization #qwen

3 min read

Pneumetron

Jul 15

Bonsai-27B: A 1-Bit LLM for On-Device Inference with Llama.cpp and MLX

#llm #quantization #1bit #gguf

3 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.