From the available metrics I currently have, I looked at how much quantizing only the mlp.down_proj layers impacts end-to-end perplexity compared to the original float model and to quantizing all linear layers.
I used asymmetric W4A16 quantization and evaluated on the same held-out dataset used for the sensitivity profiling.

Here’s a summary of the results:

Observations:

• Gemma-2B: Quantizing only the mlp.down_proj layers already causes a large perplexity increase (from 16.80 -> 58.29), showing they’re a major bottleneck for precision loss.
• LLaMA-3.2-3B: The impact is smaller (13.73 -> 15.11), but still noticeable.
• Qwen-3-4B: The increase is modest (20.10 -> 22.73), and here the sensitivity is more evenly distributed — other linear layers are nearly as fragile as mlp.down_proj, so the relative contribution is less pronounced.

It would also be interesting to run evaluations of these models on downstream benchmarks to see how these perplexity changes translate into real-task performance differences.

For a detailed breakdown of the divergence-based method used to measure layer sensitivity, you can read my blog post here: https://huggingface.co/blog/badaoui/sensitivity-aware-mixed-precision-quantizer-v1