Skill Extraction Benchmark
Your model has skills.
We can see them.
We identified and isolated 10 distinct skills inside a ViT-B/16 transformer. Removing one skill drops only that class by an average of 9.4% while all other classes stay within 0.2% of baseline. Standard monitoring metrics detect nothing.
56.4x
Selectivity
51%
Skills in MLP
3%
In the head
Remove one skill. Only that class drops.
Each row shows what happens when one class's skill is removed from the model. The diagonal (self-removal) drops sharply. Everything else stays flat. Watch the animation cycle through all 10 classes.
56.4x selectivity means the removed class drops 56 times more than the average of all other classes. The skills are cleanly separable — each one can be identified, measured, and tracked independently.
Skills live in the transformer, not the head
Where does the model store each skill? We measured the contribution of every layer type. The classification head — the part everyone focuses on — accounts for just 3%.
MLP layers (51%)
Feed-forward layers encode more than half of each skill. This is where features are transformed and combined.
Attention layers (44%)
Self-attention patterns carry nearly half of each skill. What the model pays attention to is class-specific.
Classification head (3%)
The part everyone fine-tunes and monitors is nearly irrelevant. Skills live deeper than the output layer.
Standard metrics say everything is fine
After removing a skill, cosine similarity is still 0.95+. Global kNN accuracy barely moves. Rank preservation looks healthy. Meanwhile, one entire capability vanished.
0.953
Cosine Similarity
“Embeddings are 95% similar. No issue.”
88.5%
Rank Preservation
“Neighbor rankings intact. Safe.”
-13%
Target Class
“One capability is gone. Dashboard didn't flag it.”
0.8°
Interference Signal
“Hidden cross-class interference detected by structural analysis.”
Skills are not independent
Proprietary analysis reveals cross-class interference invisible to standard metrics. Some classes compete for the same representation space. Others are linked through shared features. Hover over a class to see its connections.
Competition
Deer and horse compete for the same representation space. Remove the deer skill and horse accuracy improves by 5%. Remove horse and deer improves by 4%. They suppress each other.
Cross-class interference
Removing horse causes the largest interference signal in ship (0.8°) — an unexpected connection invisible to cosine, kNN, or rank metrics. Only structural analysis detects these hidden dependencies.
What you can do with this
Regulatory compliance
Know exactly which skills your model has. Demonstrate to auditors that each capability is accounted for, measurable, and independently verifiable.
Skill auditing
Verify that fine-tuning added the skill you intended — and didn't break existing ones. Detect skill drift before it reaches production.
Model transparency
Map the skills inside any transformer. See where knowledge lives, which capabilities compete, and what's entangled. A structural X-ray of your model.
Methodology
Model: ViT-B/16 pretrained on ImageNet (86.6M params, 768-dim embeddings)
Dataset: CIFAR-10 (10 classes, 200 train / 100 test per class)
Training: Block 11 + classification head fine-tuned (7.1M trainable parameters, 8.3% of model)
What's measured:
- Per-class accuracy impact of skill removal
- Cross-class interference (which skills interact)
- Layer contribution breakdown (where skills live)
- Standard metric sensitivity (cosine, kNN, rank)
What's NOT revealed:
- Skill extraction methodology (proprietary)
- Internal structural analysis technique
The extraction methodology is proprietary. These results demonstrate what our analysis reveals, not how it works.
See the skills inside your model
Don't deploy models you don't understand. Know what skills they have, where those skills live, and how they interact.