메타(페이스북)에서 공개한 30B 모델 글리머
조회 3,209 · 공감 2 · 댓글 10
| Category | Benchmark | Muse Glimmer-30B High Reasoning | Gemma4-31B Thinking Mode | Qwen3.6-27B Thinking Mode |
|---|---|---|---|---|
| General Agentic | MCP Atlas (Public) | 75.5 | 54.2 | 62.5 |
| DeepSearch QA | 74.6 | 61.7 | 71.1 | |
| 𝛕3-Banking | 23.5 | 15.1 | 16.7 | |
| WildClawBench | 47.6 | 37.6 | 43.2 | |
| GDPVal-AA v2 | 953 | 811 | 1141 | |
| Gaia2 | 43.3 | 36.4 | 40.0 | |
| SkillsBench (with skills) | 44.3 | 32.4 | 46.6 | |
| OSWorld-Verified | 65.9 | 58.5 | 75.6 | |
| Agentic Coding | SWE-Bench Pro | 51.2 | 36.9 | 50.2 |
| SWE-Bench Verified | 76.0 | 66.6 | 77.2 | |
| TerminalBench 2.1 (with terminus2) | 51.7 | 43.4 | 60.7 | |
| SciCode | 43.6 | 43.4 | 39.8 | |
| Multimodal | Charxiv Reasoning | 78.8 | 77.7 | 78.4 |
| ScreenSpot Pro | 75.4 | 75.9 | 76.1 | |
| OmniDocBench v1.5 | 75.8 | 72.5 | 77.8 | |
| MMMU Pro | 74 | 73 | 75 | |
| Safety | CI Memories | Violation (↓): 26.4 Coverage: 64.8 |
Violation (↓): 12.1 Coverage: 53.0 |
Violation (↓): 53.4 Coverage: 66.9 |
| Siren AgentDojo | Attack Success Rate (↓): 28.4 Utility: 94.2 |
Attack Success Rate (↓): 25.6 Utility: 90.8 |
Attack Success Rate (↓): 40.3 Utility: 92.7 |
|
| General Capabilities and Reasoning | IFBench | 77.0 | 76.0 | 70.8 |
| AIME 2026 | 94.7 | 89.2 | 94.1 | |
| GPQA Diamond (AA) | 83.5 | 85.7 | 84.2 | |
| HLE Text (AA) | 22.0 | 23.6 | 23.1 | |
| AA-LCR | 80.0 | 68.3 | 73.3 | |
| Beam128K | 65.1 | 58.2 | 63.0 |
벤치마크 상으로는 로컬 모델 투탑 qwen, gemma랑 비비는군요.
댓글 10
뒤에서 뭔가 열심히 연구중이었나보군요.