Meta seems to have designed Ember to be a strong "all round" great value model - it dominates most of its class on cost

The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.

On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.

That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]

The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:

- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]

- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]

- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]

Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!

Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]

For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.

I certainly want to measure cost per successfully completed task, including retries and escalation.

Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.

Goign by the generic benchmarks would confuse me to no end and likely burn me out.

Great times to be in!

[1] https://artificialanalysis.ai/leaderboards/models

[2] https://www.vals.ai/benchmarks/fabv2-1

[3] https://www.vals.ai/benchmarks/legal_research

[4] https://www.vals.ai/benchmarks/hlab

[5] https://www.vals.ai/models/meta_muse-ember

[6] https://www.vals.ai/models/openai_gpt-6.1-terra

[7] https://github.com/harbor-framework/harbor

Muse Ember: Our smallest, most efficient model