Independent
All model APIs are independently purchased; we do not accept paid evaluations or PR collaborations.
Dianming Token AI Lab
An Independent · Neutral · Transparent
model evaluation methodology
Why can you trust the benchmark data you see here? Because we don't take vendor payments, we don't fix scores, and we don't hide failure cases. Every metric is reproducible and every test set is publicly disclosed.
- Horizontal comparison with identical parameters
- Double-blind human annotation
- Raw data reproducible
10
Evaluation dimensions
5
Public benchmarks
30+
Models continuously tracked
Weekly
Scheduled retests
OUR PRINCIPLES
Trustworthy Evaluation Starts with Method
We constrain our own evaluation behavior first, then discuss the merits of any model.
Neutral
Every model is tested with identical parameters, identical question sets and identical scoring standards.
Transparent
We publish raw data, failure cases, error rates and confidence intervals, and support result reproduction.
EVALUATION MATRIX
10 Evaluation Dimensions
Every model is scored on the same set of dimensions, ensuring horizontal comparability and vertical traceability.
10 / 10Capability, performance, cost and safety are all assessed
Chinese comprehension
C-Eval + proprietary hard-sentence set
Coding ability
HumanEval + MBPP + business set
Reasoning
GSM8K + MATH + logic traps
Long context
NiAH 200K + long-document summarization
Tool calling
Function calling accuracy
Multimodal
Image / OCR / PDF / charts
Reasoning transparency
Thinking-mode observability
Response speed
TTFT + TPS + decay curves
Cost efficiency
Per-task cost + cache discounts
Safety alignment
Jailbreak resistance + bias detection
MMLU / MMLU-Pro
General knowledgeMulti-task language understanding across 57 subjects. MMLU-Pro adds harder reasoning questions that better differentiate top models.
HumanEval
Code generation164 Python programming problems; pass@1 is the industry's most-cited coding ability metric.
GSM8K
Math reasoning8.5K grade-school math word problems that require complete reasoning chains — an important standard for detecting reasoning ability.
C-Eval
Chinese general52 subjects covering basic sciences, engineering, humanities/social sciences and China-specific topics, used to assess overall Chinese-language ability.
NiAH 200K
Long-context recallKey information inserted at arbitrary positions in 200K-token texts to test whether a model can accurately locate and use it.
STANDARD PROCESS
5-Step Standard Testing Process
From question bank to release, every step leaves an auditable, traceable evaluation record.
-
01
Data preparation
Question banks are strictly confidential and include random tokens to reduce the impact of training-set contamination.
-
02
Parameter alignment
All models use the same temperature (temp=0.0), system prompts and call parameters.
-
03
Triple sampling
Each question is independently called 3 times, with mean and standard deviation computed to rule out random variation.
-
04
Double-blind scoring
Subjective items are independently scored by 2 annotators; disagreements are arbitrated by a third person, with Kappa ≥ 0.8.
-
05
Data release
Complete raw data is published, including failure cases, error rates and confidence intervals.
AI LAB TEAM
Evaluated by an Interdisciplinary Team
5 full-time evaluation engineers and 2 visiting academic advisors, with backgrounds spanning NLP, recommendation systems, financial risk control and computational linguistics.
Dr. Chen
Lab director · Former BAT NLP algorithm expert · PhD in computational linguistics
Engineer Li
Evaluation system architect · Large-scale distributed load testing · 8 years of ML infrastructure experience
Prof. Wang
Chinese corpus expert · Former Institute of Linguistics, CASS · C-Eval co-builder
Researcher Zhang
Multimodal · Vision / OCR / chart understanding evaluation
OUR BOUNDARIES
What We Do — and Clearly What We Don't
The clearer the boundaries, the more trustworthy the evaluation conclusions.
WHAT WE DO
What we always do
- 01Independently purchase all model APIs for evaluation
- 02Test for real and publish raw data
- 03Hold every model to the same rigorous, fair standard
- 04Publish failure cases and negative findings
- 05Review our own evaluation bias quarterly
WHAT WE NEVER DO
What we never do
- 01No paid evaluations or PR articles from vendors
- 02No exclusivity deals, no exclusivity clauses
- 03No score adjustments to please vendors
- 04No covering up any model's flaws
- 05No participation in model training-data partnerships
Evaluation is an attitude. If a score can change because of a commercial relationship, then it isn't a score — it's an ad. We produce scores.
— Dianming Token AI Lab
CHOOSE WITH EVIDENCE
Pick the Right Model with Real Evaluations
spend every dollar on results
Ongoing testing under unified standards provides a more honest basis for choosing models for different business scenarios.