A new China localization benchmark has put a dent in the default assumption that human translators beat machines. Blind evaluation of 774 English-to-Chinese outputs found professional translators outside the top five in four of six content types—and 10th of 15 on marketing copy.
The study, a joint project from localization provider EC Innovations and digital consultancy Jademond Digital, compared seven workflow models across translation, transcreation and creation tasks. Native Chinese localizers scored each deliverable on accuracy and consistency, fluency and language quality, and style and cultural adaptation, with each dimension worth one third.
Where the AI workflows actually won
Post-edited Qwen won three content categories outright, tied another and placed second on the rest. On marketing copy, humans finished 22.2 points behind the best AI workflow. The only places humans won—SEO and informational content—were much tighter, with a 2.8-point lead over the best machine workflow.
The marketing miss has a specific cause. EC Innovations argues that professional translators tend to over-correct, smoothing copy toward formal correctness and stripping out the contemporary register marketing needs. The instinct that makes a linguist strong at terminology can make them a poor fit for writing that needs to sound like the internet.
Don’t trust the category average
The most useful warning for marketers is not “use Qwen.” It’s that category averages hide variance. On SEO content, “Chinese LLMs” averaged 60.7, but the four underlying models spread by 9.2 points—and on technical content the same group span 22.3 points, from Qwen at 70.4 to Kimi at 48.1.
That means model choice matters more than whether you add post-editing. The study found post-editing is not a uniform quality layer: it added 30.6 points to raw Google Translate on user-generated content but reduced a ChatGPT draft on marketing copy by 3.7 points.
What this means for your localization stack
- Stop buying “Chinese LLMs” as a category. Test Qwen, Doubao and DeepSeek on your own content.
- For flagship SEO and informational pages, human translation or post-edited Qwen/Doubao are close enough that cost and turnaround should decide.
- For marketing, UI, technical and social copy, reconsider paying human rates; post-edited Qwen led all four in this study.
- If you run a mature machine translation pipeline, add post-editing before replatforming. Post-edited Google Translate beat every raw model on SEO content at 66.7.
The ranking caveat you cannot skip
This benchmark measured localization quality only—no rankings, traffic or conversions. The study itself cautions against presenting the 22.2-point quality gap as a ranking outcome, and points to Portent’s crawl of 756,297 pages that found no correlation between readability and Google position.
For Chinese SEO specifically, Baidu leans on site-level signals such as domain history, ICP filing and hosting geography. Getting those in order can matter more than moving a translation score from 60.7 to 74.1.
The real takeaway: run your own bake-off
Every model was tested at its December 2025 version, and the major models have all shipped since. That is the strongest argument for a two-week internal bake-off on your own content, scored blind, across the three or four models you would actually deploy.
Then tag localized pages by workflow in your analytics and watch what happens for two quarters. Your own verticals and competitive set will answer the ranking question this study deliberately did not.
Source: Search Engine Journal
