Evaluation, benchmark and RL-environment work for ML engineers

Reviewing code asks whether it runs. ML postings ask whether an experiment is sound. Everything below comes from open postings, read 13 October 2026.

The error you are hired to catch looks fine

Mercor’s standing Machine Learning Expert listing names its targets. They include a loss that does not match the objective, an evaluation set that leaks and an ablation that does not support its conclusion. It also names approaches that train cleanly and still fail in production, through distribution shift, label noise or a metric that rewards the wrong behaviour.

Leakage comes up in 4 postings. SME Careers’ Python and data science lead roles list it beside reproducibility and statistical validity. None of this is a failing unit test. For plain code correctness, see the software engineers guide.

Three kinds of work

Of the 22 open roles, 2 are reinforcement-learning environment roles and 9 carry “data scientist” in the title. The rest are ML engineer and systems roles. Platforms posting them: micro1 (6), Terac (4), Mercor (3), AfterQuery (3), SME Careers (3) and Ethos (3).

Benchmarks and evals. micro1’s Competitive Data Scientist posting wants Kaggle-style experience. You build a reproducible environment, a verified reference solution and checks that separate correct work from plausible work. You then run coding agents against the task and refine it from their failures.

RL environments. Terac’s scenario-building role asks ML and simulation engineers to construct structured environments and evaluate how an agent behaves in them. Its world-building role covers professional workflows in various domains. Both postings are only a couple of sentences, so the detail of the work is unpublished.

Systems and infrastructure. AfterQuery’s Software/ML Engineer role covers inference serving and GPU kernels. AfterQuery calls the work research and evaluation, not production engineering. micro1’s ML Engineer posting lists PyTorch, JAX, vLLM and Hugging Face tooling and calls itself very coding heavy.

Pay, and one unit to check

The median ceiling is $120, the upper quartile $200 and the highest $280, across 20 hourly roles. The 9 data scientist roles have a median ceiling of $110. Ethos lists its data scientist roles at one stated rate.

Check the unit. micro1’s ML Engineer posting shows an hourly band but describes output-based pay. You are paid for each task that meets the specification, and a minimum submission rate applies. The 2 fixed-fee roles are Terac prompt-design studies for domain experts in general, so they say little about ML work.

Conflicts with your employer

Who the client is. In 6 postings the client is described only as a leading AI lab. Mercor says a specific project listing names the rate, hours and client. You may learn who it is only after joining a pool. If your employer builds or fine-tunes models, or competes with labs that buy this data, your contract may bar the work. Ask in writing before you accept.

What you build from. Mercor’s standing listing asks for a modelling problem from a system you shipped, with its data, objective and constraint. That system and its data may be your employer’s. Use public datasets and problems you own outright. Kaggle and other competition data carry their own licence terms, so read them.

AI tools. micro1 expects you to use coding agents. Sovrano, outside this list, says work produced with AI tools is rejected in its assessment. Each platform sets its own rule, so follow the one on the project in front of you.

Credentials, hours and country

micro1’s Data Scientist posting prefers a top-ranked undergraduate university and a year at a top company within seven years. It also prefers a base in an English-speaking country. That is a stated preference. Mercor asks for experience putting models in front of real users, not a named school.

Mercor pools that say they are not specific openings account for 3 of these roles. Hours run from up to 20 a week on the Ethos postings to 15 to 30 on Mercor’s network. AfterQuery assigns work in two- to three-week blocks.

Roles limited to the US: 2 of 22. A reader in the UK is shown 19 and one in Germany 20. The Terac screening guide and the Mercor interview guide explain how to get in.

What the postings do not say

• Whether an RL environment you build is reused, and who owns it.
• How a task is scored against your reference.
• Which lab is behind most of the listings.
• How often a pool member is matched.

This site is not affiliated with any platform named here, and passing a screening is not a promise of work.