Concept consistency score
CCS measures how CLIP attention heads align with concepts; high-CCS heads preserve performance but can amplify social bias.
...
Read more
Why it matters
Too much of the market still treats evaluation, interpretability and reliability as secondary concerns. That works until teams need to confidently choose a model, mitigate hallucinations or explain how an AI decision was reached.
Our approach
We tackle these challenges from the ground up. Using a first-principles approach to evaluation, we build methods that rigorously assess AI systems at every stage: input, output and internal decision-making.
Blog
p-less Sampling: A robust LLM decoding strategy
In this research paper, we introduce a novel approach of sampling named 𝜌-less sampling for text generation in autoregressive models.
Blog
The next frontiers in AI — according to industry leaders
This research paper explores the key trends we think are particularly important in generative AI
From the labs
P-less Sampling: A robust hyperparameter-free approach for LLM decoding
Discover P-less Sampling, a hyperparameter-free LLM decoding method that improves robustness, coherence and reliability in AI-generated text....
Read more
p-less Sampling: A robust LLM decoding strategy
𝜌-less sampling improves autoregressive text generation by offering a new alternative to existing sampling techniques for LLMs.
...
Read more
Evaluating LLM-generated summaries using the Lie algebra framework
We model summarization as a geometric flow, using Lie algebra to detect incompleteness via source-text contribution vectors....
Read more
The next frontiers in AI — according to industry leaders
AI is evolving fast. Keynotes and events reveal the most important emerging trends and narratives shaping generative AI today....
Read more
Calculating uncertainty in generative AI
Dropout estimates model uncertainty cost-effectively using prediction variance or softmax entropy while improving generalization and reducing overfitting....
Read more
Evaluating LLMs using semantic entropy
Semantic entropy detects LLM confabulations better than naive entropy by evaluating meaning, generalizing across tasks and datasets....
Read more
LLM benchmarks, evals and tests
Evaluating LLM apps is hard due to open-ended outputs, variable behavior, complex reasoning and diverse real-world tasks....
Read more
Decoding LLM uncertainties for better predictability
We present two model uncertainty measures: structural uncertainty (using token entropy) and conceptual uncertainty (via embedding cosine distances)....
Read more
A surprisingly effective way to estimate token importance in LLM prompts
We explore a proof-of-concept tool to gauge individual token importance within prompts using text embeddings, excited to build on it with the community....
Read more
Probabilistic machine learning and weak supervision
We demonstrate a proof of principle for human-machine collaboration to label training data and build ML models using probabilistic rather than hard labels....
Read more
A gentle introduction to machine teaching
While ML focuses on optimizing algorithms, Machine Teaching addresses AI's domain expertise bottleneck by making humans more effective at teaching models....
Read more
TinySQL
Discover TinySQL, a progressive text-to-SQL dataset from Thoughtworks Research designed for mechanistic interpretability research....
Read more
Beyond linear steering: Unified multi-attribute control for language models
A unified approach to controlling multiple language-model attributes at once, moving beyond linear steering for more precise, flexible text generation....
Read more
Turning up the heat: Min-p samling for creative and coherent creative outputs
Learn how Min-p sampling improves LLM creativity and coherence by dynamically adapting token selection for higher-quality AI-generated text....
Read more
Beyond I am sorry, I can’t: dissecting large language model refusal
Explore how large language models generate refusals, why they happen and what they reveal about AI safety, alignment and model behavior....
Read more
Distribution-aware feature selection for SAEs
Learn how distribution-aware feature selection improves sparse autoencoders by identifying more meaningful and reliable latent features in AI models....
Read more
Towards transparent AI grading: Entropy as a signal for human-AI disagreement
Discover how semantic entropy helps detect human-AI grading disagreements, improving transparency and trust in AI-assisted assessment....
Read more
Steering smarter
Activation engineering steers LLM behavior at inference time by perturbing intermediate activation vectors during forward passes....
Read more
Partners and collaborations
Thoughtworks AI labs sit within a wider network of organizations spanning public AI research, semiconductor innovation, cloud platforms, open source and AI engineering.
These relationships strengthen the lab’s ability to contribute to the methods, tools and technical standards shaping reliable AI.
For partnerships and collaboration inquiries
email ai-labs@thoughtworks.com