Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Paper โข 2606.14516 โข Published โข 7
AI & ML interests
Democratizing access to useful AI tools and resources for journalists
irenesolaimanย
authored a
paper 2 months ago
yjerniteย
authored a
paper 7 months ago
Post
1621
The #1 trending AI/ML dataset today ๐
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Post
887
The new King ๐has arrived!
Moonshot AI now the top model on Hugging Face ๐ฅ
moonshotai/Kimi-K2-Thinking
Moonshot AI now the top model on Hugging Face ๐ฅ
moonshotai/Kimi-K2-Thinking
Post
2926
๐ธ๐คYou donโt need 100 GPUs to train something amazing!
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on ๐ค :
HuggingFaceTB/smol-training-playbook
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on ๐ค :
HuggingFaceTB/smol-training-playbook
thomwolfย
authored a
paper 11 months ago
Post
4684
๐ AI ethics and sustainability are two sides of the same coin.
In our new blog post with Dr. Sasha Luccioni, we argue that separating them (as is too often the case) means missing the bigger picture of how AI systems impact both people and the planet.
Ethical and sustainable AI development canโt be pursued in isolation. The same choices that affect who benefits or is harmed by AI systems also determine how much energy and resources they consume.
We explore how two key concepts, evaluation and transparency, can serve as bridges between these domains:
๐ Evaluation, by moving beyond accuracy or performance metrics to include environmental and social costs, as weโve done with tools like the AI Energy Score.
๐ Transparency, by enabling reproducibility, accountability, and environmental reporting through open tools like the Environmental Transparency Space.
AI systems mirror our priorities. If we separate ethics from sustainability, we risk building technologies that are efficient but unjust, or fair but unsustainable.
Read our blog post here: https://huggingface.co/blog/sasha/ethics-sustainability
AIEnergyScore/Leaderboard
sasha/environmental-transparency
In our new blog post with Dr. Sasha Luccioni, we argue that separating them (as is too often the case) means missing the bigger picture of how AI systems impact both people and the planet.
Ethical and sustainable AI development canโt be pursued in isolation. The same choices that affect who benefits or is harmed by AI systems also determine how much energy and resources they consume.
We explore how two key concepts, evaluation and transparency, can serve as bridges between these domains:
๐ Evaluation, by moving beyond accuracy or performance metrics to include environmental and social costs, as weโve done with tools like the AI Energy Score.
๐ Transparency, by enabling reproducibility, accountability, and environmental reporting through open tools like the Environmental Transparency Space.
AI systems mirror our priorities. If we separate ethics from sustainability, we risk building technologies that are efficient but unjust, or fair but unsustainable.
Read our blog post here: https://huggingface.co/blog/sasha/ethics-sustainability
AIEnergyScore/Leaderboard
sasha/environmental-transparency
- 1 reply
Post
2394
Cool stuff these past weeks on huggingface! ๐ค ๐ !
โข ๐Trackio, local-first W&B alternative
https://github.com/gradio-app/trackio/issues
โข ๐EmbeddingGemma, 300M-param, multilingual embeddings, on-device
https://huggingface.co/blog/embeddinggemma
โข ๐ปOpen LLMs in VS Code (Inference Providers)
https://x.com/reach_vb/status/1966185427582497171
โข ๐คSmol2Operator GUI agents
https://huggingface.co/blog/smol2operator
โข ๐ผ๏ธGradio visible watermarking
https://huggingface.co/blog/watermarking-with-gradio
โข ๐Trackio, local-first W&B alternative
https://github.com/gradio-app/trackio/issues
โข ๐EmbeddingGemma, 300M-param, multilingual embeddings, on-device
https://huggingface.co/blog/embeddinggemma
โข ๐ปOpen LLMs in VS Code (Inference Providers)
https://x.com/reach_vb/status/1966185427582497171
โข ๐คSmol2Operator GUI agents
https://huggingface.co/blog/smol2operator
โข ๐ผ๏ธGradio visible watermarking
https://huggingface.co/blog/watermarking-with-gradio
Post
11194
One of the hardest challenges in AI safety is finding the right balance: how do we protect people from harm without undermining their agency? This tension is especially visible in conversational systems, where safeguards can sometimes feel more paternalistic than supportive.
In my latest piece for Hugging Face, I argue that open source and community-driven approaches offer a promising (though not exclusive) way forward.
โจ Transparency can make safety mechanisms into learning opportunities.
โจ Collaboration with diverse communities makes safeguards more relevant across contexts.
โจ Iteration in the open lets protections evolve rather than freeze into rigid, one-size-fits-all rules.
Of course, this isnโt a silver bullet. Top-down safety measures will still be necessary in some cases. But if we only rely on corporate control, we risk building systems that are safe at the expense of trust and autonomy.
Read the blog post here: https://huggingface.co/blog/giadap/preserving-agency
In my latest piece for Hugging Face, I argue that open source and community-driven approaches offer a promising (though not exclusive) way forward.
โจ Transparency can make safety mechanisms into learning opportunities.
โจ Collaboration with diverse communities makes safeguards more relevant across contexts.
โจ Iteration in the open lets protections evolve rather than freeze into rigid, one-size-fits-all rules.
Of course, this isnโt a silver bullet. Top-down safety measures will still be necessary in some cases. But if we only rely on corporate control, we risk building systems that are safe at the expense of trust and autonomy.
Read the blog post here: https://huggingface.co/blog/giadap/preserving-agency
- 8 replies
Post
3304
Tremendous quality of life upgrade on the Hugging Face Hub - we now have auto-complete emojis ๐ค ๐ฅณ ๐ ๐ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Post
432
I've noticed something. While we're careful about what we post on social media, we're sharing our deepest and most intimate thoughts with AI chatbots -- health concerns, financial worries, relationship issues, business ideas...
With OpenAI hinting at ChatGPT advertising, this matters more than ever. Unlike banner ads, AI advertising happens within the conversation itself. Sponsors could subtly influence that relationship advice or financial guidance.
The good news? We have options.
๐ค Open source AI models let us keep conversations private, avoid surveillance-based business models, and build systems that actually serve users first.
Read more about it in our latest blog post, co-written with
@frimelle
https://huggingface.co/blog/giadap/privacy-conversational-ai
With OpenAI hinting at ChatGPT advertising, this matters more than ever. Unlike banner ads, AI advertising happens within the conversation itself. Sponsors could subtly influence that relationship advice or financial guidance.
The good news? We have options.
๐ค Open source AI models let us keep conversations private, avoid surveillance-based business models, and build systems that actually serve users first.
Read more about it in our latest blog post, co-written with
@frimelle
https://huggingface.co/blog/giadap/privacy-conversational-ai
Post
352
๐ We benchmark models for coding, reasoning, or safetyโฆ but what about companionship?
At Hugging Face, weโve been digging into this question because many of you know how deeply I care about how people build emotional bonds with AI.
Thatโs why, building on our ongoing research, my amazing co-author and colleague @frimelle created the AI Companionship Leaderboard ๐ฆพ
frimelle/companionship-leaderboard
Grounded in our INTIMA benchmark, the leaderboard evaluates models across four dimensions of companionship:
๐ค Assistant Traits: the โvoiceโ and role the model projects
๐ท Relationship & Intimacy: whether it signals closeness or bonding
๐ Emotional Investment: the depth of its emotional engagement
๐คฒ User Vulnerabilities: how it responds to sensitive disclosures
This work builds on our paper with @frimelle and @yjernite .
๐ข Now weโd love your perspective: which open models should we test next for the leaderboard? Drop your suggestions in the comments or reach out! Together we can expand the leaderboard and build a clearer picture of what companionship in AI really looks like.
Paper: INTIMA: A Benchmark for Human-AI Companionship Behavior (2508.09998)
INTIMA Benchmark: AI-companionship/INTIMA
At Hugging Face, weโve been digging into this question because many of you know how deeply I care about how people build emotional bonds with AI.
Thatโs why, building on our ongoing research, my amazing co-author and colleague @frimelle created the AI Companionship Leaderboard ๐ฆพ
frimelle/companionship-leaderboard
Grounded in our INTIMA benchmark, the leaderboard evaluates models across four dimensions of companionship:
๐ค Assistant Traits: the โvoiceโ and role the model projects
๐ท Relationship & Intimacy: whether it signals closeness or bonding
๐ Emotional Investment: the depth of its emotional engagement
๐คฒ User Vulnerabilities: how it responds to sensitive disclosures
This work builds on our paper with @frimelle and @yjernite .
๐ข Now weโd love your perspective: which open models should we test next for the leaderboard? Drop your suggestions in the comments or reach out! Together we can expand the leaderboard and build a clearer picture of what companionship in AI really looks like.
Paper: INTIMA: A Benchmark for Human-AI Companionship Behavior (2508.09998)
INTIMA Benchmark: AI-companionship/INTIMA
- 1 reply
giadapย
authored a
paper about 1 year ago
yjerniteย
authored 3
papers about 1 year ago
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Paper โข 2406.16746 โข Published
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Paper โข 2503.16861 โข Published โข 1
A Different Approach to AI Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AI Safety
Paper โข 2506.22183 โข Published โข 1
Post
3867
We've crossed 1 million repositories backed by Xet storage on Hugging Face! ๐๐๐
You can follow along our progress converting the Hub from Git LFS to Xet at jsulz/ready-xet-go
We have a lot of repos left to migrate, which means I have plenty of time to add more animations ๐คช
You can follow along our progress converting the Hub from Git LFS to Xet at jsulz/ready-xet-go
We have a lot of repos left to migrate, which means I have plenty of time to add more animations ๐คช
Post
3738
๐ฌ From Replika to everyday chatbots, millions of people are forming emotional bonds with AI, sometimes seeking comfort, sometimes seeking intimacy. But what happens when an AI tells you "I understand how you feel" and you actually believe it?
At Hugging Face, together with @frimelle and @yjernite , we dug into something we felt wasn't getting enough attention: the need to evaluate AI companionship behaviors. These are the subtle ways AI systems validate us, engage with us, and sometimes manipulate our emotional lives.
Here's what we found:
๐ Existing benchmarks (accuracy, helpfulness, safety) completely miss this emotional dimension.
๐ We mapped how leading AI systems actually respond to vulnerable prompts. ๐ We built the Interactions and Machine Attachment Benchmark (INTIMA): a first attempt at evaluating how models handle emotional dependency, boundaries, and attachment (with a full paper coming soon).
Check out the blog post: https://huggingface.co/blog/giadap/evaluating-companionship
๐ข We also shipped two visualization tools with Gradio to see how different models behave when things get emotionally intense:
- AI-companionship/intima-responses-2D
- giadap/INTIMA-responses
At Hugging Face, together with @frimelle and @yjernite , we dug into something we felt wasn't getting enough attention: the need to evaluate AI companionship behaviors. These are the subtle ways AI systems validate us, engage with us, and sometimes manipulate our emotional lives.
Here's what we found:
๐ Existing benchmarks (accuracy, helpfulness, safety) completely miss this emotional dimension.
๐ We mapped how leading AI systems actually respond to vulnerable prompts. ๐ We built the Interactions and Machine Attachment Benchmark (INTIMA): a first attempt at evaluating how models handle emotional dependency, boundaries, and attachment (with a full paper coming soon).
Check out the blog post: https://huggingface.co/blog/giadap/evaluating-companionship
๐ข We also shipped two visualization tools with Gradio to see how different models behave when things get emotionally intense:
- AI-companionship/intima-responses-2D
- giadap/INTIMA-responses
Post
4335
๐๐ถ๐ฟ๐๐ ๐๐ฃ๐๐ ๐ ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ต ๐๐จ ๐๐ฎ๐๐ฎ ๐ง๐ฟ๐ฎ๐ป๐๐ฝ๐ฎ๐ฟ๐ฒ๐ป๐ฐ๐ ๐ง๐ฒ๐บ๐ฝ๐น๐ฎ๐๐ฒ? ๐ช๐บ
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
Post
1289
๐ค Technology means power, and whoever owns the technology owns the power.
Thrilled to share insights from my recent interview with MIT Technology Review about the growing movement toward local LLMs and what it means for AI democratization. Read here: https://www.technologyreview.com/2025/07/17/1120391/how-to-run-an-llm-on-your-laptop/
๐ค Why this matters: When we use "free" online AI services, we're often the product. Our conversations become training data, our personal stories get "cooked into" models, and our privacy becomes a commodity. But there's an alternative path forward.
๐ก The power shift is real: Local LLMs aren't just about privacy; they're about redistributing AI power away from a handful of tech giants. When individuals, organizations, and even entire nations can run their own models, we're democratizing access to AI capabilities.
๐ค At Hugging Face, we're proud to be at the center of this transformation. Our platform hosts the world's largest library of freely downloadable models, making cutting-edge AI accessible to everyone -- from researchers and developers to curious individuals who want to experiment on their laptops or even smartphones.
The technical barriers that once required $$$ server racks are crumbling. Today, anyone with basic computer skills can download a model, run it locally, and maintain complete control over their AI interactions. No sudden algorithm changes, no data harvesting, no corporate gatekeeping.
This is about technical convenience, but especially about technological sovereignty. When AI power is concentrated in a few hands, we risk creating new forms of digital dependency. Local models offer a path toward genuine AI literacy and independence.
๐ The future of AI should be open, accessible, and in the hands of the many, not the few. What are your thoughts on AI democratization? Have you experimented with local models yet?
Thrilled to share insights from my recent interview with MIT Technology Review about the growing movement toward local LLMs and what it means for AI democratization. Read here: https://www.technologyreview.com/2025/07/17/1120391/how-to-run-an-llm-on-your-laptop/
๐ค Why this matters: When we use "free" online AI services, we're often the product. Our conversations become training data, our personal stories get "cooked into" models, and our privacy becomes a commodity. But there's an alternative path forward.
๐ก The power shift is real: Local LLMs aren't just about privacy; they're about redistributing AI power away from a handful of tech giants. When individuals, organizations, and even entire nations can run their own models, we're democratizing access to AI capabilities.
๐ค At Hugging Face, we're proud to be at the center of this transformation. Our platform hosts the world's largest library of freely downloadable models, making cutting-edge AI accessible to everyone -- from researchers and developers to curious individuals who want to experiment on their laptops or even smartphones.
The technical barriers that once required $$$ server racks are crumbling. Today, anyone with basic computer skills can download a model, run it locally, and maintain complete control over their AI interactions. No sudden algorithm changes, no data harvesting, no corporate gatekeeping.
This is about technical convenience, but especially about technological sovereignty. When AI power is concentrated in a few hands, we risk creating new forms of digital dependency. Local models offer a path toward genuine AI literacy and independence.
๐ The future of AI should be open, accessible, and in the hands of the many, not the few. What are your thoughts on AI democratization? Have you experimented with local models yet?