CV

Education

(2023-Present) Assistant Professor at the Information Systems Department, University of Haifa.
(2020-2023) Postdoctoral Researcher at the Cognitive Computation Group at the Department of Computer and Information Science, University of Pennsylvania, working with Prof. Dan Roth.
(2016-2020) Ph.D. at the faculty of Industrial Engineering and Management at the Technion, working under the supervision of Asst. Professor Roi Reichart. Thesis title: Structured Solutions in Natural Language Processing: Algorithms and Statistically Sound Evaluation. GPA: 93.5.
(2014-2016) Master in Operation Research for completing the requirements for the Master’s degree during the PhD studies, Technion.
(2010-2014) B.Sc. in Industrial Engineering and Management, Business Information Systems major, Technion. Graduated Cum Laude. 4 Dean’s Honor Rolls. GPA: 87.9.

Publications

State of What Art? A Call for Multi-Prompt LLM Evaluation

"State of what art? a call for multi-prompt llm evaluation." Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, Gabriel Stanovsky. arXiv preprint arXiv:2401.00595. Accepted to TACL

DMLR: Data-centric Machine Learning Research – Past, Present and Future

DMLR: Data-centric Machine Learning Research -- Past, Present and Future." Luis Oala, Manil Maskey, Lilith Bat-Leah, Alicia Parrish, Nezihe Merve Gürel, Tzu-Sheng Kuo, Yang Liu, Rotem Dror, Danilo Brajovic, Xiaozhe Yao, Max Bartolo, William A Gaviria Rojas, Ryan Hileman, Rainier Aliment, Michael W. Mahoney, Meg Risdal, Matthew Lease, Wojciech Samek, Debojyoti Dutta, Curtis G Northcutt, Cody Coleman, Braden Hancock, Bernard Koch, Girmaw Abebe Tadesse, Bojan Karlaš, Ahmed Alaa, Adji Bousso Dieng, Natasha Noy, Vijay Janapa Reddi, James Zou, Praveen Paritosh, Mihaela van der Schaar, Kurt Bollacker, Lora Aroyo, Ce Zhang, Joaquin Vanschoren, Isabelle Guyon, Peter Mattson. arXiv preprint arXiv:2311.13028 (2023).

140 Characters of Justice? The Promise and Perils of Using Social Media to Reveal Lay Punishment Perspectives

140 Characters of Justice? The Promise and Perils of Using Social Media to Reveal Lay Punishment Perspectives." Itay Ravid and Rotem Dror. 2023 U. Ill. L. Rev. 1473

The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics

"The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics." Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, Steffen Eger. arXiv preprint arXiv:2310.19792.

Human-in-the-Loop Schema Induction

"Human-in-the-Loop Schema Induction." Tianyi Zhang, Isaac Tham, Zhaoyi Hou, Jiaxuan Ren, Liyang Zhou, Hainiu Xu, Li Zhang, Lara J. Martin, Rotem Dror, Sha Li, Heng Ji, Martha Palmer, Susan Brown, Reece Suchocki, Chris Callison-Burch. In proceedings of the ACL 2023 Demo Track.

Zero-Shot On-the-Fly Event Schema Induction

"Zero-Shot On-the-Fly Event Schema Induction." Rotem Dror, Haoyu Wang, Dan Roth. In Proceedings of the Findings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

On the Limitations of Reference-Free Evaluations of Generated Text

"On the Limitations of Reference-Free Evaluations of Generated Text." Daniel Deutsch, Rotem Dror, and Dan Roth. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

"Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics." Daniel Deutsch, Rotem Dror, and Dan Roth. In Proceedings of the 2022 Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL).

RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios

"RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios." Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, and Heng Ji. The North American Chapter of the Association for Computational Linguistics in the Demostration Track (NAACL). 2022.

A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods." Daniel Deutsch, Rotem Dror, Dan Roth. of the Association for Computational Linguistics 2021; 9 1132–1146.

Statistical Significance Testing for Natural Language Processing

"Statistical Significance Testing for Natural Language Processing." Rotem Dror, Lotem Peled-Cohen, Segev Shlomov and, Roi Reichart. Book published in the Human Language Technology series of Morgan&Claypool.

The Structured Weighted Violations MIRA

"The Structured Weighted Violations MIRA." Dor Ringel, Rotem Dror and Roi Reichart. arXiv preprint arXiv:2005.04418 (2020).

Deep Dominance - How to Properly Compare Deep Neural Models

"Deep Dominance - How to Properly Compare Deep Neural Models." Rotem Dror, Segev Shlomov and Roi Reichart. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) 2019.

Appendix - Recommended Statistical Significance Tests for NLP Tasks

"The Hitchhikers Guide to Testing Statistical Significance in Natural Language Processing." Rotem Dror, Gili Baumer, Segev Shlomov and Roi Reichart. Association for Computational Linguistics (ACL 2018).

The Hitchhikers Guide to Testing Statistical Significance in Natural Language Processing

Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

"Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets." Rotem Dror, Gili Baumer, Marina Bogomolov and Roi Reichart. Transactions of the Association for Computational Linguistics (TACL), vol. 5, pp. 471–486, 2017.

Set-SMAA for Finding Preferable Multi-Objective Solutions

"Set-SMAA for finding preferable multi-objective solutions." Rotem Dror, Amir Kantor, Michael Masin, Segev Shlomov. GECCO (Companion) 2017: 149-150.

The Structured Weighted Violations Perceptron Algorithm

"The Structured Weighted Violations Perceptron Algorithm." Rotem Dror and Roi Reichart. EMNLP 2016.

Awards

2022 - Women’s Postdoctoral Career Development Award in Science.
2019 - Eric and Wendy Schmidt Postdoctoral Award for Women in Mathematical and Computing Sciences.
2018 - Google PhD fellowship.
2017-2018 - Gutwirth fellowship.
2017 - Award of academic excellence and research funded by the faculty of Industrial Engineering and Management at the Technion.
2015 – Award of academic excellence and research funded by the faculty of Industrial Engineering and Management at the Technion.
2014, 2018, 2019 - Excellent TA award.

Activities

Co-organizer of the NLP4Science Workshop at EMNLP2024.
Co-organizer of Data-centric Machine Learning Research (DMLR) Workshop at ICML2024.
Organizer and Chair of the Statistical Challenges in Model-based Data Science Session at CFE-CMStatistics 2023.
Co-organizer of Evaluation & Comparison of NLP Systems (Eval4NLP) Workshop at AACL2023.
Panelist at the SustaiNLP Workshop at ACL2023.
Co-organizer of Data-centric Machine Learning Research (DMLR) Workshop at ICML2023.
Founder and operator of the NLP Rolling Statistics Clinic: https://cogcomp.seas.upenn.edu/
Keynote speaker at the Machine Learning Evaluation Standard Workshop at ICLR2022.
E-visiting scholar at the Erasmus Mundus European Masters Program in Language and Communication Technologies (LCT) Consortium 2021.
Conference reviewer: since 2018 reviewing on a regular basis for ACL, EMNLP, NAACL, CoNLL, EACL, NLPCC, COLING. Occasionally reviewing for AAAI, ICML, IJCAI, NIPS. Reviewer on ACL Rolling Review.
Journal and Monograph Reviewer: Communications of the ACM (2021) and a manuscript review for the Synthesis Lectures in Human Language Technologies published by Morgan&Claypool Publishers (2021).
Establishment and organization of the Computational Data Science Seminar at the Faculty of Industrial Engineering and Management at the Technion (2018 – 2020)
Student chair in the Student Research Workshop (SRW) at ACL 2020.
Co-organizer of the Israeli Seminar on Computational Linguistics (ISCOL) 2018.

Work Experience

Research Intern, Amazon, 8.2019-11.2019.
Head teaching assistent, Technion, 2013-2019.
- Head teaching assistant in courses “Digital Simulation”, “Discrete Math”, “Probability” (undergraduate course), and “Machine Learning and Optimization” (undergraduate and graduate course).
- Skills required: responsibility, teaching skills, expertise in relevant material.
- Excellent TA award 1/2014.
Algorithm developer, RAFAEL - Advanced Defence Systems, 2013-2015.
- The job involved developing and adapting algorithms to systems produced in Rafael.
- Skills required: problem solving, simulations, programming in varied programming languages, algorithms, data analysis, statistics, operation research.
Military service, 2007-2010.
- Service in a highly classified position in the Intelligence corps unit 8200.
- Excellence recognition.
- Skills required: languages acquisition, command, problem solving, research, operation modes.

Skills

Languages – Hebrew (native), English (fluent verbal and written).
Programming courses – C, C++, Python, Pytorch, C#, HTML, PHP, MATLAB, JAVA.