The ACL Anonymity Embargo Period is exclusionary, actually: an early-career researcher's perspective
Fri 15 September 2023Preprinting and promoting one's work online is one of the most impactful and accessible career advancement methods for early-career researchers. Policies that hinder this process are thus disproportionately harmful to early career scientists, relative to their senior peers. As online promotion is more accessible than traditional routes of namebuilding, we contend that depriving "low-prestige institution"-affiliated early-career researchers of this avenue for self-advancement futher disproportionately harms them relative to their "high-prestige" peers. Finally, a broad culture of posting quality preprints has community-level and scientific benefits. We contend that the ACL anonymity policy is harmful for the ways in which it forces early-career researchers (ECRs) to choose between participating in ACL or doing what's best for their careers.
In case you're unfamiliar, the Association for Computational Linguistics (ACL), the premier venue for natural language processing (NLP) research, has an extremely strict "anonymity embargo policy," restricting the ability of researchers to publicly share or discuss their latest work online within a time window starting 1 month prior to a given conference's submission deadline, through the date when acceptance decisions are conveyed (about a four month period). The ACL Executive committe is currently considering abolishing this policy, and I strongly believe removing it is the right thing for both our science and community, and in particular I think removing it will, on the balance, have inclusionary outcomes for early-career researchers.
This essay is my attempt at laying bare the harms this policy visits on early career researchers as an inclusion-oriented argument for its removal. In particular, I will argue:
- That social media self-promotion is a crucial career development tool that all early-career researchers benefit from and has a leveling effect across existing hierarchies of institutional prestige
- That discussing one's most recent research using preprints is integral to online self-promo
- That the reduction in preprinting achieved under the current policy comes by disproportionately stifling the participation of the least privileged ECRs in the public self-promotion process
- That these exclusionary effects are disproportionately visited on cross-disciplinary researchers in AI, ML, and computer vision-adjacent subfields who we really ought to keep in the *ACL community
I believe in many other arguments for why the embargo policy should be repealed, beyond those presented here. In particular, I think claims that it significantly protects the integrity of double-blind review are unsubstantiated, or at the very least too questionable to justify the documented costs the policy has on the community. In any case, we leave those points to be argued by others or in future posts. Here the focus is on the "exclusionary harm to ECRs" angle.
In allowing preprints only for those who can meet an arbitrary and difficult filter, it produces a class of authors who can choose to operate without anonymity, and a class who cannot (as their papers are incomplete by the onerous month-in-advance embargo deadline). In trying to split the difference between two irreconcilable states of equality (preprints for all or preprints forbidden) it produces a novel sort of inequality where only some authors are even given the ability to decide. This worst-of-all-worlds compromise is akin to King Solomon's offer to give half of the baby to each self-proclaimed mother—we must choose one option or the other. By communicating why so many ECRs are so opposed to this policy, I hope to aid the community in finding a consensus option. All I ask is for is that everyone consider this angle, the way in which ECRs are hindered in traversing the prestige hierarchy when they are shut out of online promotion and preprinting, when doing the fairness calculus to decide how to balance the concerns of anonymity and its consequences.
Seeing as the hierarchies of privilege that DEI initiatives are meant to address are often embedded within the prestige hierarchies at the forefront of discussions of career fairness, I sincerely believe that policies which uphold or worsen disparities along institutional prestige hierarchies represent a fundamental failure of inclusion. Putting early-career researchers in a double bind where they cannot have both broadly accessible hierarchy-leveling online self-promotion and prompt conference participation, the ACL embargo policy is thus counter-inclusionary, and must be repealed at once.
I will now provide more lengthy explanations for why I believe each of the four points above to be true.
Disclaimers: Parts of this essay are written in a raw style; I seek to convey both the rationale and emotions behind my support for the swift removal of the embargo policy. Apologies if you feel I have unfairly characterized your position or motivations. Although I consider this policy exclusionary, I recognize the well-intentionedness of its creators and supporters, and do not hold malice towards them. Sorry it's so wordy, if I had more time I would have written it shorter.
Social media self-promotion is key for early career researcher (ECR) success and leveling across prestige hierarchies
Name- and reputation-building are necessary for a successful scientific career. Although NLP has become a much more crowded and "faster" field than it was before the late 2010s, the "traditional" avenues by which previous generations of scientists have built reputations—including participating in conferences, publishing in journals, and being invited to deliver talks and share one's work—remain integral steps on the path of career advancement for the current generation of ECRs. As always, active effort and strategy are necessary to translate quality research into career-boosting readership. However, in this fast-paced research environment the capacity of these traditional avenues is strained, leading to increasingly noisy acceptance/award decisions on an ever growing mass of submissions. Overwhelmed readers rely on the time-honored heuristics of author prestige and word of mouth to decide which works are deserving of attention, and as always these poor signals favor entrenched privileged groups by default. However, there is a silver lining in the modern research climate: the internet has given powerful and accessible tools to modify one's own position within the status hierarchies that drive these heuristics.
Online self-promotion is an exciting new way to build one's reputation, free from the constraints of proximity within physical space and institutional networks. Productively contributing to online discussions and marketing one's fresh work is a great way to become "known" within the research community that is more rapid and accessible than the traditional avenues. It isn't a surprise that many scientists across disciplines are enthusiastically taking advantage of this new avenue for career advancement.
The individual benefits of namebuilding—in any form—produce a virtuous cycle for the individual as opportunities beget more opportunities. However, this virtuous cycle for the individual can be a driver of field-level inequality of opportunity when starting points are taken in to account; after all, geographic and institutional biases continue to set the initial conditions for reputational hierarchies, as they always have. A consequence of the compounding nature of this cycle is a "rich get richer" process for early career researchers who are already well-positioned relative to their peers. The fundamentally open nature of online self-promotion is both fairer for individuals and better for science, as it enables a truer meritocracy by building an alternative to traditional physical prescense- and network-reliant namebuilding.
Let's consider internships and fellowships as two illustrative examples. For as long as these opportunities have existed, they have favored candidates in stronger positions in the aforementioned priviledge hierarchies. As ever, private channels favoring those connected in well-networked institutions tilt the scales. Intern hiring managers can and do reach out to their advisor friends for recommended candidates (often their own students), and fellowship opportunities are often promoted through department mailing lists.
This leads to predictable outcomes, such as students from top universities disproportionately winning NSF fellowships as the application coaching networks within these institutions organize to help their applicants prepare stronger proposals. These networks certainly exist in non-elite institutions—I owe my own NSF grant to the advice of a small network of previous winners I knew personally at Arizona State and my MS advisor Visar Berisha's time and advice in reviewing my application—but building these advising networks without a well-organized institutional advising machine is both non-trivially difficult and requires a lot of luck.
Online self-promotion is an exciting disruption to this career advancement hierarchy status quo. Hiring managers now solicit applicants directly on Twitter, and "namebuilt" ECRs can gain a connectedness advantage in the application process over the internet, regardless of institutional affiliation. Similarly, where fellowship aspirants previously could solely rely on institutional and advisory networks for application coaching, they can now find that advice over the internet, and provide an "adviceworthiness" signal through their publicly available work and by gaining followership. I strongly believe that on the balance the democratization of in-community namebuilding that social media has brought about has a strong leveling effect across axes of institutional prestige and advisor connectedness, at least within visa jurisdictions.
On the balance, social media participation improves the fairness and inclusion of career-level outcomes within scientific communities.
Consequently, ACL's embargo policy negatively impacts career outcome fairness and inclusion in our community by interfering with early career researchers' ability to speak freely and openly about their work. In hindering the most accessible avenue to name-building, it entrenches the powerful position of the more institutionally privileged in attaining opportunities.
Career-impacting self-promo is time-sensitive for ECRs, warranting preprinting
So far all points have only argued for general participation in namebuilding through online self-promotion. But why is a pro-preprinting culture important? We believe there are two key benefits for individual authors, and one for the community:
- Individual authors can "namebuild" faster and more effectively when they are able to discuss their latest and most freshest work, and preprinting provides a critical multi-month head start in doing so
- Scooping is prevented in a materially beneficial way that is not provided by anonymized preprints
- The whole community gets higher-quality, fresher preprints
Why is high speed namebuilding important for early-career researchers? Many ECRs are students, who have a limited number of summers to expend on critical research opportunities such as internships. A monthslong period muzzled from discussing a student's most recent work can literally mean the difference between a career trajectory-defining internship opportunity and an empty summer.
The aforementioned actively online internship hirers are paying attention to their subcommunity. They see preprints and follow discussions of the important problems in their area, particularly those related to the very topics they are hiring interns for. The less well-known a student is in their topic area, the more important it is that they get preprints online for a chance at this attention, thereby starting their virtuous growth cycle. Thus, it is imperative for greener students to get quality preprints demonstrating their work up as fast as possible.
However, the *ACL anonymity policy creates a double-bind: taking the time to get a better preprint out conflicts with the ability to publish.
Consider the case (that I regrettably know well) in which a student switches topics mid-PhD. This student is subject to the dual pressures of publish-or-delay-graduation-milestones (which would require submission to the earliest possible *ACL deadline) and the imperatives to build credibility in their new topic area as fast as possible. When they've produced a significant work that fits the topic for a job opportunity they're looking for, the stakes get even higher. After all, they'll be the most likely to recieve fair consideration for the role if their work is publicly available, and perhaps a team member or recruiter who's hiring will see the work; but skipping a deadline could cost the work's "countability" toward graduation milestones in a worst case scenario of review purgatory. Are we really getting sufficient benefit for the pain of forcing our ECRs to make this choice?
This calculus becomes even more challenging when "scooping" (the risk of someone else independently announcing an extremely similar result to yours) is considered. Obviously, unrestricted, just-in-time-open-preprinting is the best way to protect against being otherwise scooped during the review embargo period. Some suggest that anonymized preprints such as those implicitly produced by the OpenReview process ICLR uses are a preferable compromise to fully free preprints as they reduce scooping risk. While people do read (and find inspiration!) in these anonymized preprints, the aforementioned namebuilding benefits aren't there. Furthermore, authors who are inspired by anonymized preprints are forced to cite the anonymous preprint, and those citations do not propagate through attribution systems such as Google Scholar. (Add that as a relatively trivial reason to not use it on top of the trans-exclusion it perpetuates!)
For example, my labmate Weixi produced an awesome work on improving compositionality and attribute binding in text-to-image models that went to ICLR 2023, but declined to immediately preprint his ICLR draft at submission time in September, opting to instead preprint in December. Consequently, when my friend Royi was inspired in part by this work in his study of duplications in DALL-E 2, he was forced to cite it as "Anonymous (2022)." That review didn't count towards Weixi's citation count or h-index on Google Scholar (but it did on Semantic Scholar so maybe you should switch đ). Seeing as this paper has been very successful (60 Google Scholar-attributed citations in 2023 alone, better than any of mine so far!), and Weixi was able to eventually put up a deanonymized preprint during the decision window, the impact on his career-driving bibliometrics was minimal. But imagine the counterfactual where this preprinting was not allowed. How many more of 60 of those attributed citations of Weixi's work would instead be attributed to "Anonymous," and his highest cited first-author paper would only be 2. I think it's reasonable to believe that the sweet internship he landed working on text-to-image models this summer would have been more difficult to get without this significant degree of quantified impact.
While Weixi's decision to post a preprint in December 2022 only cost him a very small amount of attributable citations in the greater scheme of things, I also think that if he had rushed to push out a half-formed preprint of his paper in August 2022 rather than a polished, finished one in December 2022, the impact of his work might have been severely hampered. I certainly observe that my month-in-advance preprints are worse than my submission-time papers, and I think it's reasonable to assume both that others feel this way about their own work, and that these worse papers have blunted impact both in terms of readership and namebuilding.
Wouldn't it be better for the *ACL community as a whole if we stopped incentivizing rushed and sloppy preprints?
Why the documented impact on anonymity is an exclusionary double-bind that disproportionately impacts low-prestige ECRs
At this point, many students explicitly prefer to skip an ACL deadline in order to get on arXiv. Of course, situations like this only arise when a project is completed so last-minute that preprinting is completely impossible under the embargo policy. But what kinds of researchers are most likely to have a project that finishes last minute, which is rendered completely unpreprintable by the embargo policy?
I contend that most researchers who are finishing their project right at the deadline are students. After all, corporate researchers work from project timelines that typically require a complete paper draft for internal review prior to submission anyway.
Further, I contend that earlier stage students and students who have recently switched topics are more likely to be in this situation consequentially (as discussed above) than more experienced students who are established in their topic area. In other words, the most likely authors to be stuck in the aforementioned namebuilding double-bind are also the individuals who need the benefits of namebuilding the most.
Higher-prestige or more senior authors don't have reasons to worry about this double-bind; missing 6 months of publicity for a single project, or skipping a single project's chance to get in to the soonest conference is not as consequential for them. It should be easy to see how this creates an exclusionary dynamic.
Furthermore, the uniqueness of our community's embargo policy is confusing enough to newcomers that it borders on being hostile to aspiring new members. Even for community members, the constant changes the policy undergoes are a burden. For example, ACL 23 uniquely had a "posting date" rather than "upload date" anonymity deadline, effectively shifting enforcement of it to the whims of various preprinting servers, such as arXiv's archaic once-a-day mailing list cutoff requiring a server-side tex compile, further compounding the headache of complying to the deadline.
In the alternative "upload date" cutoff that is typical, confusions like Zack's off-by-half-an-hour error in arXiv submission time led to this undergraduate's first-ever submission being desk rejected after the rebuttal period, despite this upload time error having no materially significant differences for deanonymization relative to embargo-compliant papers uploaded only 30 minutes before, as they were all made publicly available by the arXiv at the same time. Weird interactions between the strict cutoffs inherent to a preprint embargo and the vagaries of preprint servers are making our field hostile to newcomers.
In short, we are disproportionately shutting ECRs out of the benefits of preprinting and self-promotion.
It is broadly agreed that to the extent it succeeds, the *ACL Anonymity Embargo Policy "protects double-blind review" simply by reducing the total amount of preprints that are available before acceptance decisions are made.
However, we contend that the reduction in preprinting that is produced by the policy is largely a de-facto muzzling of public discussion of the freshest ideas in our community from some less-advantaged authors. These authors are thus forced to choose between missing out on the benefits of participation in *ACL venues or suffering the many ACL-external consequences to not preprinting. such as missing out on engagement for their work at its freshest, missing out on media opportunities, missing out on citations, and missing out on career advancement/fellowship/job opportunities that would otherwise be available. In short, the ACL Anonymity Embargo Policy, while well-intentioned, is a de-facto burden on our least advantaged authors, particularly early career researchers, and further disadvantages them relative to their peers in other fields who are able to preprint early. As ECRs we believe the only sensible move is to abolish this policy completely, ASAP.
Why this policy is driving away researchers in a disciplinary overlap, and why that's a bad thing
Not all subfields within ACL are equally enthusiastic about the preprinting and online discussion-centered mode of conduct we have been describing above. In particular, the more "artificial intelligence"-oriented subfields within NLP, such as those concerned with understanding and developing applications using large language models (LLMs), those studying core machine learning techniques as applied to NLP problems, and those who work on multimodal systems such as speech, vision, and robotics seem to be the most gung-ho about preprinting.
I think a big reason for this is that the authors in this disciplinary overlap are bound to norms and incentive structures that are outside of *ACL's exclusive control. All of the aforementioned subfields overlap substantially with non-NLP disciplines, and their research is in dialogue with work that appears at non-*ACL conferences. Many *ACL participants already also publish in venues such as ICLR, NeurIPS, AAAI, CVPR, ICCV, ICML, Interspeech, etc, all of which do not have our unique restrictive embargo policy. They are thus both subject to cultural influence from the overlapping disciplines and, more importantly, subject to the same incentive structures as scientists in these overlapping disciplines.
For all the career-relevant scenarios discussed above, how can an ACL regular and ICLR regular compete for the same fellowship, or the same internship position, when the ICLR regular is at a structural advantage for namebuilding and career progress? The interaction between the embargo policy and structural incentives in the CS-interdisciplinary NLP subfields is slowly but surely driving promising ECRs away through this career outcome handicap. All the aforementioned disparities with how this handicap hurts students across hierarchies continue to apply. I think this is extremely bad for our community.
While this is tangential to the core ECR fairness issue we've been addressing here, I think the reasons why the ACL community ought to not push out our overlapping-discipline-members is fundamentally driven by the namebuilding mechanims we've been discussing above.
After all, the public nature of social media means that an online following built within the scientific community is effectively dual use in its transferability to the general public! A platform built through quality layperson-oriented scientific communication is thus useful for career advancement, implicitly rewarded through the incentive structure benefits for namebuilding. In a time of serious hype and misinformation around the capabilities of large language models, is it really wise to muzzle the scientists most well-equipped to inform the public from sharing their freshest expertise?
When certain agi animals talk about *ACL as some kind of backup venue for "non-ICLR-worthy" ideas in LLMs, yes, they are being ignorant. I understand why many in *ACL are weary of a deluge of "look at this prompt I invented" papers. However, it seems clear to me that ceding the ground for all discussion of important modern issues in AI to their ilk is good for neither the *ACL research community nor the world. I sincerely believe that the distinctive population of positions our community holds on "language understanding AI" and its responsible use makes us the best venue for level-headedly and objectively responding to impacts from and driving the advances of these generation-defining, societally influential technologies.
I think the world would be served well if central discussion around LLMs happened in venues where Emily Bender and Geoff Hinton can argue about the meaning of understanding, computational social scientists are central rather than sideshow, and papers centered on Terminator doomsday scenarios are far less regularly accepted.
Call to action
This survey is the beginning of a discussion of change in our community, not a binding vote to decide future policy. Ultimately, changes are up for the ACL Executive Committee to decide. When we get there, stay tuned for posts explicitly laying out the cases for and against the only two alternatives: free discussion for all and complete preprint and discussion shutdown.
How I answered the survey
In case you're interested in how I filled out the ACL Survey, (Note, do not construe this as some kind of voting guide) here's what I put:
What do you think of the first proposed change that allows authors to freely make versions of their submitted paper available online at any time?
I support the proposed change.
What do you think of the second proposed change that allows authors to mention their preprints at any time, including on social media?
I support the proposed change.
If you think that either changing or not changing these policies would potentially harm your career, or harm you in some other way, please explain:
Early-career researchers need the ability to promptly share their preprints online for time-sensitive opportunities such as internships and fellowships. As an ECR I think the first change is absolutely necessary. The second change would further enable ECRs to freely discuss their freshest work, improving their ability to network and grow. As an ECR, I believe I would also benefit from the second change. I feel that not making the changes harms me.
Regardless of your answers to previous questions, and assuming a âmagic solutionâ in which author identities remain unknown to reviewers and introduce no biases during review: would you like to pre-print (some of) your papers during review and discuss them on social media? Please briefly explain your answer.
Yes, the reasons I support preprinting are orthogonal to acceptance, it's about career opportunites and I don't believe preprinting or not preprinting will impact my acceptance odds.
Do you have any other suggestions on how ACL could balance the desire for open scientific discourse with the risk that it may contribute to reviewing bias?
I think if we remove the limitations we should continue collecting fine-grained data on acceptance rates and deanonymization; if the policy was preventing outcome-impacting reviewer bias, it will be visible, and we can consider reinstituting an embargo in that case with the knowledge that having it actually makes a difference.
How strongly (1-5) do your future career goals require the visibility of your papers?
5 (extremely)
If reviewers were able to see de-anonymized versions of my submissions (including authors and affiliations), I think it would typically:
Neither increase nor decrease
I mainly publish at:
ACL NLP venues, ML venues
How often do you use social media (e.g., Twitter) to share or discuss your paper?
Always
Acknowledgements
I am solely responsible for the editorial content here. Do not hold any of the following acknowledged responsible for anything you take issue with in here.
Thank you to Naomi Saphra, Gowthami Sommepalli, Weixi Feng, Royi Rassin, Alon Albalak, Danish Pruthi, Sharon Levy, Ofir Press, and Liangming Pan for discussing various components (big or small) or stages (early or late) of this draft with me.
In addition to the aforementioned, thank you to Anna Rogers, Carlos GĂłmez-RodrĂguez, Yoav Goldberg, Nathan Schneider, Sebastian Schuster, Zachary Lipton, Leon Derczynski, Asad Sayeed, Andrew Drozdov, Shaily Bhatt, Manuel Mager, Thamar Solorio, Dmytro Mishkin, Alexander Hoyle, Leshem Choshen, Yoshitomo Matsubara, Julia Mendelsohn, and probably many others who have shaped my thinking on this topic by discussing it with me publicly online :)
Category: Improving AI Science; Tags: Policy, ACL, AI, Metascience, Fairness