11-667 Projects

Team Assignments

Each group has been assigned to a project TA. Please check Piazza for the TA assignments. The role of your TA is to guide and advise your projects, so please reach out to your assigned project TA first if you have questions about your approach, or if you have any issues with the project in general.

You can meet with your TA as much or as little as you want, but there are two mandatory meetings with your TA to make sure that your group is on track:

  • Week of Oct 2: TA matching announced. Your TA will reach out to you in order to schedule an introductory meeting before Fri Oct 13.
  • Oct 31: Midterm project meetings. During class, TAs and instructors will talk to each group one-on-one to discuss the project. Please note that there will be no formal in-class presentation.

Logistics:

Team: Form a team of 3-5 people. Four is recommended.

Final Deliverables Instructions

You have three deliverables in the final weeks of class.

  • Due December 7, 2-3:30 PM: In class poster session in GHC 7107.
    • The library has some good resources on designing an effective poster.
    • VERY IMPORTANT: You can print your poster for free following this link, but you should expect posters to take 2-3 days to print, so make sure to submit your poster early enough that it is ready! We will not accept excuses for posters not ready in time.
    • During the poster session, the instructors and TAs will visit your poster and ask you to go through it. They will also ask questions about your work.
  • Due December 8, at 8 PM: Deadline for submitting final report.
    • You should write your project report following the conventions of an 8-page ACL paper. Experimental details which do not fit within 8 pages may be included in an Appendix.
    • Grading: Your report will be graded by TAs instructors OTHER than your assigned mentor. They will grade following the ARR reviewer rubric. Like an ACL paper, your report should be self-contained and comprehensible to someone unfamiliar with your project.
  • Final peer feedback
    • Due December 11 at 8 PM: At the poster session, choose five posters to fill out the following form over.

If you have been given permission to not attend the final poster sessions (for example, if you will be presenting at EMNLP), please schedule a meeting with Daphne and Chenyan sometime December 4-14 to present your poster to us. This will be an individual meeting. Your team will still be expected to show up to the in-person poster session on December 7.

A note on negative results

Sometimes research doesn’t work out. It is fine to have negative results in your report so long as you discuss why your hypothesis ended up being false, using experiments to back your explanation.

Midpoint Instructions

Your report should be in the style of an ACL submission. You may (but are not required to use the template linked here. Where relevant, you may copy text from your proposal. We will evalute youe midpoint report on:

  1. Have you made progress toward the stated goals in your proposal? You should have run at least one set of experiments by now.
  2. Do you have a plan for in place for completing your project by the end of the semester?
  3. Is your midpoint report well-written in the style of an ACL paper?

You should also prepare slides for a 3-minute in-class presentation. While your report is due on October 31 at 2 PM, we need your slides by October 30 at 8 PM so we have time to assemble them into a single slide deck.

Proposal Instructions

Please use the template linked here for the project proposal. Your proposal should have all the sections shown on this template. It should be no more than 2 pages. You should also plan a 2.5 minute presenation of the idea (3 slides maximum) to present in class.

If you would like feedback on your proposed idea, submit the request for feedback form by end of day September 21st, and the TAs will be in touch.

Checkpoints:

  • Sep 12: Project is out. Start forming teams.
  • Sep 21: Last day to request a quick check of your project ideas with TAs, with one passage maximum to describe your idea. Finalize teams (if you need help with team making, also contact TAs).
  • Sep 26: Two-page project proposal write up due, in ACL format. Spotlight slide deck due (3 slide maximum).
  • Sep 28: In-class project proposals. 2.5 min spotlight style presentations.
    • Peer scoring: each of you rates other project presentations in 1-5 likert scale (5 being you love the presentation). The scoring is based on the presentation not the project proposal content.
    • Peer feedback: each of you gives short suggestions to at least three other teams.
  • Sep 30: Projects are matched with Instructor/TA mentors.
  • Oct 30: Submit your slides by 8 PM on this date. No late slide submissions allowed.
  • Oct 31: Midpoint progress report due.
    • Four pages of write up in ACL format.
    • Three minute presentation in class highlighting your progress.
  • Oct 30-Nov 4: Schedule a midpoint check-in meeting with your assigned team mentor.
  • Dec 7: Final Project Presentation poster session.
  • Dec 8: Final project due. Eight pages of write up in ACL format (besides unlimited references and appendix). Appendices are allowed but may not be considered in grading. An additional page after the appendix stating the specific contribution of each team member, with acknowledgement of all team members, is mandatory.

Project Directions:

The final project is expected to be a novel contribution, either to fundamental research on language models or to downstream applications of LLM technology. For a top grade, you must demonstrate a significant amount of technical sophistication in your project. If you have an idea and you aren’t sure if it is of sufficient scope, you are welcome to come to office hours to pitch it to us ahead of the project proposal deadline.

Grading Rubric:

  • The effort on the project of each individual team member [20%, different to each member]
  • The completeness of the project [20%]
  • Project proposal report [5%]
  • Project midterm report [10%]
  • Project final report write-up [30%]
  • Project spotlight, Midterm presentation and final presentation [5% each for a total of 15%]
  • Bonus: the potential impact of the project idea when fully flushed out. [+10 extra %]

Grades

A+: Exceptional or surprising. Could be submitted largely as-is as a paper to an academic conference. A new application or study or improvements, etc., of LLMs that expands very significantly on existing literature and is of benefit to the broader community.

A: A respectable research contribution that is novel and effective, and could be submitted largely as-is as a paper to an academic conference. Or an application with multiple well-thought out features which aren’t present in existing applications. Or a study of existing LLMs that revealed novel insights/concerns/understandings that were previously unknown to the community.

A-: A respectable research contribution that has some small incomplete parts, but is largely complete and promising. Or an application which contains promising, well-thought out features but has usability issues or insufficient technical scope. B+: An idea that is novel, but the results may not be there yet, or the analysis is short.

B or B-: Results, analysis, or novelty are lacking.

C+ or below: Clear lack of effort or incompleteness.

Negative Results: Sometimes experiments don’t work as planned. If you try hard to get positive results but are not successful, you may still get a good grade by clearly describing why you thought your methods would work, and then performing an analysis of why your initial assumptions were incorrect, leading to results that did not match your initial expectations. The bar for paper writing, experimentation, and analysis will be a bit higher in these cases, as we want to make sure that you really made a serious effort.

Example project ideas:

We will gradually share example project ideas on this doc: project ideas.

These ideas are for reference. You can choose one of these example ideas (we do not guarantee any of these ideas will work out, same with all research ideas) or build something of your own.