On this page… (hide)

Overview

For this project, you will be given tweets and your job is to correctly classify them as joy/sad/surprise/anger/fear. The training dataset consists of a collection of 18092 labeled samples. Your classifier will be tested on a validation (or “quiz”) set for the leaderboard, and a hold-out test set for final ranking, using a loss function which weights different errors differently. The features of the dataset are described in more detail in the README.txt file of the project kit. The format of the project is a competition, with live leaderboards (see below for more details).

Project Rules and Requirements

Rules and Policies

You CANNOT download or harvest any additional training data from the internet. Both of these will be considered cheating and the penalty will be very harsh. Please, you MUST ONLY use the data we provide you with. We will test your final classifier, and if it is clear that we cannot replicate your performance because you use additional data, you will get a ZERO for the project.
Except when specified otherwise, you are allowed to download additional code or toolboxes from the internet, however, you must cite everything you use in your final project report. We don’t want you to reinvent the wheel. If you’re unsure of what extra resources are allowed, please ask us.
You must work in groups of 2–3 people. No single competitors or groups with more than 3 people will be allowed.
Before you can submit to the leaderboard, you need to register your team using turnin (described below).
In the competition, you need to reach a certain absolute score to get full credit. Placing particularly well will increase your project grade even further. First place gets 10%, second 8%, third 7%, and the rest of the top 10 teams 5% extra credit added to the project grade. The top 3 teams will also get awesome prizes — or at least public recognition.
You will need to ensure that your code will run from start to finish on our server, so that we can reproduce your result for grading. We will provide a utility so that you can make sure we can run your code successfully. See below for details.

Overall requirements

The project is broken down into a series of checkpoints. There are four mandatory checkpoints and a final writeup which is due Dec. 11th. The leaderboards will be operating continuously so you can monitor your progress against other teams and towards the score based checkpoints. All mandatory deadlines are midnight. So, the deadline “Nov. 19th” means you can submit anytime before the 19th becomes the 20th.

1% - Nov. 22, run turnin -c cis520 -p proj_groups group.txt to let us know your team name. The file should contain a single line: the team name. In order to post scores to the leaderboard, you must have submitted your team name and have 2–3 members of the team total.
9% - Nov. 28, Checkpoint: Beat the baseline 1 by any margin.
20% - Dec. 4, Checkpoint: Beat the baseline 2 by any margin.
50% - Dec. 8, By the final submission (Dec 8), implement (or download and adapt an implementation of) at least one model from each of the 3 following groups. By 11:59 PM, submit your implemented models to “proj_final” on turnin. We should be able to immediately and manually run both training and testing for each of your 3 algorithms easily. (Note: turnin will not automatically run your code or email you an output for your review.) Please include a README or PDF briefly describing the models you trained and documenting which files correspond to each algorithm and instructions on how to run and test with them. (You do not need to submit the final report at this time.). From these 3 or more models submitted to proj_final, they do not necessarily have to include the model used for your leaderboard, but you may! We will have your leaderboard model submission in the turnin project submission outlined below.

NOTE: There are TWO submissions expected on Dec. 8th. ONE is turnin to proj_final and ANOTHER is turnin to project. The project submission is detailed below to verify that your leaderboard-worthy model satisfies the timing constraints (under the section header Submit your code for the final checkpoint or to test correctness).

A generative method (NB, HMMs, k-means clustering, GMMs, etc.)
A discriminative method (logistic regression, decision trees, SVMs, etc.)
An instance based method (kernel regression, k-nearest neighbors, etc.)

20% - Dec 11 Submit the final report as a PDF to Canvas by 11:59 PM on December 11. The final report should be between 2 and 5 pages and should include all of the following.
- A table with results for each method you tried (try to use checkpoints to get test set accuracy for each of your methods)
- Analysis of your experiments. What worked and what didn’t work? Why not? What did you do to try to fix it? Simply saying “I tried XX and it didn’t work” is not enough. How did you handle the cost function?
Extra credit - In the competition, placing well will increase your project grade. First place gets 10%, second 8%, third 7%, and the rest of the top 10 teams 5% extra added to the project grade.

Evaluation

Error metric

Your predictions will be evaluated based on a weighted loss function, specified in the attached notes.

Requirements for Each Checkpoint

For the second and third checkpoints, you must submit to the leaderboard(s). For the final checkpoint, you must submit ALL of your code via turnin to the correct project folder. Make sure that you submit any code that you used in any way to train and evaluate your method. We will be opening up an autograder that will check the validity of your code to ensure that we’ll be able to evaluate it at the end.

Detailed Instructions

Download the starter kit

You will be able to download the starter kit here: http://alliance.seas.upenn.edu/~cis520/fall17/final_project_kit.zip

You will be submitting your code to the auto-grader, which will execute your code on the test set and generate a vector of predictions. The auto-grader then will compare the predictions with the ground truth. You will be receiving an email enclosing the accuracy. The accuracy will be recorded on the leaderboard.

Register your team name

Before you can get results on the leaderboard, you need to submit your team name. Everyone on your team is required to do this. Simply create a text file on eniac with your team name as follows:

$ echo "My Team Name" > group.txt
$ turnin -c cis520 -p proj_groups group.txt

This group.txt file should be raw text and contain only a single line. Do not submit PDFs, word documents, rich text, HTML, or anything like that. Just follow the above commands. If you have a SEAS email address, then you will get an email confirmation.

Submit to the leaderboard

Submit all your code files including predict_labels.m and all supporting files needed (models):

turnin -c cis520 -p leaderboard *

Your team can submit once every 3 hours, so use your submissions wisely. Your submission will be checked against the reference solutions and you will get your score back via email. This score will also be posted to the leaderboard so everyone can see how awesome you are.

You can view the current leaderboard here: http://www.seas.upenn.edu/~cis520/fall17/leaderboard.html

Submit your code for the final checkpoint or to test correctness

Your code will be predict_labels.m which takes as arguments the feature files (as in the sample code from the kit).

The time constraint for making predictions on 9,310 test samples is 10 minutes.

You must submit your code for the final checkpoint. You can do so with the following:

turnin -c cis520 -p project <list of files including predict_labels.m >

You will receive feedback from the autograder, exactly like the homework.

The feedback you will get from the autograder is whether your prediction code runs within 10 minutes for 9,310 test samples and whether the submission size is less than 50 Mb. You will not get feedback about your error on the test set, which appears on the leaderboard.

The final rankings will be released on the day of the prize ceremony, Dec. 11.