Overview
For this project, you will be given tweets and corresponding images and your job is to correctly classify them as joy/sad. You will be given a training dataset of 4,500 labeled samples and 4,500 unlabeled samples. Your model will be tested on about 4,500 hold-out test examples for the leaderboard, and 4,500 separated hold-out validation set for final submission. The features of the dataset are described in more detail in the README.txt file of the project kit.
The format of the project is a competition, with live leaderboards (see below for more details).
Project Rules and Requirements
Rules and Policies
- You CANNOT download or harvest any additional training data from the internet. Both of these will be considered cheating and the penalty will be very harsh. Please, you MUST ONLY use the data we provide you with. We will test your final classifier, and if it is clear that we cannot replicate your performance because you use additional data, you will get a ZERO for the project.
- Except when specified otherwise, you are allowed to download additional code or toolboxes from the internet, however, you must cite everything you use in your final project report. We don’t want you to reinvent the wheel. If you’re unsure of what extra resources are allowed, please ask us.
- You must work in groups of 2–3 people. No single competitors or groups with more than 3 people will be allowed.
- Before you can submit to the leaderboard, you need to register your team using turnin (described below).
- In the competition, you need to reach a certain absolute score to get full credit. Placing particularly well will increase your project grade even further. First place gets 10%, second 8%, third 7%, and the rest of the top 10 teams 5% extra credit added to the project grade. The top 3 teams will also get awesome prizes — or at least public recognition.
- You will need to ensure that your code will run from start to finish on our server, so that we can reproduce your result for grading. We will provide a utility so that you can make sure we can run your code successfully. See below for details.
Overall requirements
The project is broken down into a series of checkpoints. There are four mandatory checkpoints and a final writeup which is due Dec. 12th. The leaderboards will be operating continuously so you can monitor your progress against other teams and towards the score based checkpoints. All mandatory deadlines are midnight. So, the deadline “Nov. 19th” means you can submit anytime before the 19th becomes the 20th.
- 1% - Nov. 23, run turnin -c cis520 -p proj_groups group.txt to let us know your team name. The file should contain a single line: the team name. In order to post scores to the leaderboard, you must have submitted your team name and have 2–3 members of the team total.
- 9% - Nov. 29, Checkpoint: Beat the baseline 1 by any margin.
- 20% - Dec. 5, Checkpoint: Beat the baseline 2 by any margin.
- 50% - Dec. 9, By the final submission (Dec 9), implement (or download and adapt an implementation of) at least 4 of the following. You should have at least one model from at least four of the following groups. By 11:59 PM, submit your implemented models to “proj_final” on turnin. We should be able to immediately and manually run both training and testing for each of your four algorithms easily. (Note: turnin will not automatically run your code or email you an output for your review.) Please include a README or PDF briefly describing the models you trained and documenting which files correspond to each algorithm and instructions on how to run and test with them. (You do not need to submit the final report at this time.). From these for or more models submitted to proj_final, they do not necessarily have to include the model used for your leaderboard, but you may! We will have your leaderboard model submission in the turnin project submission outlined below.
NOTE: There are TWO submissions expected on Dec. 9th. ONE is turnin to
proj_final and ANOTHER is turnin to
project. The
project submission is detailed below to verify that your leaderboard-worthy model satisfies the timing constraints (under the section header
Submit your code for the final checkpoint or to test correctness).
- A generative method (NB, HMMs, k-means clustering, GMMs, etc.)
- A discriminative method (logistic regression, decision trees, SVMs, etc.)
- An instance based method (kernel regression, k-nearest neighbors, etc.)
- Your own regularization method (other than the standard L0, L1 or L2 penalty)
- A semi-supervised dimensionality reduction of the data
- 20% - Dec 12 Submit the final report as a PDF to Canvas by 11:59 PM on December 12. The final report should be between 2 and 5 pages and should include all of the following.
- Results for each method you tried (try to use checkpoints to get test set accuracy for each of your methods)
- Analysis of your experiments. What worked and what didn’t work? Why not? What did you do to try to fix it? Simply saying “I tried XX and it didn’t work” is not enough.
- An interesting visualization of one of your models. For instance, find the words that most correlate with the outcome.
- Extra credit - In the competition, placing well will increase your project grade. First place gets 10%, second 8%, third 7%, and the rest of the top 10 teams 5% extra added to the project grade.
Evaluation
Error metric
Your predictions will be evaluated based on their {$L_0$} Err. I.e. the number of predictions you get wrong. Your code should produce an Nx1 vector of predictions, each of which is 0 or 1.
Requirements for Each Checkpoint
For the second and third checkpoints, you must submit to the leaderboard(s). For the final checkpoint, you must submit ALL of your code via turnin to the correct project folder. Make sure that you submit any code that you used in any way to train and evaluate your method. We will be opening up an autograder that will check the validity of your code to ensure that we’ll be able to evaluate it at the end.
Detailed Instructions
Download the starter kit
You will be able to download the starter kit here: http://alliance.seas.upenn.edu/~cis520/fall16/kit2.zip
You will be submitting your code to the auto-grader, which will execute your code on the test set and generate a vector of predictions. The auto-grader then will compare the predictions with the ground truth. You will be receiving an email enclosing the accuracy. The accuracy will be recorded on the leaderboard.
Register your team name
Before you can get results on the leaderboard, you need to submit your team name. Everyone on your team is required to do this. Simply create a text file on eniac with your team name as follows:
$ echo "My Team Name" > group.txt
$ turnin -c cis520 -p proj_groups group.txt
This group.txt file should be raw text and contain only a single line. Do not submit PDFs, word documents, rich text, HTML, or anything like that. Just follow the above commands. If you have a SEAS email address, then you will get an email confirmation.
Submit to the leaderboard
Submit all your code files including predict_labels.m and all supporting files needed (models):
turnin -c cis520 -p leaderboard *
Your team can submit once every 5 hours, so use your submissions wisely. Your submission will be checked against the reference solutions and you will get your score back via email. This score will also be posted to the leaderboard so everyone can see how awesome you are.
You can view the current leaderboard here: http://www.seas.upenn.edu/~cis520/fall16/leaderboard.html
Submit your code for the final checkpoint or to test correctness
Your code will be predict_labels.m which takes as arguments the feature files (as in the sample code
from the kit).
The time constraint for making predictions on 4,500 test samples is 10 minutes.
You must submit your code for the final checkpoint. You can do so with the following:
turnin -c cis520 -p project <list of files including predict_labels.m >
You will receive feedback from the autograder, exactly like the homework.
The feedback you will get from the autograder is whether your prediction code runs within 10 minutes for 4,500 test samples and whether the submission size is less than 50 Mb. You will not get feedback about your error on the test set, which appears on the leaderboard.
The final rankings will be released on the day of the prize ceremony, Dec. 12.
