General

We will update this page periodically with extra explanations for problems that we are getting a lot of questions about.

I need more probability background. Can you point me to some textbooks?

Take a look at the ML Math page that we just put up — that has some pointers to brief review notes that should hopefully be a lot more helpful than trying to plow through a whole textbook. Also, for a quick review of common probability distributions, check out the appendix in the Pattern Recognition textbook.

How do I use turnin?

NOTE: Unless you hear otherwise, we will NOT accept assignments via e-mail from students who have SEAS accounts. Students who do not have SEAS accounts, please just send an email with your code to all the TAs (jengi@cis.upenn.edu, djweiss@cis.upenn.edu, kenry@cis.upenn.edu, owensj@cis.upenn.edu).

SEAS students: At a high level, here’s what you need to do to turn in your code ---

Somehow, get your assignment to your SEAS home directory, either via SFTP (described below), or via mounting it in Windows using SAMBA.
Once it’s there, login to the eniac.seas.upenn.edu using SSH, and run the turnin command: turnin -c cis520 -p hw1code <folder with your source code>
or
turnin -c cis520 -p hw1code <file1.m> <file2.m> ...

There’s no need to zip your files before submitting, as the turnin command will do that for you. Note that each time you turn in, the previous submission is deleted. We’ll use the timestamp on the most recent submission to determine lateness. To confirm that your submission worked, you can run turnin with the “-v” verification flag:

turnin -c cis520 -p hw1code -v

If you use Windows, you need to install an SSH client, for example as described here:

http://www.seas.upenn.edu/cets/answers/remote.html

Note: I do not recommend using SecureCRT, since it can be confusing and hard to use. A much easier to use program is “Secure Shell”, which is free for non-commercial use:

http://www.dartmouth.edu/comp/soft-comp/software/downloads/windows/ssh-sftp

Then you can connect to eniac.seas.upenn.edu using your SEAS account. Once you’re connected, you can open an SFTP session (if you’re using “Secure Shell”) that will allow you to transfer files between your computer and your SEAS home directory. (In “Secure Shell”, it’s by clicking on a little folder icon at the top of the screen.) Then, once you have your file on your SEAS account, you can use turnin on the command line following the instructions we gave before.

If you’ve never used a UNIX command line interface before, I found a collection of guides and tutorials here:

http://www.seas.upenn.edu/~cit595/resources/unix/index.shtml

Here’s another quick synopsis of using scp and ssh:

You first need to get your homework files onto eniac.  One way to do
this is with the scp command.  On a command line, if your code is in a
directory called “hw1” then you can copy it to eniac doing something
like:

scp -r hw1 <pennkey>@eniac.seas.upenn.edu:.

and then when you log into eniac using

ssh <pennkey>@eniac.seas.upenn.edu

you should see the “hw1” folder there.  Then you can use turnin with
something like:

turnin -c cis520 -p hw1code hw1

and the “hw1” folder will be submitted.

Problem 1

You should be able to solve this using just the expectation rules and formulas given in the question. If you are integrating or taking derivatives for this problem, you are working harder than you have to.

Problem 2

2.1.2: When we say “compute the posterior distribution over {$\lambda$}”, what we want in the end is a statement like “the posterior is a type <insert type here> distribution, with parameters <insert any necessary parameters here>”. For example, if you thought the posterior was a Gaussian distribution, you would say so and then state its mean and variance, since these are the two parameters associated with a Gaussian distribution.

2.2.2: Don’t try to start by converting {$\hat{\eta}$} into a sum using the Taylor expansion hint. A summation that looks like the righthand side of the hint will naturally arise in the course of computing the bias of {$\hat{\eta}$}. Simply start by writing out the formula for bias, which will contain an {$E[\hat{\eta}]$}. When you write out the formula for this expectation, you will see how to use the hint.

Problem 3

3.2.1: The definition of conditional probability, {$p(y \mid x) = \frac{p(x,y)}{p(x)}$}, and marginalization {$p(y) = \sum_x p(x,y)$} will be useful here. Try switching the order of summations if you are having trouble seeing how to use marginalization.

3.3.1: “Three variables” means {$ X_1 $}, {$ X_2 $}, and {$ X_3 $}, plus the outcome {$Y$}.

3.3.2: It will probably be easiest to write the pseudocode here if you think of a recursive algorithm. Also, your code shouldn’t need to be too long; our solution is about 15 lines.

Problem 4

4.2: For a particular dataset (standard or noisy), the procedure for generating the K-NN plot should be something like the following:

for iter = 1 … 100
- randomly select 400 training instances and leave the rest as test data
- for k = 1, 5, 9, 15
  - compute the mean LOOCV error for the 400 training points
  - compute the mean (regular, non-LOOCV) error for the test points

Now you have 100 mean values for each setting of k, for both train and test (overall 100*4*2 means). For each k, find and plot the mean of the 100 train means (so, this is the average of an average) and the standard deviation of these 100 train means. Do the same for the test values.

You will end up with 4 train points with standard deviations (draw a line through the points, use err bars to plot the deviations) and 4 test points with standard deviations (draw a line through the points, use err bars to plot the deviations).

Note that you may be able to better vectorize your code than in the pseudocode shown above (explicitly using for-loops in matlab is slower than doing matrix and vector operations), but this pseudocode does outline the correct overall procedure.

Homework 1