Make sure you check the syllabus for the due date. Please use the notations adopted in class, even if the problem is stated in the book using a different notation.

SpamBase-Poluted dataset:
the same datapoints as in the original Spambase dataset, only with
a lot more columns (features) : either random values, or somewhat
loose features, or duplicated original features.

SpamBase-Poluted with missing values dataset: train,
test.
Same dataset, except some values (picked at random) have been
deleted.

Digits Dataset (Training data,
labels. Testing data,
labels): about 60,000 images, each 28x28 pixels representing
digit scans. Each image is labeled with the digit represented, one
of 10 classes: 0,1,2,...,9.

Cheng's top 15 features (IDs as column number in data, starting at 0): 52, 51, 56, 15, 6, 22, 23, 4, 26, 24, 7, 54, 5, 19, 18.

B) Spambase polluted dataset : Run Adaboost on polluted Spambase and report performance - why does it still work? Expected Accuracy: 93%.

A) Train and test Naive Bayes. Why the dramatic decrease in performance ? Expected Accuracy with Gaussian Fits: 62%

B) Run PCA first on the dataset in order to reduce dimensionality to about 100 features. You can use a PCA package or library of your choice.

Then train/test Naive Bayes on the PCA features. Explain the performance improvement. (To make it work, you have to apply PCA consistently to training and testing points: either apply for training and store the PCA transformation to apply it later for each test point; or apply PCA once for entire dataset)

Expected Accuracy on Naive Bayes with Gaussian Fits, running on PCA features: 73%.

C) Implement your own PCA, and rerun Naive Bayes on obtained features.

D)

Expected Accuracy when using Bernoulli fits: 80%.

B) [Optional no credit] Run tSNE library first on the dataset, computing distances/similarities with missing values. Then re-train and test Naive Bayes using the tSNE representations.

P

Then reconstruct the dateaset with only these selected features, and run L2-regularized classifier. Report accuracy per class.

Dataset: 1000 2-dim datapoints ThreeCircles

A) First, train a Linear Regression (library) and confirm that it doesnt work , i.e. it has a high classification error or high Root Mean Squared Error.

B) Run KernelPCA with Gaussian Kernel to obtain a representation of T features. For reference these steps we demoed in class (Matlab):

X2 = dot(X,X,2);

DIST_euclid = bsxfun(@plus, X2, X2') - 2 * X * X';

sigma = 3;

K = exp(-DIST_euclid/sigma);

%normalize the Kernel to correspond to zero-mean

U = ones(N)/ N ;

Kn = K - U*K -K*U + U*K*U ;

[V,D] = eig(Kn,'vector') ;

[D,sorteig] = sort(D,'descend') ;

V = V(:, sorteig);

XG = Kn*V';

X3G = XG(:,1:3);

X20G = XG(:,1:20);

X100G = XG(:,1:100);

C) Retrain Linear regression on the transformed D-dim data. How large D needs to be to get good performance?

Implement and run HAAR feature Extraction for each image on the
Digit Dataset. Then train and test a 10-class ECOC-Boosting on the
extracted features and report performance. You can sample the
training set (say 20% of each class), in order to scale down the
computation.

Expected Accuracy when using 200 HAAR features, 50 random ECOC,
each Adaboost trained for 200 rounds: 89%.

(**Hint:** For extracting the MNIST dataset, here are example code
for
Python,
MATLAB
Java
)

**
HAAR features for Digits Dataset** :First randomly
select/generate 100 rectangles fitting inside 28x28 image box. A
good idea (not mandatory) is to make rectangle be constrained to
have approx 130-170 area, which implies each side should be at
least 5. The set of rectangles is fixed for all images. For each
image, extract two HAAR features per rectangle (total 200
features):

- the black horizontal difference black(left-half) - black(right-half)
- the black vertical difference black(top-half) - black(bottom-half)

black(rectangle OBCD)= black(rectangle-diag(OD)) = count of black points in OBCD matrix

for i=rows

for j=columns

black(rectangle-diag(OD

- black(rectangle-diag(OD

end for

end for

Assuming all such rectangles cornered at O have their black computed and stored, the procedure for general rectangles is quite easy:

black(rectangle ABCD) = black(OTYD) - black(OTXB) - black(OZYC) + black(OZXA)

The last step is to compute the two feature (horizontal, vertical) values as differences:

vertical_feature_value(rectangle ABCD) = black(ABQR) - black(QRCD)

horizontal_feature_value(rectangle ABCD) = black(AMCN) - black(MBND)

B) Run Regularized Regression (separate runs for LASSO and RIDGE) using a package for regularization. For example use the scikit-learn (Python) or Liblinear (C++) implementation of LASSO. Compare with Logistic Regression performance. Expected Accuracy of Lasso Logistic Regression: 93%.

C) Implement your own RIDGE optimization for Logistic Regression. Expected Accuracy of Ridge Logistic Regression: 92%.

D) Implement your own LASSO optimization for linear regression.

A) Run Boosting (Adaboost or Rankboost or Gradient Boosting) to
text documents from 20 Newsgroups without extracting features in
advance. Extract features for each round of boosting based on
current boosting weights.

B) Run Boosting (Adaboost or Rankboost or Gradient Boosting) to
image datapoints from Digit Dataset without extracting features in
advance. Extract features for each round of boosting based on
current boosting weights. You can follow this paper.