Its a project exploring an issue or concept or phenomenon we have encountered in our
study of ANNs. (We studied and interacted with the MNIST dataset and a simple ANN for digit classification)
I will be attaching the pdf with the assignment’s instructions.
The Idea/Title for my project was “How do different
learning rate strategies impact the Accuracy of a MNIST
ANN?“
In it, I measured Accuracy (Performance score) of ANNs trained on different learning rate strategies (5 strategies) over varying training lengths (number of epochs in training) (5, 10, 15, 20, 25, 30) and reporting on how they influence it.
This was done for 3 different starting learning rates (0.1, 0.3, 0.5).
I have written the code (trying to attach but .ipynb file are not being accepted by Essay Pro so I’m attaching a screenshot of what I added/wrote and the size of the base ANN (really small, only 3 layers)) (I can try to write it in a word file if it is necessary)
I am using the
MNIST dataset (attached)
I’ll be attaching an excel file with all the performance scores, for all conditions and starting learning rates.
The learning rate strategies/conditions are:
– Constant (the same throughout)
– Decaying (decreases inversely to number of epochs)
– Warm-up (starts small, gradually increasing until reaching starting learning rate (e.g. 0.1, 0.3, or 0.5) over the first 10 epochs, then constant)
– Cycle up (oscillates between starting learning rate and 1.5 the starting learning rate)
– Cycle down (oscillates between starting learning rate and 0.5 the starting learning rate)
Regarding graphs and tables, for each starting learning rate (0.1, 0.3, 0.5), I want:
– 1 table with all the empirical data for that starting learning rate, including sections for condition means and standard deviation.
– 1 line graph comparing the performance of all 5 strategies over training length for that starting learning rate. (include standard deviation error bars if not visually cluttered)
– 1 line graph same as above but excluding “Decaying” condition (for better visibility of differences between conditions)(‘Decaying’ scores are much smaller than others so graph gets zoomed out)
– (check the excel file. I sort of did a draft of what I want in them)
My hypothesis for this report was:
“I
expect that a Constant learning rate will initially result in the highest Accuracy,
but a different learning rate strategy will take its place as the number of
training epochs increases“