Hyperparameter tuning of Feed Forward and LSTM neural networks

Spread the word!


The huge number of possible variations (hyperparameter) within a neural network model makes it very hard to build a complete automated testing tool. From the other hand, manual tuning hyperparameters is very time wasting.

Unlike random automated tuning, Bayesian Optimisation methods aim to choose next hyperparameter values according to past good models.

One Python implementation of this approach is called Hyperopt. If you didn’t read this general post about Hyperopt I strongly reccomand to do it.

It’s assumed that the reader knows the basics of Feed-Forward and LSTM neural networks.

Example for regression problem with Feed-Forward neural network

The explained Python code is available in this  html page, which can also be found in my Github repository as Jupyter Notebook and PDF.

Example for binary classification problem with LSTM neural network

The same applies to LSTM networks. In this case, we have to build our model a bit differently, since we are dealing with a classification problem and LSTMs. Otherwise, the principle is the same. You can see here the html version and here the Github repository with Jupyter Notebook and PDF.

Deal with errors with some combinations of Hyperparameter

I faced many errors with the LSTM network whenever some hyperparameter values were set. It’s enough to re-execute the tuning to make such errors disappear, since they depend on the specific chosen set of values. The general approach I would suggest to avoid annoying manual code executions is to use a try … except block containing the parameter tuning inside an always true while loop. If an error occurs continue with the next iteration, otherwise break and show the results. This is the section where I used this approach (see the above LSTM implementation):




  1. hey there and thank you for your information – I have certainly picked up anything new from right here. I did however expertise a few technical issues using this website, since I experienced to reload the web site many times previous to I could get it to load properly. I had been wondering if your web hosting is OK? Not that I’m complaining, but slow loading instances times will often affect your placement in google and can damage your high quality score if ads and marketing with Adwords. Anyway I’m adding this RSS to my email and could look out for much more of your respective intriguing content. Make sure you update this again very soon.| а

    • Thanks for your feedback! I’m going to upgrade my hosting plan very soon. I do my best to offer a content which is concise and useful. I’m going to work further on it during the next month.

Leave a Reply to admin Cancel reply

Your email address will not be published.