Summer School 2021

Courses

Course 1 (Wagner / Haider): Introduction to deep learning and differentiable programming in Python - This first session of the summer school provides a general introduction to machine learning with neural networks: A historical retrospective, a sketch of modern applications and the basics of their architecture and training procedure. The Python skills that are necessary to set up a simple machine learning project are reviewed. In the exercise session, participants are familiarized with the workflow and will design their first model for classification in PyTorch. Starting from an introductory level, this course levels the fundaments of all participants.

• slides: day1_introduction to DL.pdf
• tutorial: day1_AISummerSchool2021-exercises.zip , day1-material_instructions.pdf
• video: AISS21_Day1.mp4

Course 2 (Harár): Planning, execution and dissemination of a machine learning project - During the life cycle of our machine learning research projects, we interact with source code, data, and models almost daily. Therefore, it is wise to make use of various tools and tricks of the community that save a lot of time during the development (versioning), can be decisive during the peer review (reproducibility), and are invaluable for making an impact (presentation). Moreover, the community is constantly coming up with tools useful even before the beginning (to find ideas) or after a seeming end of the project (publicity). Together, in this hands-on session, we will use those tools while developing a specific ML project. After the session, you should be able to use the tools in projects of your own.

• slides: day2-a-tool-a-day.pdf
• tutorial: day2_AISummerSchool2021-tutorial.zip
• video: AISS21_Day2.mp4

Course 3 (Perraudin): Optimization in Deep Neural Networks: Basics, Methods and Challenges- Training a deep neural network corresponds to solve a minimization optimization problem. In this lecture, we will review the building blocks used to construct algorithms for this task (such as stochastic gradient descent, momentum and adaptive learning rate). Doing so, we can understand the challenges of deep network optimization. Finally, this course will also include specific techniques to overcome these challenges (such as batch norm, regularization, dropout, early stopping, data augmentation, architecture invariance and parameter sharing). This lesson will equip students with the intuition and tools to face the most common optimization challenges and design choices in Deep Learning.

• slides: day3-1_optimization.pdf , day3-2_equivariance-losses.pdf
• tutorial: day3-AISummerSchool2021-tutorial.zip , day3-material_instructions.pdf
• video: AISS21_Day3.mp4

Course 4 (Grosse-Wentrup): A Causal Perspective on Interpretable Machine Learning and explainable AI - Machine learning and AI models are increasingly deployed in high-stakes environments, e.g., in medical decision support or loan approval, where ethical and legal considerations require models to be interpretable. Prof. Grosse-Wentrup will review current methods for interpretable machine learning and explainable AI, discuss challenges in designing such methods, and then argue that we can resolve some of these issues by adopting a causal perspective on the problem. In particular, he outlines how to reason about causal relations in general and then show how a causal perspective enables us to distinguish between interpreting the model and using the model to interpret the data-generating process.

• slides: day4-iml-inference.pdf , day4_causality_and_ai_oeaw_ss21.pdf
• tutorial: day4_AISummerSchool2021-tutorial.zip
• video: AISS21_Day4.mp4

Outlook (Perraudin):

• slides: day5-outlook.pdf
• video: AISS21_Day5.mp4

Note: All slides, tutorials and videos can also be downloaded from here.

Lecturers

Univ.-Prof. Dipl.-Ing. Dr. -Ing. Moritz Grosse-Wentrup is head of the research group Neuroinformatics at the Faculty of Computer Science of the University of Vienna and board member of the university’s research network Data Science @ Uni Vienna. [https://ni.cs.univie.ac.at/team/person/107192/]

Ing. Pavol Harár, Ph.D. is a machine learning researcher at the research network Data Science @ Uni Vienna at the University of Vienna and Senior Researcher at the Brain Diseases Analysis Laboratory, a international, multidisciplinary research group with a focus on brain diseases. [https://pavol.harar.eu/]

Dr. Nathanaël Perraudin is a Senior Data Scientist at the Swiss Data Science Center in Zurich, a Cooperation of ETH Zurich and the École polytechnique fédérale de Lausanne. [https://perraudin.info/]

Dipl.-Ing. Felix Wagner is a PhD Student at the Institute of High Energy Physics (HEPHY) of the Austrian Academy of Sciences (OEAW) and member of the CRESST and COSINUS dark matter search collaborations.

Daniel Haider MSc is a PhD student in Mathematics at the Acoustics Research Institute (ARI) of the Austrian Academy of Sciences (OEAW) and member of the groups “Mathematics and Signal Processing in Acoustics” and “Machine Learning in Acoustics”.

Schedule

September 13-17, 2021

The summer school participants were expected to possess a basic understanding of the Python programming language and the numpy package for scientific computing in Python.

If the following statements apply to you, you should be fine:

• I understand how basic data types, variables and loops are defined and used in Python (e.g., strings. numbers, Dictionaries, for-loops and lists).
• I know about Classes and Objects and how they are used in Python.
• I know about numpy and its purpose.
• I know how a matrix array is created using numpy, what it looks like, and how I can apply some basic manipulations (e.g., reshaping) and mathematical operations (e.g., addition, multiplication) to them.
• Furthermore, participants were expected to be familiar with the fundamental ideas behind machine learning and deep learning and familiar with the concept of a (deep) neural network.

If you can answer the following questions with ‘yes’, you should be fine:

• Do I know what machine learning is and could I explain its general idea to a colleague?
• Do I know what deep learning is and could I explain its general idea to a colleague?
• Do I understand what a neural network is and can I describe it, as well as explain the difference between a shallow and a deep neural network, to a colleague?

More familiarity with Python and/or machine learning might make some aspects of the summer school easier to follow, but should not be strictly necessary. You can find a wealth of resources about all these topics on the internet and some below. If you are unsure about your own prior knowledge, you will have a significantly better experience at the summer school if you invest some time (1-2 working days) into preparing yourself.

We provide a few exemplary links for preparatory material. These are separated into 3 categories: Mandatory prior knowledge, helpful prior knowledge and optional in-depth or advanced material. Obviously, you are invited to do your own research instead, if you prefer.

Mandatory prior knowledge: 

• https://python.land (Chapters Introduction & Classes and Objects)
• https://numpy.org/doc/stable/user/absolute_beginners.html#welcome-to-numpy (Absolute basics for beginners)
• https://www.youtube.com/watch?v=nKW8Ndu7Mjw (Basic understanding of machine learning)
• https://www.youtube.com/watch?v=aircAruvnKk (What is a Neural Network?)

Helpful prior knowledge (read/watch as much or as little as you like):

• https://pytorch.org/tutorials/ (PyTorch Tutorial)
• https://scikit-learn.org/stable/tutorial/index.html (Scikit-Learn Tutorial)
• https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html (Pandas Introduction – for data structures and data analysis tools)
• https://www.datacamp.com/community/tutorials/seaborn-python-tutorial (Seaborn Tutorial – for  Statistical Data Visualization)  

In-depth and advanced material:

• Deep Learning (extensive) - http://d2l.ai/index.html