Some brief notes on NeurIPS 2020

Topics:

• Fairness
• Privacy
• Meta-Learning
• The Pre-registration model for ML publications
• Miscellaneous

Fairness

Investigating Anti-Muslim Bias in GPT-3 through Words, Images, & Stories

• GPT-3 is incredibly biased against Muslims, but other groups as well. It is not always easy to identify to which groups or how a model is biased. What if we try to put humans to try and break language models BEFORE production?

scikit-learn and fairness, tools and challenges | slides

• Fairness can be defined in many ways, meaning that you can find a definition of fairness for which your ML model will be fair.

• It is expected that there is a fairness vs performance (accuracy) trade-off for your model. This needs to be crystal clear to your stakeholders. You want to be fair? You need to accept a risk of worse performance (and thus less profit).

• Removing an attribute (e.g gender) from the data does not mean that the model will be more fair. It might be more fair or more biased. This is not the solution.

• Adding constraints to the classification as meta-estimators to certain attributes can help. Adding a constraint does not mean “use less this attribute”, it could be the opposite “use more this attribute to make sure it used in a fair way”

• Tools: fairlearn (based on scikit) & aif360 (big entry barrier that you need to have a ai360 object dataset)

All in all, we can summarize with the amazing quote: “Trash in, trash out”. Trying to fix the world seems pointless. But if you just give up, then you only let people who don’t care about fixing the world, to lead the world and this technology. Just trying your best, even with little hope in sight, can make a difference.

Also interesting: Fairness, Explainability, and Privacy in AI/ML Systems

Privacy

Mostly based from the PPML Workshop

The Elephant in the Room: The Problems that Privacy-Preserving ML Can´t Solve

• Great talk about really hard challenges that might be even above ML. For example, data ownership (a selfie I took with other people on the background is it my, theirs or our data? My genomic data also includes data about my relatives).

Federated Learning Tutorial

• A long but great introduction into federated learning and current challenges!

Also interesting:

• OPACUS: Differential Privacy On Pytorch

• Building AI with Security and Privacy in mind

• POSEIDON: Privacy-Preserving Federated Neural Network Learning

• Privacy mirages in Machine learning

Meta-Learning

Mostly from Meta-Learning Workshop

Representations and Objectives by Tim Hospedales

• A different but very valuable and insightful view of meta-learning. Basically a talk of their latest survey with more details and intuitive explanations. For example about MAML:

  • The Meta-representation or “What” (Search space) of MAML is the weight initialization.
  • The Meta-Optimizer or “How” (Search algorithm) of MAML is the SGD.
  • The Meta-Objective or “Why?” (Search Objective) of MAML is decreasing validation loss on a few-shot learning task.

• Inductive bias = A learner’s policy for generalizing beyond seen instances.

• Learning to learn can be seen as a search for the ideal inductive bias for the current task of the distribution

• Meta-continual learning: The goal is to do continual learning, and we use meta-learning to help us discover a good continual learning algorithm. For example by learning a good weight consolidation or rehearsal strategy. Here meta-learning simply replaces manual design of CL algorithms.

• Continual meta-learning: The goal is something else (e.g., supervised few-shot, many-shot learning), and we want to meta-learn such an algorithm. But the family of tasks that we have to learn from arrives in a continual stream. So now we need to do vanilla meta-learning but meeting the CL criteria (e.g., our learner doesn’t forget how to learn a task that it hasn’t seen for a while).

Meta-Learning Neural Architectures, Initial Weights, Hyperparameters, and Algorithm Components

• A great intuitive introduction to the field with a bit of history step-by-step.

• SEARL: Online AutoML for RL

• A brief account of meta-Learning for Bayesian Optimization, Algorithmic Configuration, Algorithmic Selection, Dynamic Algorithmic Selection

Pre-registration model

A lot of food for thought in this workshop regarding conducting ML research, and quite concerning… (More info: Pre-registration model).

Where is Machine Learning Going?

• How are we going to deal with flag planting (a trend of submitting a paper on arXiv two weeks before deadline)

• We are becoming a “fast food” type of science.

Must see:

• Incentives for Researchers

• Can pre-registration lead to better reproducibility in ML research?

• The Turing Way: Transparent research through the scientific lifecycle

Miscellaneous

You Can’t Escape Hyperparameters and Latent Variables: Machine Learning as a Software Engineering Enterprise

• Probably the most fun “talk” (more like a small film) but also quite important

Few Lessons Learned by Samy Bengio

• Don’t plan your whole life ahead. It’s good to explore and make mistakes.

• Taking risks is part of the job. Dont ignore a topic because it’s not too trendy.

• Try to open source your code. It’s worth the effort.

• There are different ways you can impact society. Publishing, advocating, coding…

A Road towards the Chain Rule by Oriol Vinyals

• When judging other people keep in mind you judge by their success not their failures. Everyone fails all the time. Failures can be motivating and important. Do not ignore failures.

• Living abroad and getting exposed to a wider and more diverse network is important.

• Networking with friends and colleagues is more important than a “good” CV.

The Real AI Revolution by Chris Bishop

Distinction between types of researchers.

• Basic / Academic: “Want to unlock the world” (e.g Marie Curie)
• Applied / Industry: “Want to run a successful business” (e.g Thomas Edison)
• Hybrid: Initial goal is practical, an application but the method used is based on academic research about the world (Bishop classifies himself here)

Where Neuroscience meets AI

• Interesting introduction to cool stuff /facts about neuroscience. Does not go too deep or too technical or into Deep Learning.

Practical Uncertainty Estimation and Out-of-Distribution Robustness

• Datashift affects both model accuracy and certainty quality (uncertainty degrades over the shift).

Something Has Been Ruined Forever from the Queer in AI workshop

• An interesting talk, quite different than all the others. Not technical and more poetic / philosophical, about a topic that is rarely being discussed in ML.