Want to help curate these resources, and build your CV? Contact us below.
News & Research
Our updated list of the latest articles and research papers on machine learning and AI. Scroll down for the newest articles, or navigate to a specific theme.
25th April 2021
An interview with Rumen Dangovksi: On AI applications and advancements
Rumen Dangovski (http://super-ms.mit.edu/rumen.html) is a PhD student at the Physics for AI Research at MIT. Some of his interests include self-supervised and meta learning, and improving machine learning through principles from fundamental science - especially physics.
He has previously worked at a start-up company centred on developing light-based computer chips for AI applications (click here to learn more about that), and has kindly accepted to have a short talk with me about his thoughts and experience in the field of AI - here is our exclusive interview with him as he sheds light on some of the most exciting advancements in this rapidly growing field.
When predictive policing backfires
16th June 2021
Predictive policing is the process of finding patterns in criminal data, to identify potential criminals and stop crimes before they even happen. To do so, many rely on artificial intelligence to derive meaning from huge amounts of data, yet this also highlights the many issues this technology still has: most police departments are clueless as to what exactly the algorithm is searching for in potential criminals, for example, and heavily biased data only exacerbates racial and socioeconomic inequalities that plague many of the areas where predictive policing is trying to be implemented.
Robert McDaniel lived in one of the most dangerous areas of Chicago, yet had no violent criminal records of his own. In 2013, police officers from the City of Chicago arrived at his house, announcing that he was on the “heat list”: a database of people that were determined to be involved in some future shooting. However, they did not know whether he would be the shooter, or the shooted. Both a potential perpetrator and a potential victim, he became closely watched by officials day and night, causing his friends and acquaintances to grow suspicious of him - wondering whether he was collaborating, or reporting to the police in some way. After many years of struggles attempting to clear his name, McDaniel finally got involved in a shooting - although not in the way the police expected: some former friends shot him out of suspicion, because he was on the heat list and was constantly trailed by the police, putting everyone at risk. Predictive policing had backfired, by essentially causing a series of events which might not have happened had the police never visited McDaniel’s house in the first place.
This illustrates the crucial question of whether relying solely on data, and following its recommendations by the letter, is a good idea at all. Especially if the quality of the available data is subpar, trying to predict criminals’ next moves can do more harm than good, sowing mistrust and doubt in communities that only encourage violence. This also perpetuates the idea that law enforcement has no empathy for the individual circumstances of each person, and the use of artificial intelligence with such heavy implications should be closely monitored - to say the least.
NLP advancements define the future “Google search”
9th June 2021
MIT Technology Review
Nowadays, online Google searches have been ingrained into our daily life, and its page-ranking technology is now ubiquitous in all types of search engines. But this might be about to change: researchers at Google have proposed a new redesign of the way we query results on the Internet, that uses a single AI language model rather than a page-ranking algorithm.
This language model would be similar to BERT, or Open AI’s GPT-3. Rather than having to find information on web pages yourself, you’d instead ask your questions to the model and it will respond to you directly, fundamentally changing our interaction with search engines. This was done to respond to several issues: for example, it is common for search engines to return results containing your question, but not the answer. It is also a time-consuming and manual process if you want to cross-reference your sources, whereas a natural language model trained on the Internet would be able to do that for you automatically.
Essentially, the new search engine would be similar to an all-knowing expert you can interact with - synthesizing, sourcing, referencing and explaining answers in the most natural way possible. Our current NLP models are partly there, but still fail at some crucial aspects: for instance, determining how valid and reliable a source is, or generalizing a model to perform all the tasks mentioned previously at the same time.
Read the proposal preprint: https://arxiv.org/pdf/2105.02274.pdf
The new type of cyberattack that targets AI
31st May 2021
MIT Technology Review
Traditional neural networks are well-known to be power-hungry, and consume plenty of energy when the network is large and deep. To combat this, a new type of neural network has sprung up in recent years: input-adaptive multi-exit networks, which separate inputs based on their perceived difficulty to solve and spend less computational power on “easier” problems. This speeds up the process up to the point of being able to perform some of these calculations on your own smartphone.
However, this also exposes a vulnerability in the system’s design: hackers are able to add noise to the inputs to make the neural network think that the problem is more difficult to solve than it actually is, thus making the computer allocate much more resources than needed and racking up expense in energy. Furthermore, this could potentially be transferred across various other types of neural networks. Researchers from the Maryland Cybersecurity Center have prepared a new paper on this, which is due to be presented at the International Conference on Learning Representations.
The entire universe is a machine learning algorithm?
26th May 2021
The Next Web
A bold statement, to say the least. Researchers and theoretical physicists, in collaboration with Microsoft, have released a new preprint that details how they believe the universe essentially learns on its own - an “Autodidactic Universe”, as the preprint title suggests.
A quote from the preprint: “For instance, when we see structures that resemble deep learning architectures emerge in simple autodidactic systems might we imagine that the operative matrix architecture in which our universe evolves laws, itself evolved from an autodidactic system that arose from the most minimal possible starting conditions?”
The main takeaway from it is that the universe tends to adapt the laws of physics, similar to a self-learning neural network. These laws, such as conservation of energy, would therefore not be fundamental laws but would rather be constantly evolving in response to the current state of the universe. This is in stark contrast to most modern physics theories which affirm that fundamental laws are set in stone - which also implies that unifying physics might be impossible, if the universe is in a perpetual cycle of self-improvement.
Read the preprint here:
DeepONet, another step towards lightning-fast solutions in physics
22nd May 2021
Traditionally, neural networks map values between spaces with finite dimensions (for example, with image classification, think of the values of pixels in an image being mapped onto a number between 0 and 9 to represent 10 different classes), but now researchers have come up with a new concept: mapping an infinite-dimensional vector space onto another infinite-dimensional space. Using a deep neural network dubbed DeepONet, the concept is to work with operators rather than functions: these are essentially a mapping from one function to another, such as taking a derivative. Its special feature is its bifurcated architecture, which processes data in two parallel networks, a “branch” and a “trunk.” An immediate application for this is in solving partial differential equations (PDEs): these equations model almost any process we can think of, from fluid dynamics to climate change, yet are incredibly difficult to solve and usually require extremely long computations. That is, before we started using deep neural nets on them.
Solving partial differential equations using neural networks was already done last year with the Fourier neural operator (FNO). This network also maps functions to functions, from infinite-dimensional space to infinite-dimensional space, and solves PDEs with incredible speed. These two methods both seem promising and highlight dramatic new approaches to solving PDEs in comparison to the old, computationally-taxing methods. What’s more, is that these do not appear to suffer from the curse of dimensionality, where too many features (or dimensions) in a training set causes the model to underperform (The Curse of Dimensionality. Why High Dimensional Data Can Be So… | by Tony Yiu).
Podcast: In Machines We Trust - When AI meets your wallet
18th May 2021
MIT Technology Review
Cash in itself is worthless - it simply is a flimsy piece of paper that somehow, guarantees you anything from food and clothing to the ability to travel around the globe. In other words, the value of money comes from the trust that we place in it, and this is usually enforced by central banks, supported by their respective governments. However, in this day and age, a digital revolution is happening in your wallet: more and more companies are attempting to leverage AI to make payments, study spending habits, exchange funds, deal with cryptocurrencies and so much more. For example, you can now use contactless on your phone with Apple Pay, or decide to use an online bank rather than an established brick-and-mortar one for your daily expenses. The question is whether this is for the better, or for the worse.
Listen to what experts from Google, J.P. Morgan and more have to say about this in this episode from the podcast series In Machines We Trust, produced by the MIT Technology Review.
GLOM, Geoffrey Hinton’s latest hunch
12th May 2021
Geoffrey Hinton - a computer scientist, cognitive psychologist and known as one of the godfathers of AI - has a new suggestion: he believes that to make neural networks for vision truly intelligent, we must first understand the brain. And how is this achieved? Through a new concept for an intelligent system named GLOM, a computer vision system that includes transformers, neural fields, contrastive representation learning, distillation and capsules. In an arXiv preprint, he outlines the way it would be implemented, but warns that it is still purely theoretical: “This paper does not describe a working system. Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system.” Fun fact: GLOM is derived from the slang ”glom together” which may derive from the word ”agglomerate”.
The key goal of GLOM is to be able to model our human intuition - how we can make sense of our surroundings and draw parallels with our past experiences in an intuitive manner. This human perception is one of the major differences between AI and human intelligence, and Hinton believes that GLOM might play a part in solving it.
Don’t feel like reading a 44-page preprint? This Youtube video nicely breaks down the key concepts in GLOM in an understandable way:
Read the paper: https://arxiv.org/pdf/2102.12627.pdf
Bipedal robots learn to walk with arms
6th May 2021
Humans tend to walk on two legs - that is, until they encounter obstacles or situations that require them to use their arms as well for balance, for example, if you were walking on a narrow bridge. This isn’t because your legs are failing you - rather, it’s due to how precarious your environment is without extra security, and the consequences of you tripping and falling. This inspired researchers at TUM (Technical University of Munich) in Germany to leverage this feature in humanoid robots: the latter are known to be notoriously unstable in unstructured environments. Yet, they are also complex and resource-intensive to build, and thus any damage caused by falls would be extremely costly.
Using more than two limbs for locomotion in robots had been a challenge due to software and hardware limitations, but this advancement has proven to be the most human-like mobility seen in robots to date. Dubbed multi-contact locomotion, in the future, this could enable scientists to design both proactive and reactive behaviours which would allow robots to both predict and respond to uneven terrain with hand or upper limb stabilisation. Read this article for a fascinating interview with Philipp Seiwald, who works with LOLA (the first bipedal robot to have this kind of multi-contact locomotion upgrades) at TUM, to delve into more facts and detail regarding this subject matter.
Facial recognition has bled into civil rights
1st May 2021
MIT Technology Review
Detroit, January 9th, 2020. Robert Williams had been wrongfully arrested for stealing watches because AI facial recognition software had wrongfully identified him as the criminal. Now, more than a year later, the American Civil Liberties Union and University of Michigan Law School’s Civil Rights Litigation Initiative filed a lawsuit on behalf of Williams, claiming that this had infringed on his civil rights. “By employing technology that is empirically proven to misidentify Black people at rates far higher than other groups of people,” the lawsuit states, ”the DPD denied Mr. Williams the full and equal enjoyment of the Detroit Police Department’s services, privileges, and advantages because of his race or color.”
Thus, using AI-powered technology in criminal justice also has become an issue of discrimination: bias easily infiltrates systems that are not robust enough, and an over-reliance on software alone, without any human judgment, could worsen the chasm with targeted minorities. Between a rushed investigation, technological errors and the low-quality data that is fed into the algorithm, pushing for the use of AI facial recognition before it is ready could have damaging consequences on the livelihood of many.
Machine learning’s ongoing struggle with causality
21st April 2021
For humans, determining causal relationships instead of simple correlation, such as knowing that a bat moves because an arm is swinging it (and not the other way around!), seems simple and intuitive; this is thanks to our experience and intuition of the world. Yet even the most sophisticated deep learning neural networks struggle to do this, despite being able to perform impressive and detailed pattern recognition.
In a paper titled “Towards Causal Representation Learning,” researchers at the Max Planck Institute for Intelligent Systems, the Montreal Institute for Learning Algorithms (Mila), and Google Research, discuss the challenges arising from the lack of causal representations in machine learning models and provide directions for creating artificial intelligence systems that can learn causal representations.
In this article, Ben Dickson delves into why we are still failing this intuitive task and the different ways researchers are trying to remedy this. Read the original paper here:
The fault in our data
16th April 2021
MIT Technology Review
Junk in, junk out. You might have heard this saying countless times, and a dozen more if you are studying or interested in machine learning and training artificial intelligence models on labelled data. A new study from MIT has revealed an eye-opening fact: many of the most cited and widely used datasets are in fact riddled with errors and mislabeling, and this could fundamentally change our view of the success of models trained in them. For instance, “2916 label errors comprise 6% of the ImageNet validation set”. This might seem little compared to the sheer size of the data, but researchers realized that some models that performed relatively poorly on the faulty validation set actually performed better than state-of-the-art models used for example by Google, and implies that valuable models could have been wrongly discarded and lost. Has our confidence in current model performance been inflated because of this?
Read the paper here:
GPT-Neo, GPT-3’s open-source sibling
12th April 2021
In 2020, OpenAI released the famous, massive, NLP algorithm called GPT-3. It is an autoregressive language model that uses deep learning to produce jarringly human-like text, datasets, and is able to perform translation, question-answering, as well as several tasks that require on-the-fly reasoning, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. From there, many spin-offs have been created, such as DALL E which creates images from text captions. However, gaining commercial access to this product requires a paid license and massive computing resources and power.
Eleuther is an open-source alternative to the GPT-3 project. Its model is still some way from matching the full capabilities of the latter, but last week the researchers released a new version of their model, called GPT-Neo, which is about as powerful as the least sophisticated version of GPT-3. In today’s race to bigger, better NLP models, the democratization of AI models could further accelerate the path to progress and reduce the chance of AI being solely reserved for high-tech companies. Additionally, the engineers at Eleuther have devised a way to make use of spare cloud computing resources to keep the computational power accessible for all. Another interesting note is that the dataset that Eleuther is using is more diverse than GPT-3, and it avoids some sources such as Reddit that are more likely to include dubious material.
A thought-provoking matter: AI advances in emotion detection
9th April 2021
Queen Mary University
We used to think that our thoughts are the one place where we could truly be private: after all, mind-reading is fiction - or is it? Researchers at Queen Mary University in London have successfully leveraged a deep neural network that can determine a person’s emotional state by analyzing wireless signals that are used like radar. In their article, they describe how radio waves can be used to analyse the breathing and heart rates of a person, even in the absence of any other visual cues, such as facial expressions. This could then be scaled to infer emotion in large gatherings, for example at work, and collect information on how people react to different activities.
Although this breakthrough is fascinating, it raises some important questions around ethics, privacy and morals: is it right to infer the mood of an individual, let alone large groups, and act solely based on those subjective concepts? Can one truly categorize a state of mind, or is it a continuous, dynamic spectrum?
Read the paper here:
Can we have true artificial intelligence without first understanding the brain?
1st April 2021
MIT Technology Review
Jeff Hawkins is one of the most successful computer architects in Silicon Valley - and what differentiates him from most others in the field is his dedication to understanding just how neuroscience and artificial intelligence are linked, not just if AI can replicate a human mind with large enough models. After working as a software engineer, he decided to pursue a PhD in neuroscience at Berkeley to better understand the big picture: “What is intelligence and how does it work?”. Next, he ventured into entrepreneurship and founded many highly-regarded companies, namely Palm Computing and Numenta, a research company in neuroscience. Journalists from the MIT Technology Review recently interviewed him on how alike biological and artificial intelligence really are - or should be.
Dark matter and astrophysics: how AI can help you see the unobservable
27th March 2021
In astronomy, gravitational lenses are images of distant galaxies that appear bent or circular in shape: as the light emitted by these distant galaxies passes by massive objects in the universe, their gravity can distort or “pull” the light towards them. This phenomenon, called “gravitational lensing”, also occurs when galaxies are close to large amounts of dark matter - a hyperdense, invisible constituent that makes up most of our universe and that has been fascinating astrophysicists for years.
However, it is extremely tedious and difficult to find those gravitational lenses in images from observatories. But recently, researchers from several universities joined forces and designed an AI model based on deep residual neural networks to scan survey data on images of gravitational lenses, and non-lenses.
Then using this model they attempted to find additional gravitational lenses from the DESI Legacy Imaging Surveys, which are enormous datasets of images of the observable universe. Impressively, the model detected more than 1200 new potential lenses, in stark contrast to the 300 that were already found when the project first started.
Read the preprint of the study here:
Self supervised learning, the dark matter of intelligence
24th March 2021
Supervised learning, or training AI on already labelled data, has been one of the most popular paradigms in machine learning for many years. It has proven to be extremely useful, however, there is a limit to this: in real life, not everything can be labelled, and the latter is a resource-intensive, tedious task prone to error and bias.
We, humans, learn differently: through observation, inference and common sense. This final element has stumped machine learning engineers who are still trying to find ways to translate these concepts, and thus consider it as the “dark matter of AI”. For instance, children can recognize images of an animal after being shown very few examples and then recognize them in images, yet AI models require hundreds of images and still be likely to misclassify images of cows with horses. This is because, in short, humans rely on their previously acquired background knowledge of how the world works.
One way of leveraging this in AI would be through self-supervised learning. FacebookAI describes it as follows: “Self-supervised learning obtains supervisory signals from the data itself, often leveraging the underlying structure in the data. The general technique of self-supervised learning is to predict any unobserved or hidden part (or property) of the input from any observed or unhidden part of the input. For example, as is common in NLP, we can hide part of a sentence and predict the hidden words from the remaining words. We can also predict past or future frames in a video (hidden data) from current ones (observed data). Since self-supervised learning uses the structure of the data itself, it can make use of a variety of supervisory signals across co-occurring modalities (e.g., video and audio) and across large data sets — all without relying on labels.”
A computer chip ... that works on light
21st March 2021
AI runs on computers, and for a computer, hardware and software go hand-in-hand: even the most sophisticated algorithms cannot perform well if they are running on insufficient computing power during training and testing. That is why more and more people are attempting to find innovative ways to reinvent computers, in order to keep up with the fast pace of AI progress.
Traditionally, the flow of electrons in semiconductors is the foundational concept of modern computers which enables them to perform boolean operations, the building blocks of software and logic. Now, a new concept has been introduced by Lightmatter, a startup founded at MIT - light-based computer chips.
These chips can be faster than conventional chips for some types of AI calculations, as the different wavelengths of light are used to encode information, which is less greedy than controlling electrons in terms of power usage. Furthermore, they can be directly compatible with most AI software and data centres. A notable use for these chips would be deep learning, although there are some limitations. For instance, the calculations are analogue rather than digital, leading to less precision, and companies might be reluctant to use this design on a large scale as it has not yet been proven as clearly superior. However, this still shows great potential, by leveraging light and physics to unlock new ways of doing AI. As Aydogan Ozcan, a professor at UCLA, explains: “We might see major advances in computing speed, power and parallelism, which will further feed into and accelerate the success of AI.”
This is how we lost control of our faces
14th March 2021
MIT Technology Review
The applications of facial recognition are becoming increasingly ubiquitous: from telesurveillance to improving search suggestions, larger and larger datasets are being generated to draw insights from our faces. In recent years, however, a growing issue has been of the quality of the datasets: as researchers scramble for more images, asking for consent and regulated data collection has become a tedious task which more and more gloss over. This can lead to messier, non-consensual and possibly erroneous data, which has drastic repercussions on the model’s output in practice such as labelling with racist stereotypes.
The history of facial recognition is fascinating: what started out as trying to match images to people and painstakingly verifying the outputs manually, now has turned to auto labelling and attempting to infer ethnicity, gender and other characteristics from a single image.
Read more about the study that presents those findings here:
10 Breakthrough Technologies of 2021
10th March 2021
MIT Technology Review
You’ve probably heard this too many times to count: the past year has been like no other. Yet, technology and progress have still continued to improve, and are still set on redefining our future in all fields, from healthcare to finance. With selections such as data trusts or messenger RNA vaccines, the editors of the MIT Technology Review have compiled a list of 10 technologies, each with corresponding featured articles, which they believe have the potential to change our lives this year.
A couple of notable ones related to AI:
GPT-3: some of the world’s largest family of Natural Language-based computer models that can generate images, autocomplete sentences and even invent entire short stories that rival those of a human author.
Multi-skilled AI: even today, most models and robots are only able to complete tasks that they have been trained to do explicitly, or solve problems they have encountered before. Transfer learning is still in its infancy and required specific calibrations. This is in clear contrast to us humans, who have been able to adapt and transfer skills from one domain to another with ease. A key breakthrough as of late has been AI models that are able to combine different senses, for example, computer vision with audio recognition, making them better suited to understand their environment and interact with us.
Fractals in image pretraining
7th March 2021
MIT Technology Review
When using AI models for image processing, often they will be pre-trained on general image datasets sourced from the internet. This can be useful to help improve performance and speed up training time by essentially grabbing an “off-the-shelf” model to then tailor to a specific type of image or theme. However, there are a few issues with this: for example, labels can contain biased, stereotypical or even racist words, and gathering data firsthand is both costly and time-intensive.
Thus, being able to use computer-generated datasets for pre-training is an emerging trend likely to gain momentum in the future. Recently, Japanese researchers have come up with FractalDB, a database containing “an endless number of computer-generated fractals”. Since fractals are pervasive throughout nature, in places ranging from snowflakes to trees, these abstract fractals have been grouped and used as a pre-training dataset for convolutional neural networks (CNNs). The result? The performance was nearly as good as models pre-trained on actual images from state-of-the-art datasets. However, caution is still needed: abstract images have been shown to possibly confuse image recognition systems, so at this stage, computer-generated images are not yet completely able to replace manual ones.
(Artificial) Intelligence on Mars
3rd March 2021
As you might have heard, on February 18th, NASA’s Perseverance robot made its triumphant landing on the red planet to begin searching for traces of life. With cutting-edge equipment and hardware, this robot is undoubtedly state-of-the-art - but its software is not to be underestimated either. Perseverance has the most AI capabilities of any rover, which has been essential in ensuring everything runs as smoothly as possible during its mission. For example, in helping Perseverance land safely with little information about local terrain millions of kilometres away. In contrast to its predecessor Curiosity, Perseverance was able to land in a more dangerous location, Jezero Crater, largely thanks to the fact that “if it recognizes it's coming down on a place that's not safe, it will autonomously steer during its supersonic descending-to-zero speed descent to Mars”, as explained by Raymond Francis, a science operations engineer at NASA’s Jet Propulsion Laboratory.
Furthermore, other applications of AI include use in targeting instruments and improved autonomous navigation. Due to the distance, information takes up to 40 minutes to travel between the rover and ground control, and time is of the essence. Therefore, when Perseverance travels to unknown locations, it will use AI to decide autonomously which are the best areas and rocks to investigate rather than waiting until humans are able to see these locations and give instructions, which emphasizes the crucial role AI plays in the mission. Francis adds: “Space missions to outer planets or harsh environments are going to depend on AI-based autonomy more and more”.
Beyond qubits: Next big step to scale up quantum computing
28th February 2021
ScienceDaily & Nature
You may be familiar with the concept of a “bit”: the smallest binary unit of information stored in a computer, usually represented by a 1 or 0. You may also be familiar with quantum physics, where one of the fundamental principles is superposition: when particles can exist in multiple states at once. But what happens when you combine the two concepts? You get a quantum computer, where a bit becomes a “quantum bit”, or “qubit”, and can have more than two distinct states.
Quantum computers are sometimes considered the “supercomputers of tomorrow”: they have the potential to run calculations exponentially faster than today’s most powerful processors, and despite them still being in their infancy tech giants like IBM and Google are racing to find ways to make the technology stable and scalable.
One challenge in the creation of quantum computers is that it is difficult to coordinate and stabilize these qubits. Current machines are bulky and impractical, involving more than hundreds of connections and wiring. Now, scientists and engineers at the University of Sydney and Microsoft Corporation have invented a single chip that can control thousands of qubits at once. Microsoft Senior Hardware Engineer, Dr Kushal Das, a joint inventor of the chip, says: "Our device does away with all those cables. With just two wires carrying information as input, it can generate control signals for thousands of qubits.”
Nature paper: http://dx.doi.org/10.1038/s41928-020-00528-y
How to train a roomba
24th February 2021
To create functional robots, the algorithms behind them should be able to navigate through complex environments with various physical obstacles and structures. However, to do so, large image datasets are required but must be tailored to the physical properties of the robot. Take for example the popular Roomba (an automatic vacuum cleaner): traditionally, images would be taken from the vantage point of the robot in various different environments and “stitched back” manually to create a virtual layout of an interior, as images taken at human height failed to produce satisfying results. However, this sort of manual capture is inefficient and incredibly slow.
Researchers at the University of Texas at Austin are looking to take advantage of a form of deep learning known as “generative adversarial networks, or GANs, where two neural networks contest with each other in a game until the 'generator' of new data can fool a 'discriminator.'”.
This would enable the generation of any sort of possible environments, able to be tweaked to the user’s preferences, and that the robot could use to recognize and detect objects and obstacles. Mohammad Samiul Arshad, a graduate involved in the research, adds: "Manually designing these objects would take a huge amount of resources and hours of human labour while, if trained properly, the generative networks can make them in seconds."
AI and Architecture, how well do they pair?
21st February 2021
In recent years, AI models have increasingly been replacing quantifiable, formulaic and repetitive tasks “susceptible to analysis and reproduction by machine learning systems”. However, does this also apply to disciplines combining both design analysis and artistic expression, such as architecture?
Some uses include semi-automated design generation, composing both internal and external spaces, and tools which are based on neural networks that gradually learn an architect’s habits and preferences when generating designs over time and gradually adapts its processes and methods to better suit those (an example of which is Finch).
Furthermore, with the popularization of GPUs and image recognition, research is underway in a new field called “architectural biometrics”, initially stemming from a glitch where facial recognition software tended to confuse patterns in buildings with faces. This may lead us to better understand “the anthropomorphic aspect of the way that humans create and relate to buildings”.
The range of applications is vast and diverse, yet architects are, according to The Economist, some of the people least likely to be replaced by AI in the future. This is because it also incorporates social, cultural and even political elements, all of which are difficult to capture with only calculations and represent more concrete concepts. Yet the above also demonstrates that AI can be a powerful tool to help simplify and enhance the work of architects, showing how there is a potential for collaboration between AI and the human mind rather than a battle for supremacy.
Men wear suits, women wear bikinis?
16th February 2021
MIT Technology Review
It is a well-known fact that natural language models can perpetuate racism and bias because of the text it was trained on. Now, researchers at Carnegie Mellon University have also found out that image-generating algorithms that were pre-trained in an unsupervised manner (i.e. without humans labelling images) also contain human-like biases. These models base their knowledge and training on the internet - which is often full of harmful stereotypes that overrepresent those biases. For instance, images of women, even US Representative Alexandria Ocasio-Cortez, were autocompleted by the algorithm wearing low-cut tops or bikinis 53% of the time - a startling and eye-opening proportion.
This issue also transcended the more technological methodological differences for each model: for both OpenAI’s i-GPT (an image version of GPT-2) and Google’s SimCLR, photos of men and ties and suits appear more related, whereas those of women appear farther apart from the latter. The implications of this are important: it suggests that anything from video-based candidate assessment algorithms to facial recognition could perpetuate those biased views, especially when there is no human in the loop to correct them.
OpenAI’s latest image-generation capabilities
14th February 2021
“Give me an illustration of a baby daikon radish in a tutu walking a dog.” This might not be the first thing you’d like to ask an AI model, but OpenAI’s latest innovation, DALL·E, sure is capable of doing it. Using a “12-billion parameter version of GPT-3”, and trained on image-text pairs, this model takes in any regular sentence, it is able to generate a plausible image to fit the description. On a more technical level, it is a decoder-only transformer that receives both the text and the image as a single stream of 1280 tokens.
A fascinating feature that the DALL·E also displayed some capacity for “zero-shot reasoning”: essentially, it was able to apply image transformations based on textual instructions, for instance, “the exact same cat on top as a sketch on the bottom” (possibly useful in illustration and product design), with an image prompt. More simple transformations such as a photo coloured pink were performed with higher accuracy.
Furthermore, DALL·E also displayed, to some extent, geographic and temporal knowledge: you could ask it for a picture of a phone from the ’20s, or an image of food from China. However, it tended to show superficial stereotypes for choices like “food” and “wildlife,” rather than fully representing the real-life diversity of these themes.
Reducing bias in healthcare outcomes with AI
11th February 2021
MIT Technology Review
We often hear about machine learning models perpetuating bias in practical, real-life settings, whether it be racial, social or economic. But in the field of healthcare, machines might not be the only ones at fault: for instance, the US National Institute of Health found that Black patients with osteoarthritis are more likely to report higher levels of pain than their white counterparts, even though they appear to have the same KLG score. A KLG score is a pain level score determined by radiologists based on X-rays and is used by doctors to assign treatments instead of their self-reported pain. This prompted researchers to ask themselves why - are the Black patients exaggerating their pain, feel pain differently, or is the KLG score entirely unsuitable and biased towards the pain levels of white patients?
After running experiments with a deep-learning algorithm, researchers discovered that the ML model was much more accurate in predicting levels of self-reported pain based on patient X-rays than KLG - regardless of ethnicity - to the extent of reducing “racial disparity at each pain level by nearly half”. This startingly reveals that standard ways of determining pain might be flawed and tailored to certain populations (similar to overfitting in machine learning) and thus could be an area where more objective algorithms can help out.
Natural language predicts COVID viral escape
3rd February 2021
MIT Technology Review
In this day and age, COVID-19 is at the centre of current research. This is no different in the field of AI and ML: Bonnie Berger, a computational biologist, and her colleagues have released a new paper, leveraging the disciplines of biology and computer science to explain how natural-language processing (NLP) algorithms can “generate protein sequences and predict virus mutations, including key changes that help the coronavirus evade the immune system.”
Interestingly, mutations and genetic characteristics of a virus can be interpreted through grammar and semantics - for example, an unfit virus will be “grammatically incorrect”.
Using a type of neural network called LSTM, they were able to identify possible mutations for the virus that would be viable, genetically speaking. This is extremely important knowledge for healthcare researchers and authorities, as knowing in advance future incoming variations can help them plan ahead faster and better prepare their defences. A fascinating read
How to make AI a greater force for good in 2021?
30th January 2021
MIT Technology Review
January is almost over, and we are all slowly getting used to the new year. Undoubtedly, there has been massive progress in AI and machine learning in the past 12 months, such as with OpenAI’s GPT-3, or with its applications in healthcare and clinical research, and its influence will continue growing this year. But how do we ensure that it is not used maliciously, or cross ethical boundaries that are yet to be defined? With the rising concern around data protection and privacy, the regulation of this relatively new field is becoming increasingly urgent.
Karen Hao, a senior AI reporter at MIT Technology Review, delves into the five ways she hopes AI can improve on this year: by reducing corporate influence in research, refocusing on common-sense understanding, empowering marginalized researchers, centering the perspectives of impacted communities, and codifying guard rails into regulation.
Google trained a trillion-parameter AI language model
27th January 2021
In machine learning, training on data historical data fine-tunes the parameters (essentially the variables) of the algorithm. In practice, the number of parameters seems to correlate with the sophistication of a model - the higher number of parameters, the more complex tasks a model is able to perform. Researchers also noticed that simpler, more straightforward models trained on large datasets and with a high number of parameters tend to outperform more complex architectures with fewer parameters.
Using this knowledge, Google endeavoured to train a 1.6 trillion parameter model, the largest of its size to date. This improves upon their previous model with a speedup factor of 4. Since training on so much data is both time and resource-intensive, researchers used a technique that uses “only a subset of a model’s weights, or the parameters that transform input data within the model” (aka ‘sparsely activated’). However, it is important to note they did not take into account the bias that would result on training from such an extensive dataset - in real life, such models also can amplify biases present in the training data such as stereotypes and discrimination. Nevertheless, this can still benefit our understanding of the link between training data, parameters and overall model performance for the future.
23rd January 2021
In quantum chemistry, working out Schrödinger’s equation is essential in order to “predict chemical and physical properties of molecules based solely on the arrangement of their atoms in space”. However, like most partial differential equations, it remains a challenge to solve and usually requires huge amounts of brute force computational power. Recently, researchers at Freie Universität Berlin have combined accuracy and computational efficiency in creating an AI model capable of doing this much more efficiently.
Schrödinger’s equation is based on the wave equation, which determines how electrons move in a molecule. The standard approach to expressing the wave equation for a particular molecule used to consist of mathematical approximations to map the behaviour of each individual atom, but the complexity of combining predictions for each atom, especially given the number of dimensions in the wave equation, makes it nearly impossible for larger molecules.
The team leader, Dr Frank Noé, explains: “[W]e designed an artificial neural network capable of learning the complex patterns of how electrons are located around the nuclei”. Although the model is not yet ready for industrial use, it opens up promising new opportunities in reducing “the need for resource-intensive and time-consuming laboratory experiments”.
Check out the original paper here:
Language neural networks - made lightweight
19th January 2021
BERT is a heavyweight, powerful and complex algorithm in natural language processing, and can be used in applications ranging from text generation to online chatbots (it is also used in Google’s search engine). However, running this state-of-the-art algorithm is no easy feat - it requires massive computational power, something that ordinary users cannot easily get hold of.
Recently, however, Jonathan Frankle and his colleagues at MIT have discovered subnetworks in BERT that could be repurposed for easier natural language processing tasks. This links to his “lottery ticket hypothesis”: essentially, “hidden within massive neural networks, leaner subnetworks can complete the same task more efficiently. The trick is finding those “lucky” subnetworks, dubbed winning lottery tickets”. Although some subnetworks are context-specific and cannot be transferred over to different tasks, they also found that most of the time fine-tuning was not necessary to identify these “lottery ticket” subnetworks. This could become part of the solution to make artificial intelligence and machine learning more accessible to all, and even opens up the possibility for future users to train models on the go, or even on their smartphones.
Learning on the fly with reinforcement-learning
15th January 2021
In a continuous effort to push the boundaries of machine learning and deep reinforcement learning, researchers at the University of Edinburgh and Zhejiang University have combined efforts to merge several deep neural networks developed for different applications to create a new system with the benefits of all of its individual neural networks. Deep neural networks are algorithms that train on multiple examples over and over again, in order to solve problems such as classification. They are becoming more widespread, with useful applications including credit risk analysis, facial recognition and text recognition.
Now things are getting even better - this newly-developed neural net combination has the ability to learn new functions that none of the individual components could perform on their own. The researchers call it a multi-expert learning architecture (MELA). They then implemented it in a robot and discovered that it used each neural network with others and through trial and error, made them work in ways that it had not been taught, such as walking on slippery surfaces.
An AI that helps you summarize the latest in AI
22nd December 2020
MIT Technology Review
Tl;dr that research paper? The new algorithm from the Allen Institute for Artificial Intelligence (AI2) might help. On November 16, it rolled out the model onto its flagship product, Semantic Scholar, an AI-powered scientific paper search engine. It offers a short, one-sentence long summary under every computer science paper (for now) when users use the search function or go to an author’s page.
While many other research efforts have tackled the task of summarization, this one stands out for the level of compression it can achieve. Semantic Scholar can compress on average to 238 times the size of an average research paper. The next best abstractive method is able to compress scientific papers by an average of “only” 36.5 times. In the testing phase, human reviewers also judged the model’s summaries to be more informative and accurate than previous methods.
Why our ML model training is flawed.
16th December 2020
MIT Technology Review
There may be several reasons why a top-quality machine learning model, with high accuracy and precision in testing conditions, performs less than ideally in real-life situations. For example, the training and testing data may not match reality - this is known as data mismatch. Now, researchers at Google have brought to light another phenomenon that causes this gap in performance: underspecification.
From Natural Language Processing to disease prediction, underspecification is a common, and arguably one of the most significant issues of modern machine learning training. Essentially, the same models trained on the same dataset can have infinitesimal variations although they all pass the testing phase. This can be caused by random assignment of variables, for example. However, these small differences can make or break the model’s performance in the real world - in other words, we may be too lax with the testing criteria of our models, but even we, as human beings, cannot know which version would perform better.
The key here is to find a way to better specify our requirements to models and is essential in order for us to regain trust in the beneficial impact AI could have outside the labs.
Podcast: Can you teach a machine to think?
9th December 2020
MIT Technology Review
Currently, AI is advancing in leaps and bounds, yet some might wonder how close we are to a “general artificial intelligence” - one that is not limited to only text, like GPT-3 but can multitask and reason on their own via various modes of communication. This also raises a burning question: who really benefits from the replication of human intelligence in an artificial mind? Where is the incentive? Is it the Big Tech companies, who can then use it to generate more revenue? Is AI currently democratised enough so that ordinary people can access it and not only multi-billion-dollar companies? These interesting questions, and more, are raised in this episode of Deep Tech, MIT’s podcast on helping people understand and gain insight into the interlinking of technology and society.
AI is wrestling with a replication crisis
27th November 2020
MIT Technology Review
As AI is overtaking the world, so is scientific research related to it. This is exciting, but comes with a whole new set of challenges: artificial intelligence is a relatively new concept, thus there has not yet been strict regulations and frameworks implemented to structure its research. Because of this, there is a lack of transparency, such as in papers or published code (usually very little is revealed), which is often criticised since it leads to issues in reproducibility.
As Joelle Pineau, a computer scientist at Facebook AI Research explains, “It used to be theoretical, but more and more we are running experiments. And our dedication to sound methodology is lagging behind the ambition of our experiments.”
This is why Joelle, along with several other scientists in Nature, wrote a scathing critique of Google Health’s paper on breast cancer detection, which reveals a deeper underlying trend in journals publishing AI research that has little concrete evidence, preventing replication and advancement.
A System Hierarchy for Brain-Inspired Computing
26th November 2020
Neuromorphic computing - computing on architecture inspired by the brain rather than traditional von Neumann architecture - promises to usher in a new age of computing, and with it the means to develop improved forms of artificial intelligence.
Many experimental platforms for neuromorphic computing have been developed, but thus far have been limited by their lack of interoperability - no one algorithm can run on all neuromorphic architectures.
Zhang et al provide a solution to this problem in the form of a generalised system hierarchy comparable to the current Turing completeness-based hierarchy, but instead proposing 'neuromorphic completeness'.
The authors show that this hierarchy can support programming-language portability, hardware completeness and compilation feasibility. They hope that it will increase the efficiency and compatibility of brain-inspired systems, even quickening the pace toward development of artificial general intelligence.
From uncertain to unequivocal, deep learning’s future according to AI pioneer Geoff Hinton.
21st November 2020
MIT Technology Review
A decade ago, artificial intelligence was an obscure idea, and the former truly began its revolution only recently. Geoffrey Hinton, one of the winners of the Turing award last year for his foundational work in this field, talks about his thoughts on how AI and deep learning will develop in the next few years:
On the AI field’s gaps: "There’s going to have to be quite a few conceptual breakthroughs...we also need a massive increase in scale."
On neural networks’ weaknesses: "Neural nets are surprisingly good at dealing with a rather small amount of data, with a huge numbers of parameters, but people are even better."
On how our brains work: "What’s inside the brain is these big vectors of neural activity."
AI has cracked a key mathematical puzzle for understanding our world
17th November 2020
MIT Technology Review
You might remember - or are studying - partial differential equations. These types of equations are extremely useful in modelling real-world situations, such as fluid motion or planetary orbits, for example. The only problem? They are extremely hard to solve and researchers often use supercomputers, with their enormous processing power, to find their solutions.
However, now the field of AI has entered the stage: researchers at Caltech have developed a generalizable deep learning technique, capable of solving entire families of PDEs—such as the Navier-Stokes equation for any type of fluid—without needing retraining. It also is hundreds of thousands of times faster than any mathematical formula. This makes sense, as fundamentally, the job of an AI algorithm is to fund a function that will give the correct output for a given input, such as predicting “cat” from an image of a cat. Similarly, this sort of function approximation is linked to how we solve partial differential equations.
These new findings could have wide-ranging implications - from modelling weather patterns rapidly and accurately for climate change response to predicting air turbulence patterns. As Caltech professor Anandkumar puts it, “the sky’s the limit”.
But what is AI anyway?
This might help.
12th November 2020
MIT Technology Review
The emerging field of artificial intelligence can be tricky to navigate, especially to newcomers who are not familiar with programming nor the buzzwords used. Even for experts in the field, defining what artificial intelligence truly and concretely means is not as simple as it sounds, and there is no clear-cut answer.
This short podcast explains what AI is (and isn’t) - a great listen to have as a pause between study sessions, or anytime just for fun!
AI songs: easy music to our ears, but a struggle to create
7th November 2020
MIT Technology Review
AI can produce delightfully catchy music, as can be shown by the Australian Team Uncanny Valley’s submission to the AI Song Contest, titled “Beautiful the World”. However, it is still a challenge to be able to communicate with the model, and fine-tuning the track to your preferences. For example, you might want the chorus to be repeated a set number of times - but how do you tell that to AI?
As Carrie Cai, who works at Google Brain and studies human-computer interaction puts it, “AI can sometimes be an assistant, merely a tool. Or AI could be a collaborator, another composer in the room. AI could even level you up, give you superpowers. It could be like composing with Mozart.”
Can we make AI learn like children?
3rd November 2020
MIT Technology Review
A common view in AI and data science is “the larger the dataset, the better the model”. In contrast, once a human recognises something, that’s it - no need to make them look at thousands of other pictures of the same object. Similarly, children learn in the same way, and more usefully, they don’t need to see an image to recognize it: if we tell them a persimmon is like an orange tomato, they’d instantly be able to locate one if they see it in the future.
What if AI models could do the same? Researchers at the University of Waterloo (Ontario) have come up with a technique. Dubbed as the “less-than-one-shot” learning (LO-shot), this would allow models to recognize more objects than the number it was trained on. This is achieved by condensing large datasets into much smaller ones, optimised to contain the same amount of information as the original. With time, this could prove to be a groundbreaking technique which could save millions in data acquisition.
How AI discriminates in speech recognition
28th October 2020
If you have a non-American accent and attempted to use a speech-recognition software such as Siri or Alexa, perhaps you’ve already had the misfortune of experiencing the algorithm completely misunderstanding what you were saying - sometimes to a ridiculously funny extent. In fact, this is an extremely widespread problem and a depiction of bias, where the software used for voice recognition is skewed towards catering to the upper to middle-class, educated, English-speaking category.
One of the most likely culprits is the data used for training, which is predominantly from “white”, native speakers of American English. This simple “inconvenience” for some of us can become a deeper issue to others, for example for the visually impaired, who rely on speech recognition to complete everyday tasks.
This article explores the gaps in diversity within datasets and how we can start bridging them, as well as the greatest challenges we might face in doing so.
Defining key terms in AI in 1 page: informative and suitable for non-specialists
27th October 2020
Stanford Institute for Human-
Centered Artificial Intelligence
Do you really understand the core concepts in the field of artificial intelligence in the current era when 'AI' is everywhere?
Just now, Christopher Manning - Prof. CS & Linguistics at Stanford University, director of the Stanford Artificial Intelligence Laboratory (SAIL), and Associate Director of HAI - used one page to define the key terms in AI. He expressed the hope that these definitions can help non-specialists understand AI.
Serve and Replace: Covid's impact on AI automatization
21st October 2020
MIT Technology Review
The Innovation Issue
Supermarkets staffed with robots reshelving goods might be your idea of the future of helper robots, but the covid-19 pandemic has brought to light just how far-reaching their uses could be. From spraying disinfectant to walking dogs, their presence has grown tremendously. More importantly, they are able to perform riskier tasks in the presence of infected patients or deliver lab samples, which would free up nurses for more essential tasks.
On the flip side, the same robots are a threat to the workforce, stealing away potential jobs from millions who are already in financial difficulty due to the crisis. And now, workers are potential carriers of the virus, accelerating the shift to an automated world. As Hayasaki explains, “(b)efore covid-19 hit, many companies—not just in logistics or medicine—were looking at using robots to cut costs while protecting humans from dangerous tasks. Today humans are the danger, potentially infecting others with the coronavirus. Now the challenge is that a minimum-wage laborer might actually be a carrier”.
The future of AI in robotics is not black or white, and this gripping article shows just how much the pandemic is going to change the technology landscape.
Bonus: Page 71 features a fascinating story written by an AI bot!
A.I. creativity is improving fast. This hilarious GPT3-generated film is proof
20th October 2020
Since its release in June this year, OpenAI's text-generating AI tool GPT-3 has received great attention. It is used to comment on forums, write poems, and even publish articles in The Guardian.
Today, GPT-3 has achieved another feat: it wrote a script — students from Chapman University in the United States used GPT-3 to create a script and produced a new short film: Solicitors.
Check out the script created by GPT-3 and watch the short film (link 1).
However, despite the many high hopes for GPT-3 in this boom, two scientists questioned sharply and reasonably the implementation logic behind it after a series of tests.
On August 22, Gary Marcus - Professor Emeritus at New York University, the founder and CEO of Robust.AI - and Ernest Davis - Professor of Computer Science at NYU - published an article entitled GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about (link 2) on MIT Technology Review.
They told Technology Review that GPT-3 itself does not have 'revolutionary' changes, nor can it truly understand semantics. The key things still depend on human judgment. Amidst the enthusiastic voices, this undoubtedly brings more objective and rational thinking.