Discover our latest curated articles on natural language processing.

NLP advancements define the future “Google search”

9th June 2021

MIT Technology Review

Joy Xu

Nowadays, online Google searches have been ingrained into our daily life, and its page-ranking technology is now ubiquitous in all types of search engines. But this might be about to change: researchers at Google have proposed a new redesign of the way we query results on the Internet, that uses a single AI language model rather than a page-ranking algorithm.


This language model would be similar to BERT, or Open AI’s GPT-3. Rather than having to find information on web pages yourself, you’d instead ask your questions to the model and it will respond to you directly, fundamentally changing our interaction with search engines. This was done to respond to several issues: for example, it is common for search engines to return results containing your question, but not the answer. It is also a time-consuming and manual process if you want to cross-reference your sources, whereas a natural language model trained on the Internet would be able to do that for you automatically.


Essentially, the new search engine would be similar to an all-knowing expert you can interact with - synthesizing, sourcing, referencing and explaining answers in the most natural way possible. Our current NLP models are partly there, but still fail at some crucial aspects: for instance, determining how valid and reliable a source is, or generalizing a model to perform all the tasks mentioned previously at the same time.

Read the proposal preprint:

Google trained a trillion-parameter AI language model

27th January 2021

Venture Beat

Joy Xu

In machine learning, training on data historical data fine-tunes the parameters (essentially the variables) of the algorithm. In practice, the number of parameters seems to correlate with the sophistication of a model - the higher number of parameters, the more complex tasks a model is able to perform. Researchers also noticed that simpler, more straightforward models trained on large datasets and with a high number of parameters tend to outperform more complex architectures with fewer parameters.

Using this knowledge, Google endeavoured to train a 1.6 trillion parameter model, the largest of its size to date. This improves upon their previous model with a speedup factor of 4. Since training on so much data is both time and resource-intensive, researchers used a technique that uses “only a subset of a model’s weights, or the parameters that transform input data within the model” (aka ‘sparsely activated’). However, it is important to note they did not take into account the bias that would result on training from such an extensive dataset - in real life, such models also can amplify biases present in the training data such as stereotypes and discrimination. Nevertheless, this can still benefit our understanding of the link between training data, parameters and overall model performance for the future.

Language neural networks - made lightweight

19th January 2021

MIT News

Joy Xu

BERT is a heavyweight, powerful and complex algorithm in natural language processing, and can be used in applications ranging from text generation to online chatbots (it is also used in Google’s search engine). However, running this state-of-the-art algorithm is no easy feat - it requires massive computational power, something that ordinary users cannot easily get hold of.

Recently, however, Jonathan Frankle and his colleagues at MIT have discovered subnetworks in BERT that could be repurposed for easier natural language processing tasks. This links to his “lottery ticket hypothesis”: essentially, “hidden within massive neural networks, leaner subnetworks can complete the same task more efficiently. The trick is finding those “lucky” subnetworks, dubbed winning lottery tickets”. Although some subnetworks are context-specific and cannot be transferred over to different tasks, they also found that most of the time fine-tuning was not necessary to identify these “lottery ticket” subnetworks. This could become part of the solution to make artificial intelligence and machine learning more accessible to all, and even opens up the possibility for future users to train models on the go, or even on their smartphones.

An AI that helps you summarize the latest in AI

22nd December 2020

MIT Technology Review

Joy Xu

Tl;dr that research paper? The new algorithm from the Allen Institute for Artificial Intelligence (AI2) might help. On November 16, it rolled out the model onto its flagship product, Semantic Scholar, an AI-powered scientific paper search engine. It offers a short, one-sentence long summary under every computer science paper (for now) when users use the search function or go to an author’s page.

While many other research efforts have tackled the task of summarization, this one stands out for the level of compression it can achieve. Semantic Scholar can compress on average to 238 times the size of an average research paper. The next best abstractive method is able to compress scientific papers by an average of “only” 36.5 times. In the testing phase, human reviewers also judged the model’s summaries to be more informative and accurate than previous methods.

AI songs: easy music to our ears, but a struggle to create

7th November 2020

MIT Technology Review

Joy Xu

AI can produce delightfully catchy music, as can be shown by the Australian Team Uncanny Valley’s submission to the AI Song Contest, titled “Beautiful the World”. However, it is still a challenge to be able to communicate with the model, and fine-tuning the track to your preferences. For example, you might want the chorus to be repeated a set number of times - but how do you tell that to AI?

As Carrie Cai, who works at Google Brain and studies human-computer interaction puts it, “AI can sometimes be an assistant, merely a tool. Or AI could be a collaborator, another composer in the room. AI could even level you up, give you superpowers. It could be like composing with Mozart.”

How AI discriminates in speech recognition

28th October 2020

Scientific American

Joy Xu

If you have a non-American accent and attempted to use a speech-recognition software such as Siri or Alexa, perhaps you’ve already had the misfortune of experiencing the algorithm completely misunderstanding what you were saying - sometimes to a ridiculously funny extent. In fact, this is an extremely widespread problem and a depiction of bias, where the software used for voice recognition is skewed towards catering to the upper to middle-class, educated, English-speaking category. 

One of the most likely culprits is the data used for training, which is predominantly from “white”, native speakers of American English. This simple “inconvenience” for some of us can become a deeper issue to others, for example for the visually impaired, who rely on speech recognition to complete everyday tasks.

This article explores the gaps in diversity within datasets and how we can start bridging them, as well as the greatest challenges we might face in doing so.

A.I. creativity is improving fast. This hilarious GPT3-generated film is proof

20th October 2020

Digital Trends


Since its release in June this year, OpenAI's text-generating AI tool GPT-3 has received great attention. It is used to comment on forums, write poems, and even publish articles in The Guardian.

Today, GPT-3 has achieved another feat: it wrote a script — students from Chapman University in the United States used GPT-3 to create a script and produced a new short film: Solicitors.

Check out the script created by GPT-3 and watch the short film (link 1).

However, despite the many high hopes for GPT-3 in this boom, two scientists questioned sharply and reasonably the implementation logic behind it after a series of tests.

On August 22, Gary Marcus - Professor Emeritus at New York University, the founder and CEO of Robust.AI - and Ernest Davis - Professor of Computer Science at NYU - published an article entitled GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about (link 2) on MIT Technology Review.

They told Technology Review that GPT-3 itself does not have 'revolutionary' changes, nor can it truly understand semantics. The key things still depend on human judgment. Amidst the enthusiastic voices, this undoubtedly brings more objective and rational thinking.