Discover our latest curated articles on natural language processing.
Google trained a trillion-parameter AI language model
27th January 2021
In machine learning, training on data historical data fine-tunes the parameters (essentially the variables) of the algorithm. In practice, the number of parameters seems to correlate with the sophistication of a model - the higher number of parameters, the more complex tasks a model is able to perform. Researchers also noticed that simpler, more straightforward models trained on large datasets and with a high number of parameters tend to outperform more complex architectures with fewer parameters.
Using this knowledge, Google endeavoured to train a 1.6 trillion parameter model, the largest of its size to date. This improves upon their previous model with a speedup factor of 4. Since training on so much data is both time and resource-intensive, researchers used a technique that uses “only a subset of a model’s weights, or the parameters that transform input data within the model” (aka ‘sparsely activated’). However, it is important to note they did not take into account the bias that would result on training from such an extensive dataset - in real life, such models also can amplify biases present in the training data such as stereotypes and discrimination. Nevertheless, this can still benefit our understanding of the link between training data, parameters and overall model performance for the future.
BERT is a heavyweight, powerful and complex algorithm in natural language processing, and can be used in applications ranging from text generation to online chatbots (it is also used in Google’s search engine). However, running this state-of-the-art algorithm is no easy feat - it requires massive computational power, something that ordinary users cannot easily get hold of.
Recently, however, Jonathan Frankle and his colleagues at MIT have discovered subnetworks in BERT that could be repurposed for easier natural language processing tasks. This links to his “lottery ticket hypothesis”: essentially, “hidden within massive neural networks, leaner subnetworks can complete the same task more efficiently. The trick is finding those “lucky” subnetworks, dubbed winning lottery tickets”. Although some subnetworks are context-specific and cannot be transferred over to different tasks, they also found that most of the time fine-tuning was not necessary to identify these “lottery ticket” subnetworks. This could become part of the solution to make artificial intelligence and machine learning more accessible to all, and even opens up the possibility for future users to train models on the go, or even on their smartphones.
Tl;dr that research paper? The new algorithm from the Allen Institute for Artificial Intelligence (AI2) might help. On November 16, it rolled out the model onto its flagship product, Semantic Scholar, an AI-powered scientific paper search engine. It offers a short, one-sentence long summary under every computer science paper (for now) when users use the search function or go to an author’s page.
While many other research efforts have tackled the task of summarization, this one stands out for the level of compression it can achieve. Semantic Scholar can compress on average to 238 times the size of an average research paper. The next best abstractive method is able to compress scientific papers by an average of “only” 36.5 times. In the testing phase, human reviewers also judged the model’s summaries to be more informative and accurate than previous methods.
AI songs: easy music to our ears, but a struggle to create
7th November 2020
MIT Technology Review
AI can produce delightfully catchy music, as can be shown by the Australian Team Uncanny Valley’s submission to the AI Song Contest, titled “Beautiful the World”. However, it is still a challenge to be able to communicate with the model, and fine-tuning the track to your preferences. For example, you might want the chorus to be repeated a set number of times - but how do you tell that to AI?
As Carrie Cai, who works at Google Brain and studies human-computer interaction puts it, “AI can sometimes be an assistant, merely a tool. Or AI could be a collaborator, another composer in the room. AI could even level you up, give you superpowers. It could be like composing with Mozart.”
If you have a non-American accent and attempted to use a speech-recognition software such as Siri or Alexa, perhaps you’ve already had the misfortune of experiencing the algorithm completely misunderstanding what you were saying - sometimes to a ridiculously funny extent. In fact, this is an extremely widespread problem and a depiction of bias, where the software used for voice recognition is skewed towards catering to the upper to middle-class, educated, English-speaking category.
One of the most likely culprits is the data used for training, which is predominantly from “white”, native speakers of American English. This simple “inconvenience” for some of us can become a deeper issue to others, for example for the visually impaired, who rely on speech recognition to complete everyday tasks.
This article explores the gaps in diversity within datasets and how we can start bridging them, as well as the greatest challenges we might face in doing so.
A.I. creativity is improving fast. This hilarious GPT3-generated film is proof
20th October 2020
Since its release in June this year, OpenAI's text-generating AI tool GPT-3 has received great attention. It is used to comment on forums, write poems, and even publish articles in The Guardian.
Today, GPT-3 has achieved another feat: it wrote a script — students from Chapman University in the United States used GPT-3 to create a script and produced a new short film: Solicitors.
Check out the script created by GPT-3 and watch the short film (link 1).
However, despite the many high hopes for GPT-3 in this boom, two scientists questioned sharply and reasonably the implementation logic behind it after a series of tests.
On August 22, Gary Marcus - Professor Emeritus at New York University, the founder and CEO of Robust.AI - and Ernest Davis - Professor of Computer Science at NYU - published an article entitled GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about (link 2) on MIT Technology Review.
They told Technology Review that GPT-3 itself does not have 'revolutionary' changes, nor can it truly understand semantics. The key things still depend on human judgment. Amidst the enthusiastic voices, this undoubtedly brings more objective and rational thinking.