“Our free AI tool can save G42, Microsoft and Facebook millions of dollars” – Dmitry Masyuk, Yandex

CNME Editor Mark Forker spoke to Dmitry Masyuk, Director of Search and Advertising Technologies Business Group, at Yandex, in an effort to find out more about how the company has established itself as one of the most diversified technology companies in the world, the impact its game-changing Yandex YaFSDP is having in terms of optimising the training process of Large Language Models – and its unwavering commitment to the developer community across the Middle East.

Dmitry Masyuk has enjoyed a remarkable career in the field of IT, technology and telecommunications.

He started as a developer at Netcracker, before transitioning away from a developer role and moving into management and IT consulting roles.

Masyuk has worked for Deloitte, McKinsey and A.T. Kearney, and has worked on key projects in Southeast Asia, Europe and the United States.

In 2018, he joined Yandex Taxi, and in 2020, he became the CEO of Yandex Eats. Under his leadership Yandex Eats became one of the first commercially successful food tech services in the world.

In 2023, he assumed responsibility for Search and Advertising at Yandex.

CNME managed to secure an exclusive interview with Masyuk, and he began the conversation by providing a brief overview of the company.

“I would characterise Yandex as one of the most diversified technology companies in the world. We are the market leader in our region in four key domains. The first domain we are focused in, is actually the area in which I manage, and that is the largest business vertical we have in terms of revenue and profit, and that is Informational Services, such as search engine, navigation and advertising services. The second domain is related to transactional services, and again, we are a market leader here in Russia for ride-hailing services and e-commerce. The third domain is content streaming, and the fourth is B2B technology, and the classic example here is cloud services,” said Masyuk.

As Masyuk explains, the company began as a search engine business around 25 years ago, and is one of the few companies in the world that has a greater market share than Google in the search engine space in its home region.

“In terms of search, people here in Russia can freely use Google and Yandex, but we’re bigger than Google here with 65% of the market. That is a testament to us as a technology leader,” said Masyuk.

Since the advent of ChatGPT by Open AI in November 2022, there has been a democratisation of AI across the board. It has become mainstream.

Masyuk described the relatively recent phenomenon of ChatGPT as a ‘renaissance’ in artificial intelligence.

“I would describe what has happened over the last couple of years as a renaissance in artificial intelligence and machine learning through the advent of ChatGPT 3.5 by Open AI. The miracle from my perspective comes from the fact that these models are trained and specifically designed to be able to support a very natural dialogue with the user, and it has undoubtedly democratised AI. Fundamentally, language and text are the most effective and efficient means of transferring information between people, and when I say text and language, I’m really referring to software developers, they use language, and fundamentally they use text to explain to a machine what they want it to do,” said Masyuk.

However, according to Masyuk, the challenge now lies in the fact that the machine learning models that are used have been trained on terabytes of data, which was never done before.

“All these models are essentially trained on a huge volume of existing data on openly available media and forums, but in order to process all this vast data you need thousands of GPUs, and the cost to do this is excessive, you’re talking millions of dollars as a starting point. For the largest and leading technology players then you’re talking in the region of billions of dollars in terms of the pure cost of training models. In addition to the training costs, the duration of the training process is another factor. Typically, modern LLMs are trained for several months, so you’re talking anywhere from 2-5 months of uninterrupted GPUs running, which inevitably just leads to astronomical costs,” said Masyuk.

Yandex, which develops and trains its own LLMs, has created a solution to the problem through its new technology entitled Yandex YaFSDP that can optimise the training process by 26%.

Masyuk explains in greater detail what YaFSDP is, and how it is a gamechanger for the machine learning community from a fiscal point of view.

“Yandex YaFSDP is a technology that drastically speeds up the process of training these LLM models.  One of the most popular LLMs, which is available on an open-source basis, is Llama3, which has been developed by Meta. If you take a 70 billion parameter model, which is a sort of mid-to-large size model, the deployment of our YaFSDP can optimise the training process by 26%. That is hugely significant, now granted, we’re not the only company attempting to optimise the process, but these results are outstanding and unrivaled in the market. As a result, we have seen huge interest from the machine learning community since we outsourced the Yandex YaFSDP, and it’s freely available on GitHub.  In summary, with the major challenge being the cost of training these models, the fact that we can optimise the process by 26% is a gamechanger for companies from an economic perspective,” said Masyuk.

Interestingly, Masyuk also said that despite this potential competitive advantage, he wants to foster an ecosystem in which he can help his rivals get up to speed in order to accelerate the training process on a much bigger scale across the board. Enter open-source.

“As aforementioned earlier in our conversation, when you train the models then you need thousands of GPUs and that’s costly, but during the training process each GPU contains memory, and you need to constantly synchronise the learning process between the GPUs. What we did that gave ourselves an advantage over our market rivals was in relation to the approach that we adopted. We optimised the processes and protocols of communicating between those GPUs, and I think what also differentiates us is our desire in this particular field to open source it. However, despite the fact we’ve stolen a march on our competitors, and we could keep it as our competitive advantage, we actually want others within this ecosystem to be able to help speed up the process of the training models,” said Masyuk.

Masyuk then moved the dial of the conversation towards their relationship with the developer community in the Middle East, and highlighted the intricacies and sensitivities that are at play culturally across the region.

“We do believe that one of the biggest challenges in AI and LLMs specifically is to maintain its relevance for local communities. What we see through the limitation of current models and technology is how hard it is in terms of the models understanding the local intricacies and sensitivities, and this is even true for the large hyper-scalers such as Microsoft. It goes without saying that Western culture is clearly different to Eastern cultures, but specifically what I am talking about is that for a single model it is hard to keep all the historical aspects and local language instore, and it’s hard to be specific about how people think and what they are used to in the region,” said Masyuk.

Masyuk conceded that in terms of scale Yandex is not in a position to compete with tech behemoths like Google or Microsoft, however, he does believe that their ability to provide a very tailored and niche offering to customers is another key differentiator for the company.

“It isn’t easy for us to compete with Microsoft and Google in terms of scale. However, we believe that our models are the best in terms of how specific we can be. For example, we are expanding our models to Kazakhstan, and our models are much, much better at understanding the local specifics and culture. We are seeing that companies within our geographic location are seeing huge opportunities within our models. In the Middle East, there are a number of entities trying to create their own LLMs, generally to train them for specific scenarios, which are not universal, and much more specialised. Yandex YaFSDP is essentially trying to help the local ML communities across the region to create their own LLMs, that are much more relevant to their own local audience and local businesses. We have open-sourced AQLM, which is another technology that allows you to run inference on LLMs on the GPUs that have less memory, and we have created that technology in conjunction with a number of Western Universities. In summary, our fundamental goal is to diversify AI globally to empower local engineering communities to create their own specific solutions,” said Masyuk.

A new concept that has emerged in the LLM space is that of RAG (Retrieval Augmented Generation).

Masyuk believes it is a hugely significant concept and provided some more context around the demand for Retrieval Augmented Generation.

“RAG is another exciting technology dimension, and there has been a lot of discussion on its potential impact. LLMs have amazing power, but it’s also important to highlight that there are several technological limitations within LLMs. It’s not a question of their maturity, but instead how do LLMs fundamentally work? Hallucinations is something that many users have experienced when interacting with LLMs. Essentially, what that means is it fantasizes about something it doesn’t know specifically. The model is trained to approximate the textual data it has seen before, but when you ask the model for something really specific, which it fundamentally can’t store. As I mentioned earlier, the models are trained on terabytes of data, but the output is typically gigabytes of data. Fundamentally, the model doesn’t know all the specifics, but it tries to approximate the reality and it hallucinates,” said Masyuk.

Masyuk added that the second challenge facing developers is the fact that LLMs are a static file.

“Hallacunations is the first challenge, the second challenge is the fact that LLMs are a static file, that you can store on your hard drive. However, if something happened an hour ago it just doesn’t know a thing about it. Those two challenges can be breached with this RAG concept. It’s not a specific technology, it’s just a concept. The unique capabilities of RAG are the combination of the unbelievable power of reasoning capabilities of a LLM and some external source of information. You basically feed something from an external database which can be a search engine. You can create a LLM with RAG, which allows generally trained models to answer specific questions within a particular domain using that external database. This enables a process in which you can feed very specific data into the LLM, which was generalised during the training process,” said Masyuk.

“There are several different factors that differentiate us from our market competition. Everyone talks about the economy of scale, and it would be great to be huge because you have more resources, but as I say to my employees all the time, every zero in terms of the number of employees you have, and every zero in your target audience creates challenges in terms of how quickly you can create and develop things. Yandex is smaller, but that allows us to be quicker. Secondly, we can also tailor our approach to be much more relevant to the markets that we are operating in. For example, what we do better is that when we explore Kazakhstan, that’s a higher priority for us than whatever global player is also operating there, and that’s just natural, it’s not because the global player is bad, it just can’t deliver the boutique hands-on customer-centric approach we can. Recently, for instance, we launched Kazakh-speaking Neuro, an AI-powered search product that brings a completely new experience to a mass audience free of charge.

Masyuk concluded the exchange by reiterating that what really makes them stand out from the crowd is their ability to attract and retain talent at Yandex.

“What we are really trying to do is nourish talent. In 14 out of the last 20 years of the ICPC (International Collegiate Programming Contest) local teams have won the competition, and that is something we are contributing to. We have a lot of initiatives that support young talent, and coding is a universal language, so we promote it globally. Talent is the fundamental factor in our longevity and success,” concluded Masyuk.