What Are Foundation Models, and How Can They Help Implement AI at Scale?

While organizations around the globe have long gone on an AI investment spree, the number of artificial intelligence projects that make it from prototypes to production still fluctuates at around 53%.

Experts believe this often happens due to lacking tech skills, human resources, and tools to scale isolated AI proof of concepts (PoCs) across other use cases. And, of course, the presumably high cost of training sets AI models apart for different tasks.

Foundation models — i.e. large machine learning models that have been trained to use vast volumes of unlabeled data under the guidance of skilled AI consultants — may be the ultimate answer to the daunting problems of AI scalability and costs.

Your company could use such models as a starting point to enhance or automate various tasks, from converting paper-based documents into editable text files to uncovering customer sentiment in social media reviews. And you could build on your AI excellence from there, adapting foundation models for future tasks and use cases.

What Are Foundation Models, and How Could They Help Your Company Excel at AI?

Unless you’ve been living under a rock, you’ve heard about OpenAI’s ChatGPT. This language model program has absorbed tremendous volumes of conversational text using supervised learning and, at the fine-tuning stage, the reinforcement learning from human feedback (RLHF) approaches.

This generative AI solution can analyze input data against 175 billion parameters and profoundly understand written language. It can also answer questions, summarize and translate text, produce articles on a given topic, and write code, among other tasks. All you need is to provide ChatGPT with the right prompts.

OpenAI’s groundbreaking product is just one example of foundation models that can transform AI application development as we know it.

Foundation models disrupt AI development as we know it. Instead of training multiple models for separate use cases, you can now leverage a pre-trained AI solution to enhance or fully automate tasks across multiple departments and job functions.

With foundation AI models like ChatGPT, companies no longer have to train algorithms from scratch for every task they want to enhance or automate. Instead, you only need to select a foundation model that best fits your use case — and fine-tune its performance for a specific objective you’d like to achieve.

Foundation models are perfect for industries where training data can be too hard or expensive to acquire. These industries include healthcare, life science, biotechnology, and manufacturing, to name a few.

What Types of Foundation AI models Are There?

Several types of foundation AI models are commonly used in business applications.

• Semi-supervised learning models are trained on a dataset that contains a mixture of labeled and unlabeled data. The goal is to use the labeled data to improve a model’s performance on unlabeled data. AI experts turn to semi-supervised learning when training data is difficult to obtain or would cost a company an arm and a leg. This, for instance, may happen in medical settings where various healthcare IT regulations are enacted. Some common examples of semi-supervised models include pre-trained text document and web content classification algorithms.
• Unsupervised learning models are fully trained on unlabeled datasets. They discover patterns in training data or structure it on their own. Such models, among other things, can segment information into clusters based on the parameters they’ve uncovered in a training dataset. Machine Learning (ML) engineers turn to auto-encoders, K-Means, hierarchical clustering, and other techniques to create unsupervised machine learning solutions and improve their accuracy.
• Reinforcement learning models interact with their environment without specific training. When achieving a desired outcome — i.e. making a prediction developers have hoped for — the models get rewarded. On the contrary, reinforcement learning models are penalized when they make wrongful assumptions. The approach allows AI algorithms to make more complex decisions than their supervised and semi-supervised counterparts. An example of reinforcement learning in action would be autonomous vehicles or game-playing AI programs, such as AlphaGo.
• Generative AI models produce new data similar to the data they’ve been trained on. This data may include text, images, audio clips, and videos. The ChatGPT solution mentioned in the previous section belongs to this category of foundation AI models. Other examples of generative AI include the DALL-E 2 tool, which creates images based on descriptions written in natural language, and the Synthesia.io video platform, which uses text-based inputs to produce video content.
• Transfer learning models can solve tasks other than they’ve been trained on. For instance, computer vision engineers may leverage pre-trained image classification algorithms for object detection. The may also harness existing NLP solutions for more knowledge-intensive tasks, such as customer sentiment analysis. Some popular pre-trained machine learning solutions include OpenCV, a computer vision library containing robust models for object classification and image detection, and Hugging Face’s Transformers library offerings, such as generative pre-trained transformer (GPT) — i.e. a rich language model whose third generation (GPT-3) powers the ChatGPT service.
• Meta-learning models, unlike their task-orientated equivalents, literally learn to learn. Instead of devouring data to solve a specific problem, such models develop general strategies for problem-solving. This way, meta-learning solutions can easily adapt to new challenges while using their resources, such as memory and computing power, more efficiently. ML experts tap into meta-learning when training data is scarce, or a company lacks definitive plans regarding AI implementation in business. TensorFlow, PyTorch, and other open-source machine learning libraries and frameworks offer tools that allow developers to explore meta-learning techniques. Finally, cloud computing providers, such as Google, help ML experts and newbies train custom machine learning models using AutoML.

Depending on the specific application and the type of data you have, one foundation model may be more appropriate than another. Your company is free to choose between an open-source solution, which needs a bit of tweaking, or a ready-to-use third-party product , provided that it meets your business targets.

Top 3 Reasons to Leverage Foundation AI Models for Your Next Project

Compared to standalone, task-oriented machine learning models, foundation models help create reliable AI solutions faster and cheaper, with less data involved and minimal fine-tuning. And that’s not to mention that, being trained on more data than a single organization could ever obtain, foundation models display high accuracy from day one.

Below you will find a rundown of foundation AI models’ advantages.

• Foundation models will help you implement AI faster, cheaper, and with fewer resources involved. Creating and deploying an AI solution requires considerable time and resources. For every new application, you need a separate well-labeled data set. And if you don’t have it, you’ll need a team of data experts to find, cleanse, and label that information. According to Dakshi Agrawal, CTO of IBM AI, foundation models help cut down on data labeling requirements by 10–200 times, depending on a given use case, which translates into significant cost savings. On the business side, you should also consider rising cloud computing expenses. Google, for instance, spent millions to acquire DeepMind. And while your AI project may not be half as ambitious, you could easily spend $300,000 in cloud server costs alone to get your AI app up and running. Another reason to use foundation models, such as generative AI solutions, is the opportunity to quickly prototype and test different concepts without investing heavily in R&D.
• You can reuse foundation AI models to create different applications. As their name implies, AI foundation models can serve as a basis for multiple AI applications. Think about driving a car. Once you’ve got a driver’s license, you don’t need to pass the exam every time you buy another vehicle. Similarly, you can use a smaller amount of labeled data to train a general-purpose foundation model that summarizes texts to process domain-specific content. And foundation models possess “emergence” capabilities, too, which means that a model, once trained, may either learn to solve problems it was not supposed to address or glean unexpected insights from training data.
• Foundation AI models help achieve your company’s sustainability goals. Training one large machine learning model can have the same environmental footprint as running five cars over their lifetime. Such a heavy carbon footprint stands in sharp contrast with the fact that 66% of businesses are increasing the efficiency of their energy use while 49% of other businesses are developing new climate-friendly services and products. With foundation AI models, you can train intelligent algorithms faster and utilize computing resources wisely — not least thanks to these models’ architecture that takes advantage of hardware parallelism, which allows you to execute several tasks simultaneously.

Deemed “the future of AI,” foundation models lower the threshold for tapping into AI and could potentially end the failed AI proof of concept cycle by helping businesses scale models across other use cases and company wide.

But with every opportunity comes a challenge.

Things to Consider When Using Foundation Models

The only glaring drawback of foundation AI models is the difficulty in explaining some of their operations.

Large foundation models can use so much training data and have so many deep layers that it’s sometimes hard to determine how algorithms arrive at their conclusions.

The black-box nature of foundation models leaves a backdoor for cybercriminals, too. Hackers can launch data poisoning attacks and introduce AI bias, further exacerbating AI’s ethical issues.

Technology companies should join forces with governments to set up an infrastructure for public AI projects to avoid controversy  surrounding the use of AI foundation models. AI vendors should also disclose what datasets they use and how they train their models.

As Percy Liang, Stanford HAI faculty and computer science professor, opined during his recent interview with Venture Beat, “We’re very much in the early days, so the professional norms for AI are underdeveloped. It’s therefore imperative that we, as a community, act now to ensure that this technology is developed and deployed in an ethically and socially responsible fashion.”

What Does It Take to Start Using Foundation Models in Your Organization

As someone who’s spent the last ten years helping companies implement AI systems, the Itrex team is witnessing a shift in AI.

Systems that execute specific tasks in a single domain give way to broad AI that learns more generally and works across industries and use cases. Foundation models, trained on large, unlabeled datasets and fine-tuned for various applications, are driving this transformation.

If your company is ready to leapfrog your competitors and get ROI from your AI systems faster, here’s a high-level strategy for implementing foundation models.

1. Collect and pre-process your data. The first step involves collecting and pre-processing the data you will feed to a foundation AI model. The quality and diversity of this data are critical for ensuring that the fine-tuned model is accurate and robust.
2. Choose a foundation model. Many pre-trained AI foundation models are available on the market. Some popular solutions include BERT, GPT, and ResNet, among others. It’s important to choose the right foundation model based on the task you want to solve and the type of data you have.
3. Tweak the model in line with your business objectives. Once your foundation model and data are ready, you can adjust the model’s parameters to your specific task. One way to achieve this goal is transfer learning, where you use the pre-trained weights of the foundation model as a starting point and adjust them based on your training data.
4. Evaluate the model. After fine-tuning, it’s crucial to determine whether the model works well and if further adjustment is necessary. To assess the foundation model’s performance, you can use standard metrics such as accuracy, precision, recall, and F1 score.
5. Deploy your AI solution. Once you’re satisfied with the performance of your fine-tuned model, you can deploy it in a production environment. Several options for deploying AI models include cloud-based platforms, on-premise servers, or edge devices.

It’s important to remember that implementing AI foundation models requires technical expertise and access to specialized hardware and software. Therefore, it may be helpful to partner with a specialized AI vendor or consult with a team of AI experts to ensure that the process is done effectively.

Drop us a line to discuss your AI needs! We’ll assess your company’s AI readiness, audit your data and prepare it for algorithmic analysis, and choose the right foundation model for getting started with artificial intelligence!

This article was originally published on the Itrex website.