How Creating an AI Proof of Concept Can Help You Minimize AI Development and Adoption Risks

itrex-logo-black
ITRex
  • Date Published
  • Categories Blog
  • Reading Time 17-Minute Read

An AI proof of concept will help you minimize business risks, get investors on board, and identify potential challenges before investing large sums.

Our client lost only a quarter of the budget they dedicated to an artificial intelligence (AI) project because they chose to start with a proof-of-concept (PoC). The PoC allowed them to test their idea and fail fast with limited spending. To avoid wasting time and effort, always ask your AI solutions consultant for a proof of concept — especially if your company is just testing the AI waters.

This article explains what an AI PoC is and elaborates on the five steps that will guide you through your first PoC, together with the challenges that you might encounter along the way. It also presents AI PoC examples from our portfolio. And you will find a happy ending of the example depicted in the opening paragraph.

What Is an AI PoC, and When Is It Essential for Your Project’s Success?

An AI PoC is a prototype or a demonstration of a proposed AI solution designed to test whether the solution is feasible and likely to be successful. The purpose of creating an AI PoC is to validate the concept, assess the proposed solution’s potential benefits, and identify any potential challenges or limitations.

An AI PoC typically involves building a small-scale version of the proposed AI solution and testing it in a controlled environment to see how it performs and whether it meets the desired objectives. The results of an AI PoC can then be used to inform further the development and implementation of the solution.

Compared to ordinary software PoCs, an AI PoC may involve more complex considerations, such as the ability of the AI solution to learn and adapt over time, and the potential ethical implications of the solution, such as an AI bias. The technology stack for AI PoC projects is different, too.

Machine Learning Algorithms

These algorithms allow an AI system to learn from structured data and make predictions or decisions based on such learning. There are many different types of machine learning algorithms, including supervised learning algorithms, unsupervised learning algorithms, and reinforcement learning algorithms.

Neural Networks

These computational models are inspired by the structure and function of the human brain. Neural networks can process and analyze large amounts of unstructured data. They can be trained to perform various tasks, such as image recognition, natural language processing, scenario modeling, and prediction.

Robotics

This technology can be used to build physical systems capable of autonomous operation and decision-making. Robotics solutions incorporate sensors, actuators, and other hardware components that allow engineers to build a robot that can interact with its environment and perform tasks.

Cloud Computing

Cloud computing platforms like Microsoft Azure, Google Cloud, and AWS provide the computing power, storage resources, and pre-configured services needed to support the development and testing of AI PoCs. These platforms can also host and deploy AI solutions once they have been developed and tested.

Creating an AI PoC involves gathering and preparing data, building and training machine learning models, and testing and evaluating the performance of the AI system. The time it takes to create an AI PoC can vary widely depending on several factors, including the complexity of the proposed AI solution, the resources and expertise available for the PoC, and the specific objectives of the PoC. Some AI PoCs can be developed in just a few days or weeks, while others may take several months or even longer to complete.

When Is There Absolutely No Way Around an AI PoC?

It’s essential to start your project with an AI PoC in the following scenarios.

  • Your project relies on an innovative idea that hasn’t yet been tested—something that has been studied at the business level, but hasn’t been attempted technically. Neither you nor your tech vendor are sure whether it can be implemented.
  • If you need to demonstrate to stakeholders, investors, and others the feasibility of your idea within a limited timeframe. An AI PoC will do the job better than an interactive prototype or something similar.

Are There Situations Where an AI PoC Is a Waste of Time?

Even though an AI PoC is beneficial in most cases, there are a few exceptions. If your project falls under one of the following categories, a AI PoC can be overkill.

  • If your idea and approach are exceptionally well documented from functional and technical perspectives. This is rare.
  • If the solution you want to develop is standard and resembles common practices in the field. We already know that this is feasible and possible from a technical perspective.
  • If you want to build software that your front-end and back-end developers understand, often because they have already worked on something identical.

What Benefits Can You Gain From an AI PoC?

Using an AI PoC brings the following benefits.

  • Identifying potential challenges before committing more resources to an endeavor. An AI PoC allows you to “fail fast, fail better.” If a team encounters challenges it can’t overcome, all stakeholders have time to regroup or maybe change the hypothesis to reach the same goals through other methods.
  • Minimizing business risks, as you test innovative ideas in small steps instead of diving into a long-term project.
  • Improving data collection practices.
  • Getting investors and other stakeholders on board.
  • Saving time and resources. An AI PoC might uncover business or process-related issues and give you time to fix everything before starting a full-scale project.
  • Building expertise and creating knowledge owners who will mentor another team member on similar projects in the future.
  • Testing the tech stack on a smaller scale to understand its suitability for the selected business case.

Examples From Our Portfolio Where an AI PoC Saved the Day

Here are a few AI PoC examples from the ITRex portfolio that will help you appreciate the PoC approach even more.

Realizing That ML Alone Is Not the Answer

A large cargo logistics company was performing 10,000–15,000 shipments per day, and every shipment was accompanied by bills of lading and invoices to cover the operations. Employees were exhausted by the manual handling of all the documentation. The company wanted to build an ML-powered solution that would use optical character recognition (OCR) to process scanned documents and identify different fields.

The client believed that machine learning was the best solution, so we proceeded with an AI PoC to test this assumption. Soon we realized that the documents were formatted differently, and the labels used for fields varied significantly— for instance, the Load ID field alone had 8 aliases. As a result, the ML model kept growing. It became slow and inefficient. Our team decided to accompany this model with a dynamic algorithm (e.g. a dictionary where different field labels were hard-coded). This modification improved the solution’s performance significantly and saved the client time and money.

If the client had decided to skip the AI PoC, they would’ve wasted seven months just to realize that their initial idea of a pure ML-based model was not the optimal solution. With an AI PoC, they came to this conclusion in merely two months. After the successful completion of this AI PoC, we built an MVP that could handle four types of documents, taking over around 25% of the manual processing load.

Taken Aback by Meta’s Restrictions on Data Usage

A client in the entertainment industry wanted to build an AI-driven analytical platform for independent musical performers. The solution was supposed to crawl social media, including Facebook and Instagram, to gather data. It would process all this information to gauge people’s sentiments toward the artists. Musicians could sign up with the platform and receive feedback on what social media behavior would be the most beneficial for their success.

We proceeded with an AI PoC to test the idea. After just two weeks, we realized that it was simply impossible to gather data from Facebook and Instagram to use it for the purpose described above. Typically, some of the data was retrievable through Graph API. Because of this, as well as a verified business account in Meta, we assumed we would gain access to the required information. However, the client couldn’t supply us with a verified business account, and the data from Graph API alone was not sufficient for this solution to work.

Had the client decided to skip the PoC, they would’ve wasted around $20,000 on the discovery project. This would have included a detailed description of the solution and the estimation of the development costs. But as they chose to start with an AI PoC, they spent only around $5,000 before figuring out that the proposed solution was impossible to execute due to data access restrictions enforced by Meta.

A Five-step Guide for Your First AI PoC

Here are five steps that you can follow to successfully go through your AI PoC. We also list the challenges associated with each step.

Step 1: Identify Which Issue(s) You Want to Tackle with AI

It is essential to specify what, exactly, you want to accomplish by implementing an AI PoC. The selected use case needs to be of high value and represent something that you can address best with this technology. If you have doubts, a good place to start is to look into what others in your field are using AI solutions for. Another way to go is to investigate the problems that your business is facing and compare it against the potential of AI.

After you’ve accumulated a list of opportunities, you can ask the following questions to determine which ones are the best fit for your project at the moment.

  • Is the problem you intend to solve specific enough? Can you evaluate the results to determine success?
  • Have you already attempted to solve this problem with other technologies?
  • Do you have the talent and the funding to support this project through to its end? If there is no suitable in-house talent, can you hire an external dedicated team?
  • How will the problem impact your business? Is this effect significant enough to warrant your efforts?
  • Will you be able to sell this project to your executives? Is your organization ready to take on such a project?
  • Does your firm already have a data strategy? If so, how will it align with this project?
  • What are the potential risks and limitations of using AI to tackle this problem?

Associated Challenge

  • Selecting a use case that doesn’t add much value or doesn’t use the full potential of AI. Artificial intelligence is an expensive technology, and choosing an insignificant case will mean you spend more than you will receive. Check our article on how much it costs to implement AI to gain a better understanding of the expenses.

Step 2: Prepare the Data

Now that you have your problem clearly defined, it’s time to aggregate and prepare the training data for the AI algorithms. You can do that by:

  • checking which data is available for use within your company
  • generating semi-synthetic data using specific ready-made applications or your own solution
  • purchasing datasets from reliable providers
  • using open-source data
  • hiring people to scrape the data that will fit your purpose.

You don’t have to limit yourself to one source. You can use a combination of several options listed above.

Turn to data scientists to run the initial data screening. They will perform the following tasks.

  • Structure the data
  • Clean it by eliminating noise
  • Add any missing data points, in case of tabular data
  • Perform feature engineering (i.e. adding and deleting data fields)
  • Apply manipulations, such as combining or filtering data

Data scientists can advise you on how to gather additional data or how to narrow the AI PoC’s scope so that you can achieve the desired results with the existing datasets.

When the data is ready for usage, split it into three sets:

  • A training set, which the model will use to learn.
  • A validation set to validate the model and iterate on training.
  • A testing set that will evaluate the algorithm’s performance.

Associated Challenges

  • The training data is not representative of the entire population. In this case, algorithms might perform well in common cases but will deliver poor results on rare occurrences. For example, a healthcare ML model that analyzes X-rays might excel at detecting common disorders, such as effusion, but will struggle to spot rare disorders, such as a hernia.
  • Class imbalance, when the number of cases representing one class is significantly larger than another, with a ratio of 99.9% to 0.1%.
  • Incorrect labeling, such as mixing classes (e.g. labeling a bike as a car).
  • High noise in the training dataset.
  • Difficulty achieving pure class separability. This happens when some data in the training set can’t be correctly classified under a particular class.

Step 3: Architect and Build or Buy the Solution

You are probably wondering whether you should build the model yourself or whether you can procure an existing solution. Here’s when it makes sense to create an AI model from the ground up.

  • Your solution is innovative and doesn’t conform to an existing standard.
  • Ready-made solutions are costly to customize.
  • The closest off-the-shelf model is overkill, and it does much more than you actually need.

Consider procuring a ready-made model if the costs of buying and customizing the model are less than building it from the ground up.

If you decide to build the AI algorithm from scratch, you will have more control over its accuracy. It will take longer to complete the task, but it will be tailored to your business problem and your internal processes. You will not need to make changes to your system to accommodate external software.

Regarding the infrastructure for algorithm training and implementation, you can rely on the cloud instead of using local resources. There are four parameters that you can consider when deciding what suits you best.

  1. Security. If your data is very sensitive when it comes to security, then you better keep everything on-premises.
  2. Workload. If the processing load is rather heavy, opt for the cloud.
  3. Costs. Evaluate what will cost you more: acquiring the resources locally or paying for cloud usage over time.
  4. Accessibility. If you will only use the solution locally, you can depend on your in-house servers. If it needs to be accessible from different geographical locations, then it’s worth considering the cloud.

Every approach has its benefits and drawbacks. If you are operating in the healthcare sector, we have those clearly explained in the cloud computing in healthcare post on our blog. Otherwise, feel free to reach out to our AI experts to choose the best technology stack for algorithm training.

Associated Challenges

  • Lack of proper training. This will cause issues, such as poor model generalizability, which means that the model won’t make accurate predictions on data that it has not seen in training. Coming back to X-ray image analysis in the medical sector, an algorithm might successfully analyze high-quality images captured by state-of-the-art scans but still fare poorly when applied to scans generated by older machines.
  • Integration with existing systems, some of which may be outdated or powered by proprietary technologies.
  • Failing to come up with a suitable model architecture (e.g. being unable to pick the right ML model for the problem at hand).
  • The selected architecture’s capability can’t match the model’s requirements.
  • The input data is volatile, which means the model has to be frequently retrained.
  • Using more resources than your model requires to perform its tasks. There is no need to invest in a powerful server to run a simple model.

Step 4: Assess the AI PoC’s Potential to Generate Value

This step is about evaluating whether the AI PoC can live up to expectations. There are several ways to perform the assessment.

  • Go back to your key performance indicators (KPIs) and test the solution against them. These factors may include accuracy, customer satisfaction, speed, flexibility, fairness and safety.
  • Collect data on how your system operated before the AI PoC deployment. This can include the time spent on a particular manual task and the number of errors. Next, you should use the information to evaluate the impact of the PoC.
  • Compare the solution’s performance to other products that are regarded as either the benchmark for this type of problem or the wider industry. For instance, a benchmark for image classification-related issues would be a model that delivers accurate results on popular datasets, such as ImageNet.
  • Gather user feedback either through focus groups or online surveys to gauge the levels of satisfaction and determine what is missing.
  • Conduct a cost-benefit analysis to understand the financial impact of this solution on the organization.

Associated Challenges

  • Making a mistake in your assessment. It can be a simple math mistake during calculations, or an error related to estimating the business potential.

Step 5: Iterate on the AI PoC for Better Results, or Scale It Up

If the results that you received in the previous step are not up to par, you might consider modifying the solution and iterating the whole process. You can make changes to the ML algorithm and measure the performance with each adjustment. You can also experiment with different hardware components or alternative cloud service models.

If you are content with the AI PoC’s performance, you can work on scaling it in different directions. Here are a few examples.

  • Apply the PoC to other business cases. Look for other applications of this new solution within your business. For instance, if you are testing AI as one application of predictive maintenance, you can try to apply it to other related scenarios.
  • Scale the infrastructure. Review the technology used to run this software. Can you dedicate more processing power or more data storage capacity? Such modifications will enable you to use more data, decrease latency, and maybe deliver results in real-time. It will also minimize the possibility of bottlenecks in the future.
  • Optimize the PoC solution. Even though you may have gotten reasonable results in the previous step, it might be worth looking for ways to improve accuracy. You can keep training your algorithms using new data or more accurately labeled data. Or you can even experiment with implementing tweaks and changes to achieve better results.

If you decide to adopt AI company-wide following the PoC phase, you can find helpful tips in our guide on how to implement AI in your organization.

Associated Challenges

  • The architecture was not carefully considered. The solution might work well with 10,000 users but crash when the audience reaches 100,000.
  • The model contains bugs that will manifest themselves when you attempt to scale the AI solution
  • Applying the model to other business cases, other than the ones it was intended for. For instance, a solution that is meant to assemble a garden wheelbarrow can’t be applied to assembling trucks, as it might build a large garden wheelbarrow with a motor.

To Conclude

When it comes to implementing AI, start small and stay manageable. Make sure you have a clear business case with defined objectives and metrics to measure success. And always consider creating an AI PoC, except for the cases presented at the beginning of this article. This will help you identify any potential obstacles before you go fully in and make a large financial investment in a solution that may not live up to expectations.

Do you want to implement AI in your organization, but aren’t sure if your business idea is feasible? Get in touch! Our team will help you conduct a PoC to test your idea in practice.


Originally published at https://itrexgroup.com on January 9, 2023.