Gaia ML & The AI GameDev Toolkit

Procedural Worlds is proud to contribute Gaia ML to the AI GameDev Toolkit in partnership with Intel.

This toolkit makes it easy to add Artificial Intelligence (AI) Inference to your Unity project, so that you can add features like Object Detection, Style Transfer and other ML inference. 

The toolkit can further serve as an entry point for learning more about AI development in Unity and creating your own Inference features using engines like Intel OpenVINO or Unity Barracuda.

The software comes with a special free edition of Gaia called Gaia ML to explore Inference in beautiful worlds, or it can also be run independently. To learn more about using the toolkit please watch this introductory quickstart guide video, and / or read on further below.
 

Prerequisites

• Unity 2020.3 or higher (alpha and beta versions are not supported)
• Built-In Render pipeline (URP and HDRP support coming soon)
• You need to have the "Allow unsafe code" setting activated in your project's player setting for the inferences to work correctly
• If you intend to use Barracuda, you need to have the Barracuda package installed.

Gaia Editions

The AI Game Development Toolkit is compatible with the following Gaia Editions:

• Gaia ML - Free
• Gaia 2021 - version 3.2.0 or higher - Subscription required
• Gaia Pro 2021 - version 3.2.0 or higher - Subscription required

No matter the edition, Gaia will recognize that the toolkit is installed in the project and display the integration features in its UI accordingly.

Usage across the different Gaia Editions remains the same, however you can create richer, and much larger environments using Gaia Pro 2021.

You can upgrade your Gaia ML project to unlock the the full power of Gaia Pro 2021 at any time by purchasing a subscription, and installing Gaia Pro 2021 into your existing project. Learn more about our subscriptions via our offers page. 

Support

• Check out the QuickStart Video Guide to get started.
• You can ask questions about Gaia ML and the AI GameDev Toolkit in the AI GameDev Toolkit forum.
• To learn more about Gaia ML / Gaia 2021 and Gaia Pro 2021 by browsing the library and the forums.
• You can subscribe to one of the paid editions of Gaia to get deeper support in the Gaia forums.
• For one on one support or custom modifications then please choose a studio subscription and contact us to clarify scope.
• Click on the link to learn more about our special Pro Subscription offer.

Installation

The toolkit is available both in a version bundled with Gaia ML as an .unitypackage and as a standalone version on this github repository by intel.

To download the bundled version (with Gaia ML) you need to sign up as a member of Canopy (it is free), and then go to the Gaia ML download page to download it.

After downloading the file, you can either

• Double Click the file to open it with an Unity Editor that is currently open
• Drag and drop the file from your file system in the project hierarchy in your Unity Editor window
• Import the package via Assets > Import Package > Custom Package...

If you have Gaia 2021 or Gaia Pro 2021, then these will also recognize the toolkit automatically when it is installed in the project. If you do not own Gaia you can install the bundled version that comes with Gaia ML, if you do own Gaia already you can install that Gaia version first from Canopy, the Unity Asset Store or PW Store and then install the standalone Version of the AI Game Development toolkit.

Once the package is installed, make sure that you have the "Allow Unsafe Code" setting active in your project's player settings:



If you intend to use Unity Barracuda, please see the installation instructions for that and install the Barracuda package. The required files to use Intel Open Vino inferences are already part of the toolkit and do not require additional installations.

Directory Structure

The toolkit installs itself in the Folder "AIGamedevToolkit". Below that folder you can find the following directory structure:

• Inference Engines: Contains code related to the supported inference engines OpenVino and Barracuda. The respective "Inference Features" folders hold the code for the implementation of a specific inference within that engine.
• Scriptable Objects: Contains configurable scriptable objects that contain settings for a certain inference, but are also responsible for drawing the UI for those parameters, and adding / removing the inference to the scene. See "Understanding the Inference Scriptable Objects" section below for more information.

Usage via the Gaia Integration

Once the toolkit is installed, Gaia will display an extra tab in the Gaia Manager standard workflow:



The inferences that you can select here are as follows:

COCO_YOLOX: Enables the detection of the COCO (Common Objects in Context) dataset with the YOLOX (You-only-look-once) detection model. This will identify things rendered by your game camera, and will draw a border around it. Note that the COCO dataset is based on real-life photographs and therefore does not detect all objects from a default Gaia scene out of the box.



Style Transfer Barracuda: A style transfer using Unity's Barracuda engine. This takes the input from the game camera and transforms it into a style resembling a painting. This inference requires the Unity Barracuda package to be installed in the project.

Style Transfer OpenVino: Another implementation for Style Transfer, using the OpenVino Engine.

With all these inferences, you have options that you can unfold. Here you can find parameters that control the behavior of the inference and also let you use different model data:



When the "Apply Now" button at the bottom of the screen is pressed, the selected inferences are set up on an Inference Manager Object in the scene. Note that the settings made under the "Options..." are being transferred to the Inference Manager as well. These settings are in fact linked, so changing them in the Gaia Manager will change them in the Inference Manager and vice versa.




When running the scene, the selected inferences will be started / processed by the inference manager in the scene. If an inference will require additional components or objects in the scene, these will be set up as well.

Please Note: The inference will be set up using the current camera in the scene - should you change the camera, e.g. by setting up a different Player Type in Gaia, you will need to re-apply the inferences.

Standalone Usage
If you do not have Gaia installed, it is still possible to add inferences in an automated fashion like from the Gaia Manager. To do so please select

Window > AI Gamedev Toolkit > Add Inference Features

to bring up a Window with the same functionality as the Gaia Manager tab:



Just as with the integration, you can select the desired inferences here and click the "Apply now" button to have them added to the scene.

Preparing for Build

The inferences should run out of the box in the unity editor. If you decide to try them out in a standalone build, you need to perform an extra step for the COCO_YOLOX and Style Transfer OpenVINO Inferences. These two inferences require the selected model data to be present in the build. The AI Game Development Toolkit makes use of the "Streaming Assets" special folder of Unity for that. To make sure the required model files are in the build, you can click the "Copy Models to StreamingAssets" button found in the options for these Inferences:

If you check the Streaming Assets folder in your project afterwards, you will find that the required model files have been copied into the folder. Unity will automatically include the files in this folder into the build while building the project.

Understanding the Inference Scriptable Objects

The Inference Features are controlled by the Inference Scriptable Objects that you can find in the AIGamedevToolkit\ScriptableObjects\InferenceFeatures
folder:



If you inspect one of those scriptable objects, you will find that it has the exact same settings that you would find in the Gaia Manager or the Standalone "Add Inference" window:



These settings are in fact stored in the scriptable object and when you change a setting in the Gaia Manager, it is directly changed on this scriptable object.
Neither the Gaia Manager nor any other places that show these options do know how to set up the inference in the scene, or do draw the UI for displaying these fields themselves. That code comes out of the scriptable object. This approach has the following implications / advantages:

Automatic Setting Saving:

Any parameters that you change on the scriptable objects will be saved and be available the next time you open the editor

Automatic Setting Syncing:

You never need to worry about keeping the settings in sync, as there is one common "master" that holds the settings for an inference (and that is the scriptable object in question)

Creation of Configuration Variants:

It is easy to create a configuration variant for an inference by creating a new instance of the scriptable object. You can do so from the "Create" Menu in your project hierarchy:

Or by duplicating an already existing scriptable object.



You can configure this new scriptable object the way you like it, and this Inference Variant will then appear in the Gaia Manager or in the "Add Inference..." Window automatically:

Developing your own Inferences:

You can use the existing scriptable objects as a blueprint for how to develop your own inferences without having to write a single line of code that deals with storing or reading settings. The next section below has more on this.

Integrating Inferences into your own applications:

If you want to integrate the inference setup into your own toolkit that you use for building your game, or if you are another asset store publisher that wants to integrate these inferences, you can very easily do so - all you need to do is call a function that draws the available inferences in the UI where you want them to appear. See the section "Integrating a list of Inferences" below for more.

Extending the System with your own Inferences

You can use the scriptable objects infrastructure of your project as blueprints / as a framework for developing your own managed inferences. You need to worry less about saving and storing settings and how to display them inside an unity editor, but can rather concentrate on the implementation of the inference yourself. By following the examples of the AI Gamedev Toolkit, you can give the users of your inference the same advantages of easy configuration, scene setup and configuration variants as well.

To develop your own inference, create a new scriptable object class that inherits from the "InferenceFeature" class. This class has various overridable methods that you can implement to give your scriptable object the same properties as the existing examples from the AI Game Dev elopment Toolkit, most importantly:

ApplyToScene() - will be called when the apply to scene button is pressed on the UI and your inference will be added to the scene. The inference manager will be added and configured with your inference automatically, but you would need to create any additional objects you would require for your inference in here.

RemoveFromScene() - here you should remove all objects that you created in apply to scene again, in case the user wants to remove your inference.

DrawUI() - called by the Editor to draw the custom configuration fields of your inference. You will need to wrap any editor code in the #if UNITY_EDITOR define, then you can draw UI fields as you would in any unity editor. 

Initialize() - called for initialization at start on runtime

Inference() - the actual implementation of the inference during runtime. Detailing this part would exceed the scope of the manual, but you can find out more about developing your own inferences by looking at the implementation of the examples and following the tutorials mentioned on the Intel page for the toolkit. 

Integrating a list of Inferences in your own Unity application

If you have your own toolkit for your game that you are developing, or you want to integrate the list of inferences into your own asset store asset like Gaia (ML) does, this is very easy to do, just put the following code snippet wherever your custom editor or editor window gets enabled (usually the "OnEnable" event would be a good place to put this):
 

#if AIGAMEDEV
    AIGamedevToolkit.InferenceFeatureListEditor.RefreshFeatureList();
#endif

This will make sure the list of inferences is being refreshed when your editor is being opened / selected. Then put this snippet into the place where you want your list of available inferences to be drawn (usually OnGUI / OnInspectorGUI😞
 

#if AIGAMEDEV
    AIGamedevToolkit.InferenceFeatureListEditor.DisplayFeatureList();
#endif

And that is all there is to do. The AIGAMEDEV scripting define will be created automatically when the AI gamedev toolkit is being installed, so if the toolkit is not there, it will not prevent your existing code from compiling.