How I made my own build pipeline (overview)

[MiTschMR]

As an asset developer myself, I want to have my assets tested against every possible version of Unity and make sure that it works with every build target. If you have made your own scripts or tools, you may know some of these struggles. Testing an asset against multiple versions of Unity requires you to make a project with each version, import it and see if there are any errors. If there are, you have to fix them of course (or simply say that you don't support this version, like the tech stream versions). In my case, I have one editor asset, one for editor and runtime stuff as well as many more scripts that just relieve me in my work with Unity. If I want to see what issues might arise if I bump a project or asset to another version or build target, I thought it would be much easier to just make my own build pipeline that can take the latest commit to the version control system, build it with all Unity versions and build targets that are assigned to the project and see if there are any errors. If there are errors thrown, I can dig specifically look at this build target and/or editor version and fix the issues that arose. 

But how do you approach something like that? I use a version control system (VCS), have my Unity installations somewhere on my computer, my projects as well and want to have as little influence on my workflow as possible. I don't want to hassle with anything just to make this work. So I thought it should be a web application, that exposes a web API for your VCS as well as several dashboard or management options to configure the projects, install new Unity versions and their associated build targets, view the builds etc. 

Until now I have never done any web development and because I use Visual Studio, I wanted to use software that can be developed in there. From what I have heard and read, ASP.NET (Core) is high in trend and uses C#, which was the main reason I went for it. The first step was to get some knowledge about how it works, what needs to be done to configure database access, how to make a web API and more. The tutorials there were extremely helpful and tought me quite a lot. The rest I needed to suck out of my fingers, so to speak. But there are enough guides and Q&A forums out there, so you will always find a solution to a problem. 

Step 1 - Requirements
Obviously, you have to start somewhere. The first step in all of this is to write down the requirements. To me, these were the following: 

• Stores data within a database (in my case I chose MS SQL, because I already know it)
• Must work with PlasticSCM (the VCS I use)
• The Unity versions and build targets can be set for a project
• Unity can be installed from the web app
• Builds are triggered automatically, but can also be triggered manually via the web app
• Reports the result of the build and the errors, if any occur
• If there are multiple builds, they are queued to not overload the server/computer

Step 2 - Preparing
In order for me to be able to have this build pipeline, I thought having a dedicated server/computer is the best option in terms of usability and productivity. I set up a Windows Server 2022 server, installed the database (MS SQL Express, free), the .NET Core runtime tools, Plastic SCM and the Internet Information Services (IIS, Microsoft webserver, similar to Apache/Tomcat or nginx). PlasticSCM has the nice feature of being able to execute scripts when certain events occur, like a checkin on the repo. The whole software suite the app runs on consists of: 

• Dedicated Windows Server 2022 (a VM in my case)
• SQL Server 2019 Express
• ASP.NET and .NET Core 6 Runtime
• Unity Hub 3.0 beta 7 (some features don't work well in the stable 2.x releases)
• PlasticSCM 10.0.16.6280 or higher
• Needed tools for the build targets to work, like the Windows Software Development Kit for the Universal Windows Platform

Step 3 - Build the web app
With the tutorials from earlier, I have gained some knowledge about databases and how to manage data within there. The biggest hurdle to overcome was the relationships between the different tables in the database. You don't want removing a Unity version to remove all builds associated with it, nor do you want to remove Unity installation entries from the database when deleting a project. Here is a rough overview of the tables I use: 

• Projects
• UnityInstallations
• Modules
• BuildTargets
• Builds

Each of them has a class associated with it, that stores the data the right table. For you to better understand, I think it is better to just show a screenshot of the app:

 

This is the project page, where I can manage my projects. As you can see, a project has a name, description, a main version and the versions it should be tested on. What you don't see on the overview page, is which build targets it will test. 

For the build logs, it looks like this: 

And finally for the Unity installations: 

The Add from Recommended fires up the Unity Hub on the machine the web app runs on and shows the ones you see as well when you open the Unity Hub itself and want to install a new version. You can then also select the modules to install which allows building for the chosen build targets. The supported modules to be installed are these here:

• Android
• iOS
• TvOS
• Linux il2cpp
• Linux Mono
• Mac Mono
• Universal Windows Platform
• WebGL
• Windows il2cpp

There are in fact more modules that you can install, like Windows Server or Lumin OS, but Lumin OS has a version cap of Unity 2020.3 LTS and Windows Server starts with 2021.2, at earliest, so I decided to not support these and stick with the ones that are stable and supported for a longer time now, with no near end in sight. 

When doing a build, the workflow for building a project and generating the report is as follows: 

1. VCS recognizes a checkin
2. A trigger on the VCS is fired that executes a script
3. The script sends a web request to the web API
4. The build server/computer downloads the repo
5. Unity is fired up and opens the project in batchmode (silent, no editor window launched)
6. When firing up Unity, a script I have added to the project is called that builds the project after launching everything
7. After building, the same script posts the results to the web API
8. Unity exits

Step 4 - Provide configuration options
The last step in this whole development process was to bring configurable settings to the settings file. I can change the connection string for the database and the connection URL for my VCS. This is in my opinion the absolute mimimum. There may be a need for more options, like support for a different VCS (i.e. Github). 

Summary
I have gained a lot of knowledge regarding web development in general with this project and I am extremely happy to now have my own build pipeline, that takes care of the tedious tasks regarding Unity version and build target testing. I can focus on the developing my assets and games without having to worry about missing anything. If you have any questions, please, just ask me here or in a DM. I am happy with every person showing interest. 

That said, if anyone wants to use it, I have written what I use for it to run, though it can basically run anywhere (.NET Core is not tied to the Windows ecosystem, I can build for any platform), just drop me a message and I will think about a distribution model. 

Thank you for reading this far, have a great day!

Edited January 6, 2022 by MiTschMR

[Vexstorm]

 You sir are a legend, and love what you do. Keep up the amazing work 🙂 

[BunnyViking]

There were definitely words there, im sure they mean something 😄

[MiTschMR]

2 hours ago, BunnyViking said:

There were definitely words there, im sure they mean something 😄

I can explain what they mean if you tell me which ones 😉

[BunnyViking]

4 hours ago, MiTschMR said:

I can explain what they mean if you tell me which ones 😉

Haha no I think most of it is over my head 😄 I got as far as understanding (i think) that it builds projects vs. a bunch of different unity versions.

[Recon]

This is one area, Unity drop the ball on, even though this is considered another  field, web, back end, etc, alot of people learning Unity are lost.  Unity has the new process to have users, be apart of the advisory panel and they reach out more to us users for feed back. 

So alot of users are lost due to it being out of there field and knowledge .. So now that Unity is adding, web base UI elements, such as HTML, J-query, CSS, XML, and so on, this can help, users expand on this knowledge and Unity added P2P, Back end aspects, so I see IT, Web guys making this stuff and its cool to see. 

I have a few very good friends, who I tend to pick there brain in these areas over the years, as I was never a back end guy, IT, etc. 

So, when posting this stuff, it can help out alot of users who are really not understanding, or who may have never used any of this before.  

 

@MiTschMR, I recommend to look into Perforce, its more designed around Game development, than GIT, and you can go pretty crazy with this stuff. 

Not to HI-jack your thread, you could expand on this, idea. Perforce is free up to 5 users. 

I have a forum called VBulletin, many many, security, apps, log in, and stores I created when I ran my community for games for a long time )  and other things, right now making it work with Perforce, in order, to users to report bugs, using simple HTTP request, so they can report from in game. and using a Qnap, to handle my VCS, but for bug reports, and for the game browser, and bug testing I have a Rasp, Pie4, to host the back end.

 

I agree with MiTschMR doing stuff like this as the bare min, VCS is a must...for any kind of work.  

 

 

In the coming years, you will be using alot more web base languages, due to the new UI tool box, UI elements for custom inspector scripts.. This will help fix alot of the horrible UX/UI in the asset store.... for sure.  its all web base...

Edited January 6, 2022 by Recon

[MiTschMR]

I agree that Git is not the perfect solution when it comes to VCS within game development, but because Unity bought Plastic (which will be the replacement for collaborate in the very near future), it is the one that will be used most soon. Perforce may be good, I don't know, haven't used that. I am happy with Plastic as it is now and don't intend to switch anytime soon. I can look into it and maybe add support for it, but only if there is any desire to make this app public. 

I know that you are one of (if not the) most experienced user(s) in the community and it means a lot to me to have you respond here 🙂. 

That said, if anyone wants to use it, I have written what I use for it to run, though it can basically run anywhere (.NET Core is not tied to the Windows ecosystem, I can build for any platform), just drop me a message and I will think about a distribution model. 

[Recon]

On 1/6/2022 at 9:23 AM, MiTschMR said:

I agree that Git is not the perfect solution when it comes to VCS within game development, but because Unity bought Plastic (which will be the replacement for collaborate in the very near future), it is the one that will be used most soon. Perforce may be good, I don't know, haven't used that. I am happy with Plastic as it is now and don't intend to switch anytime soon. I can look into it and maybe add support for it, but only if there is any desire to make this app public. 

I know that you are one of (if not the) most experienced user(s) in the community and it means a lot to me to have you respond here 🙂. 

That said, if anyone wants to use it, I have written what I use for it to run, though it can basically run anywhere (.NET Core is not tied to the Windows ecosystem, I can build for any platform), just drop me a message and I will think about a distribution model. 

 

Anyone reading this should look at what you did as inspiration, that anything is possible even if you never done it before....

 

 

Makes total sense:)   looking forward to what else you are going with this.  This for sure would be super useful, to any Unity, or even non Unity developer.:)   

This can help save users alot of time and keep things organized.  (Using VCS, is a must for sure, and yes GIT is not a perfect solution, but my point since you did this, with the work you did, was to when you can take a look at Perforce, its alot more open, and has alot more features you could leverage and it also supports GIT, so GIT users, can still leverage GIT by using Perforce as well.:)  Your able to get ,  use what you like, and still get the best out of both worlds. 

They did this as they knew alot of users, where also GIT users.:)  Which I use as well, for some companies that used GIT, so I was able to leverage Perforce still but still be able to work with companies who used GIT. 

anyways, just tossing this out, incase you where not aware.  ( I still use GIT, for some things even though I retired.)    This is fantastic for GIT users.

 

Keep it up, making backend, web base apps and so forth, you can do some amazing tools for sure.      

 

Nice work, Mit!   Pretty neat stuff, and its your first time in anything web base?  very cool.  You should be proud of it, some people when they first use web base anything or backend, really struggle, and aren't able to make anything .  So this is very cool what you did.  

 

 

 

https://www.perforce.com/products/helix4git#:~:text=Combining your Perforce %2B Git code,Helix Core changelists for auditability.

 

Edited January 8, 2022 by Recon