Gaia and git...

[Disto]

Hello,
I'm just trying to use git with on my project with gaia and i've encountered some issues with the long files names generated for the user sessions.
And if I remember well, same applies when using multiscene setup on multi-tile mode.
Anyone would have faced this issues ?

[MiTschMR]

Can you please elaborate on the issues you get? Do they occur when you pull from the repo or already when checking in?

[Disto]

normally it was just on checking before commit. I'm retrying now on a new repo

[Disto]

Gaia User Data/Sessions/GS-20220118 - 201905/Session Operations/20220118-222429 Clearing - Trees Details GOs SpawnExt Probes, CurrentTerrainOnly, AnySource.asset.meta
Gaia User Data/Sessions/GS-20220118 - 201905/Session Operations/20220118-210614 Clearing - Trees Details GOs SpawnExt Probes, CurrentTerrainOnly, AnySource.asset.meta
Gaia User Data/Sessions/GS-20220118 - 201905/Session Operations/20220118-205405 Clearing - Trees Details GOs SpawnExt Probes, CurrentTerrainOnly, AnySource.asset.meta
Gaia User Data/Sessions/GS-20220118 - 201905/Session Operations/20220118-205019 Clearing - Trees Details GOs SpawnExt Probes, CurrentTerrainOnly, AnySource.asset.meta
fatal: pathspec 'Gaia User Data/Sessions/GS-20220118 - 201905/Session Operations/20220118-222429 Clearing - Trees Details GOs SpawnExt Probes, CurrentTerrainOnly, AnySource.asset.meta' did not match any files

That's strange, the file seems to be here, and it is visible in unity

[Disto]

same here for Gaia source files:
fatal: pathspec 'Procedural Worlds/Gaia/Lighting/Content Resources/Post Processing Profiles/Post Processing - Builtin/Gaia Default Post Processing Profiles/Gaia Default Default Post Processing.asset.meta' did not match any files

Edited January 18, 2022 by Disto

[Disto]

this is was the issue, when renaming folders for shorter name in unity, this fix the issues (for the sources files).

ex: 
Procedural Worlds/Gaia/Lighting/Content Resources/Post Processing Profiles/Post Processing - Builtin/Gaia Default Post Processing Profiles/
=>
Procedural Worlds/Gaia/Lighting/Content Resources/Post Processing Profiles/Builtin/Gaia Default PPP/

It stay the generated users files (that I guess, we couldn't rename them ?)

Edited January 19, 2022 by Disto

[Peter]

It should be possible to rename those session operation files as well, but it would be a bit tiresome, because Gaia will generate new ones over time.
IIf your structure has more folders before the assets folder you could consider to change that so that the paths become shorter in general. It should also be possible to enable long path names on the git repository:
https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

[Adam]

@Peter we have solved git integration with our .gitignore and .gitattributes setup. We should write an article on this and put it in the library as it is a common problem for people who are new to it. @Manny has something that could be used with only a little bit of work.

[Disto]

Thanks for your answers. Yes it could be great to have more info about that subject.
For now i've added session operations folder to the .gitignore.

[Brian S]

On 1/19/2022 at 12:18 AM, Adam said:

@Peter we have solved git integration with our .gitignore and .gitattributes setup. We should write an article on this and put it in the library as it is a common problem for people who are new to it. @Manny has something that could be used with only a little bit of work.

@Adam I would love to see this .gitignore and .gitattributes setup.  I am having issues that I believe are related.  Cloned projects almost completely work for me other than the Gaia terrains.  Anything published that I can look at?

[Disto]

@Adam & @Peter Any news about that ?

[Ash]

@Adam & @Peter was the article published somewhere? I don't see it in the Library

[yo_mama_]

Would be really helpful to get the .gitignore and .gitattributes setup. 

[HikmetOzgurler]

Is there any .gitignore for Gaia and Gena, so i don't have to push large files?

[Bryan]

I will check with the team on this. 
Git can be used but I found it very difficult due to size limitations. 
I personally have been using Plastic SCM and its been working great. 

 

[Amasu]

Hello There,

 

Glad I found this topic on the forum, as you may guess I am experiencing the same issue, and in some way it's nice to know I am not alone in my misery 😛 

The error in question: 

 

Quote

Unknown error occurred while loading 'Assets/Gaia User Data/Sessions/GS-20221104 - 204430/Terrain Data/Terrain_0_0-20221110 - 001057.asset'.
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

 

But first thing first, I solved the issue of space by using git-LFS and .gitignore. The logic is to avoid storing asset files in general in LF dedicated storage, rather than plain in the repo using '.gitattributes' file to explicitely exclude the desired file types from the sources git repo. Also I took a well maintained 'gitignore' from github, and added the folders that were created by Unity Asset manager (except User Data folders).

 

This speeds up A LOT the pulling and pushing operations 🙂 

 

Now about the error we get. To reproduce (in my case):

- Create a Unity project.

- Install PW dependencies: Gaia PRO + HDRP Time Of Day.

- Populate your Scene with a terrain created using Gaia Manager.

- Push your project to the git repository.

- On a different PC (or on a different path on your filesystem): Clone your repository content to a fresh folder, then import PW Gaia + HDRP ToD.

 

I have noticed two things:

1 - git client changes the EOL (End of Line) string for 'terrain_..._.asset' files created by Gaia.
 

Quote

The file will have its original line endings in your working directory
warning: CRLF will be replaced by LF in Assets/Gaia User Data/Sessions/GS-20221104 - 204430/Terrain Data/Terrain_0_0-20221110 - 001057.asset.
The file will have its original line endings in your working directory
warning: CRLF will be replaced by LF in Assets/Gaia User Data/Sessions/GS-20221104 - 204430/Terrain Heightmap Backup/Stamper Backup/Terrain_0_0-20221104 - 204752@fd23b518377876744a9bfd16e00cc2cf.asset.

   Could this alone cause the corruption of the mentioned files? Especially if you pull on a different OS than the OS used to create the original project?

 

2 - When opening one of the terrain data files (located in: 'Assets\Gaia User Data\Sessions\GS-... - blabla.asset), I notice that in addition to the 'assetPath', the full absolute path is also defined there.
 

Would it be possible that Unity/Gaia uses this full path to retrieve the terrain data files, and then throws this error when this fullpath does not exist (which is the case if you pull your project to a different path on your filesystem, or a different PC).

 

I could reproduce this on Windows, and on macOS (Metal).

 

Any help would be appreciated, thank you 🙂 

.gitattributes .gitignore

[Peter]

Hi @Amasu

 

Quote

 Could this alone cause the corruption of the mentioned files? Especially if you pull on a different OS than the OS used to create the original project?

 

When it comes to unity asset files, there are files that are "YAML" (Yet another markup language) files that can be read and edited with a text editor, but there are also binary files like the terrain data asset files. With the latter I could very well imagine that if the repository swaps out line ending characters within the binary data, it will corrupt that file because Unity most likely expects those line ending characters to be in a certain format. If you can, I would therefore try to switch off the automatic line ending changes from git.

 

Quote

Would it be possible that Unity/Gaia uses this full path to retrieve the terrain data files, and then throws this error when this fullpath does not exist

Gaia itself should not do a lookup of the terrain data file. The terrain component in the scene has a reference stored to the terrain data file via the so-called "Meta-Id" that can be found in the .meta file of each asset:



So normally it should not matter in which directory the file is, as long as it is within the "Assets" folder of the project and the .meta file stays intact so it still carries the same ID.
Gaia does not directly load the terrain data file itself, but rather will access it from the terrain component via "Terrain.TerrainData"

[Amasu]

Hi @Peter,

 

Thanks a lot for your quick reaction. 

I am going to swith off the core.autocrlf property and give it another hit.

core.autocrlf

core.autocrlf

[Steen]

So basically, we cant use github it seems. How can this be? Its kind of ridiculous that we cant get a version of gaia pro 2021 that is specifically made for being lean and mean without all the fluff when I just want to download the essentials and make custom stuff.

I have now spent hours trying to get this to work to no avail.

Plastic SCM is not a very good option as you need tp pay after 5 GB, procedural worlds folder alone is 4.5 GB and isnt getting reduced by the .gitignore.

There really should be a tutorial or best practices for this somewhere on this website. I wont even start working with this application before I know I can version control it.

I am starting a new project so please tell me how I can, with an empty project importat procedural worlds and get it pushed to github without getting all these errors, I got these whie trying all sorts of different approaches.

sometimes I got:
fatal: protocol error: bad line length 8192

sometimes:

unable to rewind rpc post data - try increasing http.postBuffer (increasing the buffer doesnt work)

I also sometimes got this:

error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500

[Bryan]

The best practice depends on what you are using Gaia for. 
If you are making the terrains and have your solution for lighting, water, etc. 
Then I would use a different project to make your terrains and texture them. 
Then you would need to go into the new project and import the terrain data, terrain layers, and textures. 

Another way of handling it would be to create everything in the scene. Then remove Gaia altogether. 
As long as you are not using any of the Gaia systems. 

Procedural worlds products still utilize the normal Unity system. 
I don't use Git hub as it is public, and two I find plastic way easier to work with. 

Plastic charges you $5 USD for every 10 gb I believe, but it's very cheap to add more developers and increase the project storage. 

 

[aldaluin]

I haven't had any trouble using git, but I only VC my own work, and even then only scripts and prefabs.  Where is the need to version GB of other people's work, that can be reimported any time from an Asset Store package? 

I use a git server on my own network instead of Github so I don't have to worry about public code. 

[Peter]

@Steen 
 

On 4/1/2023 at 7:51 PM, Steen said:

So basically, we cant use github it seems. How can this be?

Please note that Gaia does not contain any "special files" or technology that would prevent you from pushing it to Git. If you are getting errors while pushing, this has to do with the configuration of the repository and / or your Git client. Any error that you encounter while pushing to git you would get as well if the file in question was not created by Gaia, but by another application.
If you, for example, create a file with Microsoft Word that exceeds the file size limit on your git repository, you will get errors while pushing that file. That does not mean that Word or Git are broken, or that Word is a bad application that does not work with Git. It means that the repository is not configured to accept that file due to its size. 
We do use Git / GitHub for all our own projects and the development of Gaia itself without issues.

 

On 4/1/2023 at 7:51 PM, Steen said:

I am starting a new project so please tell me how I can, with an empty project importat procedural worlds and get it pushed to github without getting all these errors, I got these whie trying all sorts of different approaches.

There are mostly three things that you need to be aware of when it comes to Gaia and Git:

1. Gaia contains and produces files that may exceed the file size limit. Therefore you need to configure your repository with LFS to accept and work with those files.

2. Please note that you do not need to push the Gaia installation folder if you do not want to - it is usually enough to store the Gaia User Data folder that contains all the data that you created while using Gaia. This is essentially your work output, if you so will. (see the comment by alduin above which refers to this as well).
However when you do so, note that another team member accessing the same repo would also need to install Gaia themselves. If you want to be able to pull the whole project from the repository and be able to run it / work on it without having to do additional installation, you would need to push the install folder as well.

3. The majority of the Gaia installation size comes out of the following folders:


If you do not intend to use any of the art assets that come with Gaia, you can delete the two asset folders, if you do not intend to use any of the stamps, the stamps folder can be deleted as well.