I don't write in-house game engines, but I do write in-house devops tools and I assume the reasons are similar. Instead of reaching for things like Jenkins, Docker, Kubernetes, et. al., we developed a unified software solution that ties all of these concerns together in a way that aligns perfectly with our business objectives. Our biggest concern has always been the turnaround time of our software changes. We are at a point of doing multiple builds per day per customer, and pushing further into this realm requires some machine assistance.
For me, having 100% visibility of the entire vertical in code is the most compelling aspect. I can set a breakpoint somewhere in business logic and then step into the low-level if I need to. This sort of approach makes it really easy to spot things that are conceptually tricky. E.g. After stepping into an internal library method you discover that you are using ASCII and not UTF-8 encoding which is resulting in loss of fidelity of persisted business data. If this library method was in some external source you have no control over, it would either be entirely invisible to you, or taunting you with its unchanging gaze. For us, we simply edit the low level problem, push a commit, and its done. Knowing that we can step into the lowest level code encourages us to do so. F11 takes us all the way to the bottom of the rabbit hole. Debugging is a lot easier when you have confidence that you are passing the correct byte sequences into database/os/framework libraries. This also means that stack traces are very useful since you can view the source behind everything.
There is obviously a cost associated with maintaining this solution. We have found that it is substantially more expensive to maintain than just using a set of existing tools. For our case, 1 full-time developer is approximately what it takes to maintain all of this infrastructure. However, there is also the angle of opportunity and efficiencies. With the custom tools we have built, we are now able to cycle our software development process 4-5x faster than before. Our project managers can click a few buttons in a web UI to trigger a build of our software followed by automatic packaging and scheduled release to selected customer environments. This also includes the ability to review all errors (fully-automatic reporting) combined with snapshots of business state at time of stack trace generation. We have the ability to click on a commit hash and see a list of all errors which occurred on that tree up to that point in time. All of this links back into our source control tool. I would say that our devops process is ~95% full-auto at this point. I have not seen anything like what I am describing offered in any public marketplace.
Sometimes if you want something really nice, you are just going to have to build it yourself. Many times, the cost of in-house custom will not be justifiable at face value. You have to look beyond this and consider the higher-order consequences of your decision over longer time frames. Our solution started out very humble and gradually grew into the comprehensive monster that it is today. If we hadn't taken the leap away from Jenkins 3 years ago in favor of implementing the build logic in code, we would have never had the opportunity to incrementally add all of the other amazing stuff that we did.
Sorry to say, but this was the completely wrong approach and I’m amazed the business let you do it.
Some poor bastard is going to have to undo all your mess down the road, I hope you buy them a beer if they ever track you down. I mean, at what point do you make the call that writing your own in-house copy of _DOCKER_ is a smart move? Let alone Jenkins.. The ‘magic’ abilities you describe this platform as having are utterly standard features and are yours as soon as you install e.g gitlab...
Of course, you actually have to spend some time and learn them and how to use them properly, but that time is VASTLY less than learning how to implement them badly and minuscule compared to the amount of time it’s going to take to migrate off your in-house pile..
We push to each of our customers' environments from our centralized administration tools. Each customer environment can vary wildly (on-prem, AWS, Azure, etc) and ultimately just needs a small component installed for communication with our mothership. We do not worry about the actual hardware or hosting, but do have tools that allow for us to inspect the health of whatever host we are running on from our remote perspective.
Longer term, we have considered shipping appliances (physical or VHD) to our customers so that they do not have to take the extra step of configuring a new windows or linux host each time we want to stand up a new product environment.
For me, having 100% visibility of the entire vertical in code is the most compelling aspect. I can set a breakpoint somewhere in business logic and then step into the low-level if I need to. This sort of approach makes it really easy to spot things that are conceptually tricky. E.g. After stepping into an internal library method you discover that you are using ASCII and not UTF-8 encoding which is resulting in loss of fidelity of persisted business data. If this library method was in some external source you have no control over, it would either be entirely invisible to you, or taunting you with its unchanging gaze. For us, we simply edit the low level problem, push a commit, and its done. Knowing that we can step into the lowest level code encourages us to do so. F11 takes us all the way to the bottom of the rabbit hole. Debugging is a lot easier when you have confidence that you are passing the correct byte sequences into database/os/framework libraries. This also means that stack traces are very useful since you can view the source behind everything.
There is obviously a cost associated with maintaining this solution. We have found that it is substantially more expensive to maintain than just using a set of existing tools. For our case, 1 full-time developer is approximately what it takes to maintain all of this infrastructure. However, there is also the angle of opportunity and efficiencies. With the custom tools we have built, we are now able to cycle our software development process 4-5x faster than before. Our project managers can click a few buttons in a web UI to trigger a build of our software followed by automatic packaging and scheduled release to selected customer environments. This also includes the ability to review all errors (fully-automatic reporting) combined with snapshots of business state at time of stack trace generation. We have the ability to click on a commit hash and see a list of all errors which occurred on that tree up to that point in time. All of this links back into our source control tool. I would say that our devops process is ~95% full-auto at this point. I have not seen anything like what I am describing offered in any public marketplace.
Sometimes if you want something really nice, you are just going to have to build it yourself. Many times, the cost of in-house custom will not be justifiable at face value. You have to look beyond this and consider the higher-order consequences of your decision over longer time frames. Our solution started out very humble and gradually grew into the comprehensive monster that it is today. If we hadn't taken the leap away from Jenkins 3 years ago in favor of implementing the build logic in code, we would have never had the opportunity to incrementally add all of the other amazing stuff that we did.