First discussion of the evening was about how there is now an ongoing effort to improve the lives of cows with VR…but that was a little off topic so moving on!
This month we gathered to contemplate web technologies, and how they can or are applied to pipelines and pipeline development. The web is a large umbrella but is a constantly and rapidly evolving industry that continues to churn out new and innovative solutions to some of the toughest problems in software development and deployment.
We quickly acknowledged how relevant this topic is to all of the growing efforts and conversations around the pipeline as a service idea - and our seemingly inevitable migration to “the cloud”. Thinking back to this summer’s Siggraph, there are a number of cloud efforts going on that really go all-in on modern web technology stacks. Conduit from BlueSky Entertainment is one that came up in our discussions - especially since they still seem to intend on making it open-sourced, so it’s something that we’re all interested to play with.
An interesting topic that comes up when we start talking about we is authentication. In a private cloud or local data center it’s not as problematic, but any large web app or web-based SaaS product needs to have strong authentication and security practices these days. Thanks to the web industry as a whole this is a largely solved problem, or at least there are existing products and paradigms that can be integrated or implemented. One of the main challenges really is just in the amount of extra effort and development time it takes to incorporate these things into the software being produced. The additional layer that our industry adds is with the protection of content: where can the content be stored physically? who is allowed access and from where? where do these policies get defined and how easily can they be monitored and customized per-show or per asset?
These are some big questions, and we didn’t really have immediate answers to them, but the other side of this discussion to consider is how and where web tech can or should be applied. For those on the production side, web tooling for documentation and production tracking has been the norm for a long time now. Most studios have a number of internal tools or even paid software services that they use which are web based already. Albeit the way that we write and deploy these applications may be dated and/or not considered “modern”, but I think that speaks to the side of web that can be easy to miss nowadays: simplicity. When we ignore complex authentication and identity management; web-scale architectures; and service discovery / service meshes that have all largely come about in the last decade - creating and hosting a web site or service is actually very easy. In Python with a framework like Django one can spin up a web app in hours or minutes, but the bar seems to have raised in recent years, extending the time to market.
It’s probably (?) safe to assume that our DCC applications will continue to run on the desktop, so what does the future look like for those toolchains? Maintaining a Qt ui alongside a web interface often seems not worth the effort, but a common web service backend does allow one to create interfaces in both places. Another solution to this is a web view in Qt, which allows you to simply render your web app in a application frame. This can work well for certain cases, but often one of the issues is when the tool needs to interact with the underlying Qt application. A web based ui in Maya that wants to touch the maya api requires a strange set of data translation and connection layers that, again, just seems not to be worth it. This is especially true when you know that the tool is going to be tied to that DCC anyway. It was noted that we see this happening on the Shotgun side - that even with a web application they still spend significant effort developing desktop applications and native DCC integrations in order to deliver the best experience and workflow to artists.
Perhaps one of the greatest offering that comes out of the web is a strong ecosystem of standards. From HTTP to Protocol Buffers, OpenAPI and everything in between - any time you can build to or with a set of standards, there’s opportunity to share and take advantage of previously shared technology. For example, a REST API described through the OpenAPI standard allows client and server libraries as well as documentation become extremely easy to generate. Similarly, gRPC solves the same problem but with a protocol optimized for inter-service communication - reducing the time and headache needed on message serialization and deserialization so that you can focus on business logic.
Containers are another example of the power of standards. With such huge adoption of both, the Open Container Initiative was formed and already the community is converging, making it increasingly easy to leverage related technology without any kind of lock in or future support concerns. I think we all agree that when you are using a library, framework or technology and can find Stack Overflow answers to even your strangest questions it makes the development and support experience so much better.
While thinking about the existing areas of pipeline development that we already leverage web technologies this area came up. Similar to production tracking, but on the development side most of us use a hosted git service and probably some form of ticketing system to keep track of issues and feature requests.
One of the interesting discussion points here was actually around managing tickets, and who has access to view or submit them. A great ticketing/help desk system enables users at any level to easily get support for any aspect of their daily workflows. This is a great thing - for sure, but like any great system that quickly gains adoption sometimes the sheer scale of usage is not properly planned for. Most of us working at studios with established helpdesk systems have more or less dedicated staff to triage and handle incoming tickets. We kind of all agreed that perhaps it’s a bit of a necessary evil. Inevitably you get tickets that are not well written or inappropriate, but having systems in place so that each and every employee feels well supported is extremely valuable. On the development side, the disconnect between tickets and priorities is also important. The alternative is often that each developer creates their own priorities, or whoever is the loudest gets the best support which we believe is not an effective way to build and manage software over time (especially with respect to long term goals).
Of course, when pairing web tech with pipelines we cannot help but think about cloud. An interesting question that we pondered was: what does it mean?
What exactly does it mean to move a pipeline to the cloud? Is it simply putting our servers into someone else’s data center? Personally, I believe that the idea of the cloud is really one of decentralization - a location-aware but migratory data set. As a group we brought up the idea of fully integrated remote artists; seamlessly scaling infrastructure beyond a single data center; ephemeral or on-demand resources; as well as redundancy and uptime.
Realistically, and as we can already see amongst the many studios is that the current and future application of web technology may be different for every studio and every business model.