Small

Blendo Games

• Blendo Games was founded in 2010 by Brendon Chung, a programmer, artist, and game designer.
• While Mr. Chung is the primary employee of Blendo Games, the company has also collaborated with local area developers, musicians, and designers in Los Angeles.
• An independent gaming company, Blendo Games is located in Los Angeles and develops games for PC, Mac, and Linux systems. It can be contacted via accounts on Discord, Twitter, YouTube, Twitch, Facebook, and via email.
• The company has won the IndieCade Grand Jury, IGF Seumas McNally Grand prize, and the IGF Excellence in Design. Titles at the studio include Skin Deep, Flotilla 1& 2, Quadrilateral Cowboy, Thirty Flights of Loving, Atom Zombie Smasher, Gravity Bone, and Air Forte, among others.
• Blendo Games is a privately held gaming company for which an official statement of annual revenue is unavailable. Estimates of annual revenue from company database sites range from $62,200 to$199,000 per annum.
• Publicly available information regarding the use of DevOps at Blendo Games thus far is primarily focused on how development operations have been adapted. As a privately held company, financial information is not readily available, and information regarding the percentage of operating margins accounted for by DevOps at Blendo Games was not sourced.
• Blendo Games are still in the early stages of applying DevOps processes to game development. Mr. Chung stated in the Blendo Games blog that he was inspired by a talk given by Seth Coster of Butterscotch Shenanigans on their DevOps processes, to create a few tools to automate selected manual tasks at Blendo.
• For uploading to other sites, Blendo games has created the Steam Uploader, a graphic user interface (GUI) to upload builds to Steam, and the Itch Uploader to upload builds to itch.io. A Changelog Bot was also created to send Tortoise SVN messages to Discord.
•  Streaming tasks are automated with the Stream Suite which allows Blendo Games to post to Twitter and Discord, update dates on Twitch and operate a countdown timer during the stream pre-roll.
• Blendo Games created a Workshop Uploader during the development of the Quadrilateral Cowboy game, which has been re purposed for other games and to upload Steam Workshop items.
• A TGA viewer was created to view TGA image files, the FTP Uploader is the file uploader for the Blendo Games static site containing older blog posts, and finally a Tax Expenser was created that converts PDF transaction statements into a spreadsheet format.

Butterscotch Shenanigans

• Butterscotch Shenanigans was founded in 2012 as an independent game development studio. The company is located in Saint Louis, Missouri and has five employees.
• Butterscotch Shenanigans game portfolio includes the Quadropus Rampage, Flop Rocket, Roid Rage. Levelhead, Towelfight, and Crashlands games.
• The company is led by Seth Coster, the CEO and game programmer, who is supported by Sam Coster the CCO, node wiggler and marketing strategist, and Adam Coster, CTO and Web Slinger.
• Butterscotch Shenanigans is a privately held company and financial disclosures are not publicly available. According to Dun and Bradstreet, the company's annual revenue is estimated at $35,000.
• DevOps at Butterscotch Shenanigans
• Butterscotch Shenanigans began its DevOps process after having to address numerous bugs resulting from an untidy development cycle during the development of the game Crashlands in 2016.
• Butterscotch Shenanigans CEO Seth Coster, stated that the application of the three pillars of DevOps at Butterscotch, workflow management, amplified feedback, and continuous learning, resulted in a clockwork process for game development.
• In terms of workflow management, the team at Butterscotch built automation tools to deploy builds to QA testers, cutting delivery time from a week to one day with the added gain of error elimination. Other processes that were automated include writing patch notes and importing art assets,
• The task management system at Trello was used to visualize the workflow during the development of the game Levelhead for both the software development and business administration processes.
• Butterscotch developed tools and systems to amplify the feedback loop and ensure that feedback flowed directly opposite to the workflow visualization. The reporting procedure was automated by using Trello cards to capture the feedback and a Trello API which allowed users to convert information regarding an issue into a Trello card.
• To encourage continuous learning, Butterscotch worked to create a culture of "psychological safety," wherein employees are encouraged to experiment with efficiency methods.
• In terms of QA testing, Butterscotch hired QA testers before fully implementing DevOps. Prior to implementation, the workflow loop for testing lasted for approximately one week. When the studio implemented the GamePipe, a process derived from DevOps, the QA testers received builds automatically, which could be completed once per day. The QA team now came in once per day instead of once per week.
•  
  Source
   
• A video of Mr. Coster discussing game development at Butterscotch Shenanigans can be accessed at this link.

Medium

Brightrock Games

• Located in Brighton, East Sussex, United Kingdom, Brightrock Games is a privately held independent game studio that was founded in 2015. Brightrock Games maintains accounts on Twitter, Facebook, YouTube, and Discord.
• Creators of "genre-twisting 3D simulation and management" games, Brightrock Games is led by Josh Bishop, the managing director and designer, and has a core team of 13 employees. In addition to its core employees, Brightrock Games also collaborates with an adviser, animator, and writer/designer for its website.
•  Games from Brightrock Games include the currently published War for the Overworld, and the upcoming Project: Aftercare. There is also the Heart of Gold, Under Games, and My Pet Dungeon expansions of the War for the Overworld game.
• Brightrock Games is a privately held company and an official statement of income is unavailable. The company is a recipient of funding from the UK Games Fund as well as from the Creative Europe Programme of the European Union.  
• Despite being a commercial success, the initial launch of the flagship game, War for the Overworld, was subject to numerous issues and generated negative initial feedback.
•  
  Source
   
• Brightrock Games responded to the negative early reviews with patches, fixes, and new features. The company also implemented DevOps practices addressing the failures, with an emphasis on the amplifying feedback and continuous learning pillars of DevOps.
• DevOps at Brightrock Games
• Brightrock assumed control of the development process for the game from the publisher when the company was formed in 2015.
• Among the priorities was the creation of a development culture for product development. The team developed a method for creating development insights, and to encourage discussion and collaboration.
• Tools were made available for all persons involved in the project to express their viewpoints in a structure that was equal for all team members.
• The goal was to promote learning and sharing during the postmortem of a project.
• These activities culminated in a leveling up day, where the team collaborated to facilitate the creation the aforementioned structure, by proving tools for participants to express their feedback, in an environment that allowed for sharing information and learning. All employees were given the opportunity to have a better understanding of the development process.

Large

Riot Games

• Founded in 2006 by Brandon Neck and Marc Merrill, Riot Games is headquartered in Los Angeles, California. The company employs over 3,000 employees at more than 20 offices globally.
• Riot Games is a subsidiary of Tencent Holdings and has been certified as a Great Place to Work, and has been featured in many publications regarding positive workplaces.
• Games created by Riot Games include Ruined King, Legends of Runeterra, League of Legends, Teamfight Tactics, and Valorant.
• The company was awarded the Entrepreneur of the Year Prize from Ernst and Young in 2011, received the Outstanding Live Graphic Design Award from the National Academy of Television, Arts and Science, Sports Emmy Awards in 2018, and a Special Award from the British Academy of Film and Television Arts in 2017.
• Annual revenue for Riot Games is estimated at over $500 million per annum.
• DevOps at Riot Games
• Riot Games is a user of DevOps practices, and used packaging services heavily. In February 2020, the company ran over 14,500 containers in Riot-operated regions.
• Workflows were created to "deploy, monitor, and operate their services," and developers weren allowed to become creative in solving issues that cropped up.
•  Backend services were built by teams that ensured continuous support and triage was available when needed. These backend services for Riot Games were organized in shards for historical, scaling, and legal streamlining.
• Overtime this led to strain on the QA and load testing environments, and discrete instances of tracking down misconfigurations, and old dependencies led to a cumulative cost impact to the time that could have been spent on creating player value.
• Similar difficulties were experienced with non-Riot shards in addition to problems in on boarding distinct micro services. The incident rate for non-Riot operated shards increased due to incompatible versions of micro services and other crosscutting problems.
• The company attempted to address the problems with new league deployment tooling, increased processes, and metadata, however it was a focus on individual services to products that proved a successful attempt.
• Riot Games created an environment containing the metadata required to "fully specify, deploy, configure, run, and operate a set of distributed micro services collectively representing a product and is holistically and immutably versioned." This process was successfully tested with the launch of Legends of Runeterra.
•  The process was declarative, high level, and operator friendly. It had a shared language for services and environment, which was QA friendly when applied to the entire environment. Data contained in runbooks was elevated to the forefront of the service specification allowing partner regions to reduce the time to onboard new services, and to be informed of changes derived from updating the micro services.
• The impact to Riot Games from these activities was an improvement in operational metrics and player experience.

Electronic Arts

• Electronic Arts was founded in 1982, and is a developer of games, content, and online services for consoles, personal computers, and mobile devices.
• The company is headquartered in Redwood City, California, and in 2019 employed over 9,800 persons.
• Electronic Arts has been recognized as a top company for women, the best place to work for LGBTQ equality, as one of the world's most attractive employers in South Kora, and as one of the world's most admired companies by Fortune.
• For fiscal year 2020, net revenue was US $5.537 billion and the company had upwards of 2.68 billion players.
•  Popular titles from Electronic Arts include Plants vs. Zombies, Need for Speed, Madden NFL, FIFA Battlefield, Dragon Age, and EA Sports.
• DevOps at Electronic Arts
• Electronic Arts has been involved in DevOps processes since 2006.
• In its first iteration with DevOps, the company focused on transitioning to "meaningful CI/CD." The company began writing its own code engine, and began the process to automation-as-code. These changes led to improvements in scaling, quality, and reproducibiity, however the initial system was complex and not user-friendly.
• The second iteration of DevOps at Electronic Arts began in 2006 and focused on bridging the gap among teams, virtualization was favored over the use of physical blades, resulting in a boost to consistency and impacting on cultural changes in the organization.
• Electronic Arts also transitioned to the use of Jenkins instead of its custom tool, a move that reduced problems with the custom engine, simplified hiring as developers came into the organizations already possessing experience with the framework, and increased efforts in automation-as-code. This system worked well, however the unavailability of pipelines for Jenkins impeded wider success.
• The current iteration of DevOps at Electronic Arts began in 2015 with an embracing of open source solutions and cloud computing. The company began an engagement with CloudBees which added enterprise support and scalability while reducing the developer burden.
• With a player base of 2.68 billion, within hours of release a game from Electronic Arts would have been rigorously exercised far in excess of what internal QA processes can accommodate. Electronic Arts sees the benefits of the application of DevOps as imparting a competitive advantage to the company.

Financial Insights

Scaling DevOps with the Platform Model

• The DevOps evolution model encompasses five stages, namely normalization, standardization, expansion, automated infrastructure delivery, and self-service. Successful DevOps evolution in an organization is dependent on commensurate structural changes to the organization.
•  Progression to self-service and security integration from normalization requires that individuals be given the autonomy to work outside of the team without having to procure manual approval.
• In organizations with a single or a few products, end-to-end delivery by product teams would be the optimal model, however for organizations with multiple products and services, the platform model, helps to reduce production overheads.
• A platform model is defined as "...a foundation of self-service APIs, tools, services, knowledge and support which are arranged as a compelling internal product. Autonomous delivery teams can make use of the platform to deliver product features at a higher pace, with reduced coordination."
• The benefits to be derived from application of the platform model are increased efficiencies among teams, improved governance impacting on the management of audit, compliance, and cost, reduced context switching, and continual infrastructure upgrades.
•  Sixty-three percent of organizations use internal platforms, and 31% of developers within organizations use internal platforms. Firms with a high use of internal platforms are also ranked high in terms of placement on the DevOps evolution model, with 48% of such firms also recording a high use of internal platforms.

Talent Cost

• The most recruited job on LinkedIn is DevOps engineer, despite the contention of engineers that DevOps is a shift in culture, not a single job.
•  Compensation averages for a DevOps Engineer changes regularly in reflection of the changing skills requirements overall, as well as for open-source projects and different platforms.
• There is significant variance in how the job is defined. The team topology, project road map, software quality, and company maturity all impact on how the job is defined.
• In a 2019 DevOps Skill Report from the DevOps Institute, 39% of respondents were recruiting for a DevOps Engineer/Manager, 29% for a Software Engineer, 22% for a DevOps Consultant, 18% for Test Engineers, 17% for Automation Architects, 16% for infrastructure Engineers, and 16% for CI/CD Engineers.
• The salary range for DevOps engineers is from $91,000 to $155,000, with an average pay of $115,666.

DevOps Trends Impacting Game Development

Serverless Architecture

•  Serverless computing refers to the provision of back-end services on an as-used basis. Users are absolved of the need to have an underlying infrastructure to write and deploy code, as it is placed on architecture provided by a serverless provider.
• The service is auto scaling and charges are applied based on computations, without having to pay or reserve for fixed bandwidth or servers. Even though servers are still used in the provision of the service, developers are not required to be aware of them.
•  Advantages to be derived from serverless computing include quicker turnarounds, simplified backend code, simplified scalability, and lower costs.
•  Lower costs are derived from the elimination of upfront costs to provide the infrastructure and maintenance costs, as well as users paying for unused space and idle CPU time. Developers do not need to create a scaling policy for code as the serverless vendor will handle the scaling on demand.
• Particularly for game developers, the quicker turnaround achieved from using a serverless architecture significantly cuts time to market, as processes to deploy bug fixes and new features are eliminated in favor of a piecemeal approach.
• Applied to game development, the benefits are maximized efficiencies, reduced costs, and a more resilient workflow.

Cloud Computing and DevOps

• The cloud gaming market was valued at US $1.15 billion in 2019, with an estimated CAGR of 15.3% over the period 2020-2025. The industry is expected to have a market value of US$2.7 billion by 2025.
• In cloud gaming the processing load from game play is shifted from the user's hardware to the cloud provider's server, allowing for a better user experience and lower costs.
• For developers, Cloud computing platforms allows for the accessing of a centralized, standard platform to test, deploy, and produce code. Cloud computing providers also support DevOps tools such as continuous integration and continuous development on their platforms allowing for a lowering of the cost for on-premise DevOps automation.
• Cloud platforms have resource tracking by data, users, applications and developers embedded, hence the costs of development is easily understood, compiled and accounted for.
• Where 5G is available, higher broadband speeds and penetration rates are achieved allowing for faster delivery times and increased accessibility to the consumer.

Increased Adoption of Docker Container Registries

•  Docker container images are open-source containers that are standalone, lightweight, executable packages of software containing code, runtime, system tools, system libraries and settings.
• The containers are available for both windows-based and Linux applications, allowing for seamless running, regardless of the underlying infrastructure. The software will work seamlessly regardless of the environment.
• Docker containers can also be integrated with JFrog container registries to support Docker containers and Helm chart repositories for Kubernetes deployments. Docker containers also integrate with artifactory container registries, allowing the DevOps teams greater control over access and permissions.

Quality Assurance (QA) and DevOps

• Unlike its role as a gatekeeper in non-DevOps environments, QA is considered a vital enabler and one of the triad of DevOps processes, namely, development, operations, and testing. In a DevOps environment, the focus for QA shifts from finding bugs, to preventing their creation.
•  
  Source
   
• Given that the boundaries between development and operations are blurred in DevOps, QA functions as the tool that sends development code created in an environment supported by the infrastructure and tools provided by the DevOps team, to production.
• Because the silos between development and operations are removed in a DevOps environment, QA is infused into all processes, from the client and business, to development and operations.
• The impact of QA having an integral role in DevOps is an assumption of responsibility by all team members for maintaining quality levels throughout the project. Skill sets are also improved as developers acquire testing skills and testers acquire coding skills.
•  Best practices for QA in DevOps are included in the list below.
• Technical teams should include QA and testing teams.
• The definition of quality should be defined to accommodate the requirements of the project.
•  Metrics to measure quality should be developed with the capability of detecting defects in the software early in the development cycle.
• Team and individual goals should be optimized.
• QA teams should be embedded in the development of the requirements process.
•  Focus should be placed on automated regression testing.
• Usage of automation tools to automate testing should be commonplace.
• Work should be integrated on a daily basis to optimize continuous integration.
• The automated continuous testing process should be integrated into the SDLC.

QA Market Value

•  Proprietary research is available regarding the market value, value chain, and market forecast to 2026 for the Global Software Quality Assurance Testing market.
• One estimate pegs the global addressable market for software testing at US $60 billion in 2019. This is expected to increase to US $76 billion by 2022 at a compound annual growth rate of 8%,

Proximal Industries

• Proximal industries to QA in DevOps are game testing and software quality assurance. 
•  Game testing is a part of game development and refers to software testing processes to ensure the quality of the game. In this type of testing, the rules and requirements of the game must be understood, as well as the underlying architecture, the game flow, and other structures and dependencies impacting on the game.
• Software Quality Assurance is the process of assuring that software engineering methods, processes, work items, and activities are monitored for compliance against defined standards.