Awesome Chaos Engineering

A curated list of awesome Chaos Engineering resources.

What is Chaos Engineering?

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production. - Principles Of Chaos Engineering website.

Contents

• Culture


• Books


• Education


• Notable Tools


• Papers


• Gamedays


• Blogs & Newsletters


• Conferences & Meetups


• Forums


• Twitter

Culture

• Principles Of Chaos Engineering


• Chaos Community


• Chaos Engineering


• O'Reilly Velocity San Jose 2017: Precision Chaos


• The Discipline of Chaos Engineering


• Chaos Monkey for Fun and Profit


• Fault Injection in Production: Making the case for resilience testing


• Lord of Chaos - Becoming a Chaos Engineer


• Chaos testing - Preventing failure by instigation


• Orchestrated Chaos


• Choose your own adventure: Chaos Engineering - Video & Slides


• AMA Chaos Engineering + DiRT


• SRECON17: Principles of Chaos Engineering


• Chaos & Intuition Engineering at Netflix


• Mastering Chaos - A Netflix Guide to Microservices


• Too big to test: Breaking a production brokerage platform without causing financial devastation


• Inside Azure Search: Chaos Engineering


• Netflix, the Simian Army, and the culture of freedom and responsibility


• FIT: Failure Injection Testing


• The Netflix Simian Army


• Automated Failure Testing


• The Verification of a Distributed System by Caitie McCaffrey


• The Journey to Chaos Engineering begins with a single step - Bruce Wong and James Burns (Twilio)


• Chaos Engineering by Lorin Hochstein


• Aaron Rinehart - ChaoSlingr: Introducing Security based Chaos Testing


• Chaos Engineering - Casey Rosenthal


• The Road to Chaos - Velocity 2017- video & slides


• How Netflix DDoS’d Itself To Help Protect the Entire Internet


• 10 Years of Crashing Google


• Weathering the Unexpected


• SRECON17: Breaking Things on Purpose


• PuppetConf 2016: Chaos Patterns - Architecting for Failure in Distributed Systems


• Ship More, Sink Less - Changing Chaos Engineering and Distributed Tracing


• Cloudcast - Discipline of Chaos Engineering


• Software Engineering Daily - Failure Injection with Kolton Andrus podcast


• Responding to Failures in Playback Features with Haley Tucker podcast


• "Antics, drift, and chaos" by Lorin Hochstein


• re:invent 2017: Nora Jones Describes Why We Need More Chaos - Chaos Engineering, That Is


• Failure Friday: Four Years On


• Monkeys & Lemurs and Locusts, Oh my!


• Practical Chaos Engineering


• Chaos Day in the Met Office Cloud


• Cloud Native and Chaos Engineering


• Chaos Engineering with Kolton Andrus


• Chaos Engineering: the history, principles, and practice


• Embracing the Chaos of Chaos Engineering


• Designing Services for Resilience: Netflix Lessons


• Chaos Engineering: A cheat sheet


• How to convince your boss and make them say “Yes!” to Chaos Engineering?


• Why the World Needs More Resilient Systems


• Chaos Architecture


• Gremlin’s Tammy Bütow on the Business Side of Chaos Engineering


• Kubernetes Chaos Engineering: Lessons Learned


• Chaos Engineering: managing complexity by breaking things


• Podcast:Database Chaos with Tammy Butow


• LinkedOut: A Request-Level Failure Injection Framework


• GOTO 2018 - Breaking Things on Purpose - Kolton Andrus


• Why should Chaos be part of your Distributed Systems Engineering?


• Brian Holt - Chaos Monkeys in Your Browser What Chaos Engineering Means For the Front End


• Chaos Engineering: Why the World Needs More Resilient Systems


• QCon·Beijing 2017: The Practice of Failure Management and Fault Injection at Alibaba E-Commerce Platforms - video & speech draft (Chinese speech)


• Orchestrating Chaos using Grab's Experimentation Platform


• Breaking to Learn: Chaos Engineering Explained


• Chaos Engineering Traps


• Chaos Engineering - The Art of Breaking Things Purposefully


• Disasterpiece Theater: Slack’s process for approachable Chaos Engineering


• Taming chaos: Preparing for your next incident


• The Future of Chaos Engineering w/ Conde Nast


• Chaos Engineering For People Systems w/ Dave Rensin of Google


• Performing chaos engineering in a serverless world (AWS re:Invent 2019 CMY301)


• Building Confidence in Healthcare Systems through Chaos Engineering


• Break Your App before Someone Else Does


• Preparing for Traffic Spikes with Chaos Engineering


• Automating Chaos Engineering GameDays with Terraform


• Postmortem Culture: Learning from failure


• Problem Detection by John Allspaw


• New Paradigms for the Next Era of Security


• Cloud-Native Chaos Engineering


• Building resilient services at Prime Video with chaos engineering


• Making Chaos Part of Kubernetes/OpenShift Performance and Scalability Tests


• Lucky Lotto, chaos engineering but for teams

Books

• Chaos Engineering: Building Confidence in System Behavior through Experiment


• Site Reliability Engineering: How Google Runs Production Systems -


• The Practice Of Cloud System Administration: Designing and Operating Large Distributed Systems


• Antifragile Systems and Teams


• The InfoQ eMag: Chaos Engineering


• Learning Chaos Engineering


• Chaos Engineering: System Resilience in Practice


• Chaos Engineering: Crash test your applications


• Security Chaos Engineering: Gaining Confidence in Resilience and Safety at Speed and Scale

Education

• A Chaos Engineering Bootcamp for O'Reilly Velocity 2017 - Slides & Source code


• Your First Chaos Experiment


• Chaos Engineering 101


• A Primer on Automating Chaos


• Intro to Chaos Engineering


• Learn the basics of the Chaos Toolkit


• Build System Confidence with Chaos Engineering


• How we break things at Twitter: failure testing


• Run Chaos Experiments Without Risking Your Job


• A Guide to Your First Chaos Day


• Planning Your Own Chaos Day


• How To Install Distributed Tensorflow on GCP and Perform Chaos Engineering Experiments


• Monitoring Your Chaos Experiments


• Increasing the Resilience of APIs with Chaos Engineering


• 3 key steps for running chaos engineering experiments


• Exploring Multi-level Weaknesses using Automated Chaos Experiments


• Chaos Monkey Guide for Engineers


• Chaos Engineering for Serverless


• Network Fire Drills with Chaos Engineering


• Dev Ops Foundations: Chaos Engineering


• Resilience Engineering: Short Course


• The Chaos Engineering Collection


• PenTester Academic

Notable Tools

• Chaos Monkey - A resiliency tool that helps applications tolerate random instance failures.


• orchestrator - MySQL replication topology management and HA.


• kube-monkey - An implementation of Netflix's Chaos Monkey for Kubernetes clusters.


• Gremlin Inc. - Failure as a Service.


• Chaos Toolkit - A chaos engineering toolkit to help you build confidence in your software system.


• steadybit - A Chaos Engineering platform (SaaS or On-Prem) with auto discovery features, different attack types, user management and many more.


• PowerfulSeal - Adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.


• drax - DC/OS Resilience Automated Xenodiagnosis tool. It helps to test DC/OS deployments by applying a Chaos Monkey-inspired, proactive and invasive testing approach.


• Wiremock - API mocking (Service Virtualization) which enables modeling real world faults and delays


• MockLab - API mocking (Service Virtualization) as a service which enables modeling real world faults and delays.


• Pod-Reaper - A rules based pod killing container. Pod-Reaper was designed to kill pods that meet specific conditions that can be used for Chaos testing in Kubernetes.


• Muxy - A chaos testing tool for simulating a real-world distributed system failures.


• Toxiproxy - A TCP proxy to simulate network and system conditions for chaos and resiliency testing.


• Chaos engineering for Docker:


• Pumba - Chaos testing and network emulation for Docker containers (and clusters).


• Blockade - Docker-based utility for testing network failures and partitions in distributed applications.


• chaos-lambda - Randomly terminate ASG instances during business hours.


• Namazu - Programmable fuzzy scheduler for testing distributed systems.


• Chaos Monkey for Spring Boot - Injects latencies, exceptions, and terminations into Spring Boot applications


• Byte-Monkey - Bytecode-level fault injection for the JVM. It works by instrumenting application code on the fly to deliberately introduce faults like exceptions and latency.


• GomJabbar - ChaosMonkey for your private cloud


• Turbulence - Tool focused on BOSH environments capable of stressing VMs, manipulating network traffic, and more. It is very simmilar to Gremlin.


• chaosblade - An Easy to Use and Powerful Chaos Engineering Toolkit.


• KubeInvaders - Gamfied Chaos engineering tool for Kubernetes Clusters


• Cthulhu - Chaos Engineering tool that helps evaluating the resiliency of microservice systems simulating various disaster scenarios against a target infrastructure in a data-driven manner.


• VMware Mangle - Orchestrating Chaos Engineering.


• Byteman - A Swiss Army Knife for Byte Code Manipulation.


• Litmus - Framework for Kubernetes environments that enables users to run test suites, capture logs, generate reports and perform chaos tests.


• Perses - A project to cause (controlled) destruction to a JVM application.


• ChaosKube - chaoskube periodically kills random pods in your Kubernetes cluster.


• Chaos Mesh - Chaos Mesh is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments.


• failure-lambda - A small Node module for injecting failure into AWS Lambda using latency, exception, statuscode or diskspace.


• aws-chaos-scripts - Collection of python scripts to run failure injection on AWS infrastructure


• chaos-ssm-documents - Collection of AWS SSM Documents to perform Chaos Engineering experiments


• aws-lambda-chaos-injection - A library injecting chaos into AWS Lambda. It offers simple python decorators to do delay, exception and statusCode injection and a Class to add delay to any 3rd party dependencies.


• chaos-dingo - A tool to mess with Azure services using the Azure NodeJS SDK.


• Chaos HTTP Proxy - Introduce failures into HTTP requests via a proxy server


• Chaos Lemur - A self-hostable application to randomly destroy virtual machines in a BOSH-managed environment


• Simoorg - Linkedin’s very own failure inducer framework.


• react-chaos - A chaos engineering tool for your React apps


• vue-chaos - A chaos engineering tool for your Vue apps


• Chaos Engine - tool designed to intermittently destroy or degrade application resources running in cloud based infrastructure. Documentation


• kubedoom - Kill Kubernetes pods by playing Id's DOOM.


• kubethanos - Kills half of your randomly selected Kubernetes pods.


• go-fault - Fault injection middleware in Go


• Proofdock's Chaos Engineering Platform - A chaos engineering platform that seamlessly integrates in Azure DevOps and has a focus on the Azure cloud platform.


• Pystol - Pystol is a fault injection platform allowing users to execute fault injection Actions in cloud-native environments in a controlled and prescribed way.


• AWSSSMChaosRunner - Amazon's light-weight open-source library for chaos engineering on AWS. It can be used for EC2, ECS (with EC2 launch type) and Fargate.


• Kraken - Chaos and resiliency testing tool for Kubernetes and OpenShift.


• kube-burner - A tool aimed at stressing Kubernetes clusters by creating or deleting a high quantity of objects.


• Chaos Experimentation Framework - An extensible platform for infrastructure management including Chaos Engineering

Retired tools

• The Simian Army - A suite of tools for keeping your cloud operating in top form.


• ChaoSlingr - Introducing Security Chaos Engineering. ChaoSlingr focuses primarily on the experimentation on AWS Infrastructure to proactively instrument system security failure through experimentation.

Cloud Services

• Testing Amazon Aurora Using Fault Injection Queries


• Azure Fault Analysis Service, see also Include controlled Chaos in Service Fabric clusters


• Security Chaos Engineering for Cloud Services

Papers

• Maelstrom: Mitigating Datacenter-level Disasters by Draining Interdependent Traffic Safely and Efficiently


• Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems


• Automating Failure Testing Research at Internet Scale


• Principles of Antifragile Software


• Why is random testing effective for partition tolerance bugs?


• Chaos Engineering


• A Platform for Automating Chaos Experiments


• A Chaos Engineering System for Live Analysis and Falsification of Exception-handling in the JVM


• TripleAgent: Monitoring, Perturbation And Failure-obliviousness for Automated Resilience Improvement in Java Applications


• Lineage-driven Fault Injection


• Antifragility is a Fragile Concept


• Chaos Engineering Security


• Security Chaos Engineering: A new paradigm for cybersecurity


• Security Challenges around Chaos Engineering


• CloudStrike: Security Chaos Engineering for Cloud Services

Gamedays

• Target: What is a Gameday? - Chaos Gamedays experience by Target.


• Codecentric: Chaos Engineering Gamedays - Chaos Gamedays by Codecentric.


• New Relic: How to run a Gameday? - Chaos Gamedays experience by New Relic.


• Dius: Gamedays resources - Resources for getting started with GameDay and Chaos Engineering.


• Gremlin: Gamedays - Resources for getting started with GameDay and Chaos Engineering.


• Gremlin: What is a Chaos Day? - What is a Gameday according Gremlin.


• Gremlin: Why run a Chaos Day? - Reasons to run Gamedays according Gremlin.


• Gremlin: How to run a Gameday? - Methodology to run Gamedays according Gremlin.


• Gremlin DB: Breaking Dynamo DB - Example of a Gameday with DynamoDB by Gremlin.


• Gremlin: Introduction to Gameday - What is a Gameday according Gremlin.


• Gremlin: Planning your own Chaos Day - Example of a Gameday with DynamoDB by Gremlin.


• Gremlin: Inside Gremlin 2019 Gremlin Gamedays Roadmap - Chaos Gamedays experience by Gremlin.


• Gremlin: What I lerned running the Chaos Lab with Kafka - Example of a Gameday with Kafka by Gremlin.


• Chaos Toolkit: Chaos Engineering with Humans in the loop - Article about Chaos Gamedays.


• GooCardless: All fun and games until you start with Gamedays - Article about Chaos Gamedays.


• InfoQ: Gamedays - Achieving Resilience through Chaos Engineering - InfoQ Presentation with experiences about Chaos Gamedays.

Blogs & Newsletters

• Netflix Technology Blog - Learn more about how Netflix designs, builds, and operates our systems and engineering organizations.


• Production Ready - A mailing list about building resilient infrastructure and tools.


• SRE Weekly - Weekly Site Reliability Newsletter.


• Site Reliability Engineering resources - A curated list of awesome Site Reliability and Production Engineering resources.


• SysAdvent - One article for each day of December, ending on the 25th article.


• Gremlin Blog - Blogs on Chaos Engineering from Gremlin Inc.


• O’Reilly Systems Engineering and Operations Newsletter - Weekly systems engineering and operations news and insights from industry insiders.


• LaunchDarkly Blog - Continuous delivery and feature flags blog.


• Verica - Chaos engineering, security chaos engineering and continuous verification.


• Proofdock - Reliability, resilience and chaos engineering with a focus on MS Azure


• LitmusChaos Blog - Blogs on Chaos Engineering from LitmusChaos


• ChaosEngineering.news - Chaos Engineering newsletter. All things chaos engineering, directly to your inbox!


• Chaos Mesh Blog - Blogs on Chaos Engineering from Chaos Mesh.


• Chaos Experimentation Framework Chaos Experimentation, an open-source framework built on top of Envoy Proxy

Podcasts

• Break Things On Purpose - Monthly podcast about Chaos Engineering presented by Gremlin Inc. Also available on Spotify, Google Play, and Stitcher.

Conferences & Meetups

• Chaos Carnival - A global two-day virtual conference for Cloud Native Chaos Engineering.


• Chaos Conf - A day of Chaos Engineering demos, expert advice, and connect with your peers putting chaos into practice at their companies.


• SRECon Conferences - The official SRE conference.


• LISA Conferences - Prominent conference about SysAdmin/DevOps/SRE.


• O'Reilly Velocity Conference - Prominent conference about Systems Engineering/DevOps/SRE.


• Chaos Engineering Community Meetup Group - Bay Area Meetup group for Chaos Engineers.


• London Chaos Engineering Community _ London Area Meetup group for Chaos Engineers.


• Stockholm Chaos Engineering Meetup Stockholm Meetup group for Chaos Engineers.


• Chaos Engineering Community - A collection of meetups across the globe about Chaos Engineerings.


• Conf42.com: Chaos Engineering - Chaos Engineering for practitioners and adopters - London UK, 23 Jan 2020.


• Kubernetes Chaos Engineering Meetup Group India- India Meetup group for Chaos Engineers.

Forums

• Chaos Community Google Group


• Chaos Engineering LinkedIn Group


• Chaos Engineering Slack Community


• CNCF Chaos Engineering Working Group


• CNCF Chaos Engineering Working Group Slack: #chaosengineering (slack.cncf.io)


• CNCF Chaos Engineering Working Group Github


• Chaos Toolkit Slack Community


• Litmus Chaos Engineering Slack Community

Contributing

Please take a look at the contribution guidelines first. Contributions are always welcome!