Bottleneck Patterns - simagix/keyhole GitHub Wiki
(also available at Peek at your MongoDB Clusters like a Pro with Keyhole: Part 3)
Bottleneck Patterns
MongoDB is like a coworker you can only dream of. One who is smart, friendly, stylish but not flashy, and speaks multiple languages. Does one have a happy life? That’s on Instagram. Under stress, one shows a few syntactic patterns.
Lost in Space
You didn’t have expected performance results from properly provisioned resources even if the provisioned memory was enough to contain the working set data. In this pattern, all metrics were in healthy states except a large number of scanned objects. This could be a textbook case of using improper or even missing indexes. Even when the entire working set fits in memory, without proper indexes there could be an excessive number of object scans. Imagine a spaceship roaming in space without a navigation system; there will be more space to cover , thus resulting in a much slower response from a query.
Dream Weaver
Another commonly experienced underperforming read operation was caused by less than ideal data access use cases. Most read related metrics were flagged, such as low WiredTiger available read tickets, high WiredTiger cache used, and large number of scanned objects.
Architects and developers are attracted to MongoDB because of its modern and flexible technologies. With simple transformations, one can quickly turn XML into JSON data or to directly map relational database tables to MongoDB collections. It's like a dream come true, and Dream Weavers love to use the $lookup operator and allow many collections to get carelessly intertwined. Such implementations do not work efficiently. An important principle of using MongoDB is to have a proper schema to support your use cases. MongoDB is not "schema-less"; rather, it allows for multiple concurrent schemas to be present. These schemas, however, need to be designed appropriately.
Vikings Attack
Everything worked mostly during the day except for a short period of time. During the time, the cluster experienced high CPU utilization on I/O wait, WiredTiger dirty data ratio, and disk IOPS. Resources provisioning should be based on the loads during the peak hours. Many businesses must process a large number of transactions within a limited time. As such, the database operations burst in like a sudden Viking attack -- arriving in fast dragonships and flooding in on the crest of morning tides. Vikings were huge and enormous, and they charged fearlessly with long swords, axes, and round wooden shields. In this case, these short 'attacks' raided the village, burning up disk IOPS, and brought oplog collection to the ground before retrieval was complete.
New York, New York
Many people rush to New York to melt away little town blues, so the City can use a little bit of breathing room at times. Similarly, hosting an excessive number of collections and indexes in a MongoDB cluster creates additional overhead in WiredTiger. We often see this in multi-tenant implementations. The large number of tables maintained to support collections and indexes in the WiredTiger results in the high number of WiredTiger data handles, which cause extremely long checkpoints and block all the running operations.
Angel Has Fallen
Almost all metrics reveal that resources were all under stress. The number of queued operations was high and the cluster was restlessly catching up. All resources might be simply pegged out from many charts showing high flying plateau lines. If there is no room for tuning, consider scaling up resources or sharding.