Compiling them under one entry because they may have interrelated causes!

• Devices out of Sync

• Agent Loses Texture
• WebGL Context Lost

• Out of Memory on Digital Ocean

RAW DUMP OF THOUGHTS FROM 4/29

Texture loss may be due to:
* running out of GPU resources/memory
* * Chromebooks probably have less GPU than our laptops
* memory leak in use of GPU resources
* failure to deallocate all elements that use GPU resources

WebGL Context Loss
* This sounds like a disappearing texture buffer as a result of a PIXIJS call.
* Instrument the calls to PIXIJS to see if they get any errors
* Check the Chrome Performance monitoring tab to see GPU uses

Devices out of Sync
* Check the Chrome Performance monitoring to see if frames are colliding
* Backpressure on sockets not being processed fast enough on client

Out of Memory on Server

• Reduce inspector update frequency to once per second should reduce traffic 30x. f4279f8a2774e9cc8072c0bcc54fb2f3bb55da70
• Add PIXI texture cache reporter to debug WEBGL errors. 01d9310607e2b7c39577c15872d22996a68cbfb9
• Explicitly destroy text textures when updating text. Otherwise textures accumulate over time. (Each new text update requires a new texture, you can't just update the old texture). This should reduce texture memory increase at the rate of 1 texture per agent (with a text label) per second. 72a2d5c1f4ea10b13f42307f29d0b0ce59f9944f
• Explicitly destroy meter textures when removing them. This is relatively minor because in practice meters were rarely removed. a2328d3dd3aaa35ff549ae9709188016bf109bf9
• Explicitly destroy vobj sprites when agents are removed. The old method merely removed children but left textures in memory. So over time, the number of textures increase even if agents are removed. 69ce02366f2f66688fd27a37f1dbbfdbddd49efb
• Explicitly remove children and clear class-visual pixi objects. While destroy() does clear the texture, js objects remain. e1f1afbc791b31c976c96cb88d99c7f5e5bdd0b7
• With 0013afc90a9c46d9d170b5c97b1e46a41c152611, texture counts return to base levels after creating and removing 100 entities suggesting we are properly destroying unused pixi objects.
• Only send dobj parameters if the corresponding agent parameter has been set. This should reduce most dobjs by half. 55ff169ef18b5061dcb993529cd1b062c8f1406d

• Backpressure Explained - for a refresher
• Implementing Streams handling backpressure - Has some source code to investigate, though it doesn't quite handle our situation
• Terminal Kit - This is an ncurses-free implementation of color + row/column addressable terminal which will be useful in instrumenting our sockets to observe backpressure in real time