[ContextManager] Actually use the MemoryManger

Even though the MemoryManager is enabled for the ContextManager per
default, it never was actually used due a logic bug. Squash that bug!
51 jobs for fix-context-manager-with-memory-manager in 19 minutes and 44 seconds (queued for 4 seconds)