Try to speed up infer-loop.

@dcousineau I would be interested to know if these commits make a difference on your examples.