Custom ContentProvider caching issue

We have a custom ContentProvider that loads a document library from Sharepoint, but we have a problem with the caching - specifically of children. When caching is turned on, we see files and folders on the top level only. Clicking on a folder shows it as empty. 

Turning off the caching by overriding the method that takes children as a parameter results in all files and folders being shown for all levels. 

protected sealed override void SetCacheSettings(ContentReference contentReference,
    IEnumerable<GetChildrenReferenceResult> children, CacheSettings cacheSettings)
        {
            cacheSettings.CancelCaching = true;
            base.SetCacheSettings(contentReference, children, cacheSettings);
        }

We don't cache anything explicitly in the ContentProvider - as we understand, this should be automatically handled.

Any suggestions would be greatly appreciated.