Can't get BuildFilter<T>() to work

I have this method

public IEnumerable<TModel> GetAllStickyNewsForCurrentUser<TModel>(IEnumerable<int> categoryIds, ContentReference rootPage) where TModel : PageData, IAttachable
        {
            var categoryFilter = _searchClient.BuildFilter<TModel>();
            foreach (var categoryId in categoryIds)
            {
                categoryFilter.Or(x => x.Category.Match(categoryId));
            }
            return _searchClient.Search<TModel>()
                .Filter(x => x.IsSticky.Match(true))
                .Filter(x => x.Ancestors().Match(rootPage.ID.ToString()))
                .Filter(categoryFilter)  // returns the same result with or without this line
                .FilterForVisitor()
                .GetContentResult()
                .ToList();
        }

What I want it to do is to filter on a boolean called IsSticky where the root page is the root page parameter and where the page has the category parameters provided. Now this method works equally with or without the categoryFilter which makes me doubt I follow write correct code. Even if I change categoryFilter.Or to categoryFilter.And the result is the same, also categoryFilter.HasFilter always returns false.

What do I need to change to make this method take the category filtering into account? Thanks.