Sort by relevancy and then by other field

I have an issue with sorting in Find. I have a requirement that products in the search results should be sorted by relevancy and then by bought count. I am calculating bought count in a separate scheduled job and updating the field on the product content, so it is available for Find.

When I apply OrderBy on the bought count property, then relevancy is completely ignored. I found this thread: https://world.episerver.com/Modules/Forum/Pages/Thread.aspx?id=80937 where a custom extension method was used to force search by relevancy: https://gist.github.com/lindstromhenrik/9779858. I have tried it by ordering by score first and then by my bought count field:

search = search
    .OrderByScore()
    .ThenByDescending(x => x.BoughtCount);

Now relevancy sorting works, but sorting by bought count doesn't work anymore. If I understand correctly the reason is that sorting by score is too granular for my case. Is there a way to have 

Is there a way to fix this? It would be good if I could access the score, then I could create my own sorting "score".