To a medium saucepan placed over medium-low heat, combine the milk, sugar, vanilla bean, vanilla bean caviar, pinch of salt, whole coffee beans and 1/2 cup of heavy cream. Once the mixture is warm to the touch, cover, remove from the heat and let it to steep for 1-3 hours. (Note: if your coffee beans are a dark roast, they'll be done in an hour. However, if you're using a lighter roast (like I did), you'll want to steep the mixture for 3 hours. If you're worried about letting milk sit out for that long, transfer the pot to the refrigerator to steep.)
Rewarm the coffee-infused milk mixture. Pour the remaining 1 cup of cream into a large bowl and set a mesh strainer on top. In a separate medium bowl, whisk together the egg yolks. Slowly pour the warm coffee mixture into the egg yolks, whisking constantly, then scrape the warmed egg yolks back into the saucepan.
Stir the mixture constantly over medium heat with a heatproof spatula, scraping the bottom as you stir, until the mixture thickens and reads 170 degrees F to 175 degrees F on a digital-read thermometer. Pour the custard through the strainer and stir it into the cream. Press on the coffee beans in the strainer to extract as much of the coffee flavor as possible, then discard the beans. Mix in the vanilla and the finely ground coffee and stir until cool over an ice bath.
Chill the mixture thoroughly in the refrigerator, then freeze it in your ice cream maker according to the manufacturer's instructions. Transfer to a freezer-safe container and transfer to the freezer until firm.