Hume tells us in the conclusion to Book I of the Treatise (T 1.4.7.12) that the origin of his philosophy lies in the sentiments of curiosity and ambition; and in the History of England he describes the progress of science in terms of the satisfaction of curiosity and vanity. But while he gives us an account of how curiosity plays a role in inquiry (Treatise 2.3.10), he doesn't give us any explicit account of how vanity plays a role. The result is that we have to extrapolate from what he does say. In T 1.4.7.12 he gives us a brief sentence: "I feel an ambition to arise in me of contributing to the instruction of mankind, and of acquiring a name by my inventions and discoveries." And he does give us an account of pride in Book II. From these we can get an idea of Hume's account of the role of vanity or ambition in inquiry. A precise idea would require getting into finer details of Hume's account of pride; but we can get the general idea, I think, by comparing it to the role vanity has in our pursuit of wealth.
In discussing the esteem we have for the wealthy, Hume has a fascinating passage (T 2.2.5.21) that is useful for this purpose. (I have added the bolding.)
In general we may remark, that the minds of men are mirrors to one another, not only because they reflect each other's emotions, but also because those rays of passions, sentiments and opinions may be often reverberated, and may decay away by insensible degrees. Thus the pleasure, which a rich man receives from his possessions, being thrown upon the beholder, causes a pleasure and esteem; which sentiments again, being perceiv'd and sympathiz'd with, encrease the pleasure of the possessor; and being once more reflected, become a new foundation for pleasure and esteem in the beholder. There is certainly an original satisfaction in riches deriv'd from that power, which they bestow, of enjoying all the pleasures of life; and as this is their very nature and essence, it must be the first source of all the passions, which arise from them. One of the most considerable of these passions is that of love or esteem in others, which therefore proceeds from a sympathy with the pleasure of the possessor. But the possessor has also a secondary satisfaction in riches arising from the love and esteem he acquires by them, and this satisfaction is nothing but a second reflexion of that original pleasure, which proceeded from himself. This secondary satisfaction or vanity becomes one of the principal recommendations of riches, and is the chief reason, why we either desire them for ourselves, or esteem them in others. Here then is a third rebound of the original pleasure; after which `tis difficult to distinguish the images and reflexions, by reason of their faintness and confusion.
This striking extended metaphor gives us a starting point for considering how vanity might play a role in inquiry. We start with an original satisfaction in wealth, due to "that power...of enjoying all the pleasures of life," which itself brings about pleasure in the mind of the one has the wealth (we'll call him Rich). The passions that we feel tend to be communicated to others by sympathy and imagination; recognizing the pleasure and pride Rich has in wealth, the rest (we'll call them the Chorus), assuming no interfering factors, come to love and esteem him by sympathy, i.e., by the echoes or reflections in the Chorus of Rich's own pride and pleasure. Rich, however, comes to recognize that the Chorus loves and esteems him, and this gives us our second reflection, because by sympathy Rich shares in (and, indeed, can't help but share in, all other things being equal) the Chorus's love and esteem for himself, which adds a second level of satisfaction to his original satisfaction. As he says elsewhere, talking about the love of fame (T 2.1.11.9), "'Tis certain...that if a person consider'd himself in the same light, in which he appears to his admirer, he wou'd first receive a separate pleasure, and afterwards a pride or self-satisfaction...." This second satisfaction is what Hume calls vanity, and he very notably insists that it is the chief reason for our pusuit of wealth. From vanity we get a third reflection, dimmer than the first, but still recognizable: the Chorus doesn't just take pleasure in the idea of having Rich's enjoyment of wealth (first sympathy), it also takes pleasure in the idea of having Rich's enjoyment of the love and admiration he gets through sympathy with the Chorus's original sympathetic pleasure with himself, and this new pleasure in the chorus is the "third rebound" of sympathy. In principle, we could go on, but as Hume says, due to the "faintness and confusion" of reflections in this hall of mirror, we lose the ability to distinguish the outlines of the sentiments clearly.
So we have, to sum up, the following process:
original person
(1) first satisfaction (pride in having), which is reflected in others through
others
(2) first sympathy (as others sympathetically participate in the original person's first satisfaction), which creates in others
(3) love and admiration for the original person, which is reflected in that person through
original person
(4) second sympathy (as the original person sympathetically participates in others' admiration for him as possessor), which induces in the original person
(5) second satisfaction, or vanity (pride in being admired or loved), which is reflected in others through
others
(6) third sympathy (as others sympathetically participate in the second satisfaction of the original person), which creates in others
(7) the desire to be loved and admired as the original person is admired.
Hume tells us that wealth is the cause that produces pride most intensely and copiously (T 2.1.10), so one presumes that the status of vanity will be somewhat messier in the pursuit of scholarly distinction than it is in the pursuit of wealth. But the same process will be found there. As Hume says (T 2.2.1.9), "There are few persons, that are satisfy'd with their own character, or genius, or fortune, who are nor desirous of shewing themselves to the world, and of acquiring the love and approbation of mankind."
Thus vanity forces inquiry to take a profoundly social shape: we can't have second sympathy if there is no one other than ourselves admiring us for our genius. We need the sympathetic echo chamber, the hall of mirrors.
