Put yourself in the shoes of the aliens... would you risk life and limb, not to mention extraordinary investment in technology, energy and time, to traverse star systems armed with nothing more than anal probes, hacksaws and Rohypnol, in order to interfere with the local fauna?
How many billion dollars have the world's space agencies spent on sending people to a dead rocky planet and probes to scout the universe in the off chance of finding other forms of life, or simply to see what it looks like. Now, what would we be doing if we had interplanetary travel technology ? Just that.
And who's to say they're here for us ? Would the spider in your closet be right to assume you came for it when you're picking up a shirt in the morning ? Maybe they came looking for something they didn't find (or found, without us even noticing), and we somehow spiked their curiosity while they were there.
Look how much our technology's evolved in the past 60 years. Would you be satisfied with black and white pictures of the Grand Canyon to illustrate your geography books, or would you go back with a High Definition digital camera ? Does a gardener check on his roses just once or does he come back often to see how they're growing ?
Very often, people discrediting UFO sightings actually give the hypothetical visitors more credit and more purpose than the people claiming to have encountered them. They depict them as vastly superior to us technologically and morally and use that supposed superiority, in conjunction to human thought patterns and social behavior, to disprove theories of alien visits (what you just did when you said "put yourself in the aliens' shoes")
Maybe they're not that more advanced than we are, maybe their motives and their sense of priority are very different from ours. They might not be able to see in the same spectrum, communicate in the same frequencies, they may even be able to think and move in dimensions our brain structure doesn't even allow us to perceive, making their perspective so different from ours we couldn't comprehend it. Maybe we're as much of an enigma to them as they are to us, or maybe they're so far ahead of us they have no more consideration for us than we have for bacteria...
Until someone can prove or disprove their existence and their presence on earth, we won't know for sure, and all leads have to be explored.