I believe it is best to try minimizing harm done when possible. While it is impossible to truly live ethically or do no harm, it's best to try cutting down on it where we can. If you have two choices and one causes a lot of harm and the other causes a little harm, then all else being equal the latter should be chosen. I can cut out meat and still live a comfortable life without inconveniencing myself much, I can choose not to live a lifestyle I know is causing pointless suffering to innocent creatures for minimal benefit to me. Simple little changes can reduce the harm we cause, and they might not change much in the grand scheme of things, but they do reduce our involvement in the evils of the world. It keeps the soul clean.
The argument that vegan or vegetarian diets harm more animals than meat based diets is fundamentally rooted in the idea that you need farms to grow food for humans, but that all farm animals magically sustain themselves by drawing nutrients from the aether. If you compare the amount of farmland required to grow vegetables to feed humans to the amount of space required in a factory farm to contain animals, then yes, a farm growing produce will take up more space than a farm that's merely containing animals. This ignores that those animals need to eat, too, and you expend far more produce feeding an animal than you would need if you fed it to a human directly.