Cancerous tumors inside the nose may affect your sense of smell. But this is rare.