The second half of the stanza, which begins here, is all a metaphor. Eliot used to visit Burnt Norton with a kind-of-girlfriend called Emily who he met once a year. In the next lines he goes with her into the garden of the house.
Other echoesThe rose garden is 'our first world'. In the previous part of the poem going into the garden was the good choice we didn't make.
Inhabit the garden. Shall we follow?
Quick, said the bird, find them, find them,
Round the corner. Through the first gate,
Into our first world, shall we follow
The deception of the thrush? Into our first world.
Our 'first world' is childhood. It is also the primary existence of the present (compared to which the past and future are secondary inferences). And it means the Garden of Eden - our first world, and hence our prelapsarian innocence.
He goes into the garden chasing 'other echoes' - traces of the past. He is trying to go back to childhood and innocence, through the first gate - 'the door we never opened/into the rose garden'. He is trying to correct the previous mistake.
These lines remind me very much of The Door in the Wall, a short story by HG Wells about a man who is always searching for the door into paradise that he went through as a boy. (That link is to the full text of the story)
And the bird reminds me of Yeats' enameled bird in 'Sailing to Byzantium' which was
Set upon a golden bough to sing... Of what is past, or passing, or to come.
I think in both these poems the bird is outside of time, indifferent to human suffering, playful and deceptive. It's a sort of trickster character.
So, Eliot, knowing it is futile and deceptive, tries to simply walk back into Eden.