This ferry port appeared in Episode 10, "Remote Island Syndrome Part 1". In this episode Haruhi goes to a deserted island in a small boat from here this port which in the anime is based on the Tokushima Ferry Terminal.
Haruhi took the "Kumano" Ferry. Unfortunately this particular ferry was retired in 2007, so you won't be able to ride the same ferry. However, you will be able to experience the route that she followed.
"Let's go, people!
A mysterious island awaits us"