By default, all project-related tasks are visible to all project members.
If you want only to show the tasks where the staff member is assigned/added as a follower you need to turn off Allow all staff to see all tasks related to projects
To achieve this go to Setup->Settings->Tasks and set Allow all staff to see all tasks related to projects to NO
If the staff member has permission for tasks VIEW or is admin then this member will be still able to see all tasks.