Use a map to detect recursive types.
With this we can now typecheck fixedbugs/issue8501.go.
Updates #43088.
Change-Id: I7fad6ccf6c94268473ff72b09a3158e13a7f4cc3
Reviewed-on: https://go-review.googlesource.com/c/go/+/276374
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>