qbit/go
1
0
Fork 0
mirror of https://github.com/golang/go synced 2024-11-26 05:57:58 -07:00
Code Releases Activity

[dev.typeparams] cmd/compile: unified IR construction

Browse Source 
This CL adds a new unified IR construction mode to the frontend.  It's
purely additive, and all files include "UNREVIEWED" at the top, like
how types2 was initially imported. The next CL adds a -d=unified flag
to actually enable unified IR mode.

See below for more details, but some highlights:

1. It adds ~6kloc (excluding enum listings and stringer output), but I
estimate it will allow removing ~14kloc (see CL 324670, including its
commit message);

2. When enabled by default, it passes more tests than -G=3 does (see
CL 325213 and CL 324673);

3. Without requiring any new code, it supports inlining of more code
than the current inliner (see CL 324574; contrast CL 283112 and CL
266203, which added support for inlining function literals and type
switches, respectively);

4. Aside from dictionaries (which I intend to add still), its support
for generics is more complete (e.g., it fully supports local types,
including local generic types within generic functions and
instantiating generic types with local types; see
test/typeparam/nested.go);

5. It supports lazy loading of types and objects for types2 type
checking;

6. It supports re-exporting of types, objects, and inline bodies
without needing to parse them into IR;

7. The new export data format has extensive support for debugging with
"sync" markers, so mistakes during development are easier to catch;

8. When compiling with -d=inlfuncswithclosures=0, it enables "quirks
mode" where it generates output that passes toolstash -cmp.

--

The new unified IR pipeline combines noding, stenciling, inlining, and
import/export into a single, shared code path. Previously, IR trees
went through multiple phases of copying during compilation:

1. "Noding": the syntax AST is copied into the initial IR form. To
support generics, there's now also "irgen", which implements the same
idea, but takes advantage of types2 type-checking results to more
directly construct IR.

2. "Stenciling": generic IR forms are copied into instantiated IR
forms, substituting type parameters as appropriate.

3. "Inlining": the inliner made backup copies of inlinable functions,
and then copied them again when inlining into a call site, with some
modifications (e.g., updating position information, rewriting variable
references, changing "return" statements into "goto").

4. "Importing/exporting": the exporter wrote out the IR as saved by
the inliner, and then the importer read it back as to be used by the
inliner again. Normal functions are imported/exported "desugared",
while generic functions are imported/exported in source form.

These passes are all conceptually the same thing: make a copy of a
function body, maybe with some minor changes/substitutions. However,
they're all completely separate implementations that frequently run
into the same issues because IR has many nuanced corner cases.

For example, inlining currently doesn't support local defined types,
"range" loops, or labeled "for"/"switch" statements, because these
require special handling around Sym references. We've recently
extended the inliner to support new features like inlining type
switches and function literals, and they've had issues. The exporter
only knows how to export from IR form, so when re-exporting inlinable
functions (e.g., methods on imported types that are exposed via
exported APIs), these functions may need to be imported as IR for the
sole purpose of being immediately exported back out again.

By unifying all of these modes of copying into a single code path that
cleanly separates concerns, we eliminate many of these possible
issues. Some recent examples:

1. Issues #45743 and #46472 were issues where type switches were
mishandled by inlining and stenciling, respectively; but neither of
these affected unified IR, because it constructs type switches using
the exact same code as for normal functions.

2. CL 325409 fixes an issue in stenciling with implicit conversion of
values of type-parameter type to variables of interface type, but this
issue did not affect unified IR.

Change-Id: I5a05991fe16d68bb0f712503e034cb9f2d19e296
Reviewed-on: https://go-review.googlesource.com/c/go/+/324573
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
This commit is contained in:
Matthew Dempsky 2021-05-13 20:23:13 -07:00
parent ea438bda85
commit 79cd1687e6
12 changed files with 6164 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

126
src/cmd/compile/internal/noder/codes.go Normal file
View File

@ -0,0 +1,126 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
type code interface {
marker() syncMarker
value() int
}
type codeVal int
func (c codeVal) marker() syncMarker { return syncVal }
func (c codeVal) value() int { return int(c) }
const (
valBool codeVal = iota
valString
valInt64
valBigInt
valBigRat
valBigFloat
)
type codeType int
func (c codeType) marker() syncMarker { return syncType }
func (c codeType) value() int { return int(c) }
const (
typeBasic codeType = iota
typeNamed
typePointer
typeSlice
typeArray
typeChan
typeMap
typeSignature
typeStruct
typeInterface
typeUnion
typeTypeParam
)
type codeObj int
func (c codeObj) marker() syncMarker { return syncCodeObj }
func (c codeObj) value() int { return int(c) }
const (
objAlias codeObj = iota
objConst
objType
objFunc
objVar
objStub
)
type codeStmt int
func (c codeStmt) marker() syncMarker { return syncStmt1 }
func (c codeStmt) value() int { return int(c) }
const (
stmtEnd codeStmt = iota
stmtLabel
stmtBlock
stmtExpr
stmtSend
stmtAssign
stmtAssignOp
stmtIncDec
stmtBranch
stmtCall
stmtReturn
stmtIf
stmtFor
stmtSwitch
stmtSelect
// TODO(mdempsky): Remove after we don't care about toolstash -cmp.
stmtTypeDeclHack
)
type codeExpr int
func (c codeExpr) marker() syncMarker { return syncExpr }
func (c codeExpr) value() int { return int(c) }
// TODO(mdempsky): Split expr into addr, for lvalues.
const (
exprNone codeExpr = iota
exprConst
exprType // type expression
exprLocal // local variable
exprName // global variable or function
exprBlank
exprCompLit
exprFuncLit
exprSelector
exprIndex
exprSlice
exprAssert
exprUnaryOp
exprBinaryOp
exprCall
// TODO(mdempsky): Handle in switchStmt directly instead.
exprTypeSwitchGuard
)
type codeDecl int
func (c codeDecl) marker() syncMarker { return syncDecl }
func (c codeDecl) value() int { return int(c) }
const (
declEnd codeDecl = iota
declFunc
declMethod
declVar
declOther
)

243
src/cmd/compile/internal/noder/decoder.go Normal file
View File

@ -0,0 +1,243 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"encoding/binary"
"fmt"
"go/constant"
"go/token"
"math/big"
"os"
"strings"
"cmd/compile/internal/base"
)
type pkgDecoder struct {
pkgPath string
elemEndsEnds [numRelocs]uint32
elemEnds []uint32
elemData string
}
func newPkgDecoder(pkgPath, input string) pkgDecoder {
pr := pkgDecoder{
pkgPath: pkgPath,
}
// TODO(mdempsky): Implement direct indexing of input string to
// avoid copying the position information.
r := strings.NewReader(input)
assert(binary.Read(r, binary.LittleEndian, pr.elemEndsEnds[:]) == nil)
pr.elemEnds = make([]uint32, pr.elemEndsEnds[len(pr.elemEndsEnds)-1])
assert(binary.Read(r, binary.LittleEndian, pr.elemEnds[:]) == nil)
pos, err := r.Seek(0, os.SEEK_CUR)
assert(err == nil)
pr.elemData = input[pos:]
assert(len(pr.elemData) == int(pr.elemEnds[len(pr.elemEnds)-1]))
return pr
}
func (pr *pkgDecoder) numElems(k reloc) int {
count := int(pr.elemEndsEnds[k])
if k > 0 {
count -= int(pr.elemEndsEnds[k-1])
}
return count
}
func (pr *pkgDecoder) totalElems() int {
return len(pr.elemEnds)
}
func (pr *pkgDecoder) absIdx(k reloc, idx int) int {
absIdx := idx
if k > 0 {
absIdx += int(pr.elemEndsEnds[k-1])
}
if absIdx >= int(pr.elemEndsEnds[k]) {
base.Fatalf("%v:%v is out of bounds; %v", k, idx, pr.elemEndsEnds)
}
return absIdx
}
func (pr *pkgDecoder) dataIdx(k reloc, idx int) string {
absIdx := pr.absIdx(k, idx)
var start uint32
if absIdx > 0 {
start = pr.elemEnds[absIdx-1]
}
end := pr.elemEnds[absIdx]
return pr.elemData[start:end]
}
func (pr *pkgDecoder) stringIdx(idx int) string {
return pr.dataIdx(relocString, idx)
}
func (pr *pkgDecoder) newDecoder(k reloc, idx int, marker syncMarker) decoder {
r := pr.newDecoderRaw(k, idx)
r.sync(marker)
return r
}
func (pr *pkgDecoder) newDecoderRaw(k reloc, idx int) decoder {
r := decoder{
common: pr,
k: k,
idx: idx,
}
// TODO(mdempsky) r.data.Reset(...) after #44505 is resolved.
r.data = *strings.NewReader(pr.dataIdx(k, idx))
r.sync(syncRelocs)
r.relocs = make([]relocEnt, r.len())
for i := range r.relocs {
r.sync(syncReloc)
r.relocs[i] = relocEnt{reloc(r.len()), r.len()}
}
return r
}
type decoder struct {
common *pkgDecoder
relocs []relocEnt
data strings.Reader
k reloc
idx int
}
func (r *decoder) checkErr(err error) {
if err != nil {
base.Fatalf("unexpected error: %v", err)
}
}
func (r *decoder) sync(m syncMarker) {
if debug {
pos, err0 := r.data.Seek(0, os.SEEK_CUR)
x, err := r.data.ReadByte()
r.checkErr(err)
if x != byte(m) {
// TODO(mdempsky): Revisit this error message, and make it more
// useful (e.g., include r.p.pkgPath).
base.Fatalf("data sync error: found %v at %v (%v) in (%v:%v), but expected %v", syncMarker(x), pos, err0, r.k, r.idx, m)
}
}
}
func (r *decoder) bool() bool {
r.sync(syncBool)
x, err := r.data.ReadByte()
r.checkErr(err)
assert(x < 2)
return x != 0
}
func (r *decoder) int64() int64 {
r.sync(syncInt64)
x, err := binary.ReadVarint(&r.data)
r.checkErr(err)
return x
}
func (r *decoder) uint64() uint64 {
r.sync(syncUint64)
x, err := binary.ReadUvarint(&r.data)
r.checkErr(err)
return x
}
func (r *decoder) len() int { x := r.uint64(); v := int(x); assert(uint64(v) == x); return v }
func (r *decoder) int() int { x := r.int64(); v := int(x); assert(int64(v) == x); return v }
func (r *decoder) uint() uint { x := r.uint64(); v := uint(x); assert(uint64(v) == x); return v }
func (r *decoder) code(mark syncMarker) int {
r.sync(mark)
return r.len()
}
func (r *decoder) reloc(k reloc) int {
r.sync(syncUseReloc)
idx := r.len()
e := r.relocs[idx]
assert(e.kind == k)
return e.idx
}
func (r *decoder) string() string {
r.sync(syncString)
return r.common.stringIdx(r.reloc(relocString))
}
func (r *decoder) strings() []string {
res := make([]string, r.len())
for i := range res {
res[i] = r.string()
}
return res
}
func (r *decoder) rawValue() constant.Value {
isComplex := r.bool()
val := r.scalar()
if isComplex {
val = constant.BinaryOp(val, token.ADD, constant.MakeImag(r.scalar()))
}
return val
}
func (r *decoder) scalar() constant.Value {
switch tag := codeVal(r.code(syncVal)); tag {
default:
panic(fmt.Sprintf("unexpected scalar tag: %v", tag))
case valBool:
return constant.MakeBool(r.bool())
case valString:
return constant.MakeString(r.string())
case valInt64:
return constant.MakeInt64(r.int64())
case valBigInt:
return constant.Make(r.bigInt())
case valBigRat:
num := r.bigInt()
denom := r.bigInt()
return constant.Make(new(big.Rat).SetFrac(num, denom))
case valBigFloat:
return constant.Make(r.bigFloat())
}
}
func (r *decoder) bigInt() *big.Int {
v := new(big.Int).SetBytes([]byte(r.string()))
if r.bool() {
v.Neg(v)
}
return v
}
func (r *decoder) bigFloat() *big.Float {
v := new(big.Float).SetPrec(512)
assert(v.UnmarshalText([]byte(r.string())) == nil)
return v
}

245
src/cmd/compile/internal/noder/encoder.go Normal file
View File

@ -0,0 +1,245 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"bytes"
"encoding/binary"
"fmt"
"go/constant"
"io"
"math/big"
"cmd/compile/internal/base"
)
type pkgEncoder struct {
elems [numRelocs][]string
stringsIdx map[string]int
}
func newPkgEncoder() pkgEncoder {
return pkgEncoder{
stringsIdx: make(map[string]int),
}
}
func (pw *pkgEncoder) dump(out io.Writer) {
writeUint32 := func(x uint32) {
assert(binary.Write(out, binary.LittleEndian, x) == nil)
}
var sum uint32
for _, elems := range &pw.elems {
sum += uint32(len(elems))
writeUint32(sum)
}
sum = 0
for _, elems := range &pw.elems {
for _, elem := range elems {
sum += uint32(len(elem))
writeUint32(sum)
}
}
for _, elems := range &pw.elems {
for _, elem := range elems {
_, err := io.WriteString(out, elem)
assert(err == nil)
}
}
}
func (pw *pkgEncoder) stringIdx(s string) int {
if idx, ok := pw.stringsIdx[s]; ok {
assert(pw.elems[relocString][idx] == s)
return idx
}
idx := len(pw.elems[relocString])
pw.elems[relocString] = append(pw.elems[relocString], s)
pw.stringsIdx[s] = idx
return idx
}
func (pw *pkgEncoder) newEncoder(k reloc, marker syncMarker) encoder {
e := pw.newEncoderRaw(k)
e.sync(marker)
return e
}
func (pw *pkgEncoder) newEncoderRaw(k reloc) encoder {
idx := len(pw.elems[k])
pw.elems[k] = append(pw.elems[k], "") // placeholder
return encoder{
p: pw,
k: k,
idx: idx,
}
}
// Encoders
type encoder struct {
p *pkgEncoder
relocs []relocEnt
data bytes.Buffer
k reloc
idx int
}
func (w *encoder) flush() int {
var sb bytes.Buffer // TODO(mdempsky): strings.Builder after #44505 is resolved
// Backup the data so we write the relocations at the front.
var tmp bytes.Buffer
io.Copy(&tmp, &w.data)
// TODO(mdempsky): Consider writing these out separately so they're
// easier to strip, along with function bodies, so that we can prune
// down to just the data that's relevant to go/types.
w.sync(syncRelocs)
w.len(len(w.relocs))
for _, rent := range w.relocs {
w.sync(syncReloc)
w.len(int(rent.kind))
w.len(rent.idx)
}
io.Copy(&sb, &w.data)
io.Copy(&sb, &tmp)
w.p.elems[w.k][w.idx] = sb.String()
return w.idx
}
func (w *encoder) checkErr(err error) {
if err != nil {
base.Fatalf("unexpected error: %v", err)
}
}
func (w *encoder) sync(m syncMarker) {
if debug {
err := w.data.WriteByte(byte(m))
w.checkErr(err)
}
}
func (w *encoder) bool(b bool) bool {
w.sync(syncBool)
var x byte
if b {
x = 1
}
err := w.data.WriteByte(x)
w.checkErr(err)
return b
}
func (w *encoder) int64(x int64) {
w.sync(syncInt64)
var buf [binary.MaxVarintLen64]byte
n := binary.PutVarint(buf[:], x)
_, err := w.data.Write(buf[:n])
w.checkErr(err)
}
func (w *encoder) uint64(x uint64) {
w.sync(syncUint64)
var buf [binary.MaxVarintLen64]byte
n := binary.PutUvarint(buf[:], x)
_, err := w.data.Write(buf[:n])
w.checkErr(err)
}
func (w *encoder) len(x int) { assert(x >= 0); w.uint64(uint64(x)) }
func (w *encoder) int(x int) { w.int64(int64(x)) }
func (w *encoder) uint(x uint) { w.uint64(uint64(x)) }
func (w *encoder) reloc(r reloc, idx int) {
w.sync(syncUseReloc)
// TODO(mdempsky): Use map for lookup.
for i, rent := range w.relocs {
if rent.kind == r && rent.idx == idx {
w.len(i)
return
}
}
w.len(len(w.relocs))
w.relocs = append(w.relocs, relocEnt{r, idx})
}
func (w *encoder) code(c code) {
w.sync(c.marker())
w.len(c.value())
}
func (w *encoder) string(s string) {
w.sync(syncString)
w.reloc(relocString, w.p.stringIdx(s))
}
func (w *encoder) strings(ss []string) {
w.len(len(ss))
for _, s := range ss {
w.string(s)
}
}
func (w *encoder) rawValue(val constant.Value) {
if w.bool(val.Kind() == constant.Complex) {
w.scalar(constant.Real(val))
w.scalar(constant.Imag(val))
} else {
w.scalar(val)
}
}
func (w *encoder) scalar(val constant.Value) {
switch v := constant.Val(val).(type) {
default:
panic(fmt.Sprintf("unhandled %v (%v)", val, val.Kind()))
case bool:
w.code(valBool)
w.bool(v)
case string:
w.code(valString)
w.string(v)
case int64:
w.code(valInt64)
w.int64(v)
case *big.Int:
w.code(valBigInt)
w.bigInt(v)
case *big.Rat:
w.code(valBigRat)
w.bigInt(v.Num())
w.bigInt(v.Denom())
case *big.Float:
w.code(valBigFloat)
w.bigFloat(v)
}
}
func (w *encoder) bigInt(v *big.Int) {
b := v.Bytes()
w.string(string(b)) // TODO: More efficient encoding.
w.bool(v.Sign() < 0)
}
func (w *encoder) bigFloat(v *big.Float) {
b := v.Append(nil, 'p', -1)
w.string(string(b)) // TODO: More efficient encoding.
}

296
src/cmd/compile/internal/noder/linker.go Normal file
View File

@ -0,0 +1,296 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"io"
"cmd/compile/internal/base"
"cmd/compile/internal/ir"
"cmd/compile/internal/reflectdata"
"cmd/compile/internal/types"
"cmd/internal/goobj"
"cmd/internal/obj"
)
// This file implements the unified IR linker, which combines the
// local package's stub data with imported package data to produce a
// complete export data file. It also rewrites the compiler's
// extension data sections based on the results of compilation (e.g.,
// the function inlining cost and linker symbol index assignments).
//
// TODO(mdempsky): Using the name "linker" here is confusing, because
// readers are likely to mistake references to it for cmd/link. But
// there's a shortage of good names for "something that combines
// multiple parts into a cohesive whole"... e.g., "assembler" and
// "compiler" are also already taken.
type linker struct {
pw pkgEncoder
pkgs map[string]int
decls map[*types.Sym]int
}
func (l *linker) relocAll(pr *pkgReader, relocs []relocEnt) []relocEnt {
res := make([]relocEnt, len(relocs))
for i, rent := range relocs {
rent.idx = l.relocIdx(pr, rent.kind, rent.idx)
res[i] = rent
}
return res
}
func (l *linker) relocIdx(pr *pkgReader, k reloc, idx int) int {
assert(pr != nil)
absIdx := pr.absIdx(k, idx)
if newidx := pr.newindex[absIdx]; newidx != 0 {
return ^newidx
}
var newidx int
switch k {
case relocString:
newidx = l.relocString(pr, idx)
case relocPkg:
newidx = l.relocPkg(pr, idx)
case relocObj:
newidx = l.relocObj(pr, idx)
default:
// Generic relocations.
//
// TODO(mdempsky): Deduplicate more sections? In fact, I think
// every section could be deduplicated. This would also be easier
// if we do external relocations.
w := l.pw.newEncoderRaw(k)
l.relocCommon(pr, &w, k, idx)
newidx = w.idx
}
pr.newindex[absIdx] = ^newidx
return newidx
}
func (l *linker) relocString(pr *pkgReader, idx int) int {
return l.pw.stringIdx(pr.stringIdx(idx))
}
func (l *linker) relocPkg(pr *pkgReader, idx int) int {
path := pr.peekPkgPath(idx)
if newidx, ok := l.pkgs[path]; ok {
return newidx
}
r := pr.newDecoder(relocPkg, idx, syncPkgDef)
w := l.pw.newEncoder(relocPkg, syncPkgDef)
l.pkgs[path] = w.idx
// TODO(mdempsky): We end up leaving an empty string reference here
// from when the package was originally written as "". Probably not
// a big deal, but a little annoying. Maybe relocating
// cross-references in place is the way to go after all.
w.relocs = l.relocAll(pr, r.relocs)
_ = r.string() // original path
w.string(path)
io.Copy(&w.data, &r.data)
return w.flush()
}
func (l *linker) relocObj(pr *pkgReader, idx int) int {
path, name, tag, _ := pr.peekObj(idx)
sym := types.NewPkg(path, "").Lookup(name)
if newidx, ok := l.decls[sym]; ok {
return newidx
}
if tag == objStub && path != "builtin" && path != "unsafe" {
pri, ok := objReader[sym]
if !ok {
base.Fatalf("missing reader for %q.%v", path, name)
}
assert(ok)
pr = pri.pr
idx = pri.idx
path2, name2, tag2, _ := pr.peekObj(idx)
sym2 := types.NewPkg(path2, "").Lookup(name2)
assert(sym == sym2)
assert(tag2 != objStub)
}
w := l.pw.newEncoderRaw(relocObj)
bside := l.pw.newEncoderRaw(relocObjExt)
assert(bside.idx == w.idx)
l.decls[sym] = w.idx
l.relocCommon(pr, &w, relocObj, idx)
var obj *ir.Name
if path == "" {
var ok bool
obj, ok = sym.Def.(*ir.Name)
// Generic types and functions won't have definitions.
// For now, just generically copy their extension data.
if !ok && base.Flag.G == 0 {
base.Fatalf("missing definition for %v", sym)
}
}
if obj != nil {
bside.sync(syncObject1)
switch tag {
case objFunc:
l.relocFuncExt(&bside, obj)
case objType:
l.relocTypeExt(&bside, obj)
case objVar:
l.relocVarExt(&bside, obj)
}
bside.flush()
} else {
l.relocCommon(pr, &bside, relocObjExt, idx)
}
return w.idx
}
func (l *linker) relocCommon(pr *pkgReader, w *encoder, k reloc, idx int) {
r := pr.newDecoderRaw(k, idx)
w.relocs = l.relocAll(pr, r.relocs)
io.Copy(&w.data, &r.data)
w.flush()
}
func (l *linker) pragmaFlag(w *encoder, pragma ir.PragmaFlag) {
w.sync(syncPragma)
w.int(int(pragma))
}
func (l *linker) relocFuncExt(w *encoder, name *ir.Name) {
w.sync(syncFuncExt)
l.pragmaFlag(w, name.Func.Pragma)
l.linkname(w, name)
// Relocated extension data.
w.bool(true)
// Record definition ABI so cross-ABI calls can be direct.
// This is important for the performance of calling some
// common functions implemented in assembly (e.g., bytealg).
w.uint64(uint64(name.Func.ABI))
// Escape analysis.
for _, fs := range &types.RecvsParams {
for _, f := range fs(name.Type()).FieldSlice() {
w.string(f.Note)
}
}
if inl := name.Func.Inl; w.bool(inl != nil) {
w.len(int(inl.Cost))
w.bool(inl.CanDelayResults)
pri, ok := bodyReader[name.Func]
assert(ok)
w.sync(syncAddBody)
w.reloc(relocBody, l.relocIdx(pri.pr, relocBody, pri.idx))
}
w.sync(syncEOF)
}
func (l *linker) relocTypeExt(w *encoder, name *ir.Name) {
w.sync(syncTypeExt)
typ := name.Type()
l.pragmaFlag(w, name.Pragma())
// For type T, export the index of type descriptor symbols of T and *T.
l.lsymIdx(w, "", reflectdata.TypeLinksym(typ))
l.lsymIdx(w, "", reflectdata.TypeLinksym(typ.PtrTo()))
if typ.Kind() != types.TINTER {
for _, method := range typ.Methods().Slice() {
l.relocFuncExt(w, method.Nname.(*ir.Name))
}
}
}
func (l *linker) relocVarExt(w *encoder, name *ir.Name) {
w.sync(syncVarExt)
l.linkname(w, name)
}
func (l *linker) linkname(w *encoder, name *ir.Name) {
w.sync(syncLinkname)
linkname := name.Sym().Linkname
if !l.lsymIdx(w, linkname, name.Linksym()) {
w.string(linkname)
}
}
func (l *linker) lsymIdx(w *encoder, linkname string, lsym *obj.LSym) bool {
if lsym.PkgIdx > goobj.PkgIdxSelf || (lsym.PkgIdx == goobj.PkgIdxInvalid && !lsym.Indexed()) || linkname != "" {
w.int64(-1)
return false
}
// For a defined symbol, export its index.
// For re-exporting an imported symbol, pass its index through.
w.int64(int64(lsym.SymIdx))
return true
}
// @@@ Helpers
// TODO(mdempsky): These should probably be removed. I think they're a
// smell that the export data format is not yet quite right.
func (pr *pkgDecoder) peekPkgPath(idx int) string {
r := pr.newDecoder(relocPkg, idx, syncPkgDef)
path := r.string()
if path == "" {
path = pr.pkgPath
}
return path
}
func (pr *pkgDecoder) peekObj(idx int) (string, string, codeObj, []int) {
r := pr.newDecoder(relocObj, idx, syncObject1)
r.sync(syncSym)
r.sync(syncPkg)
path := pr.peekPkgPath(r.reloc(relocPkg))
name := r.string()
assert(name != "")
r.sync(syncTypeParamBounds)
r.len() // implicits
bounds := make([]int, r.len())
for i := range bounds {
r.sync(syncType)
bounds[i] = r.reloc(relocType)
}
tag := codeObj(r.code(syncCodeObj))
return path, name, tag, bounds
}

453
src/cmd/compile/internal/noder/quirks.go Normal file
View File

@ -0,0 +1,453 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"fmt"
"cmd/compile/internal/base"
"cmd/compile/internal/ir"
"cmd/compile/internal/syntax"
"cmd/compile/internal/types2"
"cmd/internal/src"
)
// This file defines helper functions useful for satisfying toolstash
// -cmp when compared against the legacy frontend behavior, but can be
// removed after that's no longer a concern.
// quirksMode controls whether behavior specific to satsifying
// toolstash -cmp is used.
func quirksMode() bool {
// Currently, unified IR doesn't try to be compatible with
// -d=inlfuncswithclosures=1, so we overload this as a flag for
// enabling quirks mode.
return base.Debug.InlFuncsWithClosures == 0
}
// posBasesOf returns all of the position bases in the source files,
// as seen in a straightforward traversal.
//
// This is necessary to ensure position bases (and thus file names)
// get registered in the same order as noder would visit them.
func posBasesOf(noders []*noder) []*syntax.PosBase {
seen := make(map[*syntax.PosBase]bool)
var bases []*syntax.PosBase
for _, p := range noders {
syntax.Walk(p.file, func(n syntax.Node) bool {
if b := n.Pos().Base(); !seen[b] {
bases = append(bases, b)
seen[b] = true
}
return false
})
}
return bases
}
// importedObjsOf returns the imported objects (i.e., referenced
// objects not declared by curpkg) from the parsed source files, in
// the order that typecheck used to load their definitions.
//
// This is needed because loading the definitions for imported objects
// can also add file names.
func importedObjsOf(curpkg *types2.Package, info *types2.Info, noders []*noder) []types2.Object {
// This code is complex because it matches the precise order that
// typecheck recursively and repeatedly traverses the IR. It's meant
// to be thrown away eventually anyway.
seen := make(map[types2.Object]bool)
var objs []types2.Object
var phase int
decls := make(map[types2.Object]syntax.Decl)
assoc := func(decl syntax.Decl, names ...*syntax.Name) {
for _, name := range names {
obj, ok := info.Defs[name]
assert(ok)
decls[obj] = decl
}
}
for _, p := range noders {
syntax.Walk(p.file, func(n syntax.Node) bool {
switch n := n.(type) {
case *syntax.ConstDecl:
assoc(n, n.NameList...)
case *syntax.FuncDecl:
assoc(n, n.Name)
case *syntax.TypeDecl:
assoc(n, n.Name)
case *syntax.VarDecl:
assoc(n, n.NameList...)
case *syntax.BlockStmt:
return true
}
return false
})
}
var visited map[syntax.Decl]bool
var resolveDecl func(n syntax.Decl)
var resolveNode func(n syntax.Node, top bool)
resolveDecl = func(n syntax.Decl) {
if visited[n] {
return
}
visited[n] = true
switch n := n.(type) {
case *syntax.ConstDecl:
resolveNode(n.Type, true)
resolveNode(n.Values, true)
case *syntax.FuncDecl:
if n.Recv != nil {
resolveNode(n.Recv, true)
}
resolveNode(n.Type, true)
case *syntax.TypeDecl:
resolveNode(n.Type, true)
case *syntax.VarDecl:
if n.Type != nil {
resolveNode(n.Type, true)
} else {
resolveNode(n.Values, true)
}
}
}
resolveObj := func(pos syntax.Pos, obj types2.Object) {
switch obj.Pkg() {
case nil:
// builtin; nothing to do
case curpkg:
if decl, ok := decls[obj]; ok {
resolveDecl(decl)
}
default:
if obj.Parent() == obj.Pkg().Scope() && !seen[obj] {
seen[obj] = true
objs = append(objs, obj)
}
}
}
checkdefat := func(pos syntax.Pos, n *syntax.Name) {
if n.Value == "_" {
return
}
obj, ok := info.Uses[n]
if !ok {
obj, ok = info.Defs[n]
if !ok {
return
}
}
if obj == nil {
return
}
resolveObj(pos, obj)
}
checkdef := func(n *syntax.Name) { checkdefat(n.Pos(), n) }
var later []syntax.Node
resolveNode = func(n syntax.Node, top bool) {
if n == nil {
return
}
syntax.Walk(n, func(n syntax.Node) bool {
switch n := n.(type) {
case *syntax.Name:
checkdef(n)
case *syntax.SelectorExpr:
if name, ok := n.X.(*syntax.Name); ok {
if _, isPkg := info.Uses[name].(*types2.PkgName); isPkg {
checkdefat(n.X.Pos(), n.Sel)
return true
}
}
case *syntax.AssignStmt:
resolveNode(n.Rhs, top)
resolveNode(n.Lhs, top)
return true
case *syntax.VarDecl:
resolveNode(n.Values, top)
case *syntax.FuncLit:
if top {
resolveNode(n.Type, top)
later = append(later, n.Body)
return true
}
case *syntax.BlockStmt:
if phase >= 3 {
for _, stmt := range n.List {
resolveNode(stmt, false)
}
}
return true
}
return false
})
}
for phase = 1; phase <= 5; phase++ {
visited = map[syntax.Decl]bool{}
for _, p := range noders {
for _, decl := range p.file.DeclList {
switch decl := decl.(type) {
case *syntax.ConstDecl:
resolveDecl(decl)
case *syntax.FuncDecl:
resolveDecl(decl)
if phase >= 3 && decl.Body != nil {
resolveNode(decl.Body, true)
}
case *syntax.TypeDecl:
if !decl.Alias || phase >= 2 {
resolveDecl(decl)
}
case *syntax.VarDecl:
if phase >= 2 {
resolveNode(decl.Values, true)
resolveDecl(decl)
}
}
}
if phase >= 5 {
syntax.Walk(p.file, func(n syntax.Node) bool {
if name, ok := n.(*syntax.Name); ok {
if obj, ok := info.Uses[name]; ok {
resolveObj(name.Pos(), obj)
}
}
return false
})
}
}
for i := 0; i < len(later); i++ {
resolveNode(later[i], true)
}
later = nil
}
return objs
}
// typeExprEndPos returns the position that noder would leave base.Pos
// after parsing the given type expression.
func typeExprEndPos(expr0 syntax.Expr) syntax.Pos {
for {
switch expr := expr0.(type) {
case *syntax.Name:
return expr.Pos()
case *syntax.SelectorExpr:
return expr.X.Pos()
case *syntax.ParenExpr:
expr0 = expr.X
case *syntax.Operation:
assert(expr.Op == syntax.Mul)
assert(expr.Y == nil)
expr0 = expr.X
case *syntax.ArrayType:
expr0 = expr.Elem
case *syntax.ChanType:
expr0 = expr.Elem
case *syntax.DotsType:
expr0 = expr.Elem
case *syntax.MapType:
expr0 = expr.Value
case *syntax.SliceType:
expr0 = expr.Elem
case *syntax.StructType:
return expr.Pos()
case *syntax.InterfaceType:
expr0 = lastFieldType(expr.MethodList)
if expr0 == nil {
return expr.Pos()
}
case *syntax.FuncType:
expr0 = lastFieldType(expr.ResultList)
if expr0 == nil {
expr0 = lastFieldType(expr.ParamList)
if expr0 == nil {
return expr.Pos()
}
}
case *syntax.IndexExpr: // explicit type instantiation
targs := unpackListExpr(expr.Index)
expr0 = targs[len(targs)-1]
default:
panic(fmt.Sprintf("%s: unexpected type expression %v", expr.Pos(), syntax.String(expr)))
}
}
}
func lastFieldType(fields []*syntax.Field) syntax.Expr {
if len(fields) == 0 {
return nil
}
return fields[len(fields)-1].Type
}
// sumPos returns the position that noder.sum would produce for
// constant expression x.
func sumPos(x syntax.Expr) syntax.Pos {
orig := x
for {
switch x1 := x.(type) {
case *syntax.BasicLit:
assert(x1.Kind == syntax.StringLit)
return x1.Pos()
case *syntax.Operation:
assert(x1.Op == syntax.Add && x1.Y != nil)
if r, ok := x1.Y.(*syntax.BasicLit); ok {
assert(r.Kind == syntax.StringLit)
x = x1.X
continue
}
}
return orig.Pos()
}
}
// funcParamsEndPos returns the value of base.Pos left by noder after
// processing a function signature.
func funcParamsEndPos(fn *ir.Func) src.XPos {
sig := fn.Nname.Type()
fields := sig.Results().FieldSlice()
if len(fields) == 0 {
fields = sig.Params().FieldSlice()
if len(fields) == 0 {
fields = sig.Recvs().FieldSlice()
if len(fields) == 0 {
if fn.OClosure != nil {
return fn.Nname.Ntype.Pos()
}
return fn.Pos()
}
}
}
return fields[len(fields)-1].Pos
}
type dupTypes struct {
origs map[types2.Type]types2.Type
}
func (d *dupTypes) orig(t types2.Type) types2.Type {
if orig, ok := d.origs[t]; ok {
return orig
}
return t
}
func (d *dupTypes) add(t, orig types2.Type) {
if t == orig {
return
}
if d.origs == nil {
d.origs = make(map[types2.Type]types2.Type)
}
assert(d.origs[t] == nil)
d.origs[t] = orig
switch t := t.(type) {
case *types2.Pointer:
orig := orig.(*types2.Pointer)
d.add(t.Elem(), orig.Elem())
case *types2.Slice:
orig := orig.(*types2.Slice)
d.add(t.Elem(), orig.Elem())
case *types2.Map:
orig := orig.(*types2.Map)
d.add(t.Key(), orig.Key())
d.add(t.Elem(), orig.Elem())
case *types2.Array:
orig := orig.(*types2.Array)
assert(t.Len() == orig.Len())
d.add(t.Elem(), orig.Elem())
case *types2.Chan:
orig := orig.(*types2.Chan)
assert(t.Dir() == orig.Dir())
d.add(t.Elem(), orig.Elem())
case *types2.Struct:
orig := orig.(*types2.Struct)
assert(t.NumFields() == orig.NumFields())
for i := 0; i < t.NumFields(); i++ {
d.add(t.Field(i).Type(), orig.Field(i).Type())
}
case *types2.Interface:
orig := orig.(*types2.Interface)
assert(t.NumExplicitMethods() == orig.NumExplicitMethods())
assert(t.NumEmbeddeds() == orig.NumEmbeddeds())
for i := 0; i < t.NumExplicitMethods(); i++ {
d.add(t.ExplicitMethod(i).Type(), orig.ExplicitMethod(i).Type())
}
for i := 0; i < t.NumEmbeddeds(); i++ {
d.add(t.EmbeddedType(i), orig.EmbeddedType(i))
}
case *types2.Signature:
orig := orig.(*types2.Signature)
assert((t.Recv() == nil) == (orig.Recv() == nil))
if t.Recv() != nil {
d.add(t.Recv().Type(), orig.Recv().Type())
}
d.add(t.Params(), orig.Params())
d.add(t.Results(), orig.Results())
case *types2.Tuple:
orig := orig.(*types2.Tuple)
assert(t.Len() == orig.Len())
for i := 0; i < t.Len(); i++ {
d.add(t.At(i).Type(), orig.At(i).Type())
}
default:
assert(types2.Identical(t, orig))
}
}

1970
src/cmd/compile/internal/noder/reader.go Normal file
View File

File diff suppressed because it is too large Load Diff

463
src/cmd/compile/internal/noder/reader2.go Normal file
View File

@ -0,0 +1,463 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"go/constant"
"cmd/compile/internal/base"
"cmd/compile/internal/syntax"
"cmd/compile/internal/types2"
"cmd/internal/src"
)
type pkgReader2 struct {
pkgDecoder
check *types2.Checker
imports map[string]*types2.Package
posBases []*syntax.PosBase
pkgs []*types2.Package
typs []types2.Type
}
func readPackage2(check *types2.Checker, imports map[string]*types2.Package, input pkgDecoder) *types2.Package {
pr := pkgReader2{
pkgDecoder: input,
check: check,
imports: imports,
posBases: make([]*syntax.PosBase, input.numElems(relocPosBase)),
pkgs: make([]*types2.Package, input.numElems(relocPkg)),
typs: make([]types2.Type, input.numElems(relocType)),
}
r := pr.newReader(relocMeta, publicRootIdx, syncPublic)
pkg := r.pkg()
r.bool() // has init
for i, n := 0, r.len(); i < n; i++ {
r.obj()
}
r.sync(syncEOF)
pkg.MarkComplete()
return pkg
}
type reader2 struct {
decoder
p *pkgReader2
tparams []*types2.TypeName
}
func (pr *pkgReader2) newReader(k reloc, idx int, marker syncMarker) *reader2 {
return &reader2{
decoder: pr.newDecoder(k, idx, marker),
p: pr,
}
}
// @@@ Positions
func (r *reader2) pos() syntax.Pos {
r.sync(syncPos)
if !r.bool() {
return syntax.Pos{}
}
// TODO(mdempsky): Delta encoding.
posBase := r.posBase()
line := r.uint()
col := r.uint()
return syntax.MakePos(posBase, line, col)
}
func (r *reader2) posBase() *syntax.PosBase {
return r.p.posBaseIdx(r.reloc(relocPosBase))
}
func (pr *pkgReader2) posBaseIdx(idx int) *syntax.PosBase {
if b := pr.posBases[idx]; b != nil {
return b
}
r := pr.newReader(relocPosBase, idx, syncPosBase)
var b *syntax.PosBase
filename := r.string()
_ = r.string() // absolute file name
if r.bool() {
b = syntax.NewFileBase(filename)
} else {
pos := r.pos()
line := r.uint()
col := r.uint()
b = syntax.NewLineBase(pos, filename, line, col)
}
pr.posBases[idx] = b
return b
}
// @@@ Packages
func (r *reader2) pkg() *types2.Package {
r.sync(syncPkg)
return r.p.pkgIdx(r.reloc(relocPkg))
}
func (pr *pkgReader2) pkgIdx(idx int) *types2.Package {
// TODO(mdempsky): Consider using some non-nil pointer to indicate
// the universe scope, so we don't need to keep re-reading it.
if pkg := pr.pkgs[idx]; pkg != nil {
return pkg
}
pkg := pr.newReader(relocPkg, idx, syncPkgDef).doPkg()
pr.pkgs[idx] = pkg
return pkg
}
func (r *reader2) doPkg() *types2.Package {
path := r.string()
if path == "builtin" {
return nil // universe
}
if path == "" {
path = r.p.pkgPath
}
if pkg := r.p.imports[path]; pkg != nil {
return pkg
}
name := r.string()
height := r.len()
pkg := types2.NewPackageHeight(path, name, height)
r.p.imports[path] = pkg
// TODO(mdempsky): The list of imported packages is important for
// go/types, but we could probably skip populating it for types2.
imports := make([]*types2.Package, r.len())
for i := range imports {
imports[i] = r.pkg()
}
pkg.SetImports(imports)
return pkg
}
// @@@ Types
func (r *reader2) typ() types2.Type {
r.sync(syncType)
return r.p.typIdx(r.reloc(relocType), r.tparams)
}
func (pr *pkgReader2) typIdx(idx int, tparams []*types2.TypeName) types2.Type {
if typ := pr.typs[idx]; typ != nil {
return typ
}
r := pr.newReader(relocType, idx, syncTypeIdx)
r.tparams = tparams
typ := r.doTyp()
assert(typ != nil)
if pr.typs[idx] != nil {
// See comment in pkgReader.typIdx.
return pr.typs[idx]
}
if len(tparams) == 0 {
pr.typs[idx] = typ
}
return typ
}
func (r *reader2) doTyp() (res types2.Type) {
switch tag := codeType(r.code(syncType)); tag {
default:
base.FatalfAt(src.NoXPos, "unhandled type tag: %v", tag)
panic("unreachable")
case typeBasic:
return types2.Typ[r.len()]
case typeNamed:
obj, targs := r.obj()
name := obj.(*types2.TypeName)
if len(targs) != 0 {
return r.p.check.InstantiateLazy(syntax.Pos{}, name.Type(), targs)
}
return name.Type()
case typeTypeParam:
idx := r.len()
return r.tparams[idx].Type().(*types2.TypeParam)
case typeArray:
len := int64(r.uint64())
return types2.NewArray(r.typ(), len)
case typeChan:
dir := types2.ChanDir(r.len())
return types2.NewChan(dir, r.typ())
case typeMap:
return types2.NewMap(r.typ(), r.typ())
case typePointer:
return types2.NewPointer(r.typ())
case typeSignature:
return r.signature(nil)
case typeSlice:
return types2.NewSlice(r.typ())
case typeStruct:
return r.structType()
case typeInterface:
return r.interfaceType()
case typeUnion:
return r.unionType()
}
}
func (r *reader2) structType() *types2.Struct {
fields := make([]*types2.Var, r.len())
var tags []string
for i := range fields {
pos := r.pos()
pkg, name := r.selector()
ftyp := r.typ()
tag := r.string()
embedded := r.bool()
fields[i] = types2.NewField(pos, pkg, name, ftyp, embedded)
if tag != "" {
for len(tags) < i {
tags = append(tags, "")
}
tags = append(tags, tag)
}
}
return types2.NewStruct(fields, tags)
}
func (r *reader2) unionType() *types2.Union {
terms := make([]types2.Type, r.len())
tildes := make([]bool, len(terms))
for i := range terms {
terms[i] = r.typ()
tildes[i] = r.bool()
}
return types2.NewUnion(terms, tildes)
}
func (r *reader2) interfaceType() *types2.Interface {
methods := make([]*types2.Func, r.len())
embeddeds := make([]types2.Type, r.len())
for i := range methods {
pos := r.pos()
pkg, name := r.selector()
mtyp := r.signature(nil)
methods[i] = types2.NewFunc(pos, pkg, name, mtyp)
}
for i := range embeddeds {
embeddeds[i] = r.typ()
}
typ := types2.NewInterfaceType(methods, embeddeds)
typ.Complete()
return typ
}
func (r *reader2) signature(recv *types2.Var) *types2.Signature {
r.sync(syncSignature)
params := r.params()
results := r.params()
variadic := r.bool()
return types2.NewSignature(recv, params, results, variadic)
}
func (r *reader2) params() *types2.Tuple {
r.sync(syncParams)
params := make([]*types2.Var, r.len())
for i := range params {
params[i] = r.param()
}
return types2.NewTuple(params...)
}
func (r *reader2) param() *types2.Var {
r.sync(syncParam)
pos := r.pos()
pkg, name := r.localIdent()
typ := r.typ()
return types2.NewParam(pos, pkg, name, typ)
}
// @@@ Objects
func (r *reader2) obj() (types2.Object, []types2.Type) {
r.sync(syncObject)
pkg, name := r.p.objIdx(r.reloc(relocObj))
obj := pkg.Scope().Lookup(name)
targs := make([]types2.Type, r.len())
for i := range targs {
targs[i] = r.typ()
}
return obj, targs
}
func (pr *pkgReader2) objIdx(idx int) (*types2.Package, string) {
r := pr.newReader(relocObj, idx, syncObject1)
objPkg, objName := r.qualifiedIdent()
assert(objName != "")
bounds := r.typeParamBounds()
tag := codeObj(r.code(syncCodeObj))
if tag == objStub {
assert(objPkg == nil)
return objPkg, objName
}
objPkg.Scope().InsertLazy(objName, func() types2.Object {
switch tag {
default:
panic("weird")
case objAlias:
pos := r.pos()
typ := r.typ()
return types2.NewTypeName(pos, objPkg, objName, typ)
case objConst:
pos := r.pos()
typ, val := r.value()
return types2.NewConst(pos, objPkg, objName, typ, val)
case objFunc:
pos := r.pos()
r.typeParamNames(bounds)
sig := r.signature(nil)
if len(r.tparams) != 0 {
sig.SetTParams(r.tparams)
}
return types2.NewFunc(pos, objPkg, objName, sig)
case objType:
pos := r.pos()
return types2.NewTypeNameLazy(pos, objPkg, objName, func(named *types2.Named) (tparams []*types2.TypeName, underlying types2.Type, methods []*types2.Func) {
r.typeParamNames(bounds)
if len(r.tparams) != 0 {
tparams = r.tparams
}
// TODO(mdempsky): Rewrite receiver types to underlying is an
// Interface? The go/types importer does this (I think because
// unit tests expected that), but cmd/compile doesn't care
// about it, so maybe we can avoid worrying about that here.
underlying = r.typ().Underlying()
methods = make([]*types2.Func, r.len())
for i := range methods {
methods[i] = r.method(bounds)
}
return
})
case objVar:
pos := r.pos()
typ := r.typ()
return types2.NewVar(pos, objPkg, objName, typ)
}
})
return objPkg, objName
}
func (r *reader2) value() (types2.Type, constant.Value) {
r.sync(syncValue)
return r.typ(), r.rawValue()
}
func (r *reader2) typeParamBounds() []int {
r.sync(syncTypeParamBounds)
// exported types never have implicit type parameters
// TODO(mdempsky): Hide this from public importer.
assert(r.len() == 0)
bounds := make([]int, r.len())
for i := range bounds {
r.sync(syncType)
bounds[i] = r.reloc(relocType)
}
return bounds
}
func (r *reader2) typeParamNames(bounds []int) {
r.sync(syncTypeParamNames)
r.tparams = make([]*types2.TypeName, len(bounds))
for i := range r.tparams {
pos := r.pos()
pkg, name := r.localIdent()
obj := types2.NewTypeName(pos, pkg, name, nil)
r.p.check.NewTypeParam(obj, i, nil)
r.tparams[i] = obj
}
for i, tparam := range r.tparams {
bound := r.p.typIdx(bounds[i], r.tparams)
tparam.Type().(*types2.TypeParam).SetBound(bound)
}
}
func (r *reader2) method(bounds []int) *types2.Func {
r.sync(syncMethod)
pos := r.pos()
pkg, name := r.selector()
r.typeParamNames(bounds)
sig := r.signature(r.param())
if len(r.tparams) != 0 {
sig.SetRParams(r.tparams)
}
_ = r.pos() // TODO(mdempsky): Remove; this is a hacker for linker.go.
return types2.NewFunc(pos, pkg, name, sig)
}
func (r *reader2) qualifiedIdent() (*types2.Package, string) { return r.ident(syncSym) }
func (r *reader2) localIdent() (*types2.Package, string) { return r.ident(syncLocalIdent) }
func (r *reader2) selector() (*types2.Package, string) { return r.ident(syncSelector) }
func (r *reader2) ident(marker syncMarker) (*types2.Package, string) {
r.sync(marker)
return r.pkg(), r.string()
}

40
src/cmd/compile/internal/noder/reloc.go Normal file
View File

@ -0,0 +1,40 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
// A reloc indicates a particular section within a unified IR export.
//
// TODO(mdempsky): Rename to "section" or something similar?
type reloc int
// A relocEnt (relocation entry) is an entry in an atom's local
// reference table.
//
// TODO(mdempsky): Rename this too.
type relocEnt struct {
kind reloc
idx int
}
// Reserved indices within the meta relocation section.
const (
publicRootIdx = 0
privateRootIdx = 1
)
const (
relocString reloc = iota
relocMeta
relocPosBase
relocPkg
relocType
relocObj
relocObjExt
relocBody
numRelocs = iota
)

154
src/cmd/compile/internal/noder/sync.go Normal file
View File

@ -0,0 +1,154 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
const debug = true
type syncMarker int
//go:generate stringer -type=syncMarker -trimprefix=sync
// TODO(mdempsky): Cleanup unneeded sync markers.
// TODO(mdempsky): Split these markers into public/stable markers, and
// private ones. Also, trim unused ones.
const (
_ syncMarker = iota
syncNode
syncBool
syncInt64
syncUint64
syncString
syncPos
syncPkg
syncSym
syncSelector
syncKind
syncType
syncTypePkg
syncSignature
syncParam
syncOp
syncObject
syncExpr
syncStmt
syncDecl
syncConstDecl
syncFuncDecl
syncTypeDecl
syncVarDecl
syncPragma
syncValue
syncEOF
syncMethod
syncFuncBody
syncUse
syncUseObj
syncObjectIdx
syncTypeIdx
syncBOF
syncEntry
syncOpenScope
syncCloseScope
syncGlobal
syncLocal
syncDefine
syncDefLocal
syncUseLocal
syncDefGlobal
syncUseGlobal
syncTypeParams
syncUseLabel
syncDefLabel
syncFuncLit
syncCommonFunc
syncBodyRef
syncLinksymExt
syncHack
syncSetlineno
syncName
syncImportDecl
syncDeclNames
syncDeclName
syncExprList
syncExprs
syncWrapname
syncTypeExpr
syncTypeExprOrNil
syncChanDir
syncParams
syncCloseAnotherScope
syncSum
syncUnOp
syncBinOp
syncStructType
syncInterfaceType
syncPackname
syncEmbedded
syncStmts
syncStmtsFall
syncStmtFall
syncBlockStmt
syncIfStmt
syncForStmt
syncSwitchStmt
syncRangeStmt
syncCaseClause
syncCommClause
syncSelectStmt
syncDecls
syncLabeledStmt
syncCompLit
sync1
sync2
sync3
sync4
syncN
syncDefImplicit
syncUseName
syncUseObjLocal
syncAddLocal
syncBothSignature
syncSetUnderlying
syncLinkname
syncStmt1
syncStmtsEnd
syncDeclare
syncTopDecls
syncTopConstDecl
syncTopFuncDecl
syncTopTypeDecl
syncTopVarDecl
syncObject1
syncAddBody
syncLabel
syncFuncExt
syncMethExt
syncOptLabel
syncScalar
syncStmtDecls
syncDeclLocal
syncObjLocal
syncObjLocal1
syncDeclareLocal
syncPublic
syncPrivate
syncRelocs
syncReloc
syncUseReloc
syncVarExt
syncPkgDef
syncTypeExt
syncVal
syncCodeObj
syncPosBase
syncLocalIdent
syncTypeParamNames
syncTypeParamBounds
)

152
src/cmd/compile/internal/noder/syncmarker_string.go Normal file
View File

@ -0,0 +1,152 @@
// Code generated by "stringer -type=syncMarker -trimprefix=sync"; DO NOT EDIT.
package noder
import "strconv"
func _() {
// An "invalid array index" compiler error signifies that the constant values have changed.
// Re-run the stringer command to generate them again.
var x [1]struct{}
_ = x[syncNode-1]
_ = x[syncBool-2]
_ = x[syncInt64-3]
_ = x[syncUint64-4]
_ = x[syncString-5]
_ = x[syncPos-6]
_ = x[syncPkg-7]
_ = x[syncSym-8]
_ = x[syncSelector-9]
_ = x[syncKind-10]
_ = x[syncType-11]
_ = x[syncTypePkg-12]
_ = x[syncSignature-13]
_ = x[syncParam-14]
_ = x[syncOp-15]
_ = x[syncObject-16]
_ = x[syncExpr-17]
_ = x[syncStmt-18]
_ = x[syncDecl-19]
_ = x[syncConstDecl-20]
_ = x[syncFuncDecl-21]
_ = x[syncTypeDecl-22]
_ = x[syncVarDecl-23]
_ = x[syncPragma-24]
_ = x[syncValue-25]
_ = x[syncEOF-26]
_ = x[syncMethod-27]
_ = x[syncFuncBody-28]
_ = x[syncUse-29]
_ = x[syncUseObj-30]
_ = x[syncObjectIdx-31]
_ = x[syncTypeIdx-32]
_ = x[syncBOF-33]
_ = x[syncEntry-34]
_ = x[syncOpenScope-35]
_ = x[syncCloseScope-36]
_ = x[syncGlobal-37]
_ = x[syncLocal-38]
_ = x[syncDefine-39]
_ = x[syncDefLocal-40]
_ = x[syncUseLocal-41]
_ = x[syncDefGlobal-42]
_ = x[syncUseGlobal-43]
_ = x[syncTypeParams-44]
_ = x[syncUseLabel-45]
_ = x[syncDefLabel-46]
_ = x[syncFuncLit-47]
_ = x[syncCommonFunc-48]
_ = x[syncBodyRef-49]
_ = x[syncLinksymExt-50]
_ = x[syncHack-51]
_ = x[syncSetlineno-52]
_ = x[syncName-53]
_ = x[syncImportDecl-54]
_ = x[syncDeclNames-55]
_ = x[syncDeclName-56]
_ = x[syncExprList-57]
_ = x[syncExprs-58]
_ = x[syncWrapname-59]
_ = x[syncTypeExpr-60]
_ = x[syncTypeExprOrNil-61]
_ = x[syncChanDir-62]
_ = x[syncParams-63]
_ = x[syncCloseAnotherScope-64]
_ = x[syncSum-65]
_ = x[syncUnOp-66]
_ = x[syncBinOp-67]
_ = x[syncStructType-68]
_ = x[syncInterfaceType-69]
_ = x[syncPackname-70]
_ = x[syncEmbedded-71]
_ = x[syncStmts-72]
_ = x[syncStmtsFall-73]
_ = x[syncStmtFall-74]
_ = x[syncBlockStmt-75]
_ = x[syncIfStmt-76]
_ = x[syncForStmt-77]
_ = x[syncSwitchStmt-78]
_ = x[syncRangeStmt-79]
_ = x[syncCaseClause-80]
_ = x[syncCommClause-81]
_ = x[syncSelectStmt-82]
_ = x[syncDecls-83]
_ = x[syncLabeledStmt-84]
_ = x[syncCompLit-85]
_ = x[sync1-86]
_ = x[sync2-87]
_ = x[sync3-88]
_ = x[sync4-89]
_ = x[syncN-90]
_ = x[syncDefImplicit-91]
_ = x[syncUseName-92]
_ = x[syncUseObjLocal-93]
_ = x[syncAddLocal-94]
_ = x[syncBothSignature-95]
_ = x[syncSetUnderlying-96]
_ = x[syncLinkname-97]
_ = x[syncStmt1-98]
_ = x[syncStmtsEnd-99]
_ = x[syncDeclare-100]
_ = x[syncTopDecls-101]
_ = x[syncTopConstDecl-102]
_ = x[syncTopFuncDecl-103]
_ = x[syncTopTypeDecl-104]
_ = x[syncTopVarDecl-105]
_ = x[syncObject1-106]
_ = x[syncAddBody-107]
_ = x[syncLabel-108]
_ = x[syncFuncExt-109]
_ = x[syncMethExt-110]
_ = x[syncOptLabel-111]
_ = x[syncScalar-112]
_ = x[syncStmtDecls-113]
_ = x[syncDeclLocal-114]
_ = x[syncObjLocal-115]
_ = x[syncObjLocal1-116]
_ = x[syncDeclareLocal-117]
_ = x[syncPublic-118]
_ = x[syncPrivate-119]
_ = x[syncRelocs-120]
_ = x[syncReloc-121]
_ = x[syncUseReloc-122]
_ = x[syncVarExt-123]
_ = x[syncPkgDef-124]
_ = x[syncTypeExt-125]
_ = x[syncVal-126]
_ = x[syncCodeObj-127]
_ = x[syncPosBase-128]
_ = x[syncLocalIdent-129]
}
const _syncMarker_name = "NodeBoolInt64Uint64StringPosPkgSymSelectorKindTypeTypePkgSignatureParamOpObjectExprStmtDeclConstDeclFuncDeclTypeDeclVarDeclPragmaValueEOFMethodFuncBodyUseUseObjObjectIdxTypeIdxBOFEntryOpenScopeCloseScopeGlobalLocalDefineDefLocalUseLocalDefGlobalUseGlobalTypeParamsUseLabelDefLabelFuncLitCommonFuncBodyRefLinksymExtHackSetlinenoNameImportDeclDeclNamesDeclNameExprListExprsWrapnameTypeExprTypeExprOrNilChanDirParamsCloseAnotherScopeSumUnOpBinOpStructTypeInterfaceTypePacknameEmbeddedStmtsStmtsFallStmtFallBlockStmtIfStmtForStmtSwitchStmtRangeStmtCaseClauseCommClauseSelectStmtDeclsLabeledStmtCompLit1234NDefImplicitUseNameUseObjLocalAddLocalBothSignatureSetUnderlyingLinknameStmt1StmtsEndDeclareTopDeclsTopConstDeclTopFuncDeclTopTypeDeclTopVarDeclObject1AddBodyLabelFuncExtMethExtOptLabelScalarStmtDeclsDeclLocalObjLocalObjLocal1DeclareLocalPublicPrivateRelocsRelocUseRelocVarExtPkgDefTypeExtValCodeObjPosBaseLocalIdent"
var _syncMarker_index = [...]uint16{0, 4, 8, 13, 19, 25, 28, 31, 34, 42, 46, 50, 57, 66, 71, 73, 79, 83, 87, 91, 100, 108, 116, 123, 129, 134, 137, 143, 151, 154, 160, 169, 176, 179, 184, 193, 203, 209, 214, 220, 228, 236, 245, 254, 264, 272, 280, 287, 297, 304, 314, 318, 327, 331, 341, 350, 358, 366, 371, 379, 387, 400, 407, 413, 430, 433, 437, 442, 452, 465, 473, 481, 486, 495, 503, 512, 518, 525, 535, 544, 554, 564, 574, 579, 590, 597, 598, 599, 600, 601, 602, 613, 620, 631, 639, 652, 665, 673, 678, 686, 693, 701, 713, 724, 735, 745, 752, 759, 764, 771, 778, 786, 792, 801, 810, 818, 827, 839, 845, 852, 858, 863, 871, 877, 883, 890, 893, 900, 907, 917}
func (i syncMarker) String() string {
i -= 1
if i < 0 || i >= syncMarker(len(_syncMarker_index)-1) {
return "syncMarker(" + strconv.FormatInt(int64(i+1), 10) + ")"
}
return _syncMarker_name[_syncMarker_index[i]:_syncMarker_index[i+1]]
}

276
src/cmd/compile/internal/noder/unified.go Normal file
View File

@ -0,0 +1,276 @@
// UNREVIEWED
// Copyright 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package noder
import (
"bytes"
"fmt"
"internal/goversion"
"io"
"runtime"
"sort"
"cmd/compile/internal/base"
"cmd/compile/internal/inline"
"cmd/compile/internal/ir"
"cmd/compile/internal/typecheck"
"cmd/compile/internal/types"
"cmd/compile/internal/types2"
"cmd/internal/src"
)
// localPkgReader holds the package reader used for reading the local
// package. It exists so the unified IR linker can refer back to it
// later.
var localPkgReader *pkgReader
// useUnifiedIR reports whether the unified IR frontend should be
// used; and if so, uses it to construct the local package's IR.
func useUnifiedIR(noders []*noder) {
inline.NewInline = InlineCall
if !quirksMode() {
writeNewExportFunc = writeNewExport
}
newReadImportFunc = func(data string, pkg1 *types.Pkg, check *types2.Checker, packages map[string]*types2.Package) (pkg2 *types2.Package, err error) {
pr := newPkgDecoder(pkg1.Path, data)
// Read package descriptors for both types2 and compiler backend.
readPackage(newPkgReader(pr), pkg1)
pkg2 = readPackage2(check, packages, pr)
return
}
data := writePkgStub(noders)
// We already passed base.Flag.Lang to types2 to handle validating
// the user's source code. Bump it up now to the current version and
// re-parse, so typecheck doesn't complain if we construct IR that
// utilizes newer Go features.
base.Flag.Lang = fmt.Sprintf("go1.%d", goversion.Version)
types.ParseLangFlag()
assert(types.LocalPkg.Path == "")
types.LocalPkg.Height = 0 // reset so pkgReader.pkgIdx doesn't complain
target := typecheck.Target
typecheck.TypecheckAllowed = true
localPkgReader = newPkgReader(newPkgDecoder(types.LocalPkg.Path, data))
readPackage(localPkgReader, types.LocalPkg)
r := localPkgReader.newReader(relocMeta, privateRootIdx, syncPrivate)
r.ext = r
r.pkgInit(types.LocalPkg, target)
// Don't use range--bodyIdx can add closures to todoBodies.
for len(todoBodies) > 0 {
// The order we expand bodies doesn't matter, so pop from the end
// to reduce todoBodies reallocations if it grows further.
fn := todoBodies[len(todoBodies)-1]
todoBodies = todoBodies[:len(todoBodies)-1]
pri, ok := bodyReader[fn]
assert(ok)
pri.funcBody(fn)
// Instantiated generic function: add to Decls for typechecking
// and compilation.
if len(pri.implicits) != 0 && fn.OClosure == nil {
target.Decls = append(target.Decls, fn)
}
}
todoBodies = nil
// Don't use range--typecheck can add closures to Target.Decls.
for i := 0; i < len(target.Decls); i++ {
target.Decls[i] = typecheck.Stmt(target.Decls[i])
}
// Don't use range--typecheck can add closures to Target.Decls.
for i := 0; i < len(target.Decls); i++ {
if fn, ok := target.Decls[i].(*ir.Func); ok {
if base.Flag.W > 1 {
s := fmt.Sprintf("\nbefore typecheck %v", fn)
ir.Dump(s, fn)
}
ir.CurFunc = fn
typecheck.Stmts(fn.Body)
if base.Flag.W > 1 {
s := fmt.Sprintf("\nafter typecheck %v", fn)
ir.Dump(s, fn)
}
}
}
base.ExitIfErrors() // just in case
}
// writePkgStub type checks the given parsed source files and then
// returns
func writePkgStub(noders []*noder) string {
m, pkg, info := checkFiles(noders)
pw := newPkgWriter(m, pkg, info)
pw.collectDecls(noders)
publicRootWriter := pw.newWriter(relocMeta, syncPublic)
privateRootWriter := pw.newWriter(relocMeta, syncPrivate)
assert(publicRootWriter.idx == publicRootIdx)
assert(privateRootWriter.idx == privateRootIdx)
{
w := publicRootWriter
w.pkg(pkg)
w.bool(false) // has init; XXX
scope := pkg.Scope()
names := scope.Names()
w.len(len(names))
for _, name := range scope.Names() {
w.obj(scope.Lookup(name), nil)
}
w.sync(syncEOF)
w.flush()
}
{
w := privateRootWriter
w.ext = w
w.pkgInit(noders)
w.flush()
}
var sb bytes.Buffer // TODO(mdempsky): strings.Builder after #44505 is resolved
pw.dump(&sb)
// At this point, we're done with types2. Make sure the package is
// garbage collected.
freePackage(pkg)
return sb.String()
}
// freePackage ensures the given package is garbage collected.
func freePackage(pkg *types2.Package) {
// Set a finalizer on pkg so we can detect if/when it's collected.
done := make(chan struct{})
runtime.SetFinalizer(pkg, func(*types2.Package) { close(done) })
// Important: objects involved in cycles are not finalized, so zero
// out pkg to break its cycles and allow the finalizer to run.
*pkg = types2.Package{}
// It typically takes just 1 or 2 cycles to release pkg, but it
// doesn't hurt to try a few more times.
for i := 0; i < 10; i++ {
select {
case <-done:
return
default:
runtime.GC()
}
}
base.Fatalf("package never finalized")
}
func readPackage(pr *pkgReader, importpkg *types.Pkg) {
r := pr.newReader(relocMeta, publicRootIdx, syncPublic)
pkg := r.pkg()
assert(pkg == importpkg)
if r.bool() {
sym := pkg.Lookup(".inittask")
task := ir.NewNameAt(src.NoXPos, sym)
task.Class = ir.PEXTERN
sym.Def = task
}
for i, n := 0, r.len(); i < n; i++ {
r.sync(syncObject)
idx := r.reloc(relocObj)
assert(r.len() == 0)
path, name, code, _ := r.p.peekObj(idx)
if code != objStub {
objReader[types.NewPkg(path, "").Lookup(name)] = pkgReaderIndex{pr, idx, nil}
}
}
}
func writeNewExport(out io.Writer) {
l := linker{
pw: newPkgEncoder(),
pkgs: make(map[string]int),
decls: make(map[*types.Sym]int),
}
publicRootWriter := l.pw.newEncoder(relocMeta, syncPublic)
assert(publicRootWriter.idx == publicRootIdx)
var selfPkgIdx int
{
pr := localPkgReader
r := pr.newDecoder(relocMeta, publicRootIdx, syncPublic)
r.sync(syncPkg)
selfPkgIdx = l.relocIdx(pr, relocPkg, r.reloc(relocPkg))
r.bool() // has init
for i, n := 0, r.len(); i < n; i++ {
r.sync(syncObject)
idx := r.reloc(relocObj)
assert(r.len() == 0)
xpath, xname, xtag, _ := pr.peekObj(idx)
assert(xpath == pr.pkgPath)
assert(xtag != objStub)
if types.IsExported(xname) {
l.relocIdx(pr, relocObj, idx)
}
}
r.sync(syncEOF)
}
{
var idxs []int
for _, idx := range l.decls {
idxs = append(idxs, idx)
}
sort.Ints(idxs)
w := publicRootWriter
w.sync(syncPkg)
w.reloc(relocPkg, selfPkgIdx)
w.bool(typecheck.Lookup(".inittask").Def != nil)
w.len(len(idxs))
for _, idx := range idxs {
w.sync(syncObject)
w.reloc(relocObj, idx)
w.len(0)
}
w.sync(syncEOF)
w.flush()
}
l.pw.dump(out)
}

1746
src/cmd/compile/internal/noder/writer.go Normal file
View File

File diff suppressed because it is too large Load Diff