时间:2021年5月19日
前置知识
go build
的实现在 src/cmd/go/internal/work/build.go
,忽略设置编译器类型、环境信息等处理,我们只关注最核心的部分:func runBuild(ctx context.Context, cmd *base.Command, args []string) {
...
var b Builder
...
pkgs := load.PackagesAndErrors(ctx, args)
...
a := &Action{Mode: "go build"}
for _, p := range pkgs {
a.Deps = append(a.Deps, b.AutoAction(ModeBuild, depMode, p))
}
...
b.Do(ctx, a)
}
// An Action represents a single action in the action graph.
type Action struct {
Mode string // description of action operation
Package *load.Package // the package this action works on
Deps []*Action // actions that must happen before this one
Func func(*Builder, context.Context, *Action) error // the action itself (nil = no-op)
...
}
src/cmd/go/internal/work/action.go (b *Builder) CompileAction
方法:for _, p1 := range p.Internal.Imports {
a.Deps = append(a.Deps, b.CompileAction(depMode, depMode, p1))
}
a.Deps
就是 action graph 的“起点”。构造出 action graph 后,将 a 顶点作为“根”进行深度优先遍历,把依赖的 action 依次加入任务队列,最后并发执行 action.Func
。a := &Action{
Mode: "build",
Func: (*Builder).build,
...
}
a := &Action{
Mode: "link",
Func: (*Builder).link,
...
}
...
(*Builder).link
中会调用 BuildToolchain.ld
方法,(*Builder).build
会调用 BuildToolchain.symabis
、BuildToolchain.gc
、BuildToolchain.asm
、BuildToolchain.pack
等方法来实现核心功能。BuildToolchain 是 toolchain 接口类型的,定义了下列方法:// src/cmd/go/internal/work/exec.go
type toolchain interface {
// gc runs the compiler in a specific directory on a set of files
// and returns the name of the generated output file.
gc(b *Builder, a *Action, archive string, importcfg, embedcfg []byte, symabis string, asmhdr bool, gofiles []string) (ofile string, out []byte, err error)
// cc runs the toolchain's C compiler in a directory on a C file
// to produce an output file.
cc(b *Builder, a *Action, ofile, cfile string) error
// asm runs the assembler in a specific directory on specific files
// and returns a list of named output files.
asm(b *Builder, a *Action, sfiles []string) ([]string, error)
// symabis scans the symbol ABIs from sfiles and returns the
// path to the output symbol ABIs file, or "" if none.
symabis(b *Builder, a *Action, sfiles []string) (string, error)
// pack runs the archive packer in a specific directory to create
// an archive from a set of object files.
// typically it is run in the object directory.
pack(b *Builder, a *Action, afile string, ofiles []string) error
// ld runs the linker to create an executable starting at mainpkg.
ld(b *Builder, root *Action, out, importcfg, mainpkg string) error
// ldShared runs the linker to create a shared library containing the pkgs built by toplevelactions
ldShared(b *Builder, root *Action, toplevelactions []*Action, out, importcfg string, allactions []*Action) error
compiler() string
linker() string
}
go build
会在程序初始化时进行选择:func init() {
switch build.Default.Compiler {
case "gc", "gccgo":
buildCompiler{}.Set(build.Default.Compiler)
}
}
func (c buildCompiler) Set(value string) error {
switch value {
case "gc":
BuildToolchain = gcToolchain{}
case "gccgo":
BuildToolchain = gccgoToolchain{}
...
}
src/cmd/go/internal/work/gc.go
。以 gc 方法为例:func (gcToolchain) gc(b *Builder, a *Action, archive string, importcfg, embedcfg []byte, symabis string, asmhdr bool, gofiles []string) (ofile string, output []byte, err error) {
// ...
// 拼接参数
// ...
args := []interface{}{cfg.BuildToolexec, base.Tool("compile"), "-o", ofile, "-trimpath", a.trimpath(), gcflags, gcargs, "-D", p.Internal.LocalPrefix}
// ...
output, err = b.runOut(a, base.Cwd, nil, args...)
return ofile, output, err
}
base.Tool("compile")
的二进制程序。这些程序可以被称为 Go 编译工具,位于 pkg/tool
目录下,源码位于 src/cmd
。同理,其他的方法也是调用了相应的编译工具完成实际的编译工作。cfg.BuildToolexec
。跟进到定义处可知它是由 go build -toolexec
参数设置的,官方释义为:-toolexec 'cmd args'
a program to use to invoke toolchain programs like vet and asm.
For example, instead of running asm, the go command will run
'cmd args /path/to/asm <arguments for asm>'.
-toolexec
指定的程序来运行编译工具。这其实可以看作是一个 hook 机制,利用这个参数来指定一个我们的程序,在编译时用这个程序调用编译工具,从而介入编译过程,下文中分析的 garble 项目就是使用了这种思路。附一段从编译过程中截取的命令( go build -n
参数可以输出执行的命令)方便理解,比如我们指定了 -toolexec=/home/atom/go/bin/garble
,那么编译时实际执行的就是:/home/atom/go/bin/garble /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b016/_pkg_.a -trimpath "/usr/local/go/src/sync=>sync;$WORK/b016=>" -p sync -std -buildid FRNt7EHDh77qHujLKnmK/FRNt7EHDh77qHujLKnmK -goversion go1.16.4 -D "" -importcfg $WORK/b016/importcfg -pack -c=4 /usr/local/go/src/sync/cond.go /usr/local/go/src/sync/map.go /usr/local/go/src/sync/mutex.go /usr/local/go/src/sync/once.go /usr/local/go/src/sync/pool.go /usr/local/go/src/sync/poolqueue.go /usr/local/go/src/sync/runtime.go /usr/local/go/src/sync/runtime2.go /usr/local/go/src/sync/rwmutex.go /usr/local/go/src/sync/waitgroup.go
go build
通过拼接命令的方式调用 compile 等编译工具来实现具体的编译工作,我们可以使用 go build -toolexec
参数来指定一个程序“介入”编译过程。go/ast
src/go/ast/ast.go
,其中的注释足够详细,但为了方便梳理关系,笔者整理了关系图,图中的分叉代表继承关系,所有类型都基于 Node 接口:工具分析
特性
-
runtime 及其依赖的包(support obfuscating the runtime package #193)
-
Go 插件
-
哈希处理符合条件的包、函数、变量、类型等的名称
-
将字符串替换为匿名函数
-
移除调试信息、符号表
-
可以设置 -debugdir 输出混淆过的 Go 代码
-
可以指定不同的种子以混淆出不同的结果
-
主动模式:当命令传入的第一个指令与 garble 的预设相匹配时,代表是被用户主动调用的。此阶段会根据参数进行配置、获取依赖包信息等,然后将配置持久化。如果指令是 build 或 test,则再向命令中添加 -toolexec=path/to/garble 将自己设置为编译工具的启动器,引出启动器模式
-
启动器模式:对 tool/asm/link 这三个工具进行“拦截”,在编译工具运行前进行源代码混淆、修改运行参数等操作,最后运行工具编译混淆后的代码
构造目标列表
// listedPackage contains the 'go list -json -export' fields obtained by the
// root process, shared with all garble sub-processes via a file.
type listedPackage struct {
Name string
ImportPath string
ForTest string
Export string
BuildID string
Deps []string
ImportMap map[string]string
Standard bool
Dir string
GoFiles []string
// The fields below are not part of 'go list', but are still reused
// between garble processes. Use "Garble" as a prefix to ensure no
// collisions with the JSON fields from 'go list'.
GarbleActionID []byte
Private bool
}
func setListedPackages(patterns []string) error {
args := []string{"list", "-json", "-deps", "-export", "-trimpath"}
args = append(args, cache.BuildFlags...)
args = append(args, patterns...)
cmd := exec.Command("go", args...)
...
cache.ListedPackages = make(map[string]*listedPackage)
for ...{
var pkg listedPackage
...
cache.ListedPackages[pkg.ImportPath] = &pkg
...
}
}
go list
命令,其中指定的 -deps
参数官方释义为:The -deps flag causes list to iterate over not just the named packages but also all their dependencies. It visits them in a depth-first post-order traversal, so that a package is listed only after all its dependencies. Packages not explicitly listed on the command line will have the DepOnly field set to true.
go build
创建 action 时的很相似。通过这条命令 garble 可以获取到项目所有的依赖信息(包括间接依赖),遍历并存入 cache.ListedPackages
。除此之外还要标记各个依赖包是否在 env.GOPRIVATE
目录下,只有此目录下的文件才会被混淆(特例是使用了 -tiny
参数时会处理一部分 runtime)。可以通过设置环境变量 GOPRIVATE="*"
来扩大范围以获得更好的混淆效果。关于混淆范围的问题,garble 的作者也在尝试优化:idea: break away from GOPRIVATE? #276。go build
,它只是作为中间商修改了源代码或者修改了命令中传给编译工具的参数,最后还是要依靠这三个编译工具来实现具体的编译工作,下面逐一分析。compile
main.go transformCompile
函数,主要工作是处理 go 文件和修改命令参数。go build -n
参数可以输出执行的命令,我们可以在使用 garble 时传入这个参数来更直观的了解编译过程。截取其中一条:/home/atom/go/bin/garble /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b016/_pkg_.a -trimpath "/usr/local/go/src/sync=>sync;$WORK/b016=>" -p sync -std -buildid FRNt7EHDh77qHujLKnmK/FRNt7EHDh77qHujLKnmK -goversion go1.16.4 -D "" -importcfg $WORK/b016/importcfg -pack -c=4 /usr/local/go/src/sync/cond.go /usr/local/go/src/sync/map.go /usr/local/go/src/sync/mutex.go /usr/local/go/src/sync/once.go /usr/local/go/src/sync/pool.go /usr/local/go/src/sync/poolqueue.go /usr/local/go/src/sync/runtime.go /usr/local/go/src/sync/runtime2.go /usr/local/go/src/sync/rwmutex.go /usr/local/go/src/sync/waitgroup.go
cond.go
等诸多文件编译成中间代码。garble 识别到当前的编译工具是 compile,于是”拦截“,在工具运行前做一些混淆等工作。下面分析一下相对重要的部分。var files []*ast.File
for _, path := range paths {
file, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
if err != nil {
return nil, err
}
files = append(files, file)
}
reflect.ValueOf
/ reflect.TypeOf
)的节点的类型名称可能会在后续逻辑中使用,所以不能对其名称进行混淆:if fnType.Pkg().Path() == "reflect" && (fnType.Name() == "TypeOf" || fnType.Name() == "ValueOf") {
for _, arg := range call.Args {
argType := tf.info.TypeOf(arg)
tf.recordIgnore(argType, tf.pkg.Path())
}
}
go:linkname
的标识符,从没被混淆的包中引入的变量和类型:// ignoreObjects records all the objects we cannot obfuscate. An object
// is any named entity, such as a declared variable or type.
//
// So far, this map records:
//
// * Types which are used for reflection; see recordReflectArgs.
// * Identifiers used in constant expressions; see RecordUsedAsConstants.
// * Identifiers used in go:linkname directives; see handleDirectives.
// * Types or variables from external packages which were not
// obfuscated, for caching reasons; see transformGo.
ignoreObjects map[types.Object]bool
ast.GenDecl
的情况为例:// RecordUsedAsConstants records identifieres used in constant expressions.
func RecordUsedAsConstants(node ast.Node, info *types.Info, ignoreObj map[types.Object]bool) {
visit := func(node ast.Node) bool {
ident, ok := node.(*ast.Ident)
if !ok {
return true
}
// Only record *types.Const objects.
// Other objects, such as builtins or type names,
// must not be recorded as they would be false positives.
obj := info.ObjectOf(ident)
if _, ok := obj.(*types.Const); ok {
ignoreObj[obj] = true
}
return true
}
switch x := node.(type) {
...
// in a const declaration all values must be constant representable
case *ast.GenDecl:
if x.Tok != token.CONST {
break
}
for _, spec := range x.Specs {
spec := spec.(*ast.ValueSpec)
for _, val := range spec.Values {
ast.Inspect(val, visit)
}
}
}
}
package obfuscate
const (
H2 string = "a"
H4 string = "a" + H2
H3 int = 123
H5 string = "a"
)
ast.GenDecl
类型,然后遍历其 Specs(每个定义),对每个 spec 遍历其 Values(等号右边的表达式),再对 val 中的元素使用 ast.Inspect()
遍历执行 visit()
,如果元素节点的类型是 ast.Ident
且指向的 obj 的类型是 types.Const
,则将此 obj 记入 tf.recordIgnore
。有点绕,我们把 AST 打印出来看:H4 string = "a" + H2
中的 H2 完全符合条件,所以应该被记入 tf.recordIgnore
。接下来要分析的功能中会涉及到大量类型断言和类型选择,看起来复杂但本质上与刚刚的分析过程类似,我们只要将写个 demo 并打印出 AST 就很容易理解了。main.go transformCompile
。接下来对当前的包名进行混淆并写入命令参数和源文件中,要求文件既不是 main 包,也不在 env.GOPRIVATE
目录之外。下一步将处理注释和源代码,这里会对 runtime 和 CGO 单独处理,我们大可忽略,直接看对普通 Go 代码的处理:// transformGo obfuscates the provided Go syntax file.
func (tf *transformer) transformGo(file *ast.File) *ast.File {
if opts.GarbleLiterals {
file = literals.Obfuscate(file, tf.info, fset, tf.ignoreObjects)
}
pre := func(cursor *astutil.Cursor) bool {...}
post := func(cursor *astutil.Cursor) bool {...}
return astutil.Apply(file, pre, post).(*ast.File)
}
astutil.Apply(file, pre, post)
进行 AST 的递归处理,其中 pre 和 post 函数分别用于访问孩子节点前和访问后。这部分的代码大都是比较繁琐的筛选操作,下面仅作简要分析:![图片](https://static.doc-tr.net/mp-images/87/87712edf4596a935e40411bd6416e452.jpg)
•transformGo pre
asm
link
混淆效果
go build .
和 go env -w GOPRIVATE="*" && garble -literals build .
两次编译。可以看到左侧很简单的代码经过混淆后变得难以阅读:总 结
往 期 热 门
(点击图片跳转)
![图片](https://static.doc-tr.net/mp-images/13/1364fefe1848b8f01d10240120befa70.jpg)
.Net 反序列化学习之 DataContractSerializer
![图片](https://static.doc-tr.net/mp-images/36/36f78b9f07785bf501948e7fcf8fa638.jpg)
谈谈国内开源的 PoC 框架及 Pocsuite
![图片](https://static.doc-tr.net/mp-images/99/99b64a2ac73be3a0a9125d76d45b0f16.jpg)
NAT 原理以及 UDP 穿透
![图片](https://static.doc-tr.net/mp-images/63/63c10cda9f9fafb3a3e82b5fefd74f0b.jpg)
![图片](https://static.doc-tr.net/mp-images/09/091ce80e3a883c6c41fddfa494317825.jpg)
文章评论