Go bindings on TensorFlow for MacOS-arm64

2022-03-23

因为 python 的性能问题，项目本身对于效率要求比较高，最近想把一些算法工作迁移到 go 实现。
然而在 go 版 TensorFlow 的安装上就花了我一天的时间解决各种问题。
记录一下在 macos-arm64 系统下该如何正确安装 golang 版 TensorFlow。

1. 安装 tensorflow 依赖环境和 Python 依赖

见上一篇，这里不再复述 >> 出门右转

这一步不知道是否必须，但是编译时是依赖 Python 环境的（比如 numpy）

2. 安装 protobuf

直接brew安装比较简单

1	brew install protobuf

3. 编译 TensorFlow 动态库

follow 这篇帖子 https://github.com/tensorflow/build/tree/master/golang_install_guide

帖子的大方向是正确的，只是在 aarm64 下有些坑需要踩踩，这里决定记录下来方便后续查阅。

3.1. 查看 GOPATH

1	go env GOPATH

3.2. 克隆源代码和打补丁

我的 GOPATH 是 ~/.go，所以这里下载到 ~/.go/src/github.com/tensorflow/tensorflow

git clone --branch v2.8.0 https://github.com/tensorflow/tensorflow.git ~/.go/src/github.com/tensorflow/tensorflow

cd ~/go/src/github.com/tensorflow/tensorflow

git cherry-pick --strategy-option=no-renames --no-commit 65a5434

3.3. 初始化一个新的 go.mod 文件

1	go mod init github.com/tensorflow/tensorflow

3.4. 构建动态库

这是上面官方文档对于 macos-arm64 缺失的一步

3.4.1 Install Bazel via Homebrew

https://docs.bazel.build/versions/main/install-os-x.html#install-on-mac-os-x-homebrew

1	brew install bazel

3.4.2 bazel build

https://github.com/tensorflow/tensorflow/issues/52160#issuecomment-933219952

cd “/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin” && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-arm64 && chmod +x bazel-4.2.1-darwin-arm64

注意：记得在运行 ./configue 之前要先切换到正确的 Python 环境，我的 TensorFlow 环境叫 tf，所以这里要先执行 conda activate tf

cd ~/go/src/github.com/tensorflow/tensorflow

conda activate tf

./configure

# check macos_sdk_version
xcrun --sdk macosx --show-sdk-path

bazel build --config opt --cpu=darwin_arm64 --host_cpu=darwin_arm64 //tensorflow/tools/lib_package:libtensorflow

# 需要指定系统sdk版本可使用这条命令
# bazel build --config opt --cpu=darwin_arm64 --host_cpu=darwin_arm64 --macos_sdk_version=12.0 //tensorflow/tools/lib_package:libtensorflow

sudo tar -C /usr/local -xzf bazel-bin/tensorflow/tools/lib_package/libtensorflow.tar.gz

3.5. 生成 protoco buffers 和 go 代码

这一步的成功才代前面动态库构建成功

1	(cd tensorflow/go/op && go generate)

3.6. 下载确实的依赖

1	go mod tidy

3.7. 测试

1	go test ./...

4. 测试 Go 语言中使用 TensorFlow

官方测试代码

5. Errors

一下部分单纯为了 SEO，如果你没有遇到以下问题请直接跳过。

libtensorflow.dylib is building for macOS-arm64 but attempting to link with file built for macOS-x86_64

$ (cd tensorflow/go/op && go generate -v)
generate.go
# github.com/tensorflow/tensorflow/tensorflow/go/genop/internal
ld: warning: ignoring file /Users/wuyanxin/lib/libtensorflow/lib/libtensorflow.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture arm64:
  "_TF_ApiDefMapGet", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_ApiDefMapGet in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_ApiDefMapGet)
  "_TF_ApiDefMapPut", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_ApiDefMapPut in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_ApiDefMapPut)
  "_TF_DeleteApiDefMap", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_DeleteApiDefMap in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_DeleteApiDefMap)
  "_TF_DeleteBuffer", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_DeleteBuffer in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_DeleteBuffer)
  "_TF_DeleteStatus", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_DeleteStatus in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_DeleteStatus)
  "_TF_GetAllOpList", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_GetAllOpList in _x003.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_GetAllOpList)
  "_TF_GetCode", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_GetCode in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_GetCode)
  "_TF_Message", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_Message in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_Message)
  "_TF_NewApiDefMap", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_NewApiDefMap in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_NewApiDefMap)
  "_TF_NewBuffer", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_NewBuffer in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_NewBuffer)
  "_TF_NewStatus", referenced from:
      __cgo_b4651de6ed15_Cfunc_TF_NewStatus in _x002.o
     (maybe you meant: __cgo_b4651de6ed15_Cfunc_TF_NewStatus)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
generate.go:18: running "go": exit status 2

目前官方没有 macOS-arm64 版本的动态库，遇到这个错误只能参考前面的教程自行构建。
中间遇到了很多其它坑已经没有记录了，这条错误反复出现了很多次。

遇到一些其它什么难搞的问题可以给我发邮件，也许我能凭记忆给一些方案。

感谢阅读，world peace 🙏

1. 安装 tensorflow 依赖环境 和 Python 依赖