从零开始构建向量数据库：Milvus 的源码编译安装（一）

git clone https://gitee.com/milvus-io/milvus.git

git clone https://gitee.com/milvus-io/milvus.git  
Cloning into 'milvus'...
remote: Enumerating objects: 160476, done.
remote: Counting objects: 100% (37509/37509), done.
remote: Compressing objects: 100% (12122/12122), done.
remote: Total 160476 (delta 29683), reused 31705 (delta 24611), pack-reused 122967
Receiving objects: 100% (160476/160476), 112.49 MiB | 15.33 MiB/s, done.
Resolving deltas: 100% (120888/120888), done.

# 进入下载完毕的仓库目录
cd milvus
# 删除仓库原始代码上游
git remote remove origin
# 根据自己情况选择，添加官方仓库地址为新的上游
git remote add origin https://github.com/milvus-io/milvus.git
# 或者添加自己的 fork 仓库（有读写权限，协议可以根据自己的情况选择 git 或者 https
# git remote add origin git@github.com:soulteary/milvus.git

git pull
From https://github.com/milvus-io/milvus
 * [new branch]          1.0                         -> origin/1.0
 * [new branch]          1.1                         -> origin/1.1
 * [new branch]          1.x                         -> origin/1.x
 * [new branch]          2.0                         -> origin/2.0
 * [new branch]          2.0.2                       -> origin/2.0.2
 * [new branch]          2.0.3                       -> origin/2.0.3
 * [new branch]          2.1.0                       -> origin/2.1.0
...
 * [new branch]          master                      -> origin/master
...
 * [new branch]          update_pytest_f73a87b1f282257aa972c3aaff9a450d864a785a -> origin/update_pytest_f73a87b1f282257aa972c3aaff9a450d864a785a
 * [new branch]          update_pytest_fc2839f254a2327520e6202f4a1fa31937628aa0 -> origin/update_pytest_fc2839f254a2327520e6202f4a1fa31937628aa0
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream-to=origin/<branch> master

git branch --set-upstream-to=origin/master master

git branch --set-upstream-to=origin/master master 
Branch 'master' set up to track remote branch 'master' from 'origin'.

git fetch && git pull origin master

git fetch && git pull origin master
From https://github.com/milvus-io/milvus
 * branch                master     -> FETCH_HEAD
Already up to date.

wget -qO- "https://cmake.org/files/v3.18/cmake-3.18.6-Linux-x86_64.tar.gz" | sudo tar --strip-components=1 -xz -C /usr/local

cmake --version
cmake version 3.18.6

CMake suite maintained and supported by Kitware (kitware.com/cmake).

https://github.com/Kitware/CMake/releases/download/v3.23.2/cmake-3.23.2-linux-aarch64.tar.gz

sudo apt update && \
sudo apt-get install -y cmake

wget -qO- "https://cmake.org/files/v3.18/cmake-3.18.6-Linux-x86_64.tar.gz" | sudo tar --strip-components=1 -xz -C /usr/local

sudo apt-get update && \
sudo apt install -y clang-format-10 clang-tidy-10

sudo apt-get update && \
sudo apt install -y clang-format clang-tidy

git clone -b ubuntu/focal https://git.launchpad.net/ubuntu/+source/llvm-toolchain-10

cd llvm-toolchain-10 && \
mkdir build && cd build

sudo apt-get update && \
sudo apt install -y ninja-build

cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=$(which python3) -DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON ../llvm

-- clang project is enabled
-- clang-tools-extra project is enabled
-- compiler-rt project is disabled
-- debuginfo-tests project is disabled
-- libc project is disabled
-- libclc project is disabled
...
...
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Configuring done
-- Generating done
-- Build files have been written to: /app/milvus/llvm-toolchain-10/build

cmake --build .
[4/4165] Building CXX object utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetEmitter.cpp.o
...
[2957/4165] Building CXX object tools/clang/lib/Frontend/CMakeFiles/obj.clangFrontend.dir/PrecompiledPreamble.cpp.
...

#ifndef BENCHMARK_REGISTER_H
#define BENCHMARK_REGISTER_H

#include <vector>

#include "check.h"

template <typename T>
void AddRange(std::vector<T>* dst, T lo, T hi, int mult) {
  CHECK_GE(lo, 0);
  CHECK_GE(hi, lo);
  CHECK_GE(mult, 2);

  // Add "lo"
  dst->push_back(lo);

  static const T kmax = std::numeric_limits<T>::max();

  // Now space out the benchmarks in multiples of "mult"
  for (T i = 1; i < kmax / mult; i *= mult) {
    if (i >= hi) break;
    if (i > lo) {
      dst->push_back(i);
    }
  }

  // Add "hi" (if different from "lo")
  if (hi != lo) {
    dst->push_back(hi);
  }
}

#endif  // BENCHMARK_REGISTER_H

#include <stdexcept>
#include <limits>
#include <vector>

#include "check.h"

[3/3] Linking CXX static library lib/libbenchmark_main.a

# ./bin/clang-format --version
clang-format version 10.0.0 (https://git.launchpad.net/ubuntu/+source/llvm-toolchain-10 6f319ce10f040c979f7bb074a7b7465b8c7bf9f7)

# ./bin/clang-tidy --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver1

sudo cp bin/clang-format /usr/local/bin && \
sudo cp bin/clang-tidy /usr/local/bin

sudo apt update && \
sudo apt-get install -y g++ gcc make lcov libtool m4 autoconf automake ccache libssl-dev zlib1g-dev libboost-regex-dev libboost-program-options-dev libboost-system-dev libboost-filesystem-dev libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev pkg-config && \
sudo apt-get remove --purge -y

gvm install go1.16.9 -B && \
gvm use go1.16.9 --default

Installing go1.16.9 from binary source
Now using version go1.16.9

RUN ...
    curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b ${GOPATH}/bin v1.43.0 && \
    export GO111MODULE=on && go get github.com/quasilyte/go-ruleguard/cmd/ruleguard@v0.2.1 && \
    go get -v github.com/ramya-rao-a/go-outline && \
    go get -v golang.org/x/tools/gopls && \
    go get -v github.com/uudashr/gopkgs/v2/cmd/gopkgs && \
    go get -v github.com/go-delve/delve/cmd/dlv && \
    go get -v honnef.co/go/tools/cmd/staticcheck && \
    go clean --modcache && \
    chmod -R 777 "$GOPATH" && chmod -R a+w $(go env GOTOOLDIR)
...
RUN ln -s /go/bin/dlv /go/bin/dlv-dap

cd milvus
make milvus

Building Milvus cpp library ...
/app/milvus/milvus/cmake_build /app/milvus/milvus
==System:Linux
cmake -DBUILD_UNIT_TEST=OFF -DCMAKE_INSTALL_PREFIX=/app/milvus/milvus/scripts/../internal/core/output -DCMAKE_BUILD_TYPE=Release -DOpenBLAS_SOURCE=AUTO -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DBUILD_COVERAGE=OFF -DMILVUS_DB_PATH=/tmp/milvus -DENABLE_CPU_PROFILING=OFF -DMILVUS_GPU_VERSION=OFF -DMILVUS_WITH_PROMETHEUS=ON -DMILVUS_CUDA_ARCH=DEFAULT -DCUSTOM_THIRDPARTY_DOWNLOAD_PATH= -DEMBEDDED_MILVUS=OFF /app/milvus/milvus/scripts/../internal/core
-- Building using CMake version: 3.18.6
-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
...
[100%] Built target milvus_segcore
make[2]: Leaving directory '/app/milvus/milvus/cmake_build'
Install the project...
...
make[1]: Leaving directory '/app/milvus/milvus/cmake_build'
Summary:
  Hits:               0 /  914 (0.00 %)
    Direct:           0 /  914 (0.00 %)
    Preprocessed:     0 /  914 (0.00 %)
  Misses:           914
    Direct:         914
    Preprocessed:   914
  Uncacheable:       31
Primary storage:
  Hits:               0 / 1828 (0.00 %)
  Misses:          1828
  Cache size (GB): 0.20 / 5.00 (4.06 %)

Use the -v/--verbose option for more details.
/app/milvus/milvus
Build Tag: v2.1.0-testing-20220704-15-g44024c1b1
Build Time: 2022年 07月 06日 星期三 07:36:49 UTC
Git Commit: 44024c1b1
Go Version: go version go1.16.9 linux/amd64
Building Milvus ...
if build fails on Mac M1 machines, you probably need to rerun scripts/install_deps.sh and then run: `export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@3/lib/pkgconfig"`
...
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/gofrs/flock v0.8.1
go: downloading go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489
go: downloading go.uber.org/automaxprocs v1.4.0
go: downloading go.uber.org/zap v1.17.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading google.golang.org/grpc v1.44.0
...

./bin/milvus 
Usage:

milvus run [server type] [flags]
	Start a Milvus Server.
	Tips: Only the server type is 'mixture', flags about starting server can be used.
[flags]
	-rootcoord 'true'
		Start the rootcoord server.
	-querycoord 'true'
		Start the querycoord server.
	-indexcoord 'true'
		Start the indexcoord server.
	-datacoord 'true'
		Start the datacoord server.
	-alias ''
		Set alias


milvus stop [server type] [flags]
	Stop a Milvus Server.
[flags]
	-alias ''
		Set alias


milvus mck run [flags]
	Milvus data consistency check.
	Tips: The flags are optional.
[flags]
	-etcdIp ''
		Ip to connect the ectd server.
	-etcdRootPath ''
		The root path of operating the etcd data.
	-minioAddress ''
		Address to connect the minio server.
	-minioUsername ''
		The username to login the minio server.
	-minioPassword ''
		The password to login the minio server.
	-minioUseSSL 'false'
		Whether to use the ssl to connect the minio server.
	-minioBucketName ''
		The bucket to operate the data in it

milvus mck cleanTrash [flags]
	Clean the back inconsistent data
	Tips: The flags is the same as its of the 'milvus mck [flags]'


[server type]
	indexcoord
	datanode
	embedded
	proxy
	querycoord
	querynode
	standalone
	rootcoord
	indexnode
	datacoord
	mixture