使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

人工智能78

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

; 1 对于单个相机拍摄的图像序列

很简单,将自己的图片放入一个文件夹,命令输入指向这个文件夹就好了:

1.1 Sequential & Incremental SfM pipeline

$ cd openMVG_Build/software/SfM/
$ python SfM_SequentialPipeline.py [full path image directory] [resulting directory]
$ python SfM_SequentialPipeline.py ~/home/user/data/ImageDataset_SceauxCastle/images ~/home/user/data/ImageDataset_SceauxCastle/Castle_Incremental_Reconstruction

1.2 Global SfM pipeline

$ cd openMVG_Build/software/SfM/
$ python SfM_GlobalPipeline.py [full path image directory] [resulting directory]
$ python SfM_GlobalPipeline.py ~/home/user/data/ImageDataset_SceauxCastle/images ~/home/user/data/ImageDataset_SceauxCastle/Castle_Global_Reconstruction

注意:一般拍摄图像中会有相机的参数信息,OpenMVG将尝试检索图像的像素焦距。如果没有,在 Scene Initialization stage我们可以使用 -f X * 1.2 提供一个近似焦距,其中X = Max(image

1.3 DIY

在sfm过程中可能有一些额外的参数需要设置,可以分步骤,下面是自己用的.bash文件:

cd reconstruction
mkdir reconstruction_work
cd reconstruction_work
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_SfMInit_ImageListing.exe -i ..\images\ -d D:\Reconstruction\openmvg_v20r\share\openMVG\sensor_width_camera_database.txt -o .\matches
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_ComputeFeatures.exe -i .\matches\sfm_data.json -o .\matches
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_PairGenerator.exe -i .\matches\sfm_data.json -o .\matches\pairs.bin
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_ComputeMatches.exe -i .\matches\sfm_data.json -p .\matches\pairs.bin -o .\matches\matches.putative.bin
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_GeometricFilter.exe -i .\matches\sfm_data.json -m .\matches\matches.putative.bin -g f -o .\matches\matches.f.bin
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_SfM.exe -s INCREMENTAL -i .\matches\sfm_data.json -M .\matches\matches.f.bin -o .\output
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_ComputeSfM_DataColor.exe -i .\output\sfm_data.bin -o .\output\sfm_data_colorized.ply
mkdir .\mvs
D:\Reconstruction\openmvg_v20r\bin\openMVG_main_openMVG2openMVS.exe -i .\output\sfm_data.bin -d .\mvs\undistortedImages -o .\mvs\scene.mvs

本人使用的是某车载摄像头的数据,图片上没有内参信息,运行上指令会在重建部分报错:

ERROR: [sequential_SfM.cpp:110] Unable to choose an initial pair, since there is no defined intrinsic data.

INFO: [sequential_SfM.cpp:173] Cannot find a valid initial pair - stop reconstruction.

因此需要在第一步openMVG_main_SfMInit_ImageListing生成sfm_data.json文件前手动添加的额外参数:

  • [-f|–focal] (value in pixels)
  • [-k|–intrinsics] Kmatrix: "f;0;ppx;0;f;ppy;0;0;1"
  • [-c|–camera_model] Camera model type:
  • 1: Pinhole
  • 2: Pinhole radial 1
  • 3: Pinhole radial 3 (default)
  • [-g|–group_camera_model]
  • 0-> each view have it's own camera intrinsic parameters
  • 1-> (default) view can share some camera intrinsic parameters

查看图像的参数:
使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)
设置上述内参,因为是同一个相机的图像设置g为1注意内参用分号隔开:

D:\Reconstruction\openmvg_v20r\bin\openMVG_main_SfMInit_ImageListing.exe -i ..\images\ -d D:\Reconstruction\openmvg_v20r\share\openMVG\sensor_width_camera_database.txt -o .\matches -k "3656.311177; 0.0; 1225.812961; 0.0; 3671.211662; 993.655164; 0.0; 0.0; 1.0" -g 1

格式输入错误运行bash参数会报错:

ERROR: [main_SfMInit_ImageListing.cpp:45]
 Missing ';' character
ERROR: [main_SfMInit_ImageListing.cpp:238] Invalid K matrix input

正确格式运行应该会完成重建,数据集有问题可能无法重建:

ERROR: [sequential_SfM.cpp:534]  /!\ Robust estimation failed to compute E for this pair: {4,5}

这里应该是车载数据集的视差太小了,连续帧前后估计有困难。自制数据集应尽量保证在重复区域足够多的前提下扩大视角。

openMVG的sfm pipeline成功运行后,输出三个子文件夹:
使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)
分别包含特征匹配部分、重建部分和mvs格式转换的内容
output中文件sfm_data.bin就是重建的数据

1.4 openMVS

在进行OpenMVS步骤之前还需要将上一步生成的sfm_data.bin转化成mvs格式,该步在1.3bash最后一行已经执行,这一步会产生两个输出

  • 一个就是scene.mvs这个文件
  • 另一个就是一个undistorted_images文件夹,里面是经过畸变校正的图像

下面使用MVS进行网格化和纹理贴图:
为了方便,直接把mvs文件夹复制到MVS的exe所在文件夹下
使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)
可以先查看mvs文件中的稀疏点云:

Viewer.exe mvs\scene.mvs

生成稠密点云:

DensifyPointCloud.exe mvs\scene.mvs

查看稠密点云:

Viewer.exe mvs\scene_dense.mvs

建立粗网格:

ReconstructMesh.exe mvs\scene_dense.mvs

查看粗网格:

Viewer.exe mvs\scene_dense_mesh.mvs

细化网格(可选,比较耗时:

RefineMesh.exe mvs\scene_dense_mesh.mvs

查看细化网格:

Viewer.exe mvs\scene_dense_mesh_refine.mvs

加纹理:

TextureMesh.exe mvs\scene_dense_mesh_refine.mvs

查看结果:

Viewer.exe mvs\scene_dense_mesh_refine_texture.mvs

1.5 汇总:

完整的流程,设置好输入输出就可以了:

#!/usr/bin/python3

"""
This script is for an easy use of OpenMVG and OpenMVS

usage: MvgMvs_Pipeline.py [-h] [--steps STEPS [STEPS ...]] [--preset PRESET]
                          [--0 0 [0 ...]] [--1 1 [1 ...]] [--2 2 [2 ...]]
                          [--3 3 [3 ...]] [--4 4 [4 ...]] [--5 5 [5 ...]]
                          [--6 6 [6 ...]] [--7 7 [7 ...]] [--8 8 [8 ...]]
                          [--9 9 [9 ...]] [--10 10 [10 ...]] [--11 11 [11 ...]]
                          [--12 12 [12 ...]] [--13 13 [13 ...]]
                          [--14 14 [14 ...]] [--15 15 [15 ...]]
                          [--16 16 [16 ...]] [--17 17 [17 ...]]
                          input_dir output_dir

Photogrammetry reconstruction with these steps:
    0. Intrinsics analysis             openMVG_main_SfMInit_ImageListing
    1. Compute features                openMVG_main_ComputeFeatures
    2. Compute pairs                   openMVG_main_PairGenerator
    3. Compute matches                 openMVG_main_ComputeMatches
    4. Filter matches                  openMVG_main_GeometricFilter
    5. Incremental reconstruction      openMVG_main_IncrementalSfM
    6. Global reconstruction           openMVG_main_GlobalSfM
    7. Colorize Structure              openMVG_main_ComputeSfM_DataColor
    8. Structure from Known Poses      openMVG_main_ComputeStructureFromKnownPoses
    9. Colorized robust triangulation  openMVG_main_ComputeSfM_DataColor
    10. Control Points Registration    ui_openMVG_control_points_registration
    11. Export to openMVS              openMVG_main_openMVG2openMVS
    12. Densify point-cloud            DensifyPointCloud
    13. Reconstruct the mesh           ReconstructMesh
    14. Refine the mesh                RefineMesh
    15. Texture the mesh               TextureMesh
    16. Estimate disparity-maps        DensifyPointCloud
    17. Fuse disparity-maps            DensifyPointCloud

positional arguments:
  input_dir                 the directory which contains the pictures set.

  output_dir                the directory which will contain the resulting files.

optional arguments:
  -h, --help                show this help message and exit
  --steps STEPS [STEPS ...] steps to process
  --preset PRESET           steps list preset in
                            SEQUENTIAL = [0, 1, 2, 3, 4, 5, 11, 12, 13, 14, 15]
                            GLOBAL = [0, 1, 2, 3, 4, 6, 11, 12, 13, 14, 15]
                            MVG_SEQ = [0, 1, 2, 3, 4, 5, 7, 8, 9]
                            MVG_GLOBAL = [0, 1, 2, 3, 4, 6, 7, 8, 9]
                            MVS = [12, 13, 14, 15]
                            MVS_SGM = [16, 17]
                            default : SEQUENTIAL

Passthrough:
  Option to be passed to command lines (remove - in front of option names)
  e.g. --1 p ULTRA to use the ULTRA preset in openMVG_main_ComputeFeatures
  For example, running the script
  [MvgMvsPipeline.py input_dir output_dir --steps 0 1 2 3 4 5 11 12 13 15 --1 p HIGH n 8 --3 n HNSWL2]
  [--steps 0 1 2 3 4 5 11 12 13 15] runs only the desired steps
  [--1 p HIGH n 8] where --1 refer to openMVG_main_ComputeFeatures,
  p refers to describerPreset option and set to HIGH, and n refers
  to numThreads and set to 8. The second step (Compute matches),
  [--3 n HNSWL2] where --3 refer to openMVG_main_ComputeMatches,
  n refers to nearest_matching_method option and set to HNSWL2
"""

import os
import subprocess
import sys
import argparse

DEBUG = False

if sys.platform.startswith('win'):
    PATH_DELIM = ';'
    FOLDER_DELIM = '\\'
else:
    PATH_DELIM = ':'
    FOLDER_DELIM = '/'

os.environ['PATH'] += PATH_DELIM + os.path.dirname(os.path.abspath(__file__))

os.environ['PATH'] += PATH_DELIM + os.getcwd()

def whereis(afile):
"""
        return directory in which afile is, None if not found. Look in PATH
"""
    if sys.platform.startswith('win'):
        cmd = "where"
    else:
        cmd = "which"
    try:
        ret = subprocess.run([cmd, afile], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True)
        return os.path.split(ret.stdout.decode())[0]
    except subprocess.CalledProcessError:
        return None

def find(afile):
"""
        As whereis look only for executable on linux, this find look for all file type
"""
    for d in os.environ['PATH'].split(PATH_DELIM):
        if os.path.isfile(os.path.join(d, afile)):
            return d
    return None

OPENMVG_BIN = whereis("openMVG_main_SfMInit_ImageListing")
OPENMVS_BIN = whereis("ReconstructMesh")

CAMERA_SENSOR_DB_FILE = "sensor_width_camera_database.txt"
CAMERA_SENSOR_DB_DIRECTORY = find(CAMERA_SENSOR_DB_FILE)

if not OPENMVG_BIN:
    OPENMVG_BIN = input("openMVG binary folder?\n")
if not OPENMVS_BIN:
    OPENMVS_BIN = input("openMVS binary folder?\n")
if not CAMERA_SENSOR_DB_DIRECTORY:
    CAMERA_SENSOR_DB_DIRECTORY = input("openMVG camera database (%s) folder?\n" % CAMERA_SENSOR_DB_FILE)

PRESET = {'SEQUENTIAL': [0, 1, 2, 3, 4, 5, 11, 12, 13, 14, 15],
          'GLOBAL': [0, 1, 2, 3, 4, 6, 11, 12, 13, 14, 15],
          'MVG_SEQ': [0, 1, 2, 3, 4, 5, 7, 8, 9, 11],
          'MVG_GLOBAL': [0, 1, 2, 3, 4, 6, 7, 8, 9, 11],
          'MVS': [12, 13, 14, 15],
          'MVS_SGM': [16, 17]}

PRESET_DEFAULT = 'SEQUENTIAL'

BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE = range(8)
NO_EFFECT, BOLD, UNDERLINE, BLINK, INVERSE, HIDDEN = (0, 1, 4, 5, 7, 8)

def has_colours(stream):
    '''
        Return stream colours capability
    '''
    if not hasattr(stream, "isatty"):
        return False
    if not stream.isatty():
        return False
    try:
        import curses
        curses.setupterm()
        return curses.tigetnum("colors") > 2
    except Exception:

        return False

HAS_COLOURS = has_colours(sys.stdout)

def printout(text, colour=WHITE, background=BLACK, effect=NO_EFFECT):
"""
        print() with colour
"""
    if HAS_COLOURS:
        seq = "\x1b[%d;%d;%dm" % (effect, 30+colour, 40+background) + text + "\x1b[0m"
        sys.stdout.write(seq+'\r\n')
    else:
        sys.stdout.write(text+'\r\n')

class ConfContainer:
"""
        Container for all the config variables
"""
    def __init__(self):
        pass

class AStep:
    """ Represents a process step to be run """
    def __init__(self, info, cmd, opt):
        self.info = info
        self.cmd = cmd
        self.opt = opt

class StepsStore:
    """ List of steps with facilities to configure them """
    def __init__(self):
        self.steps_data = [
            ["Intrinsics analysis",
             os.path.join(OPENMVG_BIN, "openMVG_main_SfMInit_ImageListing"),
             ["-i", "%input_dir%", "-o", "%matches_dir%", "-d", "%camera_file_params%"]],
            ["Compute features",
             os.path.join(OPENMVG_BIN, "openMVG_main_ComputeFeatures"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-o", "%matches_dir%", "-m", "SIFT"]],
            ["Compute pairs",
             os.path.join(OPENMVG_BIN, "openMVG_main_PairGenerator"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-o", "%matches_dir%"+FOLDER_DELIM+"pairs.bin"]],
            ["Compute matches",
             os.path.join(OPENMVG_BIN, "openMVG_main_ComputeMatches"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-p", "%matches_dir%"+FOLDER_DELIM+"pairs.bin", "-o", "%matches_dir%"+FOLDER_DELIM+"matches.putative.bin", "-n", "AUTO"]],
            ["Filter matches",
             os.path.join(OPENMVG_BIN, "openMVG_main_GeometricFilter"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-m", "%matches_dir%"+FOLDER_DELIM+"matches.putative.bin", "-o", "%matches_dir%"+FOLDER_DELIM+"matches.f.bin"]],
            ["Incremental reconstruction",
             os.path.join(OPENMVG_BIN, "openMVG_main_SfM"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-m", "%matches_dir%", "-o", "%reconstruction_dir%", "-s", "INCREMENTAL"]],
            ["Global reconstruction",
             os.path.join(OPENMVG_BIN, "openMVG_main_SfM"),
             ["-i", "%matches_dir%"+FOLDER_DELIM+"sfm_data.json", "-m", "%matches_dir%", "-o", "%reconstruction_dir%", "-s", "GLOBAL", "-M", "%matches_dir%"+FOLDER_DELIM+"matches.e.bin"]],
            ["Colorize Structure",
             os.path.join(OPENMVG_BIN, "openMVG_main_ComputeSfM_DataColor"),
             ["-i", "%reconstruction_dir%"+FOLDER_DELIM+"sfm_data.bin", "-o", "%reconstruction_dir%"+FOLDER_DELIM+"colorized.ply"]],
            ["Structure from Known Poses",
             os.path.join(OPENMVG_BIN, "openMVG_main_ComputeStructureFromKnownPoses"),
             ["-i", "%reconstruction_dir%"+FOLDER_DELIM+"sfm_data.bin", "-m", "%matches_dir%", "-f", "%matches_dir%"+FOLDER_DELIM+"matches.f.bin", "-o", "%reconstruction_dir%"+FOLDER_DELIM+"robust.bin"]],
            ["Colorized robust triangulation",
             os.path.join(OPENMVG_BIN, "openMVG_main_ComputeSfM_DataColor"),
             ["-i", "%reconstruction_dir%"+FOLDER_DELIM+"robust.bin", "-o", "%reconstruction_dir%"+FOLDER_DELIM+"robust_colorized.ply"]],
            ["Control Points Registration",
             os.path.join(OPENMVG_BIN, "ui_openMVG_control_points_registration"),
             ["-i", "%reconstruction_dir%"+FOLDER_DELIM+"sfm_data.bin"]],
            ["Export to openMVS",
             os.path.join(OPENMVG_BIN, "openMVG_main_openMVG2openMVS"),
             ["-i", "%reconstruction_dir%"+FOLDER_DELIM+"sfm_data.bin", "-o", "%mvs_dir%"+FOLDER_DELIM+"scene.mvs", "-d", "%mvs_dir%"+FOLDER_DELIM+"images"]],
            ["Densify point cloud",
             os.path.join(OPENMVS_BIN, "DensifyPointCloud"),
             ["scene.mvs", "--dense-config-file", "Densify.ini", "--resolution-level", "1", "--number-views", "8", "-w", "\"%mvs_dir%\""]],
            ["Reconstruct the mesh",
             os.path.join(OPENMVS_BIN, "ReconstructMesh"),
             ["scene_dense.mvs", "-w", "\"%mvs_dir%\""]],
            ["Refine the mesh",
             os.path.join(OPENMVS_BIN, "RefineMesh"),
             ["scene_dense_mesh.mvs", "--scales", "1", "--gradient-step", "25.05", "-w", "\"%mvs_dir%\""]],
            ["Texture the mesh",
             os.path.join(OPENMVS_BIN, "TextureMesh"),
             ["scene_dense_mesh_refine.mvs", "--decimate", "0.5", "-w", "\"%mvs_dir%\""]],
            ["Estimate disparity-maps",
             os.path.join(OPENMVS_BIN, "DensifyPointCloud"),
             ["scene.mvs", "--dense-config-file", "Densify.ini", "--fusion-mode", "-1", "-w", "\"%mvs_dir%\""]],
            ["Fuse disparity-maps",
             os.path.join(OPENMVS_BIN, "DensifyPointCloud"),
             ["scene.mvs", "--dense-config-file", "Densify.ini", "--fusion-mode", "-2", "-w", "\"%mvs_dir%\""]]
            ]

    def __getitem__(self, indice):
        return AStep(*self.steps_data[indice])

    def length(self):
        return len(self.steps_data)

    def apply_conf(self, conf):
        """ replace each %var% per conf.var value in steps data """
        for s in self.steps_data:
            o2 = []
            for o in s[2]:
                co = o.replace("%input_dir%", conf.input_dir)
                co = co.replace("%output_dir%", conf.output_dir)
                co = co.replace("%matches_dir%", conf.matches_dir)
                co = co.replace("%reconstruction_dir%", conf.reconstruction_dir)
                co = co.replace("%mvs_dir%", conf.mvs_dir)
                co = co.replace("%camera_file_params%", conf.camera_file_params)
                o2.append(co)
            s[2] = o2

    def replace_opt(self, idx, str_exist, str_new):
        """ replace each existing str_exist with str_new per opt value in step idx data """
        s = self.steps_data[idx]
        o2 = []
        for o in s[2]:
            co = o.replace(str_exist, str_new)
            o2.append(co)
        s[2] = o2

CONF = ConfContainer()
STEPS = StepsStore()

PARSER = argparse.ArgumentParser(
    formatter_class=argparse.RawTextHelpFormatter,
    description="Photogrammetry reconstruction with these steps: \r\n" +
    "\r\n".join(("\t%i. %s\t %s" % (t, STEPS[t].info, STEPS[t].cmd) for t in range(STEPS.length())))
    )
PARSER.add_argument('input_dir',
                    help="the directory which contains the pictures set.")
PARSER.add_argument('output_dir',
                    help="the directory which will contain the resulting files.")
PARSER.add_argument('--steps',
                    type=int,
                    nargs="+",
                    help="steps to process")
PARSER.add_argument('--preset',
                    help="steps list preset in \r\n" +
                    " \r\n".join([k + " = " + str(PRESET[k]) for k in PRESET]) +
                    " \r\ndefault : " + PRESET_DEFAULT)

GROUP = PARSER.add_argument_group('Passthrough', description="Option to be passed to command lines (remove - in front of option names)\r\ne.g. --1 p ULTRA to use the ULTRA preset in openMVG_main_ComputeFeatures\r\nFor example, running the script as follows,\r\nMvgMvsPipeline.py input_dir output_dir --1 p HIGH n 8 --3 n ANNL2\r\nwhere --1 refer to openMVG_main_ComputeFeatures, p refers to\r\ndescriberPreset option which HIGH was chosen, and n refers to\r\nnumThreads which 8 was used. --3 refer to second step (openMVG_main_ComputeMatches),\r\nn refers to nearest_matching_method option which ANNL2 was chosen")
for n in range(STEPS.length()):
    GROUP.add_argument('--'+str(n), nargs='+')

PARSER.parse_args(namespace=CONF)

def mkdir_ine(dirname):
    """Create the folder if not presents"""
    if not os.path.exists(dirname):
        os.mkdir(dirname)

CONF.input_dir = os.path.abspath(CONF.input_dir)
CONF.output_dir = os.path.abspath(CONF.output_dir)

if not os.path.exists(CONF.input_dir):
    sys.exit("%s: path not found" % CONF.input_dir)

CONF.reconstruction_dir = os.path.join(CONF.output_dir, "sfm")
CONF.matches_dir = os.path.join(CONF.reconstruction_dir, "matches")
CONF.mvs_dir = os.path.join(CONF.output_dir, "mvs")
CONF.camera_file_params = os.path.join(CAMERA_SENSOR_DB_DIRECTORY, CAMERA_SENSOR_DB_FILE)

mkdir_ine(CONF.output_dir)
mkdir_ine(CONF.reconstruction_dir)
mkdir_ine(CONF.matches_dir)
mkdir_ine(CONF.mvs_dir)

STEPS.apply_conf(CONF)

if CONF.steps and CONF.preset:
    sys.exit("Steps and preset arguments can't be set together.")
elif CONF.preset:
    try:
        CONF.steps = PRESET[CONF.preset]
    except KeyError:
        sys.exit("Unknown preset %s, choose %s" % (CONF.preset, ' or '.join(
展开收缩
))) elif not CONF.steps: CONF.steps = PRESET[PRESET_DEFAULT] print("# Using input dir: %s" % CONF.input_dir) print("# output dir: %s" % CONF.output_dir) print("# Steps: %s" % str(CONF.steps)) if 4 in CONF.steps: if 6 in CONF.steps: STEPS.replace_opt(4, FOLDER_DELIM+"matches.f.bin", FOLDER_DELIM+"matches.e.bin") STEPS[4].opt.extend(["-g", "e"]) if 15 in CONF.steps: if 14 not in CONF.steps: STEPS.replace_opt(15, "scene_dense_mesh_refine.mvs", "scene_dense_mesh.mvs") for cstep in CONF.steps: printout("#%i. %s" % (cstep, STEPS[cstep].info), effect=INVERSE) opt = getattr(CONF, str(cstep)) if opt: for o in range(0, len(opt), 2): if len(opt[o]) > 1: opt[o] = '-' + opt[o] opt[o] = '-' + opt[o] else: opt = [] for anOpt in STEPS[cstep].opt: if anOpt in opt: idx = STEPS[cstep].opt.index(anOpt) if DEBUG: print('#\tRemove ' + str(anOpt) + ' from defaults options at id ' + str(idx)) del STEPS[cstep].opt[idx:idx+2] cmdline = [STEPS[cstep].cmd] + STEPS[cstep].opt + opt print('Cmd: ' + ' '.join(cmdline)) if not DEBUG: try: pStep = subprocess.Popen(cmdline) pStep.wait() if pStep.returncode != 0: break except KeyboardInterrupt: sys.exit('\r\nProcess canceled by user, all files remains') else: print('\t'.join(cmdline)) printout("# Pipeline end #", effect=INVERSE)

reference:

https://openmvg.readthedocs.io/en/latest/software/SfM/SfMInit_ImageListing/

Original: https://blog.csdn.net/weixin_44671418/article/details/125066732
Author: 人工智睿
Title: 使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)



相关阅读1

Title: 生成式模型之 GAN

生成对抗网络(Generative Adversarial Networks,GANs),由2014年还在蒙特利尔读博士的Ian Goodfellow引入深度学习领域。2016年,GANs热潮席卷AI领域顶级会议,从ICLR到NIPS,大量高质量论文被发表和探讨。Yann LeCun曾评价GANs是"20年来机器学习领域最酷的想法"。

Generative Adversarial Nets(GAN)

Generative Adversarial Networks论文提出了一种通过对抗过程来评估生成模型。其训练两个模型:仿照原始数据分布生成数据的模型G和评估数据来源(原始数据/生成数据)的模型D。训练G的目标是最大化D犯错的概率,训练D的目标是最大化区分真实训练样本与G生成的样本的能力。

如果能够知道训练样本的分布(p(x)),那么就可以在分布中随机采样得到新样本,大部分的生成式模型都采用这种思路,GAN则是在学习从随机变量z到训练样本x的映射关系,其中随机变量可以选择服从正太分布,那么就能得到一个由多层感知机组成的生成网络(G(z;\theta_g)),网络的输入是一个一维的随机变量,输出是一张图片。

GAN的优化是一个极小极大博弈问题,公式如下:

[\underset{G}{\min} \: \underset{D}{\max}V(D,G) =\mathbb E_{x\sim p_{data}(x)}[logD(x)]+\mathbb E_{z\sim p_{z}(z)}[log(1-D(G(z)))] ]

优化这个函数,使(p_z(x))接近(p_{data}).下面首先去掉期望符号:

[\begin{align} V(G,D)&=\int_x p_{data}(x)\log(D(x))dx+\int_zp_z(z)\log(1-D(g(z)))dz \ &=\int_x [p_{data}(x)\log(D(x))+p_g(x)\log(1-D(x))]dx \end{align} ]

先固定G,求(\underset{D}{\max}V(D,G)),令其导数等于0,求得D的最优解

[D^*G(x)={p{data}(x)\over p_{data}(x)+p_g(x)} ]

现在固定D,优化G:将(D^*_G)带入目标函数。

[\begin{align} \underset{G}\min V(G,D^*G) &= \int_x

dx \ &= \mathbb E_{x\sim p_{data}}[\log{p_{data}(x)\over p_{data}(x)+p_g(x)}]+\mathbb E_{x\sim p_g}[\log{p_g(x)\over p_{data}(x)+p_g(x)}] \ &= -\log 4+KL(p_{data}\|{p_{data}+p_g\over 2})+KL(p_g\|{p_{data}+p_g\over 2}) \ &= -\log 4+2JS(p_{data}\|p_g) \end{align} ]

其中KL散度:(KL(P\|Q)=\mathbb E_{x\sim P}\log{P\over Q}=\int_xP(x)\log{P(x)\over Q(x)}dx)

JS散度:(JS(P\|Q)={1\over 2}KL(P\|{P+Q\over 2})+{1\over 2}KL(Q\|{P+Q\over 2}))

JS散度具有对称性,而KL没有。

只要P和Q没有一点重叠或者重叠部分可忽略,JS散度就固定是常数, 而这对于梯度下降方法意味着——梯度为0!此时对于最优判别器来说,生成器得不到梯度信息;即使对于接近最优的判别器来说,生成器也有很大机会面临梯度消失的问题。

参考 WGAN的介绍

f-GAN

在GAN中可以使用任意的f-divergency,相关论文f-GAN(Sebastian Nowozin, Botond Cseke, Ryota Tomioka, "f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization", NIPS, 2016)

f-divergence

P和Q是两个分布,p(x),q(x)是x的分布概率

[D_f(P||Q)=\int_x q(x)f({p(x)\over q(x)})dx ]

其中f是凸函数且f(1)=0,(D_f(P||Q))衡量了P和Q之间的距离.

当(\forall x,p(x)=q(x))时,(D_f(P||Q))具有最小值0.

当(f(x)=x\log x)时,(D_f(P||Q)=\int_xp(x)\log({p(x)\over q(x)})dx),即KL divergence.

当(f(x)=-\log x)时,(D_f(P||Q)=\int_xq(x)\log({q(x)\over p(x)})dx),即reverse KL divergence.

当(f(x)=(x-1)^2)时,(D_f(P||Q)=\int_x{(p(x)-q(x))^2\over q(x)}dx)为Chi Square divergence.

Fenchel Conjugate

每个凸函数f都有一个与之相对的conjugate function f*:

(f^ (t)=\max_{x\in dom(f)}{xt-f(x)}),且(f ) * = f.

(f(x)=\max_{t\in dom(f^)}{xt-f^(t)}),带入(D_f(P||Q))得:

[\begin{align} D_f(P||Q) &=\int_x q(x)f({p(x)\over q(x)})dx \ &=\int_xq(x)(\max_{t\in dom(f^)}{{p(x)\over q(x)}t-f^(t)})dx \ &=\max_D\int_x p(x)D(x)dx-\int_x q(x)f^*(D(x))dx \ &\text{(t=D(x))} \end{align} ]

因此GAN中

[D_f(P_{data}\|P_G)=\max_D{E_{x\sim P_{data}}[D(x)]-E_{x\sim P_G}[f^*(D(x))]} ]

可以使用任何的f-divergence,如JS,Jeffrey,Pearson.

WGAN

原始版本:weight clipping,改进版本:gradient penalty.

论文:

  • Martin Arjovsky, Soumith Chintala, Léon Bottou, Wasserstein GAN, arXiv preprint, 2017
  • Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville,"Improved Training of Wasserstein GANs", arXiv preprint, 2017

主要思想:使用Earth Mover's Distance(Wasserstein Distance)来评估两个分布之间的距离.推土机距离表示将一个分布搬运变为另一个分布的最小搬运的量.

之前GAN所采用的JS divergence的缺点是当两个分布没有交集时,距离是0,梯度为0,网络很难学习.Earth Mover's Distance便可以解决这个问题.此时网络能够持续学习,但为了防止梯度爆炸,需要weight clipping等手段.

对抗样本(adversarial examples)

14年的时候Szegedy在研究神经网络的性质时,发现针对一个已经训练好的分类模型,将训练集中样本做一些细微的改变会导致模型给出一个错误的分类结果,这种虽然发生扰动但是人眼可能识别不出来,并且会导致误分类的样本被称为对抗样本,他们利用这样的样本发明了对抗训练(adversarial training),模型既训练正常的样本也训练这种自己造的对抗样本,从而改进模型的泛化能力[1]。如下图所示,在未加扰动之前,模型认为输入图片有57.7%的概率为熊猫,但是加了之后,人眼看着好像没有发生改变,但是模型却认为有99.3%的可能是长臂猿。
使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

对抗样本跟生成式对抗网络没有直接的关系,对抗网络是想学样本的内在表达从而能够生成新的样本,但是有对抗样本的存在在一定程度上说明了模型并没有学习到数据的一些内部表达或者分布,而可能是学习到一些特定的模式足够完成分类或者回归的目标而已。

GAN生成的图片能否用于CNN训练?

现在来说,应当不可以。由于GAN是从较小的分布中采样生成的,是真实世界的极小的一部分,所以拿来训练没有广泛的适用性。另外,当前的GAN生成较大的图片比较困难(32x32以上)。

参考资料

Original: https://www.cnblogs.com/makefile/p/GAN.html
Author: 康行天下
Title: 生成式模型之 GAN

相关阅读2

Title: 【3】VSCode 主题设置推荐,自定义配色方案,修改注释高亮颜色

Title: 【3】VSCode 主题设置推荐,自定义配色方案,修改注释高亮颜色

相关文章:

VSCode 主题设置推荐,自定义配色方案,修改注释高亮颜色

设置主题:

个人比较忠爱vscode的界面,感觉比pycharm要更美观点,虽然两个都装了,但还是会习惯性打开vscode进行调试。下面是我使用的主题。

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列) 使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

或者在设置--外观里面进行细致设置

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

vscode官网:Themes Extensions - Visual Studio Marketplace,大家可以选择自己喜欢的主题

主题推荐

  • *One Dark Pro:

One Dark Pro - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

最新版本3.10.12 2021.4.10更新,下载量还是很多的,也很不错几面清新舒适

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

  • *Palenight Theme

Palenight Theme - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

2.0.1最新更新版本2020.9.21,界面比较有个性。

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

  • *Monokai Pro

Monokai Pro - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

VersionLast Updated1.1.192021/4/6

  • *Dracula Official

Dracula Official - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

VersionLast Updated2.22.32020/11/24

深紫色的色调使这个主题有别于其他主题。我喜欢它!

[En]

The dark purple hue distinguishes this theme from other themes. I love it!

  • *One Monokai Theme

One Monokai Theme - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

多彩的主题! 很多颜色,无处不在

VersionLast Updated0.5.02020/10/16

  • *Material Theme

Material Theme - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

VersionLast Updated33.2.22021/3/15

干净简约。

  • *Panda Theme

Panda Theme - Visual Studio Marketplace

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

VersionLast Updated1.3.02018/6/30

自定义配色方案

首先进入安装vscode目录找到想要修改调整的主题,以我的文件路径为例:H:\VScode\Microsoft VS Code\resources\app\extensions,主题配置文件都位于 resources/app/extensions目录中,以 theme-开头的目录即为颜色主题配置:

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

每个颜色主题配置目录包含以下文件:其中 package.json我们可以用来配色方案。

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

打开

{"name":"theme-monokai","displayName":"%displayName%",
"description":"%description%","version":"1.0.0",
"publisher":"vscode",
"license":"MIT","engines":{"vscode":"*"},
"contributes":{"themes":[{"id":"Monokai","label":"%themeLabel%",
"uiTheme":"vs-dark","path":"./themes/monokai-color-theme.json"}]},
"repository":{"type":"git","url":"https://github.com/microsoft/vscode.git"}}

参数名作用name主题ID,必需在VSCode中全局唯一,即所有主题的package.json中该值均不能重复contributes -> themes -> label主题名,"文件-首选项-颜色主题"的列表中显示该值contributes -> themes -> uiThemeVSCode整体的UI主题,vs为浅色主题contributes -> themes -> path定义配色方案的文件名,如为相对路径则相对于此文件

再打开themes文件夹下monokai-color-theme.json进行配置。

参数名作用colorsVSCode各个UI组件的颜色tokenColors语法高亮颜色
colors节点的内容直接通过键值对参数描述,

以下列举几个参数的作用:

图示参数名作用2activityBar.background活动栏背景色1activityBar.foreground活动栏前景色(例如用于图标)12editor.background编辑器背景颜色13editor.foreground编辑器默认前景色editor.findMatchBackground当前搜索匹配项的颜色editor.findMatchHighlightBackground其他搜索匹配项的颜色15editor.lineHighlightBackground光标所在行高亮文本的背景颜色editor.selectionBackground编辑器所选内容的颜色editor.selectionHighlightBackground与所选内容具有相同内容的区域颜色editor.rangeHighlightBackground突出显示范围的背景颜色,例如 "Quick Open" 和"查找"功能16editorBracketMatch.background匹配括号的背景色14editorCursor.foreground编辑器光标颜色11editorGutter.background编辑器导航线的背景色,导航线包括边缘符号和行号10editorLineNumber.foreground编辑器行号颜色5sideBar.background侧边栏背景色4sideBar.foreground侧边栏前景色3sideBarSectionHeader.background侧边栏节标题的背景颜色17statusBar.background标准状态栏背景色17statusBar.noFolderBackground没有打开文件夹时状态栏的背景色17statusBar.debuggingBackground调试程序时状态栏的背景色9tab.activeBackground活动选项卡的背景色8tab.activeForeground活动组中活动选项卡的前景色7tab.inactiveBackground非活动选项卡的背景色6tab.inactiveForeground活动组中非活动选项卡的前景色

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)
tokenColors

tokenColors使用一个对象数组描述各语法高亮颜色。每个对象有如下结构:

{
    "name": "Comment",
    "scope": [
        "comment",
        "punctuation.definition.comment"
    ],
    "settings": {
        "background": "#ffffff",
        "fontStyle": "italic",
        "foreground": "#000000"
    }
}

参数名作用name规则描述,一段容易理解的描述性文字scope作用域,指定使用那些VSCode内部对象,其含义参看
Scope Naming

setting -> background背景色,可选setting -> fontStyle字体,可选,为bold、italic、underlinesetting -> foreground前景色,可选

以下列举文末的配置文件中几个name所指定的参数的作用:

参数名作用Character字符Class类名Comment注释Function函数名Keyword关键字Number数值Operator运算符Parameter函数参数Punctuation标点符号String字符串Type内置类型Variable变量名

参考链接:VSCode自定义配色方案_weixin_30755393的博客-CSDN博客
具体文件内容我就不贴了,改改颜色和高亮很简单的。
颜色:网址提供:
颜色中英文对照表 颜色名字 色彩名称-www.5tu.cn
可以根据自己的喜好进行调整

修改注释高亮颜色

下面我们将进行选择时显示高亮和注释颜色修改,首先打开settings.json文件
选择高亮:

在setting.json中添加如下字段即可,颜色可以自定义修改【参考上面我提供的颜色网址】

    "workbench.colorCustomizations": {
        "editor.selectionBackground": "#e46bc9",
        "editor.selectionHighlightBackground": "#a32c3c"
        },

修改注释

修改注释颜色,同样在setting.json中添加:

        "editor.tokenColorCustomizations": {
            "comments": "#3CB371"  //春天绿
        },

效果图如下:选择时会高亮,注释我更喜欢淡紫色

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

Original: https://blog.csdn.net/sinat_39620217/article/details/115618730
Author: 汀、
Title: 【3】VSCode 主题设置推荐,自定义配色方案,修改注释高亮颜色

Original: https://blog.csdn.net/sinat_39620217/article/details/115618730
Author: 汀、
Title: 【3】VSCode 主题设置推荐,自定义配色方案,修改注释高亮颜色

相关阅读3

Title: PyTorch之torch.utils.data.DataLoader详解

使用openMVG+openMVS对自制数据集三维重建(单相机图片序列)

1、dataset:(数据类型 dataset)

输入的数据类型,这里是原始数据的输入。PyTorch内也有这种数据结构。

2、batch_size:(数据类型 int)

批训练数据量的大小,根据具体情况设置即可(默认:1)。PyTorch训练模型时调用数据不是一行一行进行的(这样太没效率),而是一捆一捆来的。这里就是定义每次喂给神经网络多少行数据,如果设置成1,那就是一行一行进行(个人偏好,PyTorch默认设置是1)。每次是随机读取大小为batch_size。如果dataset中的数据个数不是batch_size的整数倍,这最后一次把剩余的数据全部输出。若想把剩下的不足batch size个的数据丢弃,则将drop_last设置为True,会将多出来不足一个batch的数据丢弃。

3、shuffle:(数据类型 bool)

洗牌。默认设置为False。在每次迭代训练时是否将数据洗牌,默认设置是False。将输入数据的顺序打乱,是为了使数据更有独立性,但如果数据是有序列特征的,就不要设置成True了。

4、collate_fn:(数据类型 callable,没见过的类型)

将一小段数据合并成数据列表,默认设置是False。如果设置成True,系统会在返回前会将张量数据(Tensors)复制到CUDA内存中。

5、batch_sampler:(数据类型 Sampler)

批量采样,默认设置为None。但每次返回的是一批数据的索引(注意:不是数据)。其和batch_size、shuffle 、sampler and drop_last参数是不兼容的。我想,应该是每次输入网络的数据是随机采样模式,这样能使数据更具有独立性质。所以,它和一捆一捆按顺序输入,数据洗牌,数据采样,等模式是不兼容的。

6、sampler:(数据类型 Sampler)

采样,默认设置为None。根据定义的策略从数据集中采样输入。如果定义采样规则,则洗牌(shuffle)设置必须为False。

7、num_workers:(数据类型 Int)

工作者数量,默认是0。使用多少个子进程来导入数据。设置为0,就是使用主进程来导入数据。注意:这个数字必须是大于等于0的,负数估计会出错。

8、pin_memory:(数据类型 bool)

内存寄存,默认为False。在数据返回前,是否将数据复制到CUDA内存中。

9、drop_last:(数据类型 bool)

丢弃最后数据,默认为False。设置了 batch_size 的数目后,最后一批数据未必是设置的数目,有可能会小些。这时你是否需要丢弃这批数据。

10、timeout:(数据类型 numeric)

超时,默认为0。是用来设置数据读取的超时时间的,但超过这个时间还没读取到数据的话就会报错。 所以,数值必须大于等于0。

11、worker_init_fn(数据类型 callable,没见过的类型)

子进程导入模式,默认为Noun。在数据导入前和步长结束后,根据工作子进程的ID逐个按顺序导入数据。
对batch_size举例分析:

"""
    批训练,把数据变成一小批一小批数据进行训练。
    DataLoader就是用来包装所使用的数据,每次抛出一批数据
"""
import torch
import torch.utils.data as Data

BATCH_SIZE = 5

x = torch.linspace(1, 11, 11)
y = torch.linspace(11, 1, 11)
print(x)
print(y)
# 把数据放在数据库中
torch_dataset = Data.TensorDataset(x, y)
loader = Data.DataLoader(
    # 从数据库中每次抽出batch size个样本
    dataset=torch_dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
    # num_workers=2,
)

def show_batch():
    for epoch in range(3):
        for step, (batch_x, batch_y) in enumerate(loader):
            # training
            print("steop:{}, batch_x:{}, batch_y:{}".format(step, batch_x, batch_y))

if __name__ == '__main__':
    show_batch()

输出为:

tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.])
tensor([11., 10.,  9.,  8.,  7.,  6.,  5.,  4.,  3.,  2.,  1.])
steop:0, batch_x:tensor([ 3.,  2.,  8., 11.,  1.]), batch_y:tensor([ 9., 10.,  4.,  1., 11.])
steop:1, batch_x:tensor([ 5.,  6.,  7.,  4., 10.]), batch_y:tensor([7., 6., 5., 8., 2.])
steop:2, batch_x:tensor([9.]), batch_y:tensor([3.])
steop:0, batch_x:tensor([ 9.,  7., 10.,  2.,  4.]), batch_y:tensor([ 3.,  5.,  2., 10.,  8.])
steop:1, batch_x:tensor([ 5., 11.,  3.,  6.,  8.]), batch_y:tensor([7., 1., 9., 6., 4.])
steop:2, batch_x:tensor([1.]), batch_y:tensor([11.])
steop:0, batch_x:tensor([10.,  5.,  7.,  4.,  2.]), batch_y:tensor([ 2.,  7.,  5.,  8., 10.])
steop:1, batch_x:tensor([3., 9., 1., 8., 6.]), batch_y:tensor([ 9.,  3., 11.,  4.,  6.])
steop:2, batch_x:tensor([11.]), batch_y:tensor([1.])

Process finished with exit code 0

若drop_last=True

"""
    批训练,把数据变成一小批一小批数据进行训练。
    DataLoader就是用来包装所使用的数据,每次抛出一批数据
"""
import torch
import torch.utils.data as Data

BATCH_SIZE = 5

x = torch.linspace(1, 11, 11)
y = torch.linspace(11, 1, 11)
print(x)
print(y)
# 把数据放在数据库中
torch_dataset = Data.TensorDataset(x, y)
loader = Data.DataLoader(
    # 从数据库中每次抽出batch size个样本
    dataset=torch_dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
    # num_workers=2,
    drop_last=True,
)

def show_batch():
    for epoch in range(3):
        for step, (batch_x, batch_y) in enumerate(loader):
            # training
            print("steop:{}, batch_x:{}, batch_y:{}".format(step, batch_x, batch_y))

if __name__ == '__main__':
    show_batch()

对应的输出为:


tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.])
tensor([11., 10.,  9.,  8.,  7.,  6.,  5.,  4.,  3.,  2.,  1.])
steop:0, batch_x:tensor([ 9.,  2.,  7.,  4., 11.]), batch_y:tensor([ 3., 10.,  5.,  8.,  1.])
steop:1, batch_x:tensor([ 3.,  5., 10.,  1.,  8.]), batch_y:tensor([ 9.,  7.,  2., 11.,  4.])
steop:0, batch_x:tensor([ 5., 11.,  6.,  1.,  2.]), batch_y:tensor([ 7.,  1.,  6., 11., 10.])
steop:1, batch_x:tensor([ 3.,  4., 10.,  8.,  9.]), batch_y:tensor([9., 8., 2., 4., 3.])
steop:0, batch_x:tensor([10.,  4.,  9.,  8.,  7.]), batch_y:tensor([2., 8., 3., 4., 5.])
steop:1, batch_x:tensor([ 6.,  1., 11.,  2.,  5.]), batch_y:tensor([ 6., 11.,  1., 10.,  7.])

Process finished with exit code 0

Original: https://blog.csdn.net/qq_36044523/article/details/118914223
Author: 进击的程小白
Title: PyTorch之torch.utils.data.DataLoader详解