跳到主要内容

AR视觉

AR视觉1、概述2、进入Docker3、使用方法3.1、相机内参标定3.2、启动3.3、程序启动3.4、RANSAC方案3.5、源码

1、概述​

增强现实(Augmented Reality)，简称“AR”，技术是一种将虚拟信息与真实世界巧妙融合的技术，广泛运用了多媒体、三维建模、实时跟踪及注册、智能交互、传感等多种技术手段，将计算机生成的文字、图像、三维模型、音乐、视频等虚拟信息模拟仿真后，应用到真实世界中，两种信息互为补充，从而实现对真实世界的“增强”。

AR系统具有三个突出的特点：①真实世界和虚拟世界的信息集成；②具有实时交互性；③是在三维尺度空间中增添定位虚拟物体。

增强现实技术包含了多媒体、三维建模、实时视频显示及控制、多传感器融合、实时跟踪及注册、场景融合等新技术与新手段。

2、进入Docker​

xxxxxxxxxx sh ~/ros_melodic.sh

image-20250115200043910

3、使用方法​

在使用AR案例时候，必须要相机的内参，不然无法运行（出厂的镜像已完成相机内参标定），内参文件与代码同目录。

3.1、相机内参标定​

启动单目相机

roslaunch usb_cam usb_cam-test.launch

启动标定节点

xxxxxxxxxx rosrun camera_calibration cameracalibrator.py image:=/usb_cam/image_raw --size 9x6 --square 0.02

size：标定棋盘格的内部角点个数，例如9X6，角点一共六行九列。

square：棋盘格的边长，单位是米。

image：摄像头发布的图像话题

image-20231016164051983

标定界面

X：棋盘格在摄像头视野中的左右移动

Y：棋盘格在摄像头视野中的上下移动

Size ：棋盘格在摄像头视野中的前后移动

Skew：棋盘格在摄像头视野中的倾斜转动

启动成功后，将棋盘格放入画面中心，变换不同位姿。系统会自主识别，最好的情况是【X】、【Y】、【Size】、【Skew】下面的线条随着数据的收集先由红色变黄再变绿，尽可能填充满。

image-20231016164152328

  • 点击【CALIBRATE】计算相机内参，图片越多，时间越久，等待就好。（六七十张就差不多了，太多容易卡死）。

  • 点击【SAVE】将标定结果保存至当前运行终端的【/tmp/calibrationdata.tar.gz】。

标定结束后，可以移动出来【/tmp/calibrationdata.tar.gz】文件看看内容，

xxxxxxxxxx sudo mv /tmp/calibrationdata.tar.gz ~

解压后，里面有刚刚标定的图片，一个ost.txt文件和一个ost.yaml文件。ost.yaml就是标定相机的内参。把这里的内参的内容复制一份覆盖到/home/yahboom/dofbot_ws/src/dofbot_visual/AR下的astra.yaml。

3.2、启动​

程序启动后，把棋盘格放在相机面前，注意需要看到整个棋盘格，否则程序会退出。识别到整个棋盘格后，会显示以下12种效果，

xxxxxxxxxx ["Triangle", "Rectangle", "Parallelogram","WindMill","TableTennisTable", "Ball","Arrow", "Knife", "Desk","Bench", "Stickman", "ParallelBars"]

点击图像，按下F切换显示的效果。

3.3、程序启动​

终端输入，

xxxxxxxxxx roscore rosrun yahboomcar_visual simple_AR.py

image-20231016171452040

3.4、RANSAC方案​

  • 算法原理：使用RANSAC方案从3D-2D点对应中查找对象姿势

RanSaC算法（随机采样一致）原本是用于数据处理的一种经典算法，其作用是在大量噪声情况下，提取物体中特定的成分。下图是对RanSaC算法效果的说明。图中有一些点显然是满足某条直线的，另外有一团点是纯噪声。目的是在大量噪声的情况下找到直线方程，此时噪声数据量是直线的3倍。

image-20231016171619883

xxxxxxxxxx 如果用最小二乘法是无法得到这样的效果的，直线大约会在图中直线偏上一点。

  • RANSAC的基本假设是：

    • 数据由“局内点”组成，例如：数据的分布可以用一些模型参数来解释；

    • “局外点”是不能适应该模型的数据；

    • 除此之外的数据属于噪声。

  • 局外点产生的原因有：噪声的极值；错误的测量方法；对数据的错误假设。RANSAC也做了以下假设：给定一组（通常很小的）局内点，存在一个可以估计模型参数的过程；而该模型能够解释或者适用于局内点。

3.5、源码​

源码位置：~/yahboomcar_ws/src/yahboomcar_visual/AR

​ x #!/usr/bin/env python # -*- coding: utf-8 -*- import os import sys import time import rospy import rospkg import cv2 as cv import numpy as np from cv_bridge import CvBridge from std_msgs . msg import String from sensor_msgs . msg import CompressedImage , Image class simple_AR : def __init__ ( self ): rospy . on_shutdown ( self . cancel ) self . index = 0 self . frame = None self . img_name = 'img' self . patternSize = ( 6 , 9 ) self . bridge = CvBridge () self . flip = rospy . get_param ( "~flip" , False ) # 加载相机内参矩阵、畸变系数 # Load the camera internal parameter matrix and distortion coefficient yaml_path = rospkg . RosPack (). get_path ( "dofbot_visual" ) \+ '/AR/astra.yaml' if os . path . exists ( yaml_path ): fs = cv . FileStorage ( yaml_path , cv . FileStorage_READ ) self . cameraMatrix = fs . getNode ( "camera_matrix" ). mat () self . distCoeffs = fs . getNode ( "distortion_coefficients" ). mat () else : self . distCoeffs , self . cameraMatrix = (), () self . objectPoints = np . zeros (( 6 * 9 , 3 ), np . float32 ) self . objectPoints [:, : 2 ] = np . mgrid [ 0 : 6 , 0 : 9 ]. T . reshape ( \- 1 , 2 ) self . graphics = [ "Triangle" , "Rectangle" , "Parallelogram" , "WindMill" , "TableTennisTable" , "Ball" , "Arrow" , "Knife" , "Desk" , "Bench" , "Stickman" , "ParallelBars" ] self . Graphics = self . graphics [ self . index ] self . axis = np . float32 ([ [ 0 , 0 , \- 1 ], [ 0 , 8 , \- 1 ], [ 5 , 8 , \- 1 ], [ 5 , 0 , \- 1 ], [ 1 , 2 , \- 1 ], [ 1 , 6 , \- 1 ], [ 4 , 2 , \- 1 ], [ 4 , 6 , \- 1 ], [ 1 , 0 , \- 4 ], [ 1 , 8 , \- 4 ], [ 4 , 0 , \- 4 ], [ 4 , 8 , \- 4 ], [ 1 , 2 , \- 4 ], [ 1 , 6 , \- 4 ], [ 4 , 2 , \- 4 ], [ 4 , 6 , \- 4 ], [ 0 , 1 , \- 4 ], [ 3 , 2 , \- 1 ], [ 2 , 2 , \- 3 ], [ 3 , 2 , \- 3 ], [ 1 , 2 , \- 3 ], [ 2 , 2 , \- 4 ], [ 2 , 2 , \- 5 ], [ 0 , 4 , \- 4 ], [ 2 , 3 , \- 4 ], [ 1 , 3 , \- 4 ], [ 4 , 3 , \- 5 ], [ 4 , 5 , \- 5 ], [ 1 , 2 , \- 3 ], [ 1 , 6 , \- 3 ], [ 5 , 2 , \- 3 ], [ 5 , 6 , \- 3 ], [ 3 , 4 , \- 5 ], [ 0 , 6 , \- 4 ], [ 5 , 6 , \- 4 ], [ 2 , 8 , \- 4 ], [ 3 , 8 , \- 4 ], [ 2 , 6 , \- 4 ], [ 2 , 0 , \- 4 ], [ 1 , 5 , \- 4 ], [ 3 , 0 , \- 4 ], [ 3 , 2 , \- 4 ], [ 0 , 3 , \- 4 ], [ 1 , 2 , \- 4 ], [ 4 , 2 , \- 4 ], [ 5 , 3 , \- 4 ], [ 2 , 7 , \- 4 ], [ 3 , 7 , \- 4 ], [ 3 , 3 , \- 1 ], [ 3 , 5 , \- 1 ], [ 1 , 5 , \- 1 ], [ 1 , 3 , \- 1 ], [ 3 , 3 , \- 3 ], [ 3 , 5 , \- 3 ], [ 1 , 5 , \- 3 ], [ 1 , 3 , \- 3 ], [ 1 , 3 , \- 6 ], [ 1 , 5 , \- 6 ], [ 3 , 3 , \- 4 ], [ 3 , 5 , \- 4 ], [ 0 , 0 , \- 4 ], [ 3 , 1 , \- 4 ], [ 1 , 1 , \- 4 ], [ 0 , 2 , \- 4 ], [ 2 , 4 , \- 4 ], [ 4 , 4 , \- 4 ], [ 0 , 8 , \- 4 ], [ 5 , 8 , \- 4 ], [ 5 , 0 , \- 4 ], [ 0 , 4 , \- 5 ], [ 5 , 4 , \- 4 ], [ 5 , 4 , \- 5 ], [ 2 , 5 , \- 1 ], [ 2 , 7 , \- 1 ], [ 2 , 6 , \- 3 ], [ 2 , 6 , \- 5 ], [ 2 , 5 , \- 3 ], [ 2 , 7 , \- 3 ] ]) self . sub_graphics = rospy . Subscriber ( '/Graphics_topic' , String , self . choose_Graphics ) self . pub_img = rospy . Publisher ( "/simpleAR/camera" , Image , queue_size = 1 ) def cancel ( self ): self . sub_graphics . unregister () self . pub_img . unregister () cv . destroyAllWindows () rospy . loginfo ( "Shutting down this node." ) def choose_Graphics ( self , msg ): if not isinstance ( msg , String ): return if msg . data in self . graphics : self . Graphics = msg . data else : self . graphics_update () def graphics_update ( self ): self . index += 1 if self . index >= len ( self . graphics ): self . index = 0 self . Graphics = self . graphics [ self . index ] def process ( self , img , action ): if self . flip == True : img = cv . flip ( img , 1 ) if action == ord ( 'f' ) or action == ord ( 'F' ): self . graphics_update () gray = cv . cvtColor ( img , cv . COLOR_BGR2GRAY ) # 查找每个图片的角点 # Find the corner of each image retval , corners = cv . findChessboardCorners ( gray , self . patternSize , None , flags = cv . CALIB_CB_ADAPTIVE_THRESH \+ cv . CALIB_CB_NORMALIZE_IMAGE \+ cv . CALIB_CB_FAST_CHECK ) # 查找角点亚像素 # Find corner subpixels if retval : corners = cv . cornerSubPix ( gray , corners , ( 11 , 11 ), ( \- 1 , \- 1 ), ( cv . TERM_CRITERIA_EPS \+ cv . TERM_CRITERIA_MAX_ITER , 30 , 0.001 )) # 计算对象姿态solvePnPRansac # Compute object pose solvePnPRansac retval , rvec , tvec , inliers = cv . solvePnPRansac ( self . objectPoints , corners , self . cameraMatrix , self . distCoeffs ) # 输出图像点和雅可比矩阵 # Output image points and Jacobian matrix image_Points , jacobian = cv . projectPoints ( self . axis , rvec , tvec , self . cameraMatrix , self . distCoeffs , ) img = self . draw ( img , corners , image_Points ) cv . putText ( frame , self . Graphics , ( 240 , 30 ), cv . FONT_HERSHEY_SIMPLEX , 0.9 , ( 0 , 0 , 255 ), 1 ) self . pub_img . publish ( self . bridge . cv2_to_imgmsg ( img , "bgr8" )) return img def draw ( self , img , corners , image_Points ): # drawContours函数中绘图颜色顺序是bgr # drawContours the color order of the drawing is BGR img_pts = np . int32 ( image_Points ). reshape ( \- 1 , 2 ) if self . Graphics == "Triangle" : cv . drawContours ( img , [ np . array ([ img_pts [ 14 ], img_pts [ 15 ], img_pts [ 23 ]])], \- 1 , ( 255 , 0 , 0 ), \- 1 ) elif self . Graphics == "Rectangle" : cv . drawContours ( img , [ np . array ([ img_pts [ 12 ], img_pts [ 13 ], img_pts [ 15 ], img_pts [ 14 ]])], \- 1 , ( 0 , 255 , 0 ), \- 1 ) elif self . Graphics == "Parallelogram" : cv . drawContours ( img , [ np . array ([ img_pts [ 12 ], img_pts [ 10 ], img_pts [ 15 ], img_pts [ 9 ]])], \- 1 , ( 65 , 105 , 225 ), 1 ) elif self . Graphics == "WindMill" : cv . drawContours ( img , [ np . array ([ img_pts [ 60 ], img_pts [ 38 ], img_pts [ 61 ], img_pts [ 21 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 10 ], img_pts [ 14 ], img_pts [ 58 ], img_pts [ 21 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 62 ], img_pts [ 63 ], img_pts [ 23 ], img_pts [ 21 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 25 ], img_pts [ 64 ], img_pts [ 65 ], img_pts [ 21 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) cv . line ( img , tuple ( img_pts [ 64 ]), tuple ( img_pts [ 35 ]), ( 0 , 255 , 0 ), 3 ) elif self . Graphics == "TableTennisTable" : cv . line ( img , tuple ( img_pts [ 0 ]), tuple ( img_pts [ 60 ]), ( 255 , 0 , 0 ), 3 ) for i in range ( 1 , 4 ): cv . line ( img , tuple ( img_pts [ i ]), tuple ( img_pts [ 65 \+ i ]), ( 255 , 0 , 0 ), 3 ) cv . drawContours ( img , [ np . array ([ img_pts [ 60 ], img_pts [ 66 ], img_pts [ 67 ], img_pts [ 68 ]])], \- 1 , ( 0 , 255 , 0 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 23 ], img_pts [ 69 ], img_pts [ 71 ], img_pts [ 70 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) elif self . Graphics == "Ball" : cv . circle ( img , tuple ( img_pts [ 22 ]), 30 , ( 0 , 0 , 255 ), \- 1 ) elif self . Graphics == "Arrow" : cv . drawContours ( img , [ np . array ([ img_pts [ 13 ], img_pts [ 34 ], img_pts [ 36 ]])], \- 1 , ( 0 , 255 , 0 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 37 ], img_pts [ 15 ], img_pts [ 10 ], img_pts [ 38 ]])], \- 1 , ( 0 , 255 , 0 ), \- 1 ) elif self . Graphics == "Knife" : cv . drawContours ( img , [ np . array ([ img_pts [ 58 ], img_pts [ 24 ], img_pts [ 35 ], img_pts [ 47 ]])], \- 1 , ( 160 , 252 , 0 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 40 ], img_pts [ 38 ], img_pts [ 21 ], img_pts [ 41 ]])], \- 1 , ( 30 , 144 , 255 ), \- 1 ) cv . drawContours ( img , [ np . array ([ img_pts [ 42 : 46 ]])], \- 1 , ( 0 , 0 , 255 ), \- 1 ) elif self . Graphics == "Desk" : for i in range ( 4 ): cv . line ( img , tuple ( img_pts [ 4 \+ i ]), tuple ( img_pts [ 12 \+ i ]), ( 163 , 148 , 128 ), 3 ) cv . drawContours ( img , [ np . array ([ img_pts [ 14 ], img_pts [ 12 ], img_pts [ 13 ], img_pts [ 15 ]])], \- 1 , ( 0 , 199 , 140 ), \- 1 ) elif self . Graphics == "Bench" : for i in range ( 4 ): cv . line ( img , tuple ( img_pts [ 48 \+ i ]), tuple ( img_pts [ 52 \+ i ]), ( 255 , 0 , 0 ), 3 ) cv . drawContours ( img , [ img_pts [ 52 : 56 ]], \- 1 , ( 0 , 0 , 255 ), \- 1 ) cv . drawContours ( img , [ img_pts [ 54 : 58 ]], \- 1 , ( 139 , 69 , 19 ), \- 1 ) elif self . Graphics == "Stickman" : cv . line ( img , tuple ( img_pts [ 18 ]), tuple ( img_pts [ 4 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 18 ]), tuple ( img_pts [ 6 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 18 ]), tuple ( img_pts [ 21 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 21 ]), tuple ( img_pts [ 19 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 21 ]), tuple ( img_pts [ 20 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 21 ]), tuple ( img_pts [ 22 ]), ( 0 , 0 , 255 ), 3 ) cv . circle ( img , tuple ( img_pts [ 22 ]), 15 , ( 0 , 0 , 255 ), \- 1 ) cv . line ( img , tuple ( img_pts [ 74 ]), tuple ( img_pts [ 72 ]), ( 0 , 255 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 74 ]), tuple ( img_pts [ 73 ]), ( 0 , 255 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 74 ]), tuple ( img_pts [ 37 ]), ( 0 , 255 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 37 ]), tuple ( img_pts [ 76 ]), ( 0 , 255 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 37 ]), tuple ( img_pts [ 77 ]), ( 0 , 255 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 37 ]), tuple ( img_pts [ 75 ]), ( 0 , 255 , 0 ), 3 ) cv . circle ( img , tuple ( img_pts [ 75 ]), 15 , ( 0 , 255 , 0 ), \- 1 ) elif self . Graphics == "ParallelBars" : for i in range ( 4 ): cv . line ( img , tuple ( img_pts [ 4 \+ i ]), tuple ( img_pts [ 12 \+ i ]), ( 255 , 0 , 0 ), 3 ) cv . line ( img , tuple ( img_pts [ 8 ]), tuple ( img_pts [ 9 ]), ( 0 , 0 , 255 ), 3 ) cv . line ( img , tuple ( img_pts [ 10 ]), tuple ( img_pts [ 11 ]), ( 0 , 0 , 255 ), 3 ) return img if __name__ == '__main__' : rospy . init_node ( "simple_AR" , anonymous = False ) capture = cv . VideoCapture ( 0 ) capture . set ( 6 , cv . VideoWriter_fourcc ( 'M' , 'J' , 'P' , 'G' )) capture . set ( cv . CAP_PROP_FRAME_WIDTH , 640 ) capture . set ( cv . CAP_PROP_FRAME_HEIGHT , 480 ) print ( "capture get FPS : " , capture . get ( cv . CAP_PROP_FPS )) pTime , cTime = 0 , 0 ar = simple_AR () while capture . isOpened (): ret , frame = capture . read () action = cv . waitKey ( 1 ) & 0xFF if action == ord ( 'q' ) or action == ord ( "Q" ): break frame = ar . process ( frame , action ) cTime = time . time () fps = 1 / ( cTime \- pTime ) pTime = cTime text = "FPS : " \+ str ( int ( fps )) cv . putText ( frame , text , ( 20 , 30 ), cv . FONT_HERSHEY_SIMPLEX , 0.9 , ( 0 , 0 , 255 ), 1 ) if len ( sys . argv ) != 1 : if sys . argv [ 1 ] == "true" or sys . argv [ 1 ] == "True" : cv . imshow ( 'frame' , frame ) else : cv . imshow ( 'frame' , frame ) capture . release () cv . destroyAllWindows ()

程序流程图，

image-20231016172112895