Motion-Based Image Stitching

This disclosure relates generally to the field of image processing. More particularly, but not by way of limitation, this disclosure relates to techniques for improving real-time panoramic photography processing using motion data (e.g., gyroscopic sensor input).

Panoramic photography may be defined generally as a photographic technique for capturing images with elongated fields of view. An image showing a field of view approximating, or greater than, that of the human eye, e.g., about 160° wide by 75° high, is usually termed “panoramic.” Thus, panoramic images generally have an aspect ratio of at least 2:1, meaning that the image is at least twice as wide as it is high (or, conversely, twice as high as it is wide, in the case of vertical panoramic images). In some embodiments, panoramic images may even cover fields of view of up to 360 degrees, i.e., a “full rotation” panoramic image.

There are many challenges associated with taking visually appealing panoramic images. These challenges include photographic problems such as: difficulty in determining appropriate exposure settings caused by differences in lighting conditions across the panoramic scene; blurring across the seams of images caused by motion of objects within the panoramic scene; parallax problems (i.e., problems caused by the apparent displacement or difference in the apparent position of an object in the panoramic scene as the image capture device is panned); properly aligning the various images used to construct the overall panoramic image; blending between the overlapping regions of various images used to construct the overall panoramic image; choosing an image projection correction that does not distort photographically important parts of the panoramic photograph (e.g., rectangular, cylindrical, Mercator); and correcting for perspective changes between subsequently captured images.

Some prior art panoramic photography systems may utilize gyroscopic information to assist in image analysis and registration. As the number of images within a given time period that are aligned to generate a panoramic image increases, however, the computational costs of performing image analysis and registration can become prohibitive. This may be particularly true for image capture devices embedded in handheld electronic devices such as mobile phones, personal data assistants (PDAs), portable music players, digital cameras, as well as laptop and tablet computer systems.

In one embodiment the invention provides a method to align or stitch together digital images without analyzing the image or performing standard image registration (both computationally expensive operations). The method includes obtaining a first image from an image capture device (e.g., a digital camera), the image capture device having one or more image capture parameters (e.g., focal length and principal point), the image capture parameters having first values for the first image. At substantially the same time, motion information for the first image may be obtained (e.g., from a gyroscopic and/or accelerometer sensors). A second image may then be obtained having second image capture parameter values. As with the first image, motion data for the second image may be obtained at substantially the same time that the second image is obtained. Relative motion between the two images (i.e., due to motion of the image capture device) may be determined. The second image may then be aligned with the first image using the first and second image capture parameter values and the determined relative motion. It is significant that the two images may be aligned without the need to analyze the images.

When motion data comes from a gyroscopic sensor, the motion data for each image may be integrated to obtain instantaneous position information. This position information may be used to determine the relative motion between the two images. In another embodiment, the image capture parameter values may be used to determine intrinsic matrices for each image. These matrices, and the determined relative motion between the two images, may be used to generate a perspective transform matrix. Applying the perspective transform to the second image permits it to be aligned with the first image

The disclosed methods may be embodied in program code and stored on a non-transitory storage medium. The stored program code may be executed by a programmable control device that is part of, or controls, an image capture device.

This disclosure pertains to systems, methods, and computer readable media for stitching or aligning multiple images (or portions of images) to generate a panoramic image. In general, techniques are disclosed for using motion data (captured at substantially the same time as image data) to align images rather than performing image analysis and/or registration operations. More particularly, motion data may be used to identify the rotational change between successive images. The identified rotational change, in turn, may be used to generate a transform that, when applied to an image allows it to be aligned with a previously captured image. In this way, images may be aligned in real-time using only motion data.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this description, some structures and devices may be shown in block diagram form in order to avoid obscuring the invention. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.

It will be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system- and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the image processing field having the benefit of this disclosure.

Referring to FIG. 1, image stitching operation 100 in accordance with one embodiment begins by capturing raw image sequence 105 (block 110) and corresponding motion data 115 (block 120). Motion data 115 may then be attached to individual images within raw image sequence 105 (block 125) to produce image sequence 130. It can be advantageous to capture motion data for each image in raw image sequence 105 so that each captured image has a corresponding motion datum. It can also be advantageous, and is common, for each image in an image sequence to have a timestamp indicating when the particular image was captured (e.g., during acts in accordance with block 110). The rotational change between successive images in image sequence 130 may be identified (block 135) and used to generate a perspective transform matrix (block 140). Applying the perspective transform to a “current” image (block 145) allows it to be aligned with a previously captured image (block 150) to produce aligned image sequence 155, a panoramic image.

Referring to FIG. 2A, in one embodiment image capture operation 110 may be preformed by sensor array 200 and motion data capture operation 120 may be performed by gyroscopic sensor (gyro) 205. Sensor array 200 may capture black and white or color images and use, for example, complementary metal-oxide semiconductor (CMOS) or charged-coupled device (CCD) technology. Gyro sensor 205 may be used to generate rotation rate data in three dimensions (e.g., (x, y, z) or (pitch, roll, yaw) or in a quaternion system). Gyro sensor 205 may use any desired technology such as micro-electromechanical systems (MEMS) technology.

It will be understood that images captured in accordance with block 110 (e.g., by sensor array 200) and motion data captured in accordance with block 120 (e.g., by gyro sensor 205) should be correlated. It is important that an image captured at time t₀be synchronized with motion data captured at substantially the same time. In the embodiment illustrated in FIG. 2A, image sensor 200 may signal gyro sensor 205 each time an image is captured through, for example, the V_syncsignal. Gyro sensor 205, in turn, may tag each “next captured” motion datum each time a V_syncsignal is received. This permits each image in raw image sequence 105 to be correlated or associated with the proper motion data. Use of the phrase “next captured” reflects the possibility that motion sensor 205 may operate on a different clock signal than sensor array 200. That is, sensor array 200 and gyro 205 may operate asynchronously. Referring to FIG. 2B, in another embodiment common clock 210 may drive both image sensor array 200 and gyro 205. This arrangement permits the synchronous capture of image and motion data.

Referring to FIG. 3, in one embodiment motion data 115 may be attached to image data (i.e., each image in raw image sequence 105) through process 125. First, it will be understood that when a gyro sensor such as sensor 205 is used to provide motion information 115, what is actually produced is rate information: the rate at which the image capture device is being moved in each of, for example, 3 axis. Rate information may be integrated (block 300) to produce instantaneous rotation position information 305 (also in each of 3 axis). Using image timestamp information each image in raw image sequence 105 may be associated with the appropriate rotation position information 305 (block 310). In another embodiment, operation 125 may also use accelerometer input 315 to assist in calibrating gyro sensor 205's output. Also shown in FIG. 3 is a high-level representation of a single image frame 320 from image sequence 130. As shown, image 320 includes data 325 representing the image itself and timestamp 330 provided during acts in accordance with block 110. After attach operation 310, image 320 may also include position information 305.

Referring to FIG. 4A, identifying the relative rotation between successive images 135 in accordance with one embodiment begins by identifying the rotation position of the current image (block 400) and the rotation position of a prior image (block 405) upon which rotation for the current image is to be based (e.g., the immediately prior frame). As described above, this information may be attached to each image in accordance with block 125 (see also, FIG. 3). These two positions may be used to determine the rotation of the current image relative to the prior image. This operation may then be repeated for the next image in image sequence 130 and so on. Referring to FIG. 4B, two images from image sequence 130 are shown. Prior image I_pwas captured at time t_pand has rotation position information x_p, y_p, and z_p. Current image I_cwas taken at time t_cand has rotation position information x_c, y_c, and z_c. The difference between the rotation position of image I_pand image I_crepresents the amount of rotation experienced by the image capture unit between time t_pand t_cand can be represented by a 3×3 rotation matrix denoted [R_pc] and associated with image I_c. It will be recognized that the use of Cartesian coordinates is merely illustrative, any coordinate system may be used. It will also be recognized by those of skill in the art that rotation matrix [R_pc] may be generated in accordance with any of a number of techniques but is not, in general, the algebraic difference between the rotation position of image I_pand image I_c.

Referring to FIG. 5, a perspective transform for each image in image sequence 130 in accordance with block 140 obtains various parameter values from the image capture device (block 500). Illustrative parameters include the focal length used to capture an image and the image capture device's principal point. It will be recognized that on image capture devices that provide the capability to move their lens and/or image sensor assemblies, the focal length may change from image to image. Based on the obtained parameter values, the device's intrinsic matrix may be found or generated (block 505). It will be understood that an image's intrinsic matrix links the pixel coordinates of an object point with the corresponding coordinates in the image capture device's reference frame. An image's perspective transform matrix may then be generated for a particular image using the image capture device's intrinsic matrix associated with that image (i.e., the intrinsic matrix generated using device parameter values that were in place when the image was captured) and the image's associated rotation matrix (block 510).

A perspective transform matrix for a given image may be derived as follows. First, it will be recognized by those of skill in the art that the 2D projection of real-space (which is 3D) onto a sensor array (which is 2D) may be given as—

(xyz)=Π(XYZ),EQ.1

where

(XYZ)

represents a point in real-space, Π represents the image capture device's intrinsic matrix and

(xyz)

represents the 2D projection of the real-space point onto the sensor array's plane using homogeneous coordinates. In essence, EQ. 1 represents a 3D-to-2D transformation.

A novel use of this known relationship was to recognize that—

(XYZ)=Π-1(xyz),EQ.2

where

(xyz)

is as described above

(XYZ)

represents an estimate of where that point is in real-space, and Π⁻¹represents the inverse of the image capture device's intrinsic matrix. Thus, EQ. 2 represents a 3D-to-2D transformation estimator.

Based on the discussion above regarding FIGS. 4A and 4B, it will be recognized that—

(X1Y1Z1)=[R01](X0Y0Z0),EQ.3

where

(X0Y0Z0)

represents the real-space location of a point at time t₀(as reflected in the image captured at time t₀), [R₀₁] represents the rotation matrix for image-1 from time t₀(and image I₀) to time t₁, and

(X1Y1Z1)

represents the location of the same point after being rotated as characterized by [R₀₁].

From EQ. 1 we may obtain—

(x1y1z1)=Π1(X1Y1Z1),EQ.4

where Π₁represents the image capture device's intrinsic matrix at time t₁. Substituting EQ. 3 into EQ. 4 yields—

(x1y1z1)=Π1[R01](X0Y0Z0).EQ.5

From EQ. 2, we have—

(X0Y0Z0)=Π0-1(x0y0z0).EQ.6

Substituting EQ. 6 into EQ. 5 gives—

(x1y1z1)=Π1[R01]Π0-1(x0y0z0),EQ.7

which may be rewritten as—

(x1y1z1)=[P01](x0y0z0),EQ.8

where [P₀₁] represents the perspective transform matrix for image-1. Equations 7 and 8 describe how a point in an image captured at time t₀moved to a new position at time t₁due to rotation of the image capture device as represented by rotation matrix [R₀₁]. (It is also noted [P₀₁] incorporates the image capture device's parameters (e.g., focal length) at times t₀and t₁.) It is significant to note, perspective transform [P₀₁] permits alignment of image-1 with image-0 without any image processing operations (i.e., analysis and/or registration). More particularly, perspective transform [P₀₁] is based solely on the image capture device's parameter values (e.g., focal length) and rotational information (e.g., from a gyro) for each image. It will be recognized that this information is computationally inexpensive to obtain and process, allowing image stitching operations in accordance with this disclosure to be performed quickly and at low computational cost. This, in turn, may support high image capture rates. It is also noted that the methodologies disclosed herein may be used to complement traditional image registration methods.

Referring to FIG. 6, the above operations may be reviewed in accordance with one embodiment as image stitching operation 600. As shown, operation 600 begins when a first image is captured (block 605). As noted, this image may be designated as the current image, I_C. If current image I_Cis the first image in an image sequence such as raw image sequence 105 (the “YES” prong of block 610), the image's associated intrinsic matrix Π_cis determined, whereafter processing continues at block 645. If current image I_Cis not the first image in an image sequence (the “NO” prong of block 610), rotation matrix [R_pc] between current image I_cand prior image I_pmay be determined (block 620). Next, the image capture device's intrinsic matrix is obtained corresponding to current image I_c, Π_c(block 625). The intrinsic matrix associated with prior image I_pmay then be obtained, Π_p(block 630) and perspective transform matrix [P_pc] determined (block 635) and applied to each pixel in current image I_c(block 640). If current image I_cis the last image in an image sequence (the “YES” prong of block 645), the last image (i.e., current image I_c) may be aligned with prior image I_pin the image sequence whereafter stitching operation 600 is complete (block 650). If current image I_cis not the last image in the image sequence (the “NO” prong of block 645), current image I_cmay be aligned with prior image I_pin the image sequence (block 655), whereafter current image I_cis made the prior image I_p(block 660). A next image may be captured (block 665) which becomes the current image I_Cand processing continues at block 620.

Image stitching operations have, so far, been described in terms of whole images. That is, a first image stitched to, or aligned with, another image. The disclosed techniques, however, are equally applicable to portions of images. Referring to FIG. 7, electronic device 700 may include image capture unit 705 which can deliver a sequence of images (e.g., images 710, 715 and 720) to first memory 725. A portion or slice of each image (e.g., slices 730, 735 and 740) may be extracted from each image, placed into second memory 745, and aligned or stitched as disclosed herein. It will be recognized that memory 725 and memory 740 may be different regions within a common physical memory or wholly separate memories. It should also be recognized that it is not necessary to stitch whole images or partial images only. A whole image may be aligned with a portion of a second image which, in turn, may be aligned with another slice or whole image.

Referring to FIG. 8, one electronic device incorporating digital image stitching capability in accordance with this disclosure is shown. In this particular example, device 800 represents a mobile telephone which provides preview or capture display 805. Mobile telephone 800 also includes microphone 810 and one or more speakers (not shown). It will be recognized that the disclosed image stitching capability may be incorporated in many electronic devices. Examples include, but are not limited to, stand-alone digital electronic cameras, mobile music players, personal digital assistants (PDAs), and notebook, desktop and tablet computers.

Referring to FIG. 9A, a functional view of illustrative electronic device 900 in accordance with this disclosure includes image sensor 905, gyroscopic sensor 910, and accelerometer 915. Image sensor 905 provides images to image device driver 920, gyro sensor 910 provides motion data (e.g., rate of movement) to gyro device driver 925, and accelerometer 915 provides its data to accelerometer driver 930. In the example of FIG. 9A, images and motion data are correlated through the use of a V_syncsignal as discussed above with respect to FIG. 2A. Gyro and accelerometer data may be collected to generate motion data 935 which may then be attached 940 to individual images within raw image sequence 105. Once motion data has been attached, augmented image sequence 130 may be sent to stitching processor 945 which transforms each image in accordance with its particular perspective transform matrix so that it may be aligned with prior images in memory 950.

Referring to FIG. 9B, another illustrative image capture device 955 is shown. In this embodiment, however, common clock 960 drives image 905, gyro 910 and accelerometer 915 sensors. As noted above with respect to FIG. 2B, use of common clock 960 permits synchronous capture of image and motion data.

Referring now to FIG. 10, a simplified functional block diagram of representative electronic device 1000 incorporating digital image stitching capability is shown according to one embodiment. Electronic device 1000 may include processor 1005, display 1010, device sensors 1015 (e.g., gyro, accelerometer, proximity, and ambient light sensors), microphone 1020, audio/video codecs 1025, speaker 1030, communications circuitry 1035, image sensor with associated camera and video hardware 1040, user interface 1045, memory 1050, storage device 1055, and communications bus 1060.

Processor 1005 may be any suitable programmable control device and may control the operation of many functions, such as the generation and/or processing of image metadata, as well as other functions performed by electronic device 1000. Processor 1005 may drive display 1010 and may receive user input from user interface 1045. An embedded processor, such a Cortex® A8 with the ARM® v7-A architecture, may provide a versatile and robust programmable control device that may be utilized for carrying out the disclosed techniques. (CORTEX® and ARM® are registered trademarks of the ARM Limited Company of the United Kingdom.)

Memory 1050 may include one or more different types of storage media used by processor 1005 to perform device functions. Memory 1050 may include cache, read-only memory (ROM), and/or random access memory (RAM). Communications bus 1060 may provide a data transfer path for transferring data to, from, or between at least storage device 1055, memory 1050, processor 1005, and camera circuitry 1040. User interface 1045 may allow a user to interact with electronic device 1000 through a variety of forms, such as a button, keypad, dial, click wheel, or a touch screen.

Non-transitory storage device 1055 may store media (e.g., image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage device 1055 may include one more storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM).

Software may be organized into one or more modules and be written in any suitable computer programming language (or more than one language). When executed by, for example, processor 1005 such computer program code or software may implement one or more of the methods described herein. Each module may be stored in a different storage device or memory and may be executed by one or more processors.

Various changes in the materials, components, circuit elements, as well as in the details of the illustrated operational methods are possible without departing from the scope of the following claims. For instance, processor 1005 may be implemented using two or more program control devices or processing units communicatively coupled. Each program control device may include the above-cited processors, special purpose processors such as graphics processing units (GPUs) and custom designed state machines that may be embodied in a hardware device such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

Finally, it is to be understood that the above description is intended to be illustrative, and not restrictive. For example, the above-described embodiments may be used in combination with each other. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”