From AnyFish

Motion Capture GUI Instruction

MOCAP module aims to capture the motion of the recorded fish, comparing with established fish skeleton model. This version of MOCAP deals with two fish (one male and one female) in the scene. Double click on “mocap.exe” to open the program interface as below:

Motion capture can be done by following the five steps showed in the interface. For better illustration, this instruction takes the video “demo.mp4” in folder “videos\” as an example.

Step 1 – Select a video file and convert it into image frames

Click on “Browse” button, and find the video you want to deal with. After selecting the video, its full path will show up in the text area. Alternatively, you can enter the path and name of the video in the text area directly. If the video is stored in “videos” folder, only video name is required (eg: demo.mp4). If the video is not in “videos” folder, absolute path of the video should be entered (eg: C:\AnyFish\videos\demo.mp4).

After selecting the video file, press “Extract JPGs” button to convert it to image frames. When converting is finished, the program will automatically assign a File Name (eg: demo_mp4) for the video. This File Name is used in all further compute to this video. The output folder of image extraction is “extractjpg\File Name\” (eg: “extractjpg\demo_mp4\”). The generated File Name will be showed in text area on the right side of “File Name”. In the meantime, the “Starting Frame No.” will show the index of the first frame, and “Ending Frame No.” will show the index of the last frame of the video. Example of converting demo.mp4 is showed below:

Note: Current MOCAP version can handle avi, mp4, wmv, m4v and mpg format video files.

Note: “File Name”, “Starting Frame No.” and “Ending Frame No.” are necessary parameters to Step 2-5.

Step 2 – Calibrate image frames into front views and top views

The image frames should be calibrated into front views and top views of the tank before further computation. This step requires that the image frames are already stored in a subfolder under folder “extractjpg\”. To do the calibration, “File Name” should be entered first. If the “File Name” is automatically generated by Step 1, there is no need to change it. If not, enter File Name as the folder name which contains the image frames to calibrate, the program will look for images in “extractjpg\File Name\”. For example, if “demo_mp4” is entered, it means we will calibrate images in folder “extractjpg\demo_mp4\”.

Moreover, “Starting Frame No.” and “Ending Frame No.” should be entered, if not automatically showed up. If the “Starting Frame No.” is left as “default”, the program will calibrate from frame 1. If the “Ending Frame No.” is left as “default”, the program will calibrate to the last frame in the folder. Select between “Create New Calibration” and “Use Existing Tank Data” (if calibration data to this file is generated previously), and press on “Calibration” to start calibration. When “Create New Calibration” is selected, the program will ask for some clicking. Follow the screen instruction as follows and click as accurate as possible.

1.Please click on the four corner point on the front plane of the tank front side view, as illustrated by the blue stars. The clicking should follow the order: left top -> right top -> right bottom -> left bottom

2.Please click on the two corner points on the back plane of the tank side view, as illustrated by the green stars. This clicking should follow the order: left top -> right bottom.

3.Please click on the four corner point on the top plane of the tank top view, as illustrated by blue stars. This should follow the order: left top -> right top -> right bottom -> left bottom

4.Please click on the two corner points on the bottom plane of the tank top view, as illustrated by green stars, follow the order: left top -> right bottom

After all the clicking, the calibration data to this set of image frames are generated. Output folder of this step is “calibrateimg\calibrated_File Name\”, which will be automatically opened up when calibration is finished. The calibration data (“front_frame.mat” and “top_frame.mat”) and calibrated images are stored in “calibrateimg\calibrated_File Name\” (eg: “calibrateimg\calibrated_demo_mp4\”). Whenever there is calibration data in the folder, it can be used as “Existing Tank Data” for multiple times of calibration to this set of images.

Step 3 – Segment fish area out of every frame

Use “File Name”, “Starting Frame No.” and “Ending Frame No.” in the same way with Step 2. The program will look for result of Step 2 in folder “calibrateimg\calibrated_File Name\”.

Select between “Create New Background” and “Use Existing Background” (if background is already generated previously), and press “Segmentation” to start the segmentation process.

If “Create New Background”, the program will automatically calculate two new background images and store them in the output folder as “medianimage_front.bmp” and “medianimage_top.bmp”. Whenever there are background images in the output folder, they can be used as “Existing Background” for multiple times of segmentation of this set of images.

The output folder of this step is “colorseg\segmented_File Name\” (eg: “colorseg\segmented_demo_mp4\”), which will be automatically opened up when segmentation is finished. The generated background images and segmented images are all stored in “colorseg\segmented_File Name\”.

Step 4 – Motion capture with selected fish model

Use “File Name”, “Starting Frame No.” and “Ending Frame No.” in the same way with Step 2. The program will look for result of Step 2 in folder “calibrateimg\calibrated_File Name\” and Step 3 in folder “colorseg\segmented_File Name\”.

Select a fish shape model from the popup menu in this panel. Current version has model for “swordtail” and “stickleback”. Then, press “Capture” to start motion capture. The program will ask for some clicking as follows, please click as accurate as possible.

1. Click on female fish head (left figure), and then tail end (right figure) on the front view.

2. Click on male fish head (left figure), and then tail end (right figure) on the front view.

3. Click on female fish head (left figure above), and then tail end (right figure above) on top view.

4. Click on male fish head (left figure above), and then tail end (right figure above) on top view.

After all the clicking, the capture starts to run on all input frames. This step takes quite amount of time, please wait patiently. A wait bar will show the speed.

The output folder of this step is “matchfishmodel\twofish_File Name\” (eg: “matchfishmodel\twofish_demo_mp4\”). Model parameter of best matching in every frame will be stored in this folder as “matchres_XX.mat” (XX represents the frame index). Also, visualized matching result of every frame will be stored in the folder as “XX.png”. In every resulting image, red area represents the segmented fish area from the calibrated video frames, while green area represents the modeled fish. Larger overlapping between red and green area means better motion capture result.

Step 5 – Output motion capture result to text files and visualize the matching (Optional)

Press on “Calibrated Video” button to generate a video (in .avi format) with all calibrated images. This video can be used as original fish video to compare with other computation results. The generated video is named as “original_video_A-B.avi”, where A is “Starting Frame No.” and B is “Ending Frame No.”.

Press on “Matched Video” button to combine visualized matching result images into a video. Comparing this video with the original video to see how well the motion is captured. The generated video is named as “matched_result_A-B.avi”, where A is “Starting Frame No.” and B is “Ending Frame No.”. Press on “Result to Text File” button to convert “.mat” format result to text files. The generated text files are names as “female_data_frame_A-B.txt” and “male_data_frame_A-B.txt”.

Output folder in this step is “outputtool\File Name\” (eg: “outputtool\demo_mp4\”). All the generated avi files and text files are stored in this folder, as the example below.