Life Log Technology

Our lab opens up a new research field called life log. As spreading smart phones or other wearable devices, life log was starting to get individual life data and its utilization. In this field, the technical problems are processing big data, summarizing and linking multiple logs, and visualizing data intuitively. We aim to utilize life log from general to specific application.

Especially, we are trying research of food log and its social development. We made system called FoodLog. FoodLog smartphone tool collect more than one million images. We are focusing on analysis of trend estimation between users and individuals and image recognition using large scale data.

図1

FoodLog system interface

図2

FoodLog for smartphone

図2

Top 20 FoodLog words vs. Top 20 Rakuten Recipe words

3D Video

Recently, the technology, which connects real world and three dimensional data, are in progress. In the aspect of input, there are progresses of computer vision technology and depth camera such as kinect. Also in the aspect of outputs, 3D printer is widespread now in addition to graphics. By advent of drone, camera itself gets flexibility.

However, there are still many problems, such as scaling or moving subjects. We are investigating new technologies, such as 3 dimensional reconstruction and compression, human motion analysis, motion analysis using depth cameras, and super-resolution of depth images.

図3

3D reconstruction

図4

Depth super resolution

図5

Interactive 3D modeling

Manga Image Processing

We are making media processing research for culture such as manga and animation. Especially, manga is one of the Japan’s representative contents, but they are rarely became subjects of image processing. Also manga is a difficult object for image processing as it is a binary image.

We are making our research in two points; search and drawing support. About the former, we are making manga searching system using sketch, and trying to make more interactive searching system. By using more information such as manga’s text, we are aiming multi modal system. About the latter, we work on coloring using referenced pictures, and support using our searching system. The technology lying behind the systems are segmentation and separation of screen tones. We also made manga data set "Manga109" for academic purpose by corporation of 94 authors.

図6

© Ken Akamatsu

Manga image retrieval using sketches


図6

Automatic coloring of manga images


図6

© Tetsuya Kurosawa, Hidehisa Masaki

Automatic separation of manga line drawings and screen tones

Media Processing for Visualization

  • Large scale high speed image search

    We are investigating as follows; A method to visualize and express big data to compact size, high speed searching of large scale image data base, ranking which consider its contents, high speed formation of image collage, classification of individual album data using social media, and research consider sensitivity.

  • Cinderella technology

    Cooperating with female researcher, we are making research of women’s make-up using technology.

図7

Content-reserved photo collage generation

図7

Metamorphic mirror

It is difficult for many ordinary users to make high qualified contents or put appropriate tags. Therefore, we are investigating ways to support contents making and to enhance users’ experience by big-scale data mining. We are aiming at ‘virtuous circle’ - as the big data accumulated everyday is becoming high quality and rich, big data analysis is accelerating.

Not only images or movies, we target many media, such as voices, text tags, and documents as subjects of our research. Specifically, we are making our research in next three steps.

  • Support of action design
    ex. travel support system ( route recommendation, supporting system of photographing, out-of-the-way spot recommendation)
  • Support of elements creating
    ex. supporting system to take high quality photos
    ex. supporting system or recommendation system to be popular
  • Support of contents creating
    ex. supporting system to take high quality movies
    ex. Classification, evaluation, and supporting system of presentations

図7

Travel route recommendation considering individual preferences

図7

Photo spot recommendation based on input image

図7

Vine video evaluation

図7

Estimation of image popularity by tags

図7

Presentation analysis

Object Recognition & Machine Learning

We advocate ”certainty factor” model: an error rate of machine learning depends on question’s difficulty. We are examining a way to estimate certainty factor of machine learning accurately, and a method to change the approach of machine learning when certainty factor is low. We confirmed that we could calculate “certainty factor” of deep learning, which is in fashion.

In recent image recognition and machine learning algorithm, we should do parameter tuning in order to get best results. However, if the question changed, we should do parameter tuning again, which take costs. Thus, we are investigating automatic optimization method for complicated architecture or parameter.

By applying image recognition method, we are also investigating fashion image search.

図7

Parameter optimization of CNNs using evolutionary algorithm

図7

Search and suggestion system for clothing

Fast Optimization Methods for Image Processing

Under recent increasing resolution and big data in the area of image or movie processing, two methods are needed; speeding up to calculate in realistic time, and optimization method to get superior solutions. Therefore, we are investigating topics such as, optimization using evolutionary computation, speeding up by hierarchical processing, and high-speed search method. This could be used in depth perception, motion analysis, non-linear resize, object recognition and so on.

図7

Hierarchical labelling

Others

We are making splendid studies which do not belong to the above categories. For instance, we are doing research as follows:

  • Evaluation of relationship between compression quality of three dimensional model and print quality
  • Real time generation and composition of face model using depth sensor for video chat or virtual mirror
  • Medical Image processing and recognition
  • Contents security (Digital watermark, Authenticity determination)

図7

Real 3D movie generation and compression (supported by Kanze Kyukoukai)

図7

Subjective evaluation of 3D models