We start AI Lab in the company and research “attention mechanism” in deep learning

As I said before. I completed the online course “deeplearning ai“. This is an awesome course I want to recommend to everyone. There are many topics we can learn in the course. One of the most interesting things for me is “attention mechanism” in neural translation.  So I would like to explain it in details. Do not worry as I do not use mathematics in this article.  Let us start.


The definition of attention mechanism is “The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step”. It may be natural when we consider how we translate language from one to another. Yes, human-being pays more attention to specific objects than others when they are more interesting to them. When we are hungry,  we tend to look for the sign of “restaurant” or ” food court”,  do not care the sing of “library”,  right?

We want to apply the same thing for translation by computers. Let me consider again. It is true that when we translate English to our mother tongue, such as Japanese, we look at the whole part of the sentences first, then make sure what words are important to us.  we do not perform translation one on one basis. In another word, we pay more attention to specific words than other words. So we want to introduce the same method in performing neural translation by computers.


Originally, attention mechanism was introduced (1) in Sep 2014. Since then there are many attention mechanisms introduced. One of the strongest attention models is “Transformer” by Google brain in  June 2017.  I think you use Google translation every day. It performs very well. But transformer is better than the model used in Google translation. This chart shows deference between  GNMT (Google translation) and Transformer(2).

Fortunately, Google prepares the framework to facilitate AI research.  It is called “Tensor2Tensor (T2T) “. It is open sourced and can be used without any fees. It means that you can do it by yourself! I decide to set up “AI Lab” in my company and introduce this framework to research attention mechanism. There are many pre-trained models including “Transformer”.  Why don’t you join us?


I used translations as our example to explain how attention mechanism works. But it can be applied to many other fields such as object detection which is used in face recognition and a self-driving car. It must be excited when we consider what can be achieved by attention mechanism.  I would like to update the progress.  So stay tuned!



When you need AI consulting,  do not hesitate to contact TOSHISTATS



(2) Attention Is All You Need,  By Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez, Łukasz Kaiser,Illia Polosukhin,  in June 2017




Notice: TOSHI STATS SDN. BHD. and I do not accept any responsibility or liability for loss or damage occasioned to any person or property through using materials, instructions, methods, algorithm or ideas contained herein, or acting or refraining from acting as a result of such use. TOSHI STATS SDN. BHD. and I expressly disclaim all implied warranties, including merchantability or fitness for any particular purpose. There will be no duty on TOSHI STATS SDN. BHD. and me to correct any errors or defects in the codes and the software.

What will be the flight service in the future? I write it in the air!


Now I am in the air from Kuala Lumpur to Tokyo as I have a business trip.  I always use Air Asia because it is convenient and reasonable. Since AirAsia has operated,  it is getting cheaper to flight from Kuala Lumpur to Tokyo. It is very good, especially for younger generations. I would like to welcome them in Japan very much.  Then I am wondering what the flight service will be in the future. Let us consider it with me!


1. Service on flight

Low cost carriers, including AirAisa increase the number of customers per flight compared with legacy carriers to reduce the price of the flight. Therefore services for each customer are not the sane as legacy carriers.  I think, however, it will be improved dramatically supported by digital technologies.  At each site, electronic dashboard might be equipped and all information, such as flight schedules,  emergency evacuation methods might be provided.  These are translated into many languages with machine translations so there is no need to worry about language barriers. ( In my flight of AirAsia, English, Japanese and Malay are used in the flight announcement. ) Meals in a fight will be improved, too.  We might order meals on demand through the electronic dashboard whenever you want to eat. These data can be collected customer by customer.  Therefore, preference of each customer might be known in advance.  This technology is called “personalization”. So low cost carriers might predict what kind of meals are needed in the flight based on past experience of  each customer. It enables them to widen the variety of meals served because there is less risk to have a lack of inventories of meals on the flight.  To serve meals to each customer,  robots of cabin attendant assistants might support cabin attendants so that meals are served smoothly. I am excited if I can choose many varieties of meals on demand.


2. Immigration

Before getting on the board,  it takes time to pass immigration.  I always think it might be more effective with technologies called “face recognition”.  Computers can identify who you are by comparing to your face image stored on the passport. It is good to take less time to pass immigration for everyone. If it is connected to a database of INTERPOL,  it can enhance identification of criminals.


3.  Maintenance

Airplanes have a massive amount of parts. Therefore, maintenance is critically important to keep flights safe. Especially for low cost carriers, there is less time to maintain airplanes from landing to taking off again.  It can be enhanced by technologies   called “internet of things” and “predictive analytics“.  In internet of things,  each part has sensors and provide data periodically thought the internet.  Data from the sensors are collected and analyzed by “predictive analytics” to predict which parts are likely to fail in  advance.  Maintenance can be  more effective by using the results of  predictive analytics. Data from sensors can be transmitted from airlines to airports, even though they are in the air. Therefore failed parts or potential one can be identified before air plains land.  It enables us to decrease the time of maintenance.


Beyond low cost carriers,  the airplane in the air might be connected to other industries such as hotels. For example, the flight might be delayed due to bad weather and customers need reservations of hotels as the flight will land at the midnight. In such case, We can reserve hotels thought digital dashboard of each sheet. It is good to have reservations of the hotel even if we are in the air!

I hope my flights will be more comfortable in the future!  Could you agree?




Note: Toshifumi Kuga’s opinions and analyses are personal views and are intended to be for informational purposes and general interest only and should not be construed as individual investment advice or solicitation to buy, sell or hold any security or to adopt any investment strategy.  The information in this article is rendered as at publication date and may change without notice and it is not intended as a complete analysis of every material fact regarding any country, region market or investment.

Data from third-party sources may have been used in the preparation of this material and I, Author of the article has not independently verified, validated such data. I and TOSHI STATS.SDN.BHD. accept no liability whatsoever for any loss arising from the use of this information and relies upon the comments, opinions and analyses in the material is at the sole discretion of the user. 

It is awesome if you can create your own news-broadcasting, isn’t it?


News broadcastings are well-known from everyone. For example, CNN, financial times and Bloomberg, etc.  If you can make your own news broadcasting, it is awesome and amazing. But is it possible?  One of the obstacles is how we can collect articles and information from all over the world in real-time basis.  Of course I do not have my own network of news correspondents all over the globe. Then, what should we do about that?

Last week I found the blog about “GDELT 2.0“. The GDELT Project, which monitors events driving global society, creating a free, open platform for computing in the entire world, was founded and led by Kalev H. Leetaru. The GDELT Project’s full name stands for the Global Database of Events, Language, and Tone (GDELT).  Now this project is going to a new stage of “GDELT 2.0”.  Compare with “GDELT 1.0”,  “GDELT 2.0” has a great deal of progress as follows


1.  “GDELT 2.0” can cover documents and information written in 65 languages

There is a lot of linguistic communication to be written and spoken all over the world. If we try to cover all over the Earth, we need to understand languages other than English. For example, an apple is called “Ringo” in Japanese. If computers cannot read what “Ringo”means, it is impossible to collect the information about apple in Japan because few of the articles are translated from Japanese to English. There is no need to worry about them. GDELT 2.0” can do that by using real time machine translation. This function is called “GDELT Translingual“.  It means that global news that GDELT monitors in 65 languages, representing 98.4% of its daily non-English monitoring volume, is transformed in real time into English. It is amazing because the media of the non-Western world can be included in our coverage. There are no language barriers to worry about.


2. “GDELT 2.0” can be updated in near-real time basis

A blog of  “GDELT 2.0″ says ” In essence, within 15 minutes of GDELT monitoring a news report breaking anywhere the world, it has translated it, processed it to identify all events, counts, quotes, people, organizations, locations, themes, emotions, relevant imagery, video, and embedded social media posts, placed it into global context, and made all of this available via a live open metadata firehouse enabling open research on the planet itself.”  These data use to be updated once a day. Now it is updated within 15 minutes. I think it is critically important when we try to create our own news-broadcasting.


3. “GDELT 2.0” can exercise content analysis for each article in near-real time basis

“GDELT 2.0” can also judge whether the articles are positive or negative. The blog says “GDELT 2.0” can quantify the extraordinary array of latent emotional and thematic signals subconsciously encoded in the world’s media each day. 18 content analysis systems totaling more than 2,230 dimensions are now run on each news article seen by GDELT each day and all of these scores are available. It is called “the Global Content Analysis Measures (GCAM)”.


In short,  information all over the world can be updated with real-time machine translation and content analysis.  It is definitely amazing. With this database of “GDELT 2.0”,  we might create our own news broadcasting!  Could you try it now?

If you are interested in “GDELT 2.0”, it is a nice video for an introduction.

This new toy looks so bright! Do you know why ?


Last week I found that new toy  called “CogniToys” for infants will be developed in the project of Kickstarter, one of the biggest platforms in cloud funding.  The developer is elemental path, one of the three winners of the IBM Watson competition. Let see why it is so bright!

According to the web site of this company,  this toy is connected to the internet.  When a child talks to this toy, it can reply because this toy can see what a child says and answer the question from a child.  It usually requires less than one second to answer because IBM Watson-powered system is powerful enough to calculate answers quickly.


Let us look at the descriptions of this company’s technology.

“The Elemental Path technology is built to easily license and integrate into existing product lines. Our dialog engine is able to utilize some of the most advanced language processing algorithms available driving the personalization of our platform, and keeping the interaction going between toy and child.”

Key words are 1. Dialog    2. Language processing   3. Personalization


1. Dialog

This toy communicates with children by conversation, rather than programming. Therefore technology called “speech recognition” is needed in it.  This technology is applied in real-time machine translation such as Microsoft Skype, too.


2. Language processing

In the area of machine learning, it is called “Natural language processing”. Based on the structure of sentence and phrase, the toy understands what children say.  IBM Watson is very expert in the field of natural language processing because Watson should understand the meaning of questions in Jeopardy contests before.


3. Personalization

It is beneficial when children talk to this toy, it knows children preference in advance. This technology is called “Personalization”.  Through interactions between children and the toy, it can learn what children like to cognize. This technology is oftentimes used in retailers such as Amazon and Netflix. There is no disclosure about the method of personalization as far as I know.  I am very interested in how the personalization mechanism works.


In short, machine learning enables this toy to work and be smart. Functions of Machine Learning are provided as a service by big IT companies, such as IBM and Microsoft.  Therefore, this kind of applications is expected to be put out to the market in future. This is amazing, isn’t it?  I imagine next versions of the toy can see images,  identify what they are and share images with children because technology called image recognition is also offered as a service by big companies.

I ordered one CogniToy through Kickstarter. It is expected to deliver in November this year. I will report how it works when I get it!


Note:IBM, IBM Watson Analytics, the IBM logo are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. 

What can computers do now ? It looks very smart !


Lately I found that several companies such as Microsoft and IBM provide us services by machine learning. Let us see what is going on now.

These new services are based on the progress on Machine learning recently. For example, Machine translation services between English and Spanish are provided by Microsoft skype. It uses Natural Language Processing by Machine learning. Although it started at Dec 2014, the quality of the services is expected to be improved quickly as a lot of people use and computer can learn the data from such users.


It is beneficial for you to explain what computers can do lately so that you can imagine new services in future. First, computers can see the images and videos and identify what it is. This is image recognition. Second, it can listen to our speech and interpret what you mean. This is speech recognition. It can translate one language to another, as well. This is machine translation. Third, computers can research based on concepts rather than key words. Fourth, it can calculate best choice among the potential options. This is an optimization. In short computers can see, listen to, read, speak and think.

These functions are utilized in many products and services although you cannot notice it. For example, IBM Watson Analytics provides these functions through platform as a service to developers.


I expect these functions enable computers to behave just like us. At the initial phase, it may be not so good just like a baby. However, machine learning allows computers to learn from experience. It means that the computer will perform better than we do in many fields. As you know, Shogi, one of the popular Japanese board game, artificial machine players can beat human professional teams. This is amazing!

Proceeding forward, it is recommended that you understand how computers are progressing in terms of the functions above. Many companies such as Google, Facebook invest a great deal of money in this filed. Therefore, many services are anticipated to be released in near future. Some of new services can impact our jobs, education and society a lot. Some of them may arise new industries in future.


Some day, when you are in the room, the computer can identify you by computer vision. Then ask if you want to drink a cup of coffee. The computer holds a lot of data, such as temperature, weather, time, season, your preference in it and generates the best coffee for you. If you want to know how this coffee is generated, the computer provides you a detailed report about the coffee. All settings are done automatically. It is the ultimate coffee maker by using powerful computer algorithm. Do you want it for you?



Note:IBM, IBM Watson Analytics, the IBM logo are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. 

Mobile and Machine learning can be good friends in 2015 !



Number of mobile devices will be increasing drastically in the emerging markets in 2015. One of the biggest reason why it is increasing is that good smart phones are affordable because of competitions among the suppliers such as Google, Samsun and Xiaomi.  It is good for people in the emerging countries because a lot of people can have their own personal devices and enjoy the same internet life as people in developed countries do. I hope everyone all over the world will be able to be connected to the internet in near future.

Not only the number of mobile devices but the quality of its services will be changed dramatically in 2015 because machine learning will be available for commercial purpose. Let us consider this change more details. The key idea behind this is “Shift from judgement by ourselves to judgement by machines”.


1.  Machine Learning

Machine Learning has a power to change every industry. With machine learning,  computers can identify certain objects in images and video,  understand conversations with us and read the documents written in natural languages.  It means that most of information around us can be interpreted by computers.  Not only numerical data but also other kinds of information are understood by computers.  This changes landscape of every industry completely.  Computers can make business decisions and all we have to do is just to monitor it.  It already happened in the field of assessing credit worthiness of the customers  in banks many years ago.  Same things will happen in all industries near future.


2. Data

In emerging markets, more and more mobile phones will be sold so that every person might own his or her device in near future. It means that people all over the world will be connected through the internet and more information are collected in real-time basis.  In addition to that a lot of automobiles, homes and parts are also connected through the internet and send the information in real-time basis, too.  Therefore we can realize when and where they are and what condition of each is in real-time basis.  So maintenance for parts will be done as soon as it is needed and optimizations of resources used by people can be achieved as we can get such information in real-time basis.


3. Output

Output from computers will be sent to mobile devices of each responsible personnel  in real-time basis. So there is no need to stay in office during working-time as we can be notified wherever we are. It raises productivity of our jobs a lot. No need to wait for notifications of outputs from computers in office anymore.


Yes, my company is highly interested in the progress of machine learning for the commercial purpose. I would like to watch it closely.  I also would like to develop new services based on machine learning on mobile devices going forward.

Can we talk to computers without programming language?


IBM announced that Watson analytics provides us data analysis and visualization as a service without programming at 4th Dec 2014. It said that “breakthrough natural language-based cognitive service that can provide instant access to powerful predictive and visual analytic tools for businesses, is available in beta”.  Let us consider what kind of impacts IBM Watson analytics provides us.


Watson analytics is good at doing natural language processing.  For example,  if doctors ask Watson analytics how to cure the disease, Watson analytics understand the questions from doctors, research massive data and answer the questions. There is no need to program codes by doctors. It means that we may change from “we should learn computer programming” to “we should know how to have a conversation with computers”.  It may enable a lot of non-programming persons to use computers effectively.

In addition to that,  Watson analytics is also good at handling unstructured data.  These data include text, image, voice and video.  Therefore Watson analytics can analyze e-mail, social media contents, pictures taken by consumers.  So It may be possible to recommend what we should eat at restaurants by taking pictures of menus there, because computers have our health data and they can choose the best meals for our health by analyzing the pictures of menus.

In terms of algorithm,  these functionalities above can be achieved by machine learning.  So the more people start using this service, the more accurate answers by computers are because computers learn from a lot of data and are getting better.


IBM Watson analytics may change the landscape of every industry.  Traditionally data analysis can be executed by data scientists, using numerical data and programming languages. However this new kind of data analysis by IBM Watson analytics,  data analysis can be executed by businessmen/women, using e-mail, pictures and video and natural languages.  Machine translation from one language to another will be also available therefore there are less language barrier going forward.  This must be democratization for data analysis. It is exciting when it happens in 2015 !


Note:IBM, the IBM logo are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. 

Mobile services will be enhanced by machine learning dramatically in 2015


Merry Christmas !  The end of 2014 is approaching.  It is a good time to consider what will happen in the fields of machine learning and mobile services in 2015.  This week we consider machine translation and image recognition,  next week recommender systems and internet of things as well as mobile services by machine leaning. I hope you can enjoy it !


1.  Machine translation / Text mining

Skype is a top innovator in this fields.   Microsoft already announced that machine translation between English and Spanish is available by Skype. So in 2015,  it would be possible to translate between English and other languages. Text translation is also available among 40 languages in its chat service.  So language barrier are getting lower and lower.  It is still difficult to answer to questions by computers automatically.  But it is also gradually improved.  Mizuho bank announced that it will use IBM Watson, one of the famous artificial intelligence to assist call center operators.  These technologies make global service to be developed more easily as manuscripts and frequent Q&A are translated from the language to another automatically.  I love that because my educational programs can be expanded to all over the world!


2. Image recognition

Since computers identified the image of cats automatically by deep learning, images recognition technology progresses dramatically.  Soft bank announced that Pepper, new robot for consumers, will be able to read human emotions. In my view, the most important factor to read emotions must be image recognition of  human facial expressions. Pepper could be very good at doing this therefore it can read human emotions.  Image recognition technology is very good for us as each smart phone has a nice camera and it is easy for people to take pictures and send them to clouds and social media.  Image recognition can enable us to analyze massive amount of images, which are sent through internet. That data must be a treasure for us.


These machine learning technologies must be connected to mobile phone of each customer in 2015. It means that mobile services are enhanced by machine learning dramatically. All information around us will be collected through internet and send to machine learning in real-time basis and machine learning will return the best answer for individuals. This will be standard model of mobile services as speed of calculation and communication are increasing rapidly.

Next week we consider recommender systems,  internet of things and investment technology.  See you next week!