Financial industry and artificial intelligence


UBS announced that it will deliver personalized advice to the bank’s wealthy clients by using artificial intelligence (AI). UBS plans to roll out a digital service in Asia next April.

I think this is one of the example for financial institutions to go to “digital personalized marketing”  by artificial intelligence.  In future  personalized services by AI are one of the key strategic technologies in the financial industry. Let us consider how artificial intelligence are implemented and used in marketing of  financial industries more details.


1. data

This is a basis for the analysis to predict what financial products customers want.  According to this article about UBS, in the presentation by founders of Sqreem, they said that they crawl through a wide range of openly available, unstructured data. I would like to explain unstructured data. It means the data is not organized in a database as we usually see. So I assume massive amount data could be gathered automatically.  Data might be gathered in real-time basis so final outputs such as recommendations also might be provided in real-time basis. It is a dynamic process, rather than a static process.


2. algorithm

There is no disclosure about how calculations are done in details as far as I know. So this is my assumptions based on the article.  This might be one of the recommender systems. As the article says, this focuses on the behavior of customers.  Behavior of customers could be identified in deeper level and precise recommendations to individual customers could be  provided effectively.  In my thought,  this system might be on-line learning system, too. It means that algorithm could learn new things by themselves, could be updated based on stream data in real-time basis and adjust the change of customers’ preferences.


3. output

This is also my assumptions based on the article. The articles mentioned mobile phones and other digital devices.   I think recommendations might be mainly provided to individual customers through their mobile phones. Mobile phones could be personal interface against banks and financial institutions.  One of the biggest advantage of mobile phones is that customers preference could be gathered through interaction between customers and banks without any official inquiry to the customers.


This is not the end of story but the beginning of it.  As technology is progressed,  a lot of industries will try to introduce such kind of personalized recommender systems. This is marketing of digital era so that everyone can obtain the best products and services among a lot of choices. How wonderful it is !

How about a recommender system for yourself? Computers know you better than you do!?


Since the beginning of September, I have been considering recommender systems intensively.  Now I realize that recommender systems may know you better than you do because recommender systems can memorize your behavior, such as shopping, touring and learning as many as possible.  It is more than your memory in your brain as human being forget their memory as time passes.  The more data computers have, the more accurate the recommendations are. Now that more and more people have their own devices, such as smart phones/tablets, and use them everyday in their lives. It means that data on our personal behaviors is accumulated in computers every second, even though we do not realize that.  In the future, personal devices may provide us recommendations for every choice in our lives. What are the advantages and disadvantages of  recommendations in the future? Let me think about it for a while.



You can easily obtain what you want based on the recommendations. It may be something you cannot imagine even though computers know that for you.  When I go shopping to huge department stores, I sometimes get tired in finding what I want because there are too many goods in the department stores. In such cases, the recommendation is definitely a powerful tool if it is accurate. Information about products and services can be gathered from all over the world so recommendations may be about products from foreign countries. When a new kind of bread is introduced and appears in a bakery, computers analyze the factors of this bread, such as taste, price, appearance and calculate the metrics to provide recommendations for you. I would like to have this recommendation as I love breads as breakfast.



You may loss the opportunities to realize new taste and preference of yourself because computers can calculate your preference very accurately. When recommendations by computer never miss your expectations, you may feel no need to go outside recommendations.  It means that there is no challenge to go outside of your past behavior. If you like Japanese food, computers may provide you recommendations of Japanese food only. So you eat only Japanese food, never try other kind of foods.  But I think human being needs to go to outside it to create innovations and adventures in their lives. If that is the case, I may input random numbers into my personal device so that recommendations have some noises from my past behavior. I need a little challenge against my past behaviors as it makes my life more interesting, even if I do not know whether it works or not.


I imagine that, in 2040 each personal device such as a smart phone, a smart watch can hold massive data and carry calculation power in it.  So it may calculate your preference far better than you think.  In the morning,  you may find your most favorite bread, which you want to eat at breakfast,  on the dining table before you think about it.  This will be based on recommendations by your smart phone. It may know everything about you. It may be perfect. Is it wonderful, isn’t it?

I think it is good to learn how models work on recommender systems!


For three weeks I have researched recommender systems by using many websites.  I am very surprised to see that a lot of documents, sites and videos are available to know how recommender systems work and what current topics are in order to improve them.  I especially focus on video lectures by Xavier Amatriain, working for Netflix as a Research/Engineering Director. He covered from basic methodologies of recommender system to the latest methods to achieve business objectives.  It is strongly recommended to see them if you are interested in recommender systems. When technical terms in the video lectures are difficult to understand,  I suggest you to look at  MOOCs by Dr.Andrew Ng at Stanford university in advance.  It enables you to understand the technicalities of  machine learning with ease because it provides broad knowledge of machine learning as I said before.

In my thought,  there are two major methods to calculate ratings for recommendations. One is matrix factorization and the other is neural network, even though a lot of other methods can be used in recommender engines.  The object of this project is to develop the model of recommender engine so that beginners for data analysis can develop the recommender engine by themselves.  Therefore neural network is out of my scope as it is too complex for beginners.  Matrix factorization must be good for learning to develop the models in recommender systems.  So I decided to focus on the method of matrix factorization in this project.


In my view, key points of modeling based on matrix factorization as follows

1. How are customers’ preferences represented in the models?

Someone like sweet things and someone do not.  Someone like love stories and someone do not.  Someone like rock ‘n’ roll and someone do not.  So  customers’ preferences look like vectors of the level of each preference. It is reasonable and easy to understand even for beginners.  Why don’t you make  a vector of your preferences for your favorite items?


2. How are items’ features represented in the models?

It may take time to prepare the features for each item.  The features of each item should be in line with customers’ preference as we discussed above.   So whether it is sweet or not?  Whether it is a love story or not?  Whether it is rock ‘n’ roll or not?  Items’ features also look like vectors as customers’ preferences do.  It is easy to understand, too.


3. How can we match between customers’ preferences and items’ features?

It is reasonable to make metrics by using products between customers’ preferences and items’ features.  If the customers’ preference has higher score and the items’ feature which is corresponding to the preference, also has a higher score,  products between the preference and the feature is also higher as each of them is higher. It means that this higher score of the product enables us to recommend this item to this customer because this item has features which the customer likes.  It makes sense! 


If you need more details of matrix factorization, this paper is recommended to read. It explains how the model work in recommender systems. Going forward, I would like to develop a prototype model of recommender engine with R language so that beginners understand how the models work on recommender systems.

For your information, Coursera will provide the course of “Mining Massive Datasetsby Jure Leskovec, Anand Rajaraman and Jeff Ullman at Stanford University. It will start at 29th Sep 2014 and cover recommender systems. Do not miss it!

How can we classify recommender systems now?


I have researched recommender systems for more than two weeks and I am very surprised to see there are a lot of recommender systems.  Therefore, I think I should have a proper way to classify these systems to explain the characteristic of each system.  Most of research papers and documents focus on the system called “collaborative filtering”.  But I think it is a little difficult to explain the difference between systems by using the word “collaborative filtering”. So I would like to focus data which are used in developing models.  Especially I am interested in content features, such as actors and actresses,  time of  production,  directors,  countries where it was produced, and so on in case items are movies. I hope beginners for recommender systems can understand them easily.


1. Recommender systems with content features

When we have content features in our data set,  this type of models is used. They are useful when there are less data about customers’ rating and interactions because recommendation to the customer can be produced without other customers’ data.  It means that we can avoid “cold start issues”. On the other hand,  we need data about content features for each item. It may take time and cost to prepare it, although it is worth doing so.   I think in most cases this type of recommender systems are used in businesses now.


2. Recommender systems without content features

When we have no content features in our data set,  this type of models should be considered. Without content features,  similarity between customers are used to produce recommendations.  Similarity between items are also used without contents features.  This type models may be referred as”collaborative filtering” in documents and research papers. It has advantages as there is no need to obtain content features.  On the other hand, we need customers ratings and interactions in advance to develop models.  It is not good for startups as they have less data about customers in general.


In practice,  it is good for beginners to classify recommender systems based on whether content features are used or not. Because there is no need to know the mechanisms of recommender systems.  There are a lot of methods to develop recommender engine. For example, regression, classification, clustering, collaborative filtering, etc.  So I think it is very difficult to classify recommender systems based on the way of “how to calculate recommendations”. When beginners are getting familiar with the methods above,  they can understand  how each method works in recommender systems.


When I research the documents and papers about recommender systems,  I found that Netflix prize,  a kind of programming competition where the winner was granted one million USD in 2009.  It is very interesting to go into deeper because these models discussed during the competition were superior to existing models to provide recommendations accurately and easy to learn even for beginners. I would like to discuss these methods in the next blog.  See you next time!

How can we produce recommendations based on customers information?


Most of you know what recommendations by retailers, e-commerce are.  Few people, however, know how they are produced behind the web-screens or e-mails. So I would like to explain the mechanism about production of recommendations in a series of my blog as the project is going on in the company. First, I would like to consider three points below one by one.  I focus on personalized recommendations, which can be customized to customers individually,  based on the information about them. Unpersonalized recommendations, such as recommendations based on just sales ranking are out of our interest and scope because this is expected to be provided for every customer equally.


1. Customers

Of course,  customers are the most important for our businesses.  The problem is that how customer information is obtained and how customer preference can be known based on customer information.  It is clear that the best way to know customer preference is just asking them.  But it is almost impossible to ask everything about their preference.  Fortunately websites and smart phone are widely used among customers so they make us easier to obtain customer information through “What they view longer”, “What they put into favorite items ” and “What they bought in the past”.  We can know customer preference based on this information.


2. Items

Items mean not only products, but news, information, services and anything which can be chosen by customers.  Each item can be expressed by some features, which are characteristics of the products.  When two items share the same features,  one can be recommended to customers who bought the other because both products has similarities each other.  It may be difficult to choose good features to do it.  So I would like to continue to research how to choose features effectively.


3. Relationship between customers and items

Once information of customers and items are obtained,  the relationship between customers and items should be considered.  I imagine it is very important to obtain the relationship accurately, so that recommendations can be accurate and effective. Statistical models are needed to calculate metrics in order to express the strength of the relationship between customers and items.


These three points above are critically important to construct a recommender engine because recommendations are a kind of matching between customers and items.  I would like to expand this argument to develop algorithms so that recommendations can be calculated correctly. I found that a dozen of programs of recommender engines, which are open source, are available to us. I would like to review some of them going forward.  I hope you can enjoy them, too!



Why is “Recommendation” critically important for small and medium enterprises ?


When I teach data analysis,  I always consider what the best application of statical models to the real businesses is.  I researched it several weeks and I found a recommender engine is one of the best applications to explain how statistical models or machine learning work in the real world.  Now that most people get the recommendations about products, services, news through emails and websites.  One of the famous examples is the recommendation by  So it is easy to understand what recommendations are and how useful they are.

I provided recommendations to my customers manually when I used to be an account executive at the branch of the security company more than 20 years ago.  I had more than 200 customers there and sold the financial products to them. It was an interesting  job as financial markets have been moving every day, every second. But there were problems about the way of marketing at that time.


1.  I could not take care of every customer effectively

I could contact 20 or 30 customers by phone on a daily basis (there were no e-mails at that time). It was impossible, however, to contact more than 200 customers so I might miss or overlook the needs of customers because I could not understand who the customers were and what they wanted within limited time. It led to opportunity cost for me.


2 . I could not understand all products effectively

When I used to be an account executive,  financial innovation was going on in Japan.  It means that not only traditional products, such as stocks and bonds, but also derivatives and options were available to retail investors.  There was not enough time for me to understand every product in detail.  So I might fail to satisfy customers’ needs due to lack of knowledge of products which were available at that time.


If I could have a recommender at that time, these problems above could be solved as recommender engine could make the most of the information about both customers and products at once, in a timely manner. In order to provide good recommendations, it is clear that information about both customers and products are needed. It might be time-consuming and require human resources  if we manage this information manually. But recommender engine can process it quick enough to provide recommendations in a timely manner. When companies, such as SME have small sales force, recommender engines are critically important because it enables such companies to provide recommendations to customers effectively.  This is one of the best ways to communicate to customers with reasonable cost as well.


So I want to develop a recommender engine to provide good recommendations based on information from customers and products.  My company is starting an open project to develop a recommender engine with R.  This will be open source and get public so that everyone can learn how it works if he/she is interested in it.  I will report the progress of the project on this blog going forward. I hope you can enjoy it!

Do you want to be “Analytic savvy manager”?

Data is and will be around us and it is increasing at an astonishing rate.  In the such business environment,  what should business managers do?    I do no think every manager should have an analytics skill at the same level as a data scientist because it is almost impossible.  However, I do think every manager should communicate with data scientists and make better decisions by using output from their data analysis.  Sometimes this kind of manager is called “analytic savvy manager”.  Let us consider what “analytic savvy manager”should know.


1.   What kind of data is available to us?

Business managers should know what kind of data is available for their business analysis.  Some of them are free and others are not.  Some of them are in  companies or private and others are public.  Some of them are structured and others are not.  It is noted that data which are available is increasing in terms of volume and variety.  Data is a starting point of analysis, however, data scientists may not know specific fields of business in detail. It is business managers that know what data is available to businesses.  Recently data gathering services have provided us a lot of data for free.  I recommend you to look at “Quandl” to find public data. It is easy to use and provides a lot of public data for free.  Strong recommendation!


2.  What kind of analysis method can be applied?

Business managers do not need to memorize formulas of each analysis method.  I recommend business managers to understand simple linear regression and logistic regression and get the big picture about how the statistical models work. Once you are familiar with two methods,  you can understand other complex statistical models with ease because fundamental structures are not so different among methods.  Statistical models enable us to understand what big data means without loss of information.  In addition to that,  I also recommend business managers to keep in touch with the progress of machine learning,  especially deep learning.  This method has great performances and is expected to be used in a lot of business field such as natural language processing.  It may change the landscape of businesses going forward.


3.  How can output from analysis be used to make better decisions?

This is critically important to make a better decision.  Output of data analysis should be in aligned with business needs to make decisions.  Data scientist can guarantee whether numbers of the output are accurate in terms of calculations. However, they can not guarantee whether it is relevant and useful to make better decisions.  Therefore business managers should communicate with data scientist during the process of data analysis and make the output of analysis relevant to business decisions. It is the goal of data analysis.


I do not think these points above are difficult to understand for business managers even though they do not have a quantitative analytic background.   If you are getting familiar with these points above, it would make you different from others at the age of big data.

Do you want to be “Analytic savvy manager”?



What an excellent tool ” R ” is !

R language is an incredible statistic tool and have been improved on going.  10 years ago when I tried to exercise data analysis privately at home,  I used excel because my PC already had installed Microsoft Office as its initial setting. On the other hand, I used proprietary tools such as MATLAB in the companies where I worked.  MATLAB was an excellent tool to analyze data but the problem was its cost to keep them.  I could not pay this cost by myself as it was expensive to me,  therefore I was forced to use excel in my personal data analysis in my home.  There was no choice except that. I wished I would have MATLAB in my PC many times before. Many experts in the financial industry have written books about programming of MATLAB. However, I could not program it by myself at home as no MATLAB environment existed there.  So I was very surprised when I saw how R worked three years ago.  It can be downloaded without any fee and has the powerful functions in it. I can program freely and store them as my functions.  I decided to start learning R.  Now that I know how excellent R is and always recommend R for anyone who are interested in statistics and data analytics.


R has advantages compared to other tools

1.  R is available without any fee.

This is the biggest advantage to proprietary tools, especially for beginners of data analytics. With R, beginners have opportunities to have experience of data analytics by a tool used by professionals.  R lowers the barrier to enter the world of data analytics.  Many people start data analytics from their curiosity,  In such case, it is very difficult to invest a lot of money to own statistical tools.  Now there is no need to worry,  just go to R-project site and download R.  It is easy and available to everyone as long as one has an access to the internet.


2.  R is an open source

R is an open source, therefore, it is transparent and you can make your program as you want. When you make excellent programs,  you can make your programs available to anyone all over the world through Rproject site. If you go to this site, you can find many kinds of programs,  which covered from economics, finance to biostatistics.  These programs are called “package” and prepared by professionals all over the world and you can look at each code if you want.  According to R-project site there are more than five thousand packages and they are still increasing. No one knows what the real total number of programs is in R.  Fortunately, most programs are available to anyone without any cost.  It is wonderful for anyone who are interested in data analytics.


3.  There is a lot of information about R on the internet.

R is a good tool for learning statistics because there are a lot of tutorials, instructions and documents on the internet. Most of them are free  so you do not have to buy books about R.  It is one of the reasons why I set up my start up for digital learning of statistical computing. I prepare the introduction course to R. If you are interested in R, you can look at this,  of course, without any fee.


I drew this chart by R. It is fun to do that.  Let’s start R now !

Key three things for data analytics


Big data is and will be one of the key words for business now and in the future. Then the question arises. What is the purpose of gathering data and analyze them?  We gather a lot of data  and analyze them in order to make a decision in our businesses. Without a decision,  no data has  meaning to  us.  Data should be gathered to make a decision.   The problem is how we can do that? Data is data, of course,  we cannot use the data effectively by just seeing massive amounts of data.  Vendors might provide us data warehouse, statistical models. But they are just tools, although they are important for data analytics.  We can not think each of them exits independently because they should be deeply interconnected each other. So I would like to present three key things to think about data analytics.   Let me explain each of them here.


1. Data

Data is a starting point about analytics.  There are many kinds of data, such as financial data,  temperature of regions and population of each country, etc.  Data has its meaning and structures behind it. Therefore, it is sometimes said that”Let the data speak by itself.”  In the beginning of analytics,  it is very difficult to identify which data has strong power to explain our observations in advance.  So, it is better we include as much data as possible to analyze.

2. Statistical models

Statistical models are interpreters to enable us to understand what the data means to us.   When we gather data, it is usually difficult to understand what the data mean just by seeing them because they are massive volume.  Therefore, we use statistical models so that we can understand what means the data has and identify the mechanism behind the data.  There are many kinds of statistical models.  We should be careful to choose the right one in analyzing data.

3. Outputs

Outputs are critically important for business managers because outputs directly impact their decision in their businesses.  Certainly outputs are provided as numbers or graphs.  All we have to do is to ensure these numbers and graph can support a decision by business managers.  Statistical models can produce only the numbers, not explanations about numbers.  Again, we should ensure the outputs of the models is what we need to make a decision in our businesses.


I always considered these three key things when I analyzed the risk of each borrower in the consumer finance company. When you wonder what are needed for data analytics in your own business,  I recommend  you to consider these key things above.  I am sure you can find what are needed to make your decision supported by data analytics.