Learning Machine Learning: The Infrastructure

1 Comment

braindata-370x290In 2016, all I was reading about was big data, deep learning, artifical intelligence, machine learning, etc… soon I realized I needed to do more than just read about it. So for 2017, I decided it was time to take a deep dive into Machine Learning and see what all the buzz was about.

I haven’t programmed in 20 years but figured now would be a great time to restart. From all the reading I did in 2016 it was clear that the programming language of choice for Machine Learning was Python. I didn’t want to take a bunch of disconnected courses on Coursera and Udacity to learn about Machine Learning, instead I had a project in mind. When I moved to India 12 years ago, it was to launch an algorithm/quant hedge fund and I was the guy tasked with getting all the technology infrastructure (servers, data feeds, leased lines, datacenter access, etc…) in place and then over time I would learn to build trading algorithms. One thing led to another and I never got around to build those models. Over the years, I felt the algo/quant space was over done and it would be tough to get back into it. However there has been a resurgence with all of the new technologies involving Artificial Intelligence entering the space. So that was my goal, learn Machine Learning to trade the stock market.

I spent the first couple weeks of the new year putting together a plan to accomplish the end goal. The first thing was to take an introduction course on Python from Coursera. In parallel I was researching the algo/quant side and understanding what goes into building models, trading models and risk management. Not only did I want to learn about Machine Learning but whatever I did, I wanted to build it like it was going to be a billon dollar asset management company – highly redundant architecture, quality data feeds and top-notch risk management. It soon became clear this was something that was not going to get built over the weekend!

I was able to breakdown the work into 3 stages:
1. Infrastructure – cloud provider, servers, databases, data feeds, trade execution
2. Research trading models – researching and designing algorithms to produce “alpha”
3. Risk management – once the trade is made, constantly monitoring the position and making sure it fits within the risk model that has been designed. Or as they say within the industry Value at Risk (VaR).

This blog post will talk about the infrastructure and some of the technology I learned along the way.

It quickly became apparent that many of the Machine Learning experts were using something called Jupyter which is an open-source platform to share notebooks and run live Python code. It’s like an online version of an IDE (integrated development environment) that programmers use to build applications.

The next thing was to start getting data and lots of data onto the platform that I had built. For all the crap I talk about Yahoo, they have a pretty good finance section to download historical stock data for Indian stocks. Using pandas, a Python data analysis library, I was able to pull down all the price data I needed.

Some of the technologies I learned and implemented along the way:

  • Amazon Web Services – the cloud provider
  • EC2/Ubuntu – Linux distribution on an EC2 server
  • Let’s Encrypt – secure the server with a free SSL cert
  • Python – programming language
  • Jupyter – online IDE
  • pandas – data analysis library for Python (developed by an AQR employee)
  • Python scripting – used to get the daily price updates from Yahoo
  • RDS/MySQL – database where the price data resides
  • crontab – run the Python script at 2am in the morning
  • crontab.guru – a super simple site to understand the syntax for scheduling cron jobs
  • MySQLWorkBench – Software to interact with the MySQL DB
  • SQL statements – Structured Query Language (SQL) to manage and get data from the DB

Below is a SlideShare document showing the process of setting up the server on AWS:

Part 2 will talk about the research aspect of building trading models – the traditional methods and using the newer Machine Learning tools like Apache SystemML, Caffe2, Microsoft’s CNTK,  TenserFlow and Sciket-learn to name a few.

Thank You, Mukesh Bhai

Leave a comment

55724156.cms
On September 1, 2016 the largest Indian company by market cap – Reliance, entered the telecom market…again. In 2002, Reliance launched its first mobile network and brought the per minute call rates down. This time around Mukesh Ambani launched Reliance Jio at the annual shareholders meeting and vowed to bring down the data prices.

I remember last year calling Vodafone to get a better 4G data package and they were offering plans with 2GB, 4GB and some with 8GB of data. And my cable TV provider Hathway was offering monthly plans for 60GB and 80GB of high-speed internet at 50Mbps. Mukesh bhai was not kidding about bringing down the prices. Today, Vodafone is offering me around 7GB a month and Hathway is now offering 200GB a month at the same price I was paying. I’m sure I can negotiate and get better pricing but I’ll wait and see.

When Jio was launched they gave it away for free so everyone could experience high speed internet, which was a brilliant move. It was a hugh gamble by Mukesh and team and I think it has paid off. Because, once you get a taste of high speed internet, you are hooked and it’s really tough to go back to 2G (edge) or 3G speeds.

Although Jio has many pricing plans, I think it’s all noise. They are focused on only 1 plan – the Rs. 303 monthly plan. For Rs. 303 you get unlimited voice calling across India and 28GB of data. 28GB of data per month is a hugh disruption not only for the Indian telecom scene but around the world. AT&T Wireless in the US recently announced their “unlimited” data plan that is capped at 22GB per month.

So why did Jio pick Rs. 303 as the price point that they wanted to focus on? Currently, the average mobile phone user in India spends about Rs. 141 per month, what the industry calls ARPU – average revenue per user. I think Jio is betting that the average Indian user will double their spend to over Rs. 300 a month but only if they find value in it. And from the looks of it, they are finding value in the service. Everywhere I look, people are glued to their phones and streaming content such as movies, music and TV shows. Jio has over 100 million free users and as of March 31, 2017 over 72 million people have applied for the Rs. 99 Jio Prime yearly program.

As I mentioned because of Jio the other telecom providers have dropped their prices as well. Below is a quick comparison:
Jio – Rs. 303, unlimited voice calls, 28GB
Vodafone – Rs. 346, virtually unlimited voice calls, 28GB
Airtel – Rs. 349, unlimited voice calls, 28GB

And just for fun, let’s compare how these plans stack up against the US wireless carriers:
Sprint – Rs. 3888 ($60), unlimited calling, 23GB
T-Mobile – Rs. 4536 ($70), unlimited calling, 28GB
Verizon – Rs. 5184 ($80), unlimited calling, 22GB
AT&T Wireless – Rs. 5832 ($90), unlimited calling, 22GB

I would like to take this opportunity to thank you Mukesh Bhai for changing the telecom game in India not once but twice. I run by your house almost everyday and think of two things as I pass, just how fat is your internet pipe and why do your Z level security agents not like when I try to run on your sidewalk. Or is your fiber optic cabling located under the sidewalk?

The Future of Payments

Leave a comment

Fintegrate_2017
Earlier this month I had a chance to be on a panel discussing User Experience (UX) for payments. The panel was part of the Fintegrate Zone 2017 event located at the BSE Building in Bombay hosted by the Zone Startups.

The panel was moderated by Harsimran Julka @HarsimranJulka an editor for the Economic Times. The panel included:
Anurag Sinha, Co-Founder, Walnut App
Deepak Agarwal, CDO, Barclays Wealth
Sohini Rajola, @RajolaSohini, RVP, Western Union
Tina Singh, @tinasinghj, CDO, Mahindra Finance
Malcolm Anthony, Head of User Experience Design, PayPal
Nitin Vyakaranam, @vnitinb,  Founder & CEO, ArthaYantra

As with any recent discussion involving the Indian financial markets half the time was devoted to talking about Modi’s demonetization. It was more about who benefited from it and who struggled with it, as a whole most fintech startups all benefited from it.

Although we touched on the overall user experience of payments and had much to debate about, I still feel most of the world is struggling with a seamless payment experience. Part of the issue is that people are used to physical cash and it’s been around for ages. People are familiar with it and how to use it, kids from a very young age are taught about physical money and many have piggy banks with some of that loot! Basically, cash is convenient, intuitive and effortless.

But as with everything else, we need to move forward and electronic payments are the future and most governments are behind it as a way to tackle the black money and counterfeit money. Credit/debit cards are a hybrid instrument, although the card is physical in nature it connects to an electronic platform to authorize, clear and settle the payments. Credit cards are prone to fraud since someone can steal your card, go to an online store and enter your card details and buy stuff.

This is where a whole new generation of solutions are entering the marketplace under the banner of mobile proximity payments (MPP), this includes near field communications (NFC) and quick response (QR) codes. NFC is the technology behind Apply Pay, Google Pay, Visa payWave and MasterCard contactless,  it’s a communications protocol that works with devices that are within inches of each other. With Apple Pay when you are ready to checkout, the retailers point of sale (POS) system will “talk” to your phone and then you use Touch ID to authenticate and enable the payment. That really is the way to do it. The problem with NFC is that the phone has to have an NFC chip and so does the retailers POS system. I don’t see this gaining much traction in India as many of the phones are fairly inexpensive and won’t include an NFC chip for years.

How-to-get-paytm-QR-code-175x300
Surprisingly, because of India’s demonetization the use of QR codes has gone from a niche type of application to full mainstream usage. Demonetization was a stroke of luck for Paytm and they turned it into gold. Overnight people needed to send money and many people quickly downloaded the Paytm app and started to transact.

A couple weeks ago, I used the QR code functionality to pay for parking at Phoenix Mills and it was pretty seamless. Since all smartphones have a camera they can scan this QR code and submit a payment to an individual or retailer. I really see this taking off and becoming the standard in India, it’s a low tech solution but sometimes that’s required to get high (mass) adoption in India.

BharatQR, Another Payment Option?

Leave a comment

It’s another day and yet another payment option/technology was launched in India. The newest one to the party is called BharatQR, it’s being launched by the Government of India. BharatQR is like Paytm except instead of using e-wallets, you just need a bank account. It’s pretty clear the Indian government is hell bent on getting most people to transact online. With the explosive growth of Paytm, I’m guessing the government decided it needed it’s own QR-code offering.

I think this is a great move but I think the average user will be even more confused now. Below is a list of electronic payment options that I have compiled in alphabetical order:

  1. Aadhar Enabled Payment Service (AEPS)
  2. BharatQR
  3. BHIM
  4. Apple Pay and Android Pay (coming soon…)
  5. Credit/debit card
  6. E-wallets – Paytm, Mobikwik, etc…
  7. IMPS
  8. NEFT
  9. RTGS
  10. RuPay
  11. UPI

Yeah, even the most tech savvy person would get confused. I think the government should just wrap AEPS, BharatQR and BHIM into a single app and make that the defacto standard.

10 Years of the Mumbai Marathon

Leave a comment


Yup, the first Mumbai Marathon I ran was in 2006. Back then it was really easy to register for a running bib and probably under 5,000 people ran the half-marathon. Now it’s tough to register and they have capped the number of runners at some 25,000 for the half-marathon. When I used to train 11 years ago, the streets were empty and people would look at me like I was some crazy person. Today, if you don’t pay attention you might “run” into another runner on a Saturday or Sunday morning run. It’s great to see the whole city get involved in the race and especially all the people who come out to cheer on Pedder Road and Marine Drive.

Below are my finishing times for the 10 years I’ve been running the Mumbai Marathon.

1. 2006 – 2:16
2007 – did not participate (some wedding, I think mine!)
2. 2008 – 2:16
3. 2009 – 2:07
2010 – got sick
4. 2011 – 2:01:26
5. 2012 – 1:57:01
6. 2013 – 2:02:59
7. 2014 – 2:08:37
8. 2015 – 2:04:18
9.  2016 – 1:58:31
10. 2017 – 2:08:47

Picture from the first Mumbai Marathon I ran in 2006.

Modi Marches On

Leave a comment

We live in an era of limited attention span, super short news cycles and the upcoming President of the US who uses Twitter and it’s 140 characters to talk. When PM Modi announced on November 8th that all Rs. 500 and Rs. 1000 notes would stop being legal tender as of midnight that day, it was like an earthquake and here we are almost 46 days later still talking about it.

The demonetization topic has come up at almost every party or business meeting I have attended and it’s been great to hear the pros and cons of PM Modi’s actions. First, I think we Indians can adapt to any damn thing and this exercise clearly shows that. People that had stacks and stacks of black money figured out ways to deposit their money into the banks. It remains to be seen if they will be able to get their money back or how much of a penalty they will have to pay. On the other hand, the middle class waited patiently to deposit their money and waited even more patiently to get the new currency notes.

The poor ended up being pawns in a political game where the opposition party said the poor were suffering the most. Actually, the poor have been suffering long before demonetization. The per capita income in India is about $1,500…not per week or month that’s per year. The Chief Minister (think Governor of a US State) of West Bengal, Mamta Banerjee, was one of the harshest critics of the policy and was on TV almost every night to highlight how much the poor are suffering. Because of the lines that people had to stand in line to get cash their own cash. Uhhh, we Indians are used to lines. Go to VT or Churchgate train station at 6:30pm and tell me what you see. Come to Nariman Point at 6pm to catch a bus and tell me what you see. I’ve seen these lines in Nariman Point for the past 10 years and that hasn’t changed.

The opposition party even played some of their classic hits like ex-PM Manmohan Singh. Manmohan Singh is like a one-hit wonder, he might have been the chief architect of India’s entry into the global economy in 1991 but he also was the PM during one of the most corrupt periods in recent times and was absolutely silent about it. (The joke is when he visited the dentist, the dentist said “at least open your mouth in my office”.)

I hope Modi doubles down on his drive to make the country a digital currency nation. When people say, how can you expect a poor man to buy a smart phone to take part in this new digital economy I just lose it. Have the politicians scammed this country for so many years that they have not been able to lift people out of poverty? That’s the real tragedy, not demonetization.

24 Hours of Craziness With Modi and Trump

Leave a comment

This past Tuesday around 8pm (Bombay 8pm, NYC 9:30am) I turned on CNN International to watch US voters goto the polls to elect either Trump or Clinton. It was clearly going to be the Hillary Clinton show based on all the media publications and polls who said Trump goes low and that Hillary goes high. As I was switching channels I saw that India’s Prime Minister Narendra Modi was giving a speech at the same time. I thought man, why is he trying to steal the limelight while the biggest elections in the world are happening and this guy must be talking about Swachh Bharat #2 (Swachh Bharat = Clean India). I continued to flip channels and settled back on watching CNN International.

1narcos-tmagarticleThen my phone went nuts with all the WhatsApp messages (yeah, India loves WhatsApp and not Facebook Messenger). I’m wondering if there was some announcement that CNN missed in regards to the US elections – nope. PM Modi dropped the equivalent of a nuclear bomb on the financial markets by banning the use of Rs. 500 (USD 7.40) and Rs. 1000 (USD 14.80) denomination notes in an effort to fight black money, corruption, fake currency notes and terror financing. This was not some ban that would go into effect in 1 week, 1 month or 1 year, it was going into effect in 4 hours – at the stroke of midnight Tuesday. The old notes could be redeemed at a bank over the next 2 months for new notes. Of course, if you have unaccounted wealth its going to be a very awkward conversation with the Income Tax officials to explain it. Which means most people will just burn it Pablo Escobar style like he did in Narcos. As of Wednesday morning the biggest legal tender was Rs. 100 (USD 1.48) until the new currency notes hit the market. It was a genius move by PM Modi and one that will go down as the biggest changes in the course of India’s financial markets. Yes, there is probably some hyperbole in that statement but it took real balls to pull this off by PM Modi. I guess in the end he really did launch Swachh Bharat #2 (Swachh Bharat = Clean India).

pollsAt this point, my TV remote was probably ready to retire since I was switching between 5 or 6 news channels to understand the implications of this massive announcement from PM Modi. All the while I was following the US elections on CNN and Nate Silver’s FiveThirtyEight website – which was still showing that Hillary was going to win. So, I went to bed at 1:30am in the morning (Bombay 1:30am Wednesday, NYC 3:00pm Tuesday). I got up at 5am in the morning to start watching the election results come in as the polls started to close across the country and all of a sudden the numbers for Clinton don’t look as awesome as they did just hours ago. I kept on checking FiveThirtyEight about Clinton’s chances of winning and it was like watching the stock price of a company that just announced it was being investigated by the FBI (wait, that actually did happen to Clinton). It seemed like Trump might win and the financial markets around the world were poised to open gap down.

Then it happened, Trump was able to cross the magic line of 270 electoral votes and win the US Presidential elections to be the 45th President of America. I’ll be honest, it’s still a funny thing to say, hear and type that Donald J. Trump is the next president of the United States of America. I didn’t vote for him and I really hope he only focuses on getting the economy back on track and fixing the broken political system with its special interest groups, lobbying groups and political action committees. It’s definitely wait and watch.

On another note, something that I’ve observed over the past few days is that people want to hear what they want to hear. Or in the case of online media, see or share things they have a strong viewpoint about. Of course, that’s the whole point of Facebook and Twitter to share, post or highlight content you agree with. Unfortunately, some of that content is either fake or written just to drive traffic to a page that is plastered with ads. Then there are the articles that are absolute puff pieces with zero fact checking and just published to cater to audience that likes or dislikes a particular point of view.

Below are a couple screenshots this past week from my Facebook wall.

facebook

This article claims 3 people have died because they were standing in a line. Yet, they provide no information of the deceased or interviews with their next of kin. This piece definitely caters to people that are livid that PM Modi has screwed them of their hard earned black money.

facebook-image

The narrative of this Facebook post I assume is that Trump is going to be President and the “shit has already started” in that minorities are being targeted. The problem is that this Saudi student died a week BEFORE the elections. Yet, people are sharing it and adding their two cents. I’m waiting for someone to share a story about Rosa Parks like it happened yesterday.

This past week both Modi and Trump did some historic things which might rub some people the wrong way. My advice is don’t blindly follow and do your own research and understand both sides of the coin.

Favorite lines comparing the US and India:
US counting votes, India counting notes.
In the US, black president leaving. In India, black money leaving.

Note: As if this blog post wasn’t long enough, I just realized this is my 500th blog post. I’m sure it’s Trump’s fault that I didn’t write 1,000 blog posts by now 😉