Open DataCamp – Bangalore’13
I attended and gave a part of talk at Open Data Camp, Bangalore 2013, which was scheduled on 2nd and 3rd March’13. ~200 people participated in the camp from different fields. I met lot of smart people and had a great time. Sessions I attended :-
Visualizing Politics: S Anand
Anand took the data from 2004 Lok Sabha election, which he got from ECI website and showed lot of interesting facts and stats on the India’s map. His presenstation is available here.
As usual there was lot lot to learn from his talk.
data.gov.in - Alka Mishra
Alka gave overview about data.gov.in, which is an Indian Government initiative to open-up data from different government departments. She shared the policies and processes through which data.gov.in is collecting datasets from different departments. data.gov.in is currently working with Central Government but some of the states also shown interest.
She understood the concerns of the data enthusiasts that most of time, the data on the government websites is not in usable format. She said now there are such polices that the different departments have to submit the data in a standardize format.She urged for more participation from the citizens on the data.gov.in. She said that we can request for any specific datasets. Her session was following by a panel discussion “Why Open Data is Good for Gov and Citizens alike”.
Big Data and Hadoop in Indian Context – Mohan Kumar
Speaker mentioned some interesting facts and have an overview of Big Data and Hadoop.
Panel Discussion: What has Open Data got to with UIDAI? – Sunil Abraham, Anivar Aravind and Anant Maringanti.
This was most interesting session I attended at ODC. It came as total surprise to me. The panelists discussed whats not so open about the UIDAI
An Introduction to Apache Hadoop – Mrinal Wadhwa
On day-2, I attended half day session on Apache Hadoop. Mrinal gave really good overview on Hadoop. For what tasks we should use Hadoop and when we are better off with RDBMS. I enjoyed this session as there was lot to learn.
I gave following talk with Rajesh K, Meera K, and Vineet.
Public Problem Solving using Data: Lessons from the Daksh Survey
DAKSH is an independent, professional, apolitical, not-for-profit organization based in Bangalore, which surveyed people of Karnataka to see how their respective MLA performed. Through Rajesh I came to know that they needed someone to analyse the collected data and come up with a score card for each MLA. I volunteered and did the analysis. Following shows the overall exercise we did as a group:-
The score card for each MLA was published in Vijay Karnataka, which is a leading Kanada News Paper. Rajesh started the talk by the introduction to Public Problem Solving. After that Vineet introduced the audience to Daksh. Then I share the steps which I used to analyse the collected Data. Later Meera showed how Citizen Matters is using the results collected from above exercise, which a very small part of overall effort which her team is putting to collect the stats for each MLA. The audience were surprised to how people from different fields came together and worked on this.
Following is a picture of mine showing the score-card of MLAs which was published on Vijay Karnataka. Thanks to Meera Sankar for taking the pic.
.
2012 – The Year it was for me..
Three most interesting things I did/happened during 2012 :
- Taught a full semester course
- Joined back Red Hat and moved to Banagalore
- Did not watch television for the entire year
Jan to March
Started of the year started on a very high note with a bike trip to Mahabaleshwar with my wife.
Just after that trip I had to travel to US for some official work. This was not as fun as it was with the first trip. Surprisingly I met one a school friend from 4th grade there.We had some good time there.
After after coming from US, I participated in Gnunify’12 and organised programming contest.
April – June
In April we went to memorable Matheran trip. We traveled on Toy Train (Neral to Matheran). It took more than expected time to reach destination; it looked time was stuck during that toy train journey. How great that was !!!
During that trip I decided to look out for next professional opportunity and leave STEC (my previous company). After a month or so I got about know about an opportunity in Red Hat and applied for it and got selected. Its good to apply for just one company and getting selected. Sometimes having more than one offers, just confuses and later we think if I joined that company then things might be different.
During that time I got the opportunity to teach Operating System at SICSR, Pune. So I started teaching.
July – September
I took a month long break after leaving STEC. During that time I just taught at SICSR and did some study of statistics other than just relaxing ..:). I also a talks on Linux Performance Tuning at PLUG and Pune University. After completing my teaching I moved to Bangalore in Sept. and joined Red Hat. Though it took some time to settle down but I am liking here more than Pune.
Oct – Dec
During this time I settled down in Bangalore and new work place. I visited my home town, Jabalpur during Deepawali.
And for the entire year we did not re-charge my Tata-Sky connection. Yes!! really. By doing this we got more family time, read more books/newspaper and were away from the randomness which television puts in our lives. Everyone should give it a try. Though we watched few things online.
While working on a problem I had to revise my basics of statistics/probability. While doing some search on the net I saw there is lot happening in this field or it was always happening but I was not looking for it. I slowly started developing interest in this field. I attended a hack night in Pune and Fifth Elephant Conference in Bangalore. Lets see how much I can do here…
There were few setbacks but not worth mentioning here. I am eagerly waiting 2013 to un-fold ….:)
Joined back Red Hat!!
On 3rd September’12 I joined back Red Hat 
. Earlier I worked with Red Hat for 3 years in in GSS (Global Support Services) in Pune. This time I joined the peformance team in Banagalore office and will be working on Red Hat Storage (Gluster) product.
I am really happy to join back the company I like most.
Teaching experience at SICSR, Pune
Last month I finished teaching a semester course on Operation Systems at SICSR, Pune. We Started the session in Mid June and finished on 1st September. In June and July I taught with my full time job at STEC. I left STEC on 3rd August and then teaching became my primary focus.
I taught MSc(CA) first semester course. The students had varied back-ground. When myself and SICSR team, specially Harshad Gune were putting down objective of the course, we decided to teach by giving examples rather than typical OS theory. So that students at-least get to see the tools like cscope, git, make etc, through which they can traverse the kernel coded, compile the kernel… it they wanted
After the first session I was not sure whether student would be able to cope with the content which we have lined up for them. Many times I just had to repeat the 75% of content of the previous lecture to make everyone on the same page. This was still OK but when students did not answer which I just taught I got really frustrated. It felt like why I am trying to teach such advance stuff to the students who do not care. But there were some students who followed me as expected. I think this story of almost every class… .
After couple of weeks of teaching I realized that may be I should not force students to learn each and everything I teach them. So I started focusing much on the fundamentals rather giving more details. This lowered my expectation from the class but at the same time students started grasping more.
That being said I saw good improvement in their understanding of the subject over these months. I am satisfied with kind of the results students showed me in my internal exams.
Earlier I took week-long courses but this was my first full semester course. The latter was much more challenging and fun. It was great learning learning experience for me.
I hope to get more opportunities to teach in future.
Linux Performance Analysis and Tuning talk at PLUG and University of Pune
On this month’s PLUG session I gave talk on Performance Analysis and Tuning on Linux, which has good feedback. Then I got an invitation to do the similar lecture from CMS (Centre for Modeling and Simulation) Department, University of Pune. I accepted it and took a session on 25th of this month. I covered same topics in both sessions.
I decided not to have any slides and just show output of commands like ps, top, sar and different settings in “/proc” and “/sys” filesystems. I tried to give information about differrent fields in the outputs and related that with performance issue,
like “load average” in top output, Average request size in iostat ouput.
To show the different subsystem I referred different figures from http://www.redbooks.ibm.com/redpapers/pdfs/redp4285.pdf
Following are some commands and other details which I share
ps command
$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.2 61988 5236 ? Ss Aug17 0:02 /usr/lib/systemd/systemd root 2 0.0 0.0 0 0 ? S Aug17 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S Aug17 0:05 [ksoftirqd/0]
- top Command
$top
From the top command output we discussed :–
- What is load average and what does it high value means?
- What are different fields in line starting with “CpuX”?
- What is nice value and how it effects the priority for a process?
- free command
$ free -m
total used free shared buffers cached
Mem: 1962 1306 656 0 75 508
-/+ buffers/cache: 722 1240
Swap: 2000 0 2000
With above output I shared the concept of the “buffers” and “cached”. Generally we consider “cached” memory as used memory and does not count it as free, which is not correct. Cache memory can be freed whenever needed. Later I demoed with the drop_cache setting of the VM’s (Virtual Memory) configuration(/proc/sys/vm)
- SAR (System Activity Reporter)
It is used to collect, report, or save system activity information. It can report CPU, Disk, Memory, Network, paging and other information. If configured with cron then it collects information everyday and store the statistics in “/var/log/sa” folder. The “saX” files are binary files and “sarX” file are human readable data, where is day of the month. We can collect all those stats manually as well.
Following command will collect Disk, CPU, Memory, paging, load average stats in every 5 mins and save it in “logfile” file in binary format:-
$ sar -dp -u -r -B -q 5 -o
$ sadf -d — -dp -u -r -B -q 5 >
- /proc/meminfo
- /proc/<pid>/maps
- /proc/diskstats
- /proc/sys/net/ipv4
- /proc/slabinfo
- /proc/interrupts
Then I showed how to configure sysRq and, collect stack trace for each process and memory information of the system
echo 1 > /proc/sys/kernel/sysrq echo t > /proc/sysrq-trigger echo m > /proc/sysrq-trigger
The Fifth Elephant » Pune Data Hacknight
I came to know the Pune hacknight from the twitter and after digging some more on the net I came to know about The Fifth Elephant conference in Bangalore.
I did not had any prior working background about the big data but I always found it interesting. In my current profile at STEC, I anyalize the performance numbers, so I thought hacknight would be the right place to place to get started.
I registered myself and waited eagerly for the event. On the day I reached there at the Venue on time. I saw some familiar faces like Saleem and Shreyank. For others I was meeting for the first time.
Once everyone was in we had a round of introduction and discussed the ideas posted on the website. We then listed down the ideas for which we had the datasets. I found following ideas interesting:
- Travel Recommender
- IEEE ICDM Kaggle Competition
- Automatic Twitter Movie Reviews Analyzer
- Movie Star Social Media Popularity Meter
I decided to work on the “IEEE ICDM Kaggle Competition” with Jaidev. It took some time to figure out what the exact problem is and what the training sample is.
There was a listing of the products in a json file and reviews of the products in another json file. We have to the match the reviews with products. Rather than jumping into writing any code for reading the file I decided to look at the data more care fully. After spending some time on that I saw there we have to write some machine learning tool. I do not have much idea in that area. Jaidev gave me reference material on that.
I was not how I should proceed further so I decided to not to spend more time on the dataset. I started looking for more basic definition like what is bigdata, what does data science means, why it becoming more important etc. . I found it very fascinated that what all we can do with the data.
Sometime in the night Nikhil shared some info about group databases and neo4j, which I found is very useful. Shreyank told us what we can do with the D3. He pulled out his friends list from the Facebook and their connectivity with each other. Later he showed very nice graphs using the data which he collected. I was amazed to see them.
hasgeek has done the great job in organising the event. The venue was centrally located, spacious and had good internet connection. We had plenty of snacks, soft and energy drinks.
Everyone was free to work on any datasets, which is good. For beginners like me if we had some simpler datasets with well defined problem statements then it would have much better.
Overall I liked the event very much and would like to come for more such events. Meanwhile I would try to work with simpler datasets.
Saurya-Mandal ki sair (Star-Party)
Yesterday I attended the star party organized by Jyotirvidya Parisanstha, Pune.JVP organizes an over-night star party outside the city of Pune around 30 km from the city glow.Program contains necked eye star gazing, telescopic viewing of moon, planets, nebulae, galaxies,slide shows; astro-games, etc.
I came to know about this from our Askshar Bharati group that we are invited to join it with some fee. From AB group around 30 members attended it. I went with my wife and one of her friend. There were around 80 participants other than our group. Most of them were kids.
The Party was organized at Pirangut, which is around 30 KMs from Pune. We reached there around8 PM and first thing we saw was Venus from the telescope which was looking like crescent Moon.
Then one of the member from JVP showed us different constellations like Ursa Major, Ursa Minor, Gemini,Leo, Libra ,Scorpio; Stars like Pole-Star,Svati (nakshatra),Chitra and Planets like Saturn, Mars,Venus etc from naked eyes. We were amazed to see the Saturn from the telescope. The outer rings of Saturn were clearly visible.That was a magnificent view , must watch.
We then had the break for Dinner. In between JVP team installed more telescopes; from which we saw Mars and other clusters of stars.
Then we were shown few videos and a demo of Stellarium software. Stellarium is a free open
source planetarium for your computer. It shows a realistic sky in 3D, just like what you see
with the naked eye, binoculars or a telescope.
I was very happy see such open-source software.
After this we had another session in which instructor showed us Milky way and other objects.
It was 2 AM by than. I then slept for 2 hours and after I woke up I saw kids were playing Antakshari based on solar objects. Its is really a great way of learning.
By 5 AM we left the place and reached Pune by 5:45 AM.
Sessions like these are great way of learning about our Solar-System for everyone; especially for Kids.
Akshar Bharati Community Library goes online
Today we made ABCL (Akshar Bharati Community Library) online. One can access it from http://abcl.aksharbharati.org/
We now have more than 3500 books and it is operational for last six months.
By making it online now users can:-
- Search the books online
- Put a hold on book online
- Get email notification
- Suggest a book which is not there in catalog
It is made online using softare called KOHA. It is put on Amazon cloud. Its costs around Rs 3000 per month.
ABCL is a free libary for everyone though we encourage the Rs 100 donation per month for people who can afford.
Delivery model
1. A member can come to library location and get the books any no. of times.
2. Corporate model
A company can sponsor the library for its employees or support staff by donating the amount. In that case a person from the Library will visit the office and issue the new books and take the old ones.
Enabling the corporate model would take some time. So if we can around 10 members from a company then the person from ABCL can also visit.
3. Housing socities.
Similar to the corporte model we are planning to a model at housing society. Housing society would not be able to sponsor thier members and their support staff.
So if there is any person who is willing to become the ABCL representative then we can deliver the book at the housing societies weekly.
We will be adding more than 1000 in next two weeks. Apart from reading books kids from nearby the place come to read and play some books.
Some Volunteer from also come here and teach basic Computers to the Kids. Till now ABCL is doing exceptionally well and with putting Library online we should be able to take it next level.
Gnunify’12 and Programming Contest
Gnunify was scheduled on 10th and 11th February’12. This year Gnunify has completed 10 years.It has come a long way and I am attached with it since 2006. Since last year I started to more active part by organizing the programming contest for System Programming. We declare the topics for the programming contest one month so that students can go back go back and study those topics which were declared if they haven’t read them seriously till now. The topics from this year were:-
- Process
- Process Control
- File I/O
- Signals
- Locking Primitives(Mutex/Semaphores)
- Multi Threading
This year my company STEC sponsored this contest. We got 100+ registration but around 50 people showed up for the written test.Out of them 10 qualified for the lab test. There were 30 objective questions for the written test and 3 questions for the lab test.Lab test was relatively easier as compared to last year.
Questions for Written Test
Questions for Lab Test
Shakthi Kannan and Kiran Divekar judged the lab test. Shakthi also helped me with the lab to setup the lab and answer the questions raised by participants. The lab test was for 2 hours and judges went to everyone’s desk in the last half hour to look at their programs.
We had the prize distribution at 2 PM. Apart form me,Vivek and Ritesh were present at the time of prize distribution. Vivek briefed the participants and others about STEC and its India Operation. Judges then distributed the prizes to winners and participant certificate to all participants.
Apart from organizing the programming contest I gave a talk on btrfs which I gave on FUDCon last year as well. Surprisingly the auditorium was full. There were lots of students so I spent most of the time in explaining concepts like file-system, COW, snapshots etc.
I did not get the chance to attend to other talks as programming contest kept me busy. Though in Lunch time I got time meet few my friends from FOSS community. We observed that participation in Gnunify is getting decreased from last couple of years. This might be happening because now we also very technology specific workshops/events and now people want to attend them. There were also discussion that we should now come with different approach to oraganize Gnunify. Lets see how things shape up in near future.
Long live GNUnify!!!
Setting up 26 Libraries on 26th January’12 @ Akshar Bharati
Akshar Bharati’s vision is to help achieved India’s Vision 2020 of becoming a developed nation and sustain the growth through development of tomorrow’s citizens by encouraging environment of learning and education. Keeping this in mind we have decided to setup 26 Libraries on 26th January’12. To make this a reality we have to accomplish following tasks:-
- Identifying the locations
Over the years AB team has setup 300+ libraries in different places of India. Because of such visiblity we regularly get requests to set the library at new location. We have few such requests and will set libraries in few place from this list.
We’ll pick few locations from Donor’s request. Few donors choose to donate the whole library which costs around 25K(INR) in their home town or in remembrance of their loved ones. Apart from this AB team members do frequent field visits and suggest the location where we can setup potential libraries.
From above and other sources we’ll select 26 location and will setup all of them on the same day.
- Procurement of the books
As AB always setup the library with new books and donate books according to the need, the procurement team has to go through the the selected location list and then decide the right set of books.
After selecting the books the team will place the order the to the publishers and get them in the central place. Then the team will sort them according to the locations and stamp them. Then the book set will be forwarded to the team which will setup the library at the physical location.
- Fund Raising
And Guess What!!!! Setting all the libraries require funds. We have some funds which was given to AB from individuals and companies like IBM, Symantec, STEC, and ATOS Origin. But we need to collect more funds.
For this we’ll be reaching out to more companies and individuals. As we all know it is very difficult get some money from someone’s pocket. This task will
require some extra effort. People don’t pay until they have 100% faith in the cause.
- Library setup at the location
Not everyone can go to each of 26 locations, so we’ll have a team for each location which will go setup the library. But before setting it up lot of co-ordination is required b/w each team.
I hope we’ll finish all the tasks as expected and setup all the libraries.
If any of my blog reader want to contribute to any of the tasks I mentioned then please let me know. As it is not possible for one present physically I would appreciate any donation which one can make and do one’s contribution to India 2020 Vision. Indian salaried donor can get tax exemption under Section 80G. Donation can be done online. Details are mentions here. Here is a documentary on Akshar Bharati