WikipediaHI: Offline Wikipedia in Hindi !!

Last week I spent some time working on WikipediaHI activity for Sugar Desktop Environment. I must say it is one of the awesome activities I have come across. The best part is that it can serve you with data in offline mode. That is even if don't have internet connection which is otherwise required to access Wikipedia online, then also your WikipediaHI activity will serve your purpose.

There are lot many developers and contributors who are working in collaborative form on such awesome stuff who continuously inspire you to take up new things and create something that can be used by others in the world. Sugar developers and contributors are epitome of such group.

I came across few of such developers, Anish Mangal and Gonzalo Odiard, two of them whose contributions are significant for Sugar. I took up the task of creating WikipediaHI using Wikipedia dump for Hindi available for free. I followed the steps specified on this page[ hosted by Gonzalo] for creating Wikipedia activity in your own language.

I will quickly explain the steps I took to create WikipediaHI:

1) Downloaded the Wikipedia dump file for Hindi:
http://dumps.wikimedia.org/hiwiki/20121225/hiwiki-20121225-pages-articles.xml.bz2
NOTE: [ Make sure you pick the valid latest file from here : http://dumps.wikimedia.org/hiwiki/   this location will show you listing as per dates. Pick the latest dump and proceed further.]

and downloaded WikipediaBase from this link

2) Created "hi" directory for HINDI under WikipediaBase directory and moved the downloaded dump to this folder.

3) Extracted contents of this file using:
bzip2 -d hiwiki-20121225-pages-articles.xml.bz2

4) Processed the dump using page parser:
../tools2/pages_parser.py

The result of this operation will generate these files:
hiwiki-20121225-pages-articles.xml.links
hiwiki-20121225-pages-articles.xml.page_templates
hiwiki-20121225-pages-articles.redirects
hiwiki-20121225-pages-articles.templates

5) Then you can include selective articles or all articles from this dump to your activity by using this command:
../tools2/make_selection.py
* Make sure you have favorites.txt and blacklist.txt filled with appropriate keywords.

Now if you want to include all articles use this command:
../tools2/make_selection.py --all

6) Then proceed to create the index for these articles:
../tools2/create_index.py

7) In order to test the index created in previous step you can use this command:
../tools2/test_index.py

8) Next step is to expand the templates of articles :
cd ..
./tools2/expandtemplates.py hi

9) Go back to hi directory and re-create the index :
cd hi
mv hiwiki-20121225-pages-articles.xml.processed_expanded hiwiki-20121225-pages-articles.xml.processed
../tools2/create_index.py --delete_all

10) Download the images for the articles you selected:
cd hi
../tools2/download_images.py

if you want to download the images for pages you selected in previous step:
../tools2/download_images.py --all

11) Create files specific to language:
(a)activity/activity.info.lang : activity info file for you language activity
(b)activity/activity-wikipedia-lang.svg : activity icon for your language
(c)activity_lang.py : activity file for your language
(d)static/about_lang.html : about page for wikipedia in your language.
(e)static/index_lang.html : index page for wikipedia in your language. This is the page displayed when activity is launched. So its important for you to know the articles included in the search.db ( generated when index is created) for you to create the index page.


12) Create the XO file for wikipedia in your language:
./setup_new_wiki.py hi/hiwiki-20121225-pages-articles.xml

I went through the search.db file to identify the articles present in it and create the index page accordingly.
This gave me an idea to write some script that can generate index page(part or whole) to be used as home page for activity using search.db[ Stay tuned for next blog on this idea]

Here you go.. you can see WikipediaHI

On launching this, you can see the index page listing the articles you can view offline using WikipediaHI

If you want to play with WikipediaHI, you can download it : WikipediaHI-35.xo

I must thank Gonzalo for his amazing help and guidance in getting this done. I have to mention here that Wikipedia
changed its XML format in their dumps which resulted in error when I was creating the index. I took Gonzalo's help to get it resolved.
Thanks to Anish, who motivated me to pick this up and guided me to complete it.

Thanks guys !! :D

Writing Your First Sugar Activity !

The best open source project to work with is "Sugar" primarily developed by SugarLabs and developers, contributors around the world. As a  Sugar Evangelist, I take this opportunity to share this post that describes on how you can get started with your first Sugar activity. The program that runs on Sugar Desktop environment is called as an Activity.





Prerequisites:

• Familiarity with Python and PyGTK(Python interface for GUI programming using GTK+)
• Sugar Desktop environment( installed, build or emulator on qemu)
• GTK+ packages installed on you Linux flavor.

The example Activity shared in the following section has been tested on Fedora 17( Beefy Miracle).

1. Create a directory structure :

mkdir -p KartikActivity.activity/activity

2.Create activity.info :

Create a file inside the "activity" sub directory with name "activity.info" to describe your bundle in the activity sub-directory. The Activity Bundles specification explain in detail the meaning of each field.Write downs attribute names and corresponding values in this file as :

[Activity]
name = Kartik
bundle_id = org.laptop.Kartik
exec = sugar-activity Kartik.KartikActivity
icon = myicon
activity_version = 1.0
show_launcher = yes

example :

3. Activity Icon:

Design an icon for your activity by following the instructions on making icons for Sugar and place it in the activity sub-directory. The file name should match the icon file name specified in the info file (e.g. myicon.svg).

I used the same icon as is used in paint :

4. Create setup.py:

Write the setup.py script in the top level directory (e.g. KartikActivity.activity/setup.py). The content should be like:

from sugar.activity import bundlebuilder
bundlebuilder.start()

A more advanced version, which supports building activity bundles without Sugar installed, looks like this:

 #!/usr/bin/env python
 try:
     from sugar.activity import bundlebuilder
     bundlebuilder.start()
 except ImportError:
     import os
     os.system("find ./ | sed 's,^./,KartikActivity.activity/,g' > MANIFEST")
     os.system('rm KartikActivity.xo')
     os.chdir('..')
     os.system('zip -r KartikActivity.xo KartikActivity.activity')
     os.system('mv KartikActivity.xo ./KartikActivity.activity')
     os.chdir('KartikActivity.activity')

5. Code your activity in Python:

The name you specified in the .info file as "class" is the name of the class which runs your code. For the activity.info file above, we specify a top-level module named KartikActivity.Kartik

The content in Katik.py file :

from sugar.activity import activity
import logging

import sys, os
import gtk

class KartikActivity(activity.Activity):
    def hello(self, widget, data=None):
        logging.info('Hello Kartik')      

    def __init__(self, handle):
        print "running activity init", handle
        activity.Activity.__init__(self, handle)
        print "activity running"

        # Creates the Toolbox. It contains the Activity Toolbar, which is the
        # bar that appears on every Sugar window and contains essential
        # functionalities, such as the 'Collaborate' and 'Close' buttons.
        toolbox = activity.ActivityToolbox(self)
        self.set_toolbox(toolbox)
        toolbox.show()

        # Creates a new button with the label "Hello Kartik".
        self.button = gtk.Button("Hello Kartik")

        # When the button receives the "clicked" signal, it will call the
        # function hello() passing it None as its argument.  The hello()
        # function is defined above.
        self.button.connect("clicked", self.hello, None)

        # Set the button to be our canvas. The canvas is the main section of
        # every Sugar Window. It fills all the area below the toolbox.
        self.set_canvas(self.button)

        # The final step is to display this newly created widget.
        self.button.show()

        print "AT END OF THE CLASS"

6. Create a MANIFEST:

(e.g. KartikActivity.activity/MANIFEST), containing the list of the files (relative to the directory that the MANIFEST is in) to include in the package. (Note: Be sure not to leave blank lines at the end of the file.) This script does that in linux (run it from within the KartikActivity.activity directory):

cd KartikActivity.activity
find . -type f | sed 's,^./,,g' > MANIFEST

Content of MANIFEST should look like this:

activity/myicon.svg
activity/activity.info
MANIFEST
Kartik.py
setup.py

7. Give permissions:

Make sure that all your python files have the required permissions to be used.

chmod a+x setup.py
chmod a+x Kartik.py

8. Bundle your Activity:

Setup your bundle for development (must be user olpc when you do this) to become user olpc, type: su - olpc

If you are prompted for a password, trying using: su

python setup.py dev

This just creates a symlink to your activity folder in ~/Activities, so that Sugar can find your activity.

9. Run your Activity !:

Restart Sugar using Ctrl-Alt-Erase and your activity will appear in the interface! (NOTE: By default, the Home view shows only the favorite activities. You should press Ctrl+2 or go the right-upper corner and change to the List View)

Now you can see your activity is visible within Journal: