Skip to content

Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump

License

Notifications You must be signed in to change notification settings

dbpedia-spotlight/json-wikipedia

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

88 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

json-wikipedia json-wikipedia

Json Wikipedia contains code to convert the Wikipedia XML dump into a JSON dump.

Setup

compile the project running

mvn assembly:assembly 

the command will produce a JAR file containing all the dependencies the target folder.

Convert the Wikipedia XML to JSON

java -cp target/json-wikipedia-1.0.0-jar-with-dependencies.jar it.cnr.isti.hpc.wikipedia.cli.MediawikiToJsonCLI -input wikipedia-dump.xml.bz -output wikipedia-dump.json[.gz] -lang [en|it] 		

or

./scripts/convert-xml-dump-to-json.sh [en|it] wikipedia-dump.xml.bz wikipedia-dump.json[.gz]

produces in wikipedia-dump.json the JSON version of the dump. Each line of the file contains an article of dump encoded in JSON. Each JSON line can be deserialized in an Article object, which represents an enriched version of the wikitext page.

Usage

Once you have created (or downloaded) the JSON dump (say wikipedia.json), you can iterate over the articles of the collection easily using this snippet:

RecordReader<Article> reader = new RecordReader<Article>(
		"wikipedia.json",new JsonRecordParser<Article>(Article.class)
).filter(TypeFilter.STD_FILTER);

for (Article a : reader) {
// do what you want with your articles	
}

You can also add some filters in order to iterate only on certain articles (in the example we used only the standard type filter, which excludes meta pages e.g., Portal: or User: pages.).

The RecordReader and JsonRecordParser are part of the hpc-utils package.

In order to use these classes, you will have to install json-wikipedia in your maven repository:

mvn install

and import the project in your new maven project adding the dependency:

<dependency>
    <groupId>it.cnr.isti.hpc</groupId>
	<artifactId>json-wikipedia</artifactId>
	<version>1.0.0</version>
</dependency> 

Schema

 root
 |-- integerNamespace: long (nullable = true)
 |-- lang: string (nullable = true)
 |-- links: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- description: string (nullable = true)
 |    |    |-- end: long (nullable = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- start: long (nullable = true)
 |-- namespace: string (nullable = true)
 |-- paragraphsLink: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- links: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- description: string (nullable = true)
 |    |    |    |    |-- end: long (nullable = true)
 |    |    |    |    |-- id: string (nullable = true)
 |    |    |    |    |-- start: long (nullable = true)
 |    |    |-- paraText: string (nullable = true)
 |-- redirect: string (nullable = true)
 |-- title: string (nullable = true)
 |-- type: string (nullable = true)
 |-- wid: long (nullable = true)
 |-- wikiText: string (nullable = true)
 |-- wikiTitle: string (nullable = true)

About

Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 98.9%
  • Other 1.1%