Skip to content

Commit

Permalink
RT
Browse files Browse the repository at this point in the history
  • Loading branch information
xros committed Mar 22, 2016
1 parent 17d1405 commit 7619811
Show file tree
Hide file tree
Showing 4 changed files with 33 additions and 33 deletions.
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Alexander Liu
* ```pip install jsonpyes```

> In the past we import data in this way. Too many jobs by hands...
> ![before image](static/snapshot106.jpg)
> ![before image](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot106.jpg)
>
> Using ElasticSearch Bulk API to import this data, sometimes ES only recognizes data in its API way.
>
Expand All @@ -24,7 +24,7 @@ Alexander Liu

jsonpyes
--------
* ![user interface](static/snapshot139.png)
* ![user interface](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot139.png)

#### Instructions:
There are 3 proccesses of importing raw JSON data to ElasticSearch
Expand All @@ -37,35 +37,35 @@ jsonpyes
* ```jsonpyes --data raw_data.json --check```

* If the json data file is valid:
![json valid](static/snapshot98.jpg)
![json valid](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot98.jpg)

* If the json data file is invalid:
![json invalid](static/snapshot99.jpg)
![json invalid](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot99.jpg)

##### 2. Only importing without validating
* Notice: If the raw JSON data file is invalid, ElasticSearch will not import it.
* ```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2```
* Or enable multi-threads ```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2 --thread 8```
![no threads](static/snapshot102.jpg)
![no threads](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot102.jpg)

* ```jsonpyes``` supports multi-threads when importing data to elasticsearch
![muti-threads enabled](static/snapshot132.png)
![muti-threads enabled](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot132.png)

> As you can see these two containers have same docs loaded, if we use **_--thread 8_** it could be slightly faster.
That really depends on your computer/server resources.
This was tested on a 4GB RAM / 2.4Ghz intel i5 Linux x64 laptop system.

* ![benmarks](static/snapshot133.png)
* ![benmarks](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot133.png)

* And it works.
![it works](static/snapshot105.jpg)
![it works](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot105.jpg)

##### 3. Both validating and importing
* ```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex1 --type mytype1 --check```
![validating and importing](static/snapshot135.png)
![validating and importing](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot135.png)

* And it works.
![the results](static/snapshot101.jpg)
![the results](https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot101.jpg)



Expand Down
20 changes: 10 additions & 10 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Instructions:
your computer/server resources. This was tested on a 4GB RAM /
2.4Ghz intel i5 Linux x64 laptop system.

- .. figure:: static/snapshot133.png
- .. figure:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot133.png
:alt: benmarks

benmarks
Expand All @@ -88,12 +88,12 @@ Happy hacking!
:target: https://pypi.python.org/jsonpyes
.. |GitHub license| image:: https://img.shields.io/github/license/xros/jsonpyes.svg
:target: https://github.com/xros/jsonpyes/blob/master/LICENSE
.. |before image| image:: static/snapshot106.jpg
.. |user interface| image:: static/snapshot139.png
.. |json valid| image:: static/snapshot98.jpg
.. |json invalid| image:: static/snapshot99.jpg
.. |no threads| image:: static/snapshot102.jpg
.. |muti-threads enabled| image:: static/snapshot132.png
.. |it works| image:: static/snapshot105.jpg
.. |validating and importing| image:: static/snapshot135.png
.. |the results| image:: static/snapshot101.jpg
.. |before image| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot106.jpg
.. |user interface| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot139.png
.. |json valid| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot98.jpg
.. |json invalid| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot99.jpg
.. |no threads| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot102.jpg
.. |muti-threads enabled| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot132.png
.. |it works| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot105.jpg
.. |validating and importing| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot135.png
.. |the results| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot101.jpg
24 changes: 12 additions & 12 deletions jsonpyes.egg-info/PKG-INFO
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
Metadata-Version: 1.1
Name: jsonpyes
Version: 1.2.21
Version: 1.2.22
Summary: Import JSON raw data to ElasticSearch using Python in one line of commands
Home-page: https://github.com/xros/jsonpyes
Author: Alexander Liu
Author-email: PGP E6DE7BEA
License: GPL V3
Download-URL: https://github.com/xros/jsonpyes/tarball/1.2.21
Download-URL: https://github.com/xros/jsonpyes/tarball/1.2.22
Description: json-py-es
==========

Expand Down Expand Up @@ -72,7 +72,7 @@ Description: json-py-es
your computer/server resources. This was tested on a 4GB RAM /
2.4Ghz intel i5 Linux x64 laptop system.

- .. figure:: static/snapshot133.png
- .. figure:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot133.png
:alt: benmarks

benmarks
Expand All @@ -97,15 +97,15 @@ Description: json-py-es
:target: https://pypi.python.org/jsonpyes
.. |GitHub license| image:: https://img.shields.io/github/license/xros/jsonpyes.svg
:target: https://github.com/xros/jsonpyes/blob/master/LICENSE
.. |before image| image:: static/snapshot106.jpg
.. |user interface| image:: static/snapshot139.png
.. |json valid| image:: static/snapshot98.jpg
.. |json invalid| image:: static/snapshot99.jpg
.. |no threads| image:: static/snapshot102.jpg
.. |muti-threads enabled| image:: static/snapshot132.png
.. |it works| image:: static/snapshot105.jpg
.. |validating and importing| image:: static/snapshot135.png
.. |the results| image:: static/snapshot101.jpg
.. |before image| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot106.jpg
.. |user interface| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot139.png
.. |json valid| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot98.jpg
.. |json invalid| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot99.jpg
.. |no threads| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot102.jpg
.. |muti-threads enabled| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot132.png
.. |it works| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot105.jpg
.. |validating and importing| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot135.png
.. |the results| image:: https://raw.githubusercontent.com/xros/jsonpyes/master/static/snapshot101.jpg

Keywords: elasticsearch,json,json2es,jsonpyes
Platform: Unix
Expand Down
2 changes: 1 addition & 1 deletion jsonpyes_contrib/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""
"""
# version number like 1.2.3a0 or 1.2.3, must have at least 2 parts, like 1.2
__version__ = '1.2.21'
__version__ = '1.2.22'

0 comments on commit 7619811

Please sign in to comment.