Merge "Documentation structure update"

README.rst

@@ -1,101 +1,68 @@
-YAQL - Yet Another Query Language
+YAQL: Yet Another Query Language
-=================================
+================================
 
-At the beginning of millennium the growing trend towards data formats standardization and application integrability made
+YAQL (Yet Another Query Language) is an embeddable and extensible query
-XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging
+language, that allows performing complex queries against arbitrary objects. It
-from small config files to very large datasets. As these data often had a complex structure with many levels of
+has a vast and comprehensive standard library of frequently used querying
-nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query
+functions and can be extend even further with user-specified functions. YAQL is
-these data sets. This  is how XPath and later XQL were born.
+written in python and is distributed via PyPI.
 
-With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides
+Quickstart
-being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays,
+----------
-dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related
+Install the latest version of yaql:
-technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of
-those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and
-JavaScript languages, but later on ports to other languages including Python were written.
 
-JSONPath allows navigation and querying, well, JSONs.
+.. code-block:: console
-Suppose we have JSON as in following:
 
-    {
+    pip install yaql>=1.0.0
-      "customers": [
+..
-      {
+
-        "customer_id": 1,
+Run yaql REPL:
-        "name": "John",
+
-        "orders": [{
+.. code-block:: console
-          "order_id": 1,
+
-          "item": "Guitar",
+    yaql
-         "quantity": 1
+..
-        }]
+
-      },{
+Load a json file:
-       "customer_id": 2,
+
-       "name": "Paul",
+.. code-block:: console
-       "orders": [ {
+
-          "order_id": 2,
+    yaql> @load my_file.json
-          "item": "Banjo",
+..
-          "quantity": 2
+
-         },{
+Check it loaded to current context, i.e. `$`:
-         "order_id": 3,
+
-         "item": "Piano",
+.. code-block:: console
-         "quantity": 1
+
-        }]
+    yaql> $
-      }
+
-     ]
+..
-    }
+
+Run some queries:
+
+.. code-block:: console
+
+    yaql> $.customers
+    ...
+    yaql> $.customers.orders
+    ...
+    yaql> $.customers.where($.age > 18)
+    ...
+    yaql> $.customers.groupBy($.sex)
+    ...
+    yaql> $.customers.where($.orders.len() >= 1 or name = "John")
+..
+
+Project Resources
+-----------------
+
+* `Official Documentation <http://yaql.readthedocs.org>`_
+
+* Project status, bugs, and blueprints are tracked on
+  `Launchpad <https://launchpad.net/yaql>`_
 
 
-then
+License
+-------
 
-`jsonpath(data, "$.customers[0].name") -> [‘John’]`
+Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0
-`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]`
-
-But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath:
-
-`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") ->  [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]`
-
-The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is
-replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression
-requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python
-programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to
-simplify program logic it would not be a big deal, but what if JSONPath is written by program users?
-
-JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order
-having ID = some variable value he has to dynamically construct expression string using string formatting or
-concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also
-open the doors for injection attacks (remember SQL injection?)
-
-Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text
-representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is
-one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed?
-Or the model is a graph rather than a tree?
-
-It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function
-to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient
-power to query model without manual coding and have it still secure.
-This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data
-querying.
-
-Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL:
-
-`$.customers[0].name -> $.customers[0].name (no change)`
-`$.customers[*].orders[*].order_id -> $.customers.orders.order_id`
-
-the main addition to JSONPath is functions and operators. Consider the following YAQL expressions:
-
-`$.customers.orders[$.quantity > 0].quantity.sum() -> 4`
-`$.customers.orders.select($.quantity * $.quantity).sum() -> 6`
-`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]`
-`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]`
-`$.customers.orders.order_id.orderDesc($).first() -> 3`
-
-Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large
-set of commonly used functions. But they are not built-in. All the functions and operators (which are also function:
-`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions
-and to remove standard ones.
-
-JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third
-parameter - context.
-
-Context is a repository of functions and variables that can be used in expressions. So all the functions above are just
-ordinary Python functions that are registered in Context object. But because they all need to be registered in Context
-user can always customize them, add his own model-specific ones and have full control over the expression evaluation.

doc/source/index.rst

@@ -4,16 +4,33 @@
    contain the root `toctree` directive.
 
 Welcome to yaql's documentation!
-========================================================
+================================
 
-Contents:
+Introduction
+~~~~~~~~~~~~
 
 .. toctree::
    :maxdepth: 2
 
    readme
-   installation
+   what_is_yaql
+
+Usage
+~~~~~
+
+.. toctree::
+   :maxdepth: 1
+
    usage
+   language_description
+   standard_library
+
+Contributing
+~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 1
+
    contributing
 
 Indices and tables

doc/source/installation.rst

@@ -1,12 +0,0 @@
-============
-Installation
-============
-
-At the command line::
-
-    $ pip install yaql
-
-Or, if you have virtualenvwrapper installed::
-
-    $ mkvirtualenv yaql
-    $ pip install yaql

doc/source/language_description.rst

@@ -0,0 +1,19 @@
+Language description
+====================
+
+This section is not ready yet
+
+Syntax
+~~~~~~
+
+Types
+~~~~~
+
+Operators
+~~~~~~~~~
+
+Context
+~~~~~~~
+
+$ variable
+~~~~~~~~~~

doc/source/standard_library.rst

@@ -0,0 +1,4 @@
+Standard Library
+================
+
+This section is not ready yet.

doc/source/usage.rst

@@ -1,7 +1,13 @@
-========
 Usage
-========
+=====
 
-To use yaql in a project::
+This section is not ready yet.
 
-	import yaql
+REPL
+~~~~
+
+Embedding YAQL
+~~~~~~~~~~~~~~
+
+Extending YAQL
+~~~~~~~~~~~~~~

doc/source/what_is_yaql.rst

@@ -0,0 +1,22 @@
+What is YAQL
+============
+
+YAQL is a general purpose query language, that is designed to operate on
+objects of arbitrary complexity.
+YAQL has a large standard library of functions for filtering, grouping and
+aggregation of data. At the same time YAQL allows you to extend it by
+defining your own functions.
+
+Why YAQL?
+=========
+
+So why bother and create another solution for a task, that has been addressed
+by many before us? Obviously because we were not satisfied with flexibility
+and/or quality of any existing solution. Most notably we needed a tool for json
+data, that would support some complex data transformations.
+YAQL is a pure-python library and therefore is easily embeddable in any python
+application.
+YAQL is designed to be human-readable and has a SQL-like feel and look. It is
+inspired in part by LINQ for .NET.
+Since YAQL is extensible and embeddable it makes a perfect choice for becoming
+the basis for your DSLs.