Merge pull request #276 from sabracrolleton/master

Import and export functions for dao classes

CHANGELOG.md

@@ -1,10 +1,58 @@
+# Changelog v. 1.33.1
+Dao Export and Import Functions (Postmodern v. 1.33.1 and newer)
+
+There may be times when the types of values in a dao slot do not have comparable types in Postgresql. For purposes of the following example, assume you have slots that you want to contain lists. Postgresql does not have a "list" data type. Postgresql arrays must be homogeneous but CL lists do not have that limitation. What to do?
+
+One method would be to use text columns or jsonb columns in Postgresql and have functions that convert as necessary going back and forth. In the following example we will use text columns in Postgresql and write CL list data to string when we "export" the data to Postgresql and then convert from string when we "import" the data from Postgresql into a dao-class instance.
+
+Consider the following dao-class definition. We have added additional column keyword parameters :col-export and :col-import. These parameters refer to functions which will convert the values from that slot to a valid Postgresql type (in our example, a string) on export to the database and from that Postgresql type to the type we want in this slot on import from the database.
+
+```lisp
+    (defclass listy ()
+      ((id :col-type integer :col-identity t :accessor id)
+       (name :col-type text :col-unique t :col-check (:<> 'name "")
+             :initarg :name :accessor name)
+       (rlist :col-type (or text db-null) :initarg :rlist :accessor rlist
+              :col-export list-to-string :col-import string-to-list)
+       (alist :col-type (or text db-null) :initarg :alist :accessor alist
+              :col-export list-to-string :col-import string-to-alist)
+       (plist :col-type (or text db-null) :initarg :plist :accessor plist
+              :col-export list-to-string :col-import string-to-plist))
+      (:metaclass dao-class)
+      (:table-name listy))
+```
+
+Now we need to define the import functions. When writing your import functions, pay attention to how you want to handle nil or :NULL values as well as how you might want to error check the conversion from a Postgresql datatype to a CL datatype.
+
+```lisp
+    (defun string-to-list (str)
+      "Take a string representation of a list and return a lisp list.
+    Note that you need to handle :NULLs."
+      (cond ((eq str :NULL)
+             :NULL)
+            (str
+             (with-input-from-string (s str) (read s)))
+            (t nil)))
+```
+
+And now we need to define the export function. In our example we are just going to be using format to write the CL value to a string. You are responsible for writing an export function that does what you need. This example just tells Postgresql to insert a string "unknown" if the slot value is not a list. You would need more error checking and condition handling.
+
+```lisp
+    (defun list-to-string (val)
+      "Simply uses (format ..) to write a list out as a string"
+      (if (listp val)
+          (format nil "~a" val)
+          "unknown"))
+```
+
 # Changelog v. 1.33.0
 This version of Postmodern now provides the ability to pass parameters to Postgresql in binary format IF that format is available for that datatype. Currently this means int2, int4, int8, float, double-float (except clisp) and boolean. Rational numbers continue to be passed as text.
 
 The flag is set in the database connection object. (Thank you Cyrus Harmon for suggesting that). This means it can be set either in the initial connection to the database or using the use-binary-parameters function to set it after the initial connection has been established. If you are using multiple connections, some can be set to use binary parameters, some not.
 
 If a query to Postgresql does not have a table column which would allow Postgresql to determine the correct datatype and you do not specify differently, Postgresql will treat the parameters passed with the query as text. The default text setting with results:
 
+```lisp
     (query "select $1" 1 :single)
     "1"
     (query "select $1" 1.5 :single)
@@ -15,14 +63,18 @@ If a query to Postgresql does not have a table column which would allow Postgres
     "false"
     (query "select $1" :NULL :single)
     :NULL
+```
 
 You can specify parameter type as so:
 
+```lisp
     (query "select $1::integer" 1 :single)
     1
+```
 
 Setting the use-binary slot in the database connection object to t has the following results:
 
+```lisp
     (query "select $1" 1 :single)
     1
     (query "select $1" 1.5 :single)
@@ -33,21 +85,25 @@ Setting the use-binary slot in the database connection object to t has the follo
     NIL
     (query "select $1" :NULL :single)
     :NULL
+```
 
 The default for cl-postgres/Postmodern is to continue to pass parameters to Postgresql as text (not in binary format) in order to avoid breaking existing user code. If you want to pass parameters to Postgresql in binary format and want to set that up when you are making the database connection, the following examples may help. We continue the difference in the signatures (cl-postgres using optional parameters and postmodern using keyword parameters) because of the expected downstream breakage if we shifted cl-postgres:open-database to using keyword parameters.
 
 The signature for opening a database in cl-postgres:
 
+```lisp
     (defun open-database (database user password host
                           &optional (port 5432) (use-ssl :no)
                           (service "postgres") (application-name "")
                           (use-binary nil))
         ...)
+```
 
 or your favorite macro.
 
 In postmodern you have the connect function or the with-connection macro:
 
+```lisp
     (defun connect (database-name user-name password host
                     &key (port 5432) pooled-p
                     (use-ssl *default-use-ssl*)
@@ -60,26 +116,31 @@ In postmodern you have the connect function or the with-connection macro:
       `(let ((*database* (apply #'connect ,spec)))
          (unwind-protect (progn ,@body)
            (disconnect *database*))))
+```
 
 In any case, you can set the flag after the connection is established with the use-binary-parameters function:
 
+```lisp
     (pomo:use-binary-parameters *database* t)
 
     (cl-postgres:use-binary-parameters some-database-connection t)
+```
 
 Using binary parameters does tighten type checking when using prepared queries. You will not be able to use prepared queries with varying formats. In other words, if you have a prepared query that you pass an integer as the first parameter and a string as the second parameter the first time it is used, any subsequent uses of that prepared query during that session will also have to pass an integer as the first parameter and a string as the second parameter.
 
 Benchmarking does indicate a slight speed and consing benefit to passing parameters as binary, but your mileage will vary depending on your use case.
 
 In addition, this version also adds the ability to have queries returned as vectors of vectors, using a vectors keyword.
 
+```lisp
     (query "select id, some_int, some_text from tests_data :where id = 1" :vectors)
-    or
+;; or
     (query (:select 'id 'some-int 'some-text :from 'test-data)
          :vectors)
     #(#(1 2147483645 "text one")
       #(2 0 "text two")
       #(3 3 "text three"))
+```
 
 Like :array-hash, if there is no result it will return an empty array, not nil.
 # Changelog v. 1.32.9
@@ -115,16 +176,19 @@ S-SQL Enhancements
 ## :Update
 without the :columns parameter, :update requires alternating column value like so:
 
+```lisp
     (query (:update 'weather
             :set 'temp-lo (:+ 'temp-lo 1)
                  'temp-hi (:+ 'temp-lo 15)
                  'prcp :default
             :where (:and (:= 'city "San Francisco")
                          (:= 'date "2003-07-03"))
             :returning 'temp-lo 'temp-hi 'prcp))
+```
 
 :update now accepts a :columns parameter. This allows the use of either :set or :select (both of which need to be enclosed in a form) to provide the values, allowing update queries like:
 
+```lisp
     (query (:update 'weather
             :columns 'temp-lo 'temp-hi 'prcp
                      (:set (:+ 'temp-lo 1)  (:+ 'temp-lo 15) :DEFAULT)
@@ -136,16 +200,19 @@ without the :columns parameter, :update requires alternating column value like s
                      (:select 'x.datname 'x.encoding
                      :from (:as 'pg-database 'x)
                      :where (:= 'x.oid 't1.oid))))
+```
 
 ## :Insert-into
 Insert-into also now accepts a :columns parameter which allows more precise use of select to insert values into specific row(s). A sample query could look like:
 
+```lisp
     (query (:insert-into 't11
             :columns 'region 'subregion 'country
             (:select (:as 'region-name 'region)
                      (:as 'sub-region-name 'subregion)
                      'country
              :from 'regions)))
+```
 
 ## Joins
 ### Lateral Joins
@@ -161,6 +228,7 @@ Joins are now expanded to include lateral joins. So addition join types are
 ### Ordinality
 Selects can now use :with-ordinality or :with-ordinality-as parameters. Postgresql will give the new ordinality column the name of ordinality. :with-ordinality-as allows you to set different names for the columns in the result set.
 
+```lisp
     (query (:select '*
             :from (:generate-series 4 1 -1)
             :with-ordinality))
@@ -169,7 +237,7 @@ Selects can now use :with-ordinality or :with-ordinality-as parameters. Postgres
     (query (:select 't1.*
             :from (:json-object-keys "{\"a1\":\"1\",\"a2\":\"2\",\"a3\":\"3\"}")
             :with-ordinality-as (:t1 'keys 'n)
-
+```
 
 ## New Utility copy-from-csv
 Just a convenience function. It runs the psql copy command from inside lisp using uiop:run-program

README.md

@@ -4,7 +4,7 @@ A Common Lisp PostgreSQL programming interface
 
 ---
 
-Version 1.33.0
+Version 1.33.1
 
 Postmodern is a Common Lisp library for interacting with [PostgreSQL](http://www.postgresql.org) databases. It is under active development. Features are:
 
@@ -183,7 +183,7 @@ You do not have to pull in the whole result of a query at once, you can also ite
     (format t "On this row, x = ~A and y = ~A.~%" x y))
 ```
 
-You can work directly with the database or you can use a simple database-access-class (aka dao) which would cover all the columns in a row. This is what a database-access class looks like:
+You can work directly with the database or you can use a simple [database-access-class](https://marijnhaverbeke.nl/postmodern/dao-classes.html) (aka dao) which would cover all the columns in a row. This is what a database-access class looks like:
 
 ```lisp
 (defclass country ()
@@ -342,7 +342,7 @@ described above. Using the slightly more complicated version of the country dao
 
 This defines our table in the database. execute works like query, but does not expect any results back.
 
-See [Introduction to Multi-table dao class objects](doc/postmodern.html) in the postmodern.org or postmodern.html manual for a further discussion of multi-table use of daos.
+See [Introduction to Multi-table dao class objects](doc/dao-classes.html#multi-table-dao-class-object) in the postmodern.org or postmodern.html manual for a further discussion of multi-table use of daos.
 
 ### Inserting Data
 
@@ -458,11 +458,22 @@ Other authentication methods have not been tested. Please let us know if there i
 
 The reference manuals for the different components of Postmodern are kept in separate files. For using the library in the most straightforward way, you only really need to read the Postmodern reference and glance over the S-SQL reference. The simple-date reference explains the time-related data types included in Postmodern, and the CL-postgres reference might be useful if you just want a low-level library for talking to a PostgreSQL server.
 
+- [Postmodern - Index Page](https://marijnhaverbeke.nl/postmodern/index.html)
 - [Postmodern](https://marijnhaverbeke.nl/postmodern/postmodern.html)
 - [S-SQL](https://marijnhaverbeke.nl/postmodern/s-sql.html)
 - [Simple-date](https://marijnhaverbeke.nl/postmodern/simple-date.html)
 - [CL-postgres](https://marijnhaverbeke.nl/postmodern/cl-postgres.html)
 
+Some specific topics in more detail:
+
+- [Array Notes](https://marijnhaverbeke.nl/postmodern/array-notes.html)
+- [Creating Tables](https://marijnhaverbeke.nl/postmodern/create-tables.html)
+- [Dao Classes](https://marijnhaverbeke.nl/postmodern/dao-classes.html)
+- [Dynamic Queries](https://marijnhaverbeke.nl/postmodern/dynamic-queries.html)
+- [Interval Notes](https://marijnhaverbeke.nl/postmodern/interval-notes.html)
+- [Isolation Notes](https://marijnhaverbeke.nl/postmodern/isolation-notes.html)
+
+
 ## Data Types
 
 ---

cl-postgres.asd

@@ -16,7 +16,7 @@
   :author "Marijn Haverbeke <[email protected]>"
   :maintainer "Sabra Crolleton <[email protected]>"
   :license "zlib"
-  :version "1.33.0"
+  :version "1.33.1"
   :depends-on ("md5" "split-sequence" "ironclad" "cl-base64" "uax-15"
                      (:feature (:or :sbcl :allegro :ccl :clisp :genera
                                 :armedbear :cmucl :lispworks)

cl-postgres/protocol.lisp

@@ -629,8 +629,7 @@ looks like:
 
 Obviously, row readers should not do things with the database connection
 like, say, close it or start a new query, since it still reading out the
-results from the current query.Create a row-reader, using the given name
-for the fields argument and the given body for reading the rows. A row reader
+results from the current query. A row reader
 is a function that is used to do something with the results of a query. It has
 two local functions: next-row and next-field, the first should be called
 once per row and will return a boolean indicating whether there are