Merge pull request #264 from sabracrolleton/master

s-sql enhancements

CHANGELOG.md

@@ -1,4 +1,114 @@
-# Changelog v. 1.32.1
+# Changelog v. 1.32.8
+S-SQL Enhancements
+
+## :Update
+without the :columns parameter, :update requires alternating column value like so:
+
+    (query (:update 'weather
+            :set 'temp-lo (:+ 'temp-lo 1)
+                 'temp-hi (:+ 'temp-lo 15)
+                 'prcp :default
+            :where (:and (:= 'city "San Francisco")
+                         (:= 'date "2003-07-03"))
+            :returning 'temp-lo 'temp-hi 'prcp))
+
+:update now accepts a :columns parameter. This allows the use of either :set or :select (both of which need to be enclosed in a form) to provide the values, allowing update queries like:
+
+    (query (:update 'weather
+            :columns 'temp-lo 'temp-hi 'prcp
+                     (:set (:+ 'temp-lo 1)  (:+ 'temp-lo 15) :DEFAULT)
+            :where (:and (:= 'city "San Francisco")
+                         (:= 'date "2003-07-03"))))
+
+    (query (:update 't1
+            :columns 'database-name 'encoding
+                     (:select 'x.datname 'x.encoding
+                     :from (:as 'pg-database 'x)
+                     :where (:= 'x.oid 't1.oid))))
+
+## :Insert-into
+Insert-into also now accepts a :columns parameter which allows more precise use of select to insert values into specific row(s). A sample query could look like:
+
+    (query (:insert-into 't11
+            :columns 'region 'subregion 'country
+            (:select (:as 'region-name 'region)
+                     (:as 'sub-region-name 'subregion)
+                     'country
+             :from 'regions)))
+
+## Joins
+### Lateral Joins
+Joins are now expanded to include lateral joins. So addition join types are
+
+- :join-lateral (best practice is still to be specific on what kind of join you want)
+- :left-join-lateral
+- :right-join-lateral
+- :inner-join-lateral
+- :outer-join-lateral
+- :cross-join-lateral
+
+### Ordinality
+Selects can now use :with-ordinality or :with-ordinality-as parameters. Postgresql will give the new ordinality column the name of ordinality. :with-ordinality-as allows you to set different names for the columns in the result set.
+
+    (query (:select '*
+            :from (:generate-series 4 1 -1)
+            :with-ordinality))
+
+
+    (query (:select 't1.*
+            :from (:json-object-keys "{\"a1\":\"1\",\"a2\":\"2\",\"a3\":\"3\"}")
+            :with-ordinality-as (:t1 'keys 'n)
+
+
+## New Utility copy-from-csv
+Just a convenience function. It runs the psql copy command from inside lisp using uiop:run-program
+
+# Changelog v. 1.32.7
+
+Additional capabilities for s-sql functions :insert-into and :insert-rows-into
+
+Specifically, both can now use:
+
+- overriding-system-value
+- overriding-user-value
+- on-conflict-do-nothing
+- on-conflict
+- on-conflict-on-constraint
+- on-conflict-update
+- do-nothing
+- update-set
+- from
+- where
+- returning
+
+See updated s-sql docs for examples.
+
+# Changelog v. 1.32.4
+
+Added the ability to return results as json-encoded results as follows:
+
+- :Json-strs
+Return a list of strings where the row returned is a json object expressed as a string
+
+    (query (:select 'id 'int4 'text :from 'short-data-type-tests :where (:< 'id 3)) :json-strs)
+    ("{\"id\":1,\"int4\":2147483645,\"text\":\"text one\"}"
+     "{\"id\":2,\"int4\":0,\"text\":\"text two\"}")
+
+- :Json-str
+Return a single string where the row returned is a json object expressed as a string
+
+    (query (:select 'id 'int4 'text :from 'short-data-type-tests :where (:= 'id 3)) :json-str)
+    "{\"id\":3,\"int4\":3,\"text\":\"text three\"}"
+
+- :Json-array-str
+Return a string containing a json array, each element in the array is a selected row expressed as a json object
+
+    (query (:select 'id 'int4 'text :from 'short-data-type-tests :where (:< 'id 3)) :json-array-str)
+    "[{\"id\":1,\"int4\":2147483645,\"text\":\"text one\"}, {\"id\":2,\"int4\":0,\"text\":\"text two\"}]"
+
+# Changelog v. 1.32.3
+
+Added flag to avoid SSL certificate verification if required by user
 
 ## Fix S-SQL issue 239 (:drop-table ...) expanded incorrectly
 

README.md

@@ -1,7 +1,7 @@
 # Postmodern
 A Common Lisp PostgreSQL programming interface
 ---
-Version 1.32
+Version 1.38
 
 Postmodern is a Common Lisp library for interacting with [PostgreSQL](http://www.postgresql.org) databases. It is under active development. Features are:
 
@@ -19,6 +19,13 @@ The biggest differences between this library and CLSQL/CommonSQL or cl-dbi are t
 * [License](#dependencies)
 * [Download and installation](#download-and-installation)
 * [Quickstart](#quickstart)
+* [Authentication](#authentication)
+* [Reference](#reference)
+* [Data types](#data-types)
+* [Portability](#portability)
+* [Reserved Words](#reserved-words)
+* [Feature Requests](#feature-requests)
+* [Resources](#resources)
 * [Running tests](#running-tests)
 * [Reference](#reference)
 * [Caveats and to-dos](#caveats-and-to-dos)
@@ -419,9 +426,8 @@ The reference manuals for the different components of Postmodern are kept in sep
 - [Simple-date](https://marijnhaverbeke.nl/postmodern/simple-date.html)
 - [CL-postgres](https://marijnhaverbeke.nl/postmodern/cl-postgres.html)
 
-## Data Types, Caveats and to-dos
+## Data Types
 ---
-### Data Types
 
 For a short comparison of lisp and Postgresql data types (date and time datatypes are described in the next section)
 
@@ -551,7 +557,7 @@ the same sample data looks like:
 | timestamp\_without\_timezone | 2020-05-16T05:47:33.315622-04:00 | TIMESTAMP             |
 | timestamp\_with\_timezone    | 2020-05-16T09:47:27.855146-04:00 | TIMESTAMP             |
 
-### Portability
+## Portability
 The Lisp code in Postmodern is theoretically portable across implementations,
 and seems to work on all major ones as well as some minor ones such as Genera.
 It is regularly tested on ccl, sbcl, ecl and cmucl. ABCL currently has issues with utf-8 and :null.
@@ -560,7 +566,7 @@ Please let us know if it does not work on the implementation that you normally u
 
 The library is not likely to work for PostgreSQL versions older than 8.4. Other features only work in newer Postgresql versions as the features were only introduced in those newer versions.
 
-### Reserved Words
+## Reserved Words
 It is highly suggested that you do not use words that are reserved by Postgresql as identifiers (e.g. table names, columns). The reserved words are:
 
 "all" "analyse" "analyze" "and" "any" "array" "as" "asc" "asymmetric"
@@ -577,12 +583,10 @@ It is highly suggested that you do not use words that are reserved by Postgresql
 "similar" "some" "symmetric" "table" "then" "to" "trailing" "true" "union"
 "unique" "user" "using" "variadic" "verbose" "when" "where" "window" "with"
 
-### Things that could be implemented
+## Feature Requests
 Postmodern is under active development so issues and feature requests should
 be flagged on [[https://github.com/marijnh/Postmodern](Postmodern's site on github).
 
-Some areas that are currently under consideration can be found in the ROADMAP.md file.
-
 ## Resources
 ---
 - [Mailing List](https://mailman.common-lisp.net/listinfo/postmodern-devel)

ROADMAP.md

@@ -1,4 +1,4 @@
-# Roadmap
+# Possible Roadmap
 Postmodern is a common lisp support library for the postgresql database. It makes
 no attempt to be database agnostic. You can think of postmodern as having three components
 - cl-postgres : a low level interface for communicating with a postgresql database server
@@ -13,25 +13,21 @@ welcomed, particularly by anyone willing to work on the item.
 No guarantee is given with respect to resolution or timing on any item.
 
 ## Sql support
-- [ ]   Hypothetical Set Aggregates Functions (rank, dense-rank, percent-rank, cume-dist)
 - [ ]   UUID (see e.g  https://github.com/michaeljforster/cl-postgres-plus-uuid)
         Postgresql has a uuid extension. A database owner needs to add the extension manually to the specific database, calling:
         create extension if not exists "uuid-ossp";
         A uuid can then be generated in postmodern by calling (query (:select (:uuid-generate-v1)))
-- [ ]   Generate-Series needs testing and interval work
-- [ ]   Lateral Join (postgresql 9.3)
 - [ ]   Transition tables for triggers (postgresql 10)
 - [ ]   Hash Indexes (postgresql 10, See https://blog.2ndquadrant.com/postgresql-10-identity-columns/,
         https://www.depesz.com/2017/04/10/waiting-for-postgresql-10-identity-columns/)
 - [ ]   Full text search with phrases (postgresql 9.6 and additional functionality in 12.0)
 - [ ]   WITH CHECK clause - Auto-updatable views can now specify whether an INSERT or UPDATE
         would change the state of the row so that it would no longer be visible in the view.
         Using WITH CHECK OPTION will prevent any such changes from occuring. (postgresql 9.4)
-- [ ]   WITH ORDINALITY clause (postgresql 9.4)
 - [ ]   Table Creation with different indexes (various postgresql version additions)
+- [ ]   Generated columns - see https://pgdash.io/blog/postgres-12-generated-columns.html
 - [ ]   Postgresql regular expression support - see https://www.postgresql.org/docs/current/static/pgtrgm.html
-- [ ]   Multiple row upserts
-- [ ]   Crosstabview support (postgresql 9.6)
+- [ ]   Create table by selecting from another table.
 
 ## Data type support
 - [ ]   json, jsonb (postgresql 9.4, full text search support in postgresql 10) See
@@ -76,7 +72,7 @@ No guarantee is given with respect to resolution or timing on any item.
 
 ## Security Audit
 
-## Long Range
+## Long Range (Likely Never)
 - [ ]   Consider extending dao into more ORM capability
 - [ ]   Multi-Cluster Support
 - [ ]   Replication Support

cl-postgres.asd

@@ -16,7 +16,7 @@
   :author "Marijn Haverbeke <[email protected]>"
   :maintainer "Sabra Crolleton <[email protected]>"
   :license "zlib"
-  :version "1.32.7"
+  :version "1.32.8"
   :depends-on ("md5" "split-sequence" "ironclad" "cl-base64" "uax-15"
                      (:feature (:or :sbcl :allegro :ccl :clisp :genera
                                 :armedbear :cmucl :lispworks)

cl-postgres/sql-string.lisp

@@ -75,55 +75,6 @@ the loss of precision and offering to continue or reset
 can not be expressed within 38 decimal digits (for example 1/3), it will be truncated, and lose some precision. Set this variable to nil to suppress
 that behaviour and raise an error instead.")
 
-(defun write-rational-as-floating-point (number stream digit-length-limit)
-  "DEPRECATED. The same as write-ratio-as-floating point. Note the difference between rational and ratio.
-Kept for backwards compatibility.
-Given a ratio, a stream and a digital-length-limit, if *silently-truncate-rationals* is true,
-will return a potentially truncated ratio. If false and the digital-length-limit is reached,
-it will throw an error noting the loss of precision and offering to continue or reset
-*silently-truncate-rationals* to true. Code contributed by Attila Lendvai."
-  (declare #.*optimize* (type fixnum digit-length-limit))
-  (check-type number ratio)
-  (let ((silently-truncate? *silently-truncate-rationals*))
-    (flet ((fail ()
-                 (unless silently-truncate?
-                   (restart-case
-                    (error 'database-error :message
-                           (format nil "Can not write the rational ~A as a floating point number with only ~A available digits. You may want to (setf ~S t) if you don't mind the loss of precision."
-                                   number digit-length-limit '*silently-truncate-rationals*))
-                    (continue ()
-                              :report (lambda (stream)
-                                        (write-string "Ignore this precision loss and continue" stream))
-                              (setf silently-truncate? t))
-                    (disable-assertion ()
-                                       :report (lambda (stream)
-                                                 (write-string "Set ~S to true (the precision loss of ratios will be silently ignored in this Lisp VM)." stream))
-                                       (setf silently-truncate? t)
-                                       (setf *silently-truncate-rationals* t))))))
-      (multiple-value-bind (quotient remainder)
-          (truncate (if (< number 0)
-                        (progn
-                          (write-char #\- stream)
-                          (- number))
-                      number))
-        (let* ((quotient-part (princ-to-string quotient))
-               (remaining-digit-length (- digit-length-limit (length quotient-part))))
-          (write-string quotient-part stream)
-          (when (<= remaining-digit-length 0)
-            (fail))
-          (unless (zerop remainder)
-            (write-char #\. stream))
-          (loop
-           :for decimal-digits :upfrom 1
-           :until (zerop remainder)
-           :do (progn
-                 (when (> decimal-digits remaining-digit-length)
-                   (fail)
-                   (return))
-                 (multiple-value-bind (quotient rem) (floor (* remainder 10))
-                   (princ quotient stream)
-                   (setf remainder rem)))))))))
-
 (defun write-quoted (string out)
   (write-char #\" out)
   (loop :for ch :across string :do
@@ -194,7 +145,7 @@ used by S-SQL.)")
       ;; handle more. For practical reasons we also draw the line there. If
       ;; someone needs full rational numbers then
       ;; 200 wouldn't help them much more than 38...
-      (write-rational-as-floating-point arg result 38)))
+      (write-ratio-as-floating-point arg result 38)))
   (:method ((arg (eql t)))
     "true")
   (:method ((arg (eql nil)))

cl-postgres/tests/tests.lisp

@@ -721,15 +721,6 @@ variables:~:{~%  ~A: ~(~A~), ~:[defaults to \"~A\"~;~:*provided \"~A\"~]~}~%"
               (is (equalp (exec-query connection "select array_agg(row(1,2,3));" 'list-row-reader)
                           '((#((1 2 3)))))))))
 
-(test write-rational-as-floating-point
-      (let ((old-silently-truncate *silently-truncate-rationals*))
-        (setf cl-postgres:*silently-truncate-rationals* nil)
-        (signals error (cl-postgres::write-rational-as-floating-point (/ 1321 7) *standard-output* 5))
-        (setf cl-postgres:*silently-truncate-rationals* t)
-        (is (equal  (with-output-to-string (s) (cl-postgres::write-rational-as-floating-point (/ 1321 7) s 5))
-                    "188.71"))
-        (setf cl-postgres:*silently-truncate-rationals* old-silently-truncate)))
-
 (test write-ratio-as-floating-point
       (let ((old-silently-truncate *silently-truncate-ratios*))
         (setf cl-postgres:*silently-truncate-ratios* nil)