Taking the effort of posting here what is reported on web2py. I feel that those DAL bugs are never seen as a priority if not posted here.........

https://github.com/web2py/web2py/issues/910

I'm struggling with this: https://github.com/web2py/pydal/blob/master/pydal/helpers/methods.py#L83

...as it requires identity of aliased Table instances, and hence fails if the same aliasing happens twice, i.e.:

table.with_alias("alias") is not table.with_alias("alias")

...even though it's the same table, and the same alias, and consequently the same SQL.

Why does it have to be this way - wouldn't it be good enough to ensure that the DAL name of both tables is the same (i.e. that they refer to the same original table)? Do they really have to be exactly the same Python object?

This changeset implements subselect objects that can be used in a join like a virtual table or passed to Expression.belongs(). Subselect objects can be created using Set.nested_select() which accepts the same arguments as regular select(). One additional argument, correlated, determines whether the subselect may reference tables from parent select (defaults to True which means referencing is permitted). Example usage:

total = table1.field2.sum().with_alias('total')
sub1 = db(table1.field2 != None).nested_select(table1.field1, total, groupby=table1.field1)
sub1 = sub1.with_alias('tmp')
result1 = db(table2.field1 == sub1.field1).select(table2.ALL, sub1.total)

total = table1.field2.sum().with_alias('total')
sub2 = db(table1.field2 != None).nested_select(total, groupby=table1.field1)
result2 = db(table2.field2.belongs(sub2)).select(table2.ALL)

I'm opening this pull request early for design discussion. Unit tests for Python 2.7, 3.4 & 3.5 and SQLite, MySQL and PostgreSQL are passing on my machine but there's still work to do. Comments and suggestions are appreciated.

Design changes

• Adapters and dialects must no longer resolve table names over and over through DAL.__getitem__(). They must pass Table/Select objects around.
• New methods for class Table:
  • query_name() - Full table name for query FROM clause, including possible alias.
  • query_alias() - The appropriate string that identifies the table throughout a query (e.g. for building fully qualified field names).
• Rows.__init__() now takes additional argument - list of Field/Expression objects corresponding to columns
• BaseAdapter._select() now returns a Select object instead of string
• BaseAdapter.tables() now returns a dict mapping table names to table objects instead of name list.
• BaseAdapter.common_filter() now accepts table objects as well as table names.
• many other BaseAdapter internal methods now take or return table objects instead of table names.

As introduced on the group (https://groups.google.com/forum/#!topic/web2py-developers/zw6-bFpe96s) this is a major refactoring of the code base.

Feedbacks and comments are welcome.

Intentions and the rationale:

• Stop using the BaseAdapter as a super-collector (sometimes taking place of SQLite – ed. note)
• Better using inheritance between classes
• Simplify the writing of new adapters (and increase the contributions)
• De-uglify a lot of the code making it simpler to understand and debug
• Allowing to easily implement performance optimizations

Refactoring steps:

• [x] BaseAdapter re-structuring
• [x] Better SQLAdapter and NoSQLAdapter
• [x] Base dialects structure
• [x] Base parsers structure
• [x] Base representers structure
• [x] Migrations separation from base adapter
• [x] SQLite and Spatialite adapters
• [x] Postgresql adapter
• [x] Postgresql 2/new adapter
• [x] MySQL adapter
• [x] MSSQL adapter
• [x] jdbc versions of adapters
• [x] Review of NoSQLAdapter
• [x] MongoDB Adapter
• [x] Other SQL adapters
• [x] Other NoSQL adapters
• [x] Final code cleaning

Changes from 16.03:

• Postgre adapter no longer parse datetime objects
• MSSQL2Adapter no longer exists since was the same of MSSQLNAdapter (but the uri is still valid)
• Includes fix for #296
• BaseAdapter.connection and BaseAdapter.cursor are now thread safe
• Removed _lastsql in adapters since not thread safe
• Empty values for ObjectId fields in MongoAdapter now are stored (and returned) as None instead of a fake ObjectId converted to integer 0

This changeset fixes update()/delete() on aliased tables and bugs caused by using the wrong table/field name in various queries (especially in migrator).

Changes in class Table

• Table._ot has been renamed to Table._dalname and it is always set to the original internal table name.
• Table._rname now defaults to quoted Table._dalname (except when the table is not associated with any database, only then it gets set to None).
• Added Table._raw_rname which is the unquoted version of _rname (for generating extra identifers based on the backend table name, e.g. sequence names).
• Table.sqlsafe renamed to sql_shortref.
• Table.sqlsafe_alias renamed to sql_fullref.

Example where they belong in a query: SELECT * FROM sql_fullref WHERE sql_shortref.id > 0

Changes in class Field

• Field.sqlsafe now raises exception when the field object is not associated with any table.
• Deprecated Field.sqlsafe_name, use Field._rname instead.
• Added property Field.longname which returns field reference string as 'tablename.fieldname' (as oposed to 'table_rname.field_rname' returned by sqlsafe).
• Field.clone() now returns the new Field object without link to any table and with empty _rname if it needs to be requoted.
• Added method Field.bind() which links the field to a table. When the field is already linked, it raises exception. It also sets empty Field._rname to quoted Field.name. This method is called internally by table/subselect constructor.
• Added Field._raw_rname which is the unquoted version of _rname.

Other changes

• Removed BaseAdatper.alias(), the code was move to Table.with_alias() where it belongs + bugfixes.
• SQLDialect.update() and SQLDialect.delete() now take a Table object instead of tablename. This change was necessary because MySQLDialect.delete() and also other dialects need to access both sql_shortref and sql_fullref when the table is aliased.
• New method writing_alias() added to SQLDialect. The method simply returns table.sql_fullref. It was added so that dialects like SQLite which don't support UPDATE/DELETE on aliased tables can raise exception without reimplementing the whole update()/delete() methods.
• IMAP adapter was updated to new API (passing Table objects instead of tablename). I forgot to do this in the subselect pull request. Beware that this change is untested.
• Field._rname and Field._raw_rname will now be stored in pickled table metadata files.

Don't merge this until I say is ready

This implements basic indexes support as requested in #152. Intended workflow:

# just fields
db.mytable.create_index('index_name', db.mytable.myfield_a, db.mytable.myfield_b)
# with expressions
db.mytable.create_index('index_name', db.mytable.myfield_a, db.mytable.myfield_b.coalesce(None))
# with additional parameters (where is supported only by postgres)
db.mytable.create_index('index_name', db.mytable.myfield_a, where=(db.mytable.myfield_b == False), unique=True)

db.mytable.drop_index('index_name')

@niphlod @mdipierro @ilvalle @michele-comitini feedbacks are welcome.

It seems to me that sqlform factory is broken. Git bisect traced it down to commit @03dc3a8d4cddf3c1fd92cdab62d923d64f777776

Traceback (most recent call last):
  File "/home/marin/web2py/gluon/restricted.py", line 227, in restricted
    exec ccode in environment
  File "/home/marin/web2py/applications/move/controllers/user.py", line 142, in <module>
  File "/home/marin/web2py/gluon/globals.py", line 412, in <lambda>
    self._caller = lambda f: f()
  File "/home/marin/web2py/applications/move/controllers/user.py", line 98, in login
    form = SQLFORM.factory(Field('email', 'string'), Field('password', 'string')).process(keepvalues=True)
  File "/home/marin/web2py/gluon/html.py", line 2301, in process
    self.validate(**kwargs)
  File "/home/marin/web2py/gluon/html.py", line 2238, in validate
    if self.accepts(**kwargs):
  File "/home/marin/web2py/gluon/sqlhtml.py", line 1688, in accepts
    self.vars.id = self.table.insert(**fields)
  File "/home/marin/web2py/gluon/packages/dal/pydal/objects.py", line 706, in insert
    ret = self._db._adapter.insert(self, self._listify(fields))
  File "/home/marin/web2py/gluon/packages/dal/pydal/adapters/base.py", line 737, in insert
    raise e
ValueError: INSERT INTO no_table(password,email) VALUES ('password','[email protected]');

Some adapters are not properly initialized when they reuse idle connection from pool. In case of PostgreSQL, this actually results in unnecessary migrations of json fields back and forth between TEXT and JSON SQL types because the adapter will not check whether it can use dialects with native JSON support.

This pull request introduces new after_reconnect() adapter method which will be executed every time the connection is assigned to a new adapter object, no matter whether the connection was freshly created or reused from connection pool.

Since so many errors and fundamental issues seem to have crept into the Oracle adapter over time, I took on the task of setting up the automated Travis tests to include Oracle.

Note: using Oracle on Travis requires a download of the free Express Edition Oracle platform during container build, since it cannot be redistributed, so to make it work (via this handy project) you need to add two environment variables to Travis directly: a username / password for a (free) oracle.com account that has accepted their license agreement for this product. I can email that username / password, and once added to the main pydal Travis account, the build will succeed. See the successful build here.

Once I started running the tests on Oracle, a considerable number of fundamentally breaking issues cropped up everywhere--including everything from broken python syntax to badly handled sql generation and types--but after many adjustments and improvements to the adapter, dialect, parser, etc, the entire test suite now runs successfully (except for a handful of tests that are not appropriate for Oracle, like the Time type it doesn't support, which I toggled off for this adapter in tests/sql.py).

The biggest challenge for Oracle was properly handling case-sensitive table names and field names (when not using entity_quoting=False), which requires that they be quoted correctly everywhere. I did need to add a few additional switches to the main migrator.py file to make those additional quotes possible in places where it didn't currently invoke dialect.quote on an rname. Those I wrapped in an if-oracle condition for now, but perhaps it would be best in the long run to automatically invoke the quote function on rnames for these additional cases for all adapters.

Everything now works smoothly--even complex things like correct results paging for all the test instances, with far greater stability than the prior Oracle query-paging syntax--and I'm hoping that moving it into the automated tests can finally make Oracle a first-tier citizen again for pydal, given that it had drifted into severe problems over time.

@mdipierro @gi0baro The upload code in present pyDAL is uncovered. I made a test to correct a bug that shows with uploadfs=pyfilesystem. This bug affects current web2py too.

Just spotted. I have this Row returned from pyDAL:

<Row {'users': {'id': 26470719833643260735021266671L}, '_extra': {'registration_id': '', 'last_name': 'Barillari', 'first_name': 'Giovanni', 'reset_password_key': '', 'password': '***', 'registration_key': 'b40bde36-258a-48fb-b565-ec22cabc0586', 'email': '***'}}>

In v15.03 was working as expected. In current master the problem seems caused by the difference behavior between Row.__getitem__ and Row.__getattr__. @ilvalle maybe a consequence of your refactoring with BasicStorage? I propose to add a:

__getattr__ = __getitem__

in Row class.

Details can be found here -> https://groups.google.com/g/py4web/c/f2BGHP_1-F8

In short, if you have an upload field that isn't based on a database field, it fails to build the filename.

I will submit a PR.

Number of PostgreSQL connections from apps keeps growing even to hundreds.

All connections are entering Idle state after COMMIT or ROLLBACK query. It seems that the connection pool does not pick them properly, but keeps opening new ones. Not sure if any connections are ever closed. As soon as MAX_CONNECTIONS set by PostgreSQL is reached, the db starts refusing further connections, crashing the app. Problem occurs regardless of the pool_size (tried 0 and 10).

Experienced for pydal used by py4web, not sure if it is py4web-specific.

The problem disappears if I use pool_size=0 and the custom DAL fixture:

class ClosingDAL(pydal.DAL, Fixture):

  reconnect_on_request = True

  def on_request(self, context):
    if self.reconnect_on_request:
      self._adapter.reconnect()
    threadsafevariable.ThreadSafeVariable.restore(ICECUBE)

  def __close_silently(self):
    #to ignore AttributeError: '_thread._local' object has no attribute '_pydal_db_instances_'
    try: self.close()
    except Exception as e: print(f"[DAL] closing error {e}")

  def on_error(self, context):
    self.rollback()
    self.__close_silently()

  def on_success(self, context):
    self.commit()
    self.__close_silently()

For further investigation: Since DAL from core.py performs either commit() or rollback() only, which part of pydal is responsible for closing idling connections above pool_size?

Related discussion thread: https://groups.google.com/g/py4web/c/6pyLn6MsHus

Tested only with this:

from pydal import DAL, Field
db = DAL("google:datastore")
db.define_table("thing", Field("name"), Field("age", "integer"), Field('price','float'), Field('active', 'boolean'))
print(db.thing.insert(name="table", age=1, price=3.14, active=True))
rows = db(db.thing.name=="table").select()
print(str(rows))
print(db(db.thing.name=="table").count())
rows = db(db.thing.age==1).select()
print(str(rows))
id = rows[-1].id
row = db.thing(id)
print(row)
row.update_record(name="chair")
row.delete_record()
print(db.thing(id))
rows = db(db.thing).select()
print(str(rows))
db(db.thing.name=="chair").delete()

Hi, Pydal works well with SQL databases in Google Appengine, but google:datastore and google:datastore+nbd don't work with python3. I have tried to migrate the GoogleDatastore(NoSQLAdapter) with no success till now.

It's been published new guides to move from the deprecaed google libraries:

https://github.com/GoogleCloudPlatform/appengine-python2-3-migration/tree/main/datastore

https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-#0

https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore#0

https://github.com/googlecodelabs/migrate-python2-appengine

I'm asking for some help to upgrade this pydal feature.

This warning appears when running in google app engine:

Please remove any imports of google.appengine.api.rdbms. First Generation Cloud SQL instances have been shut down, and rdbms.py will be removed in a future release. See: https://cloud.google.com/sql/docs/mysql/deprecation-notice

The rdbms library is imported in _gae.py: `# -- coding: utf-8 --

try: from new import classobj except ImportError: classobj = type

try: from google.appengine.ext import db as gae except ImportError: gae = None Key = None else: from google.appengine.ext import ndb from google.appengine.api import namespace_manager, rdbms from google.appengine.api.datastore_types import Key # for belongs on ID from google.appengine.ext.ndb.polymodel import PolyModel as NDBPolyModel`

And it is used for the google:sql connection in google.py adpater:

`if gae: from .._gae import ndb, rdbms, namespace_manager, classobj, NDBPolyModel from ..helpers.gae import NDBDecimalProperty

class GoogleMigratorMixin(object): migrator_cls = InDBMigrator

@adapters.register_for("google:sql") class GoogleSQL(GoogleMigratorMixin, MySQL): uploads_in_blob = True REGEX_URI = "^(?P.*)/(?P.+)$"

def _find_work_folder(self):
    super(GoogleSQL, self)._find_work_folder()
    if os.path.isabs(self.folder) and self.folder.startswith(os.getcwd()):
        self.folder = os.path.relpath(self.folder, os.getcwd())

def _initialize_(self):
    super(GoogleSQL, self)._initialize_()
    self.folder = self.folder or pjoin(
        "$HOME",
        THREAD_LOCAL._pydal_folder_.split(os.sep + "applications" + os.sep, 1)[1],
    )
    ruri = self.uri.split("://", 1)[1]
    m = re.match(self.REGEX_URI, ruri)
    if not m:
        raise SyntaxError("Invalid URI string in DAL")
    self.driver_args["instance"] = self.credential_decoder(m.group("instance"))
    self.dbstring = self.credential_decoder(m.group("db"))
    self.createdb = self.adapter_args.get("createdb", True)
    if not self.createdb:
        self.driver_args["database"] = self.dbstring

def find_driver(self):
    self.driver = "google"

def connector(self):
    return rdbms.connect(**self.driver_args)

`

https://cloud.google.com/appengine/docs/standard/python/refdocs/modules/google/appengine/api/rdbms#connect

Hi,

db.define_table('tablename, Field('fieldname, notnull=True)) triggers psycopg2.errors.NotNullViolation: column "fieldname__tmp" of relation "tablename" contains null values on PostgreSQL migrations when there is already some data in the table. It gives sense because py4web creates notnull fieldname__tmp with null values (supposed to be replaced by fieldname data in the next step).

Setting default to something else than DEFAULT serves as a workaround.

On PostgreSQL I would suggest setting notnull=False for temporary fields (*__tmp) even if the original field is notnull=True, unless there is a non-null default value specified.

David